Day 11 Chatbot integration- 看图学英文

Chatbot integration- 看图学英文

大致上的概念是要利用 Line 把图片传给 chatbot ，再把图片传到 Azure 认知服务，分别执行 object detection 和 image captioning，匡出图片中的物体，并且描述图片内容。

这边要注意的是，有时候新手会直接把密码、金钥和端点直接放进去程序码中，然後直接推到 Azure Web App，甚至直接推到 githib ，我想这不是一个好的习惯。虽然 Azure Web App 的使用者是自己，不是公开的情况，但如果是放到 github 的话，那自己的密码、金钥和端点就全曝光了。所以，最好还是养成好习惯，不要直接将自己的金钥和密码以明码的方式放在自己的程序码中上传。

这里介绍一个简单的做法：另外开一个 json 档案config.json，把一些比较敏感的变数放进此档案中，在上传到 Azure Web App。

上传 config

• 准备config.json

{
    "line": {
        "line_secret": "your line secret",
        "line_token": "your line token",
    },
    "azure": {
        "cv_key": "your subscription key of computer vision",
        "cv_end": "your endpoint of computer vision",
        "blob_connect": "your connect string",
        "blob_container": "your blob container name"
        }
}

• 建立连结 Web App 的 tunnel。

az webapp create-remote-connection \
-n linecv --resource-group Tibame &

• 使用scp上传config.json，这边要注意只能上传到/home，这样在 Web App 的 chatbot 才能读到此档案。

scp -P <port> config.json [email protected]:/home/config.json

Python套件

requirements.txt

Flask==1.0.2
line-bot-sdk
azure-cognitiveservices-vision-computervision
azure-storage-blob
Pillow

示范程序

程序概念

• 从 Line 的讯息取得图片
• 上传图片至 blob ，并取得图片 URL
• 以图片 URL 作为输入，执行物体侦测和影像描述
• 将物体侦测的结果存成图片上传 blob，并取得连结
• 将上述连结与影像描述结果，结合 flex message
• 回传 flex message 给使用者

application.py

import os
import json

from flask import Flask, request, abort
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from azure.storage.blob import BlobServiceClient

from msrest.authentication import CognitiveServicesCredentials
from linebot import LineBotApi, WebhookHandler
from linebot.exceptions import InvalidSignatureError
from linebot.models import (
    MessageEvent,
    FlexSendMessage,
    ImageMessage,
)

from PIL import Image, ImageDraw, ImageFont

app = Flask(__name__)


CONFIG = json.load(open("/home/config.json", "r"))

SUBSCRIPTION_KEY = CONFIG["azure"]["cv_key"]
ENDPOINT = CONFIG["azure"]["cv_end"]
CV_CLIENT = ComputerVisionClient(
    ENDPOINT, CognitiveServicesCredentials(SUBSCRIPTION_KEY)
)

CONNECT_STR = CONFIG["azure"]["blob_connect"]
CONTAINER = CONFIG["azure"]["blob_container"]
BLOB_SERVICE = BlobServiceClient.from_connection_string(CONNECT_STR)


LINE_SECRET = CONFIG["line"]["line_secret"]
LINE_TOKEN = CONFIG["line"]["line_token"]
LINE_BOT = LineBotApi(LINE_TOKEN)
HANDLER = WebhookHandler(LINE_SECRET)



# 特地留下 hello world，程序码有问题时，我会从这边开始检查
@app.route("/")
def hello():
    "hello world"
    return "Hello World!!!!!"


# 为了上传图片到 Azure blob
def upload_blob(container, path):
    """
    Upload files to Azure blob
    """
    blob_client = BLOB_SERVICE.get_blob_client(container=container, blob=path)
    with open(path, "rb") as data:
        blob_client.upload_blob(data, overwrite=True)
    data.close()
    return blob_client.url

# 影像描述
def azure_describe(url):
    """
    Output azure image description result
    """
    description_results = CV_CLIENT.describe_image(url)
    output = ""
    for caption in description_results.captions:
        output += "'{}' with confidence {:.2f}% \n".format(
            caption.text, caption.confidence * 100
        )
    return output

# 物体侦测
def azure_object_detection(url, filename):
    """
    Azure object detection, and output images with bounding boxes
    """
    img = Image.open(filename)
    draw = ImageDraw.Draw(img)
    font_size = int(5e-2 * img.size[1])
    fnt = ImageFont.truetype("static/TaipeiSansTCBeta-Regular.ttf", size=font_size)
    object_detection = CV_CLIENT.detect_objects(url)
    if len(object_detection.objects) > 0:
        for obj in object_detection.objects:
            left = obj.rectangle.x
            top = obj.rectangle.y
            right = obj.rectangle.x + obj.rectangle.w
            bot = obj.rectangle.y + obj.rectangle.h
            name = obj.object_property
            confidence = obj.confidence
            print("{} at location {}, {}, {}, {}".format(name, left, right, top, bot))
            draw.rectangle([left, top, right, bot], outline=(255, 0, 0), width=3)
            draw.text(
                [left, top + font_size],
                "{} {}".format(name, confidence),
                fill=(255, 0, 0),
                font=fnt,
            )
    img.save(filename)
    # 把画完方框的图片传至 blob
    link = upload_blob(CONTAINER, filename)
    # 为了避免一堆图档塞爆 Web App，用完就删掉
    os.remove(filename)
    return link

# Callback for Line chatbot
@app.route("/callback", methods=["POST"])
def callback():
    """
    LINE bot webhook callback
    """
    # get X-Line-Signature header value
    signature = request.headers["X-Line-Signature"]
    print(signature)
    body = request.get_data(as_text=True)
    print(body)
    try:
        HANDLER.handle(body, signature)
    except InvalidSignatureError:
        print(
            "Invalid signature. Please check your channel access token/channel secret."
        )
        abort(400)
    return "OK"

# 处理影像讯息
@HANDLER.add(MessageEvent, message=ImageMessage)
def handle_content_message(event):
    """
    Reply Image message with results of image description and objection detection
    """
    # event 是使用者与 Line 之间的互动事件，可以印出 event 的物件，观察 event。
    # message ID 会作为後续图片存档的名称， user ID 则是作为之後人脸登入的依据。
    print(event.message)
    print(event.source.user_id)
    print(event.message.id)

    # 读取为了产生 flex message 的 json 档
    with open("templates/detect_result.json", "r") as f_h:
        bubble = json.load(f_h)
    f_h.close()
    
    # 以 Line message ID 作为档案名称
    filename = "{}.jpg".format(event.message.id)
    
    # 图片讯息以 binary 的形式传输，取得之後先存成图档
    message_content = LINE_BOT.get_message_content(event.message.id)
    with open(filename, "wb") as f_h:
        for chunk in message_content.iter_content():
            f_h.write(chunk)
    f_h.close()
    
    # 开启档案，为了取得影像指寸
    img = Image.open(filename)
    
    # 上传图片到 Blob
    link = upload_blob(CONTAINER, filename)
    
    # 执行物体侦测和影像描述
    link_ob = azure_object_detection(link, filename)
    output = azure_describe(link)
    link = link_ob

    # 以 flex message 输出结果
    bubble["body"]["contents"][0]["text"] = output
    bubble["header"]["contents"][0]["url"] = link
    # 以影像原尺寸，作为 Flex Message 输出的依据
    bubble["header"]["contents"][0]["aspectRatio"] = "{}:{}".format(
        img.size[0], img.size[1]
    )

    LINE_BOT.reply_message(
        event.reply_token, [FlexSendMessage(alt_text="Report", contents=bubble)]
    )

将上述的Python程序，连同相对应的requirements.txt，透过git push部署到 Azure Web App ， chatbot 就能依照图片的内容，以通顺的英文句子产生说明罗～～

接下来，让 chatbot 也可以认出图片中的文字！