how to run mqtt in flask application simultaneously over wsgi - flask

I created a flask application that consists of MQTT client which simultaneously subscribe data from esp32 and store it in database.
from flask import Flask, redirect, render_template, request, session, abort,make_response,jsonify, flash, url_for
import paho.mqtt.client as mqtt
import json
import config
import db_access
#mqtt code
def on_message(client, userdata, message):
topic = message.topic
print("line 12 - topic checkpoint - ",topic)
msgDecode=str(message.payload.decode("utf-8","ignore"))
msgJson=json.loads(msgDecode) #decode json data
print("line 15 - json checkpoint - ",type(msgJson))
# deviceID = msgJson["DeviceID"]
# currentCounter = msgJson["Counter"]
# status = msgJson["Status"]
db_access.updateStatus(msgJson["DeviceID"],msgJson["Status"])
app = Flask(__name__,template_folder='templates')
'''Web portal routes'''
#app.route('/device/switch', methods=['POST'])
def switch():
#parameter parsing
deviceID = request.args.get('deviceID')
status = request.args.get('status')
statusMap = {"on":1,"off":0}
#MQTT publish
mqtt_msg = json.dumps({"deviceID":int(deviceID),"status":statusMap[status]})
client.publish(config.MQTT_STATUS_CHANGE_TOPIC,mqtt_msg)
time_over_flag = 0
loop_Counter = 0
while status != db_access.getDeviceStatus(deviceID):
time.sleep(2)
loop_Counter+=1
if loop_Counter ==2:
time_over_flag = 1
break
if time_over_flag:
return make_response(jsonify({"statusChange":False}))
else:
return make_response(jsonify({"statusChange":True}))
if __name__ == "__main__":
db_access.createUserTable()
db_access.insertUserData()
db_access.createDeviceTable()
db_access.insertDeviceData()
print("creating new instance")
client = mqtt.Client("server") #create new instance
client.on_message=on_message #attach function to callback
print("connecting to broker")
client.connect(config.MQTT_BROKER_ADDRESS)
client.loop_start()
print("Subscribing to topic","esp/#")
client.subscribe("esp/#")
app.run(debug=True, use_reloader=False)
This is code in ____init____.py
db_access.py is consist of database operations and config.py consist of configurations.
Will this work in apache?
also, I have no previous experience with WSGI

The problem with launching your code (as included) with a WSGI server, is the part in that last if block, only gets run when you execute that file directly with the python command.
To make this work, I'd try moving that block of code to the top of your file, around here:
app = Flask(__name__,template_folder='templates')
db_access.createUserTable()
db_access.insertUserData()
db_access.createDeviceTable()
db_access.insertDeviceData()
print("creating new instance")
client = mqtt.Client("server") #create new instance
client.on_message=on_message #attach function to callback
print("connecting to broker")
client.connect(config.MQTT_BROKER_ADDRESS)
client.loop_start()
print("Subscribing to topic","esp/#")
client.subscribe("esp/#")
I'd also rename that __init__.py file, to something like server.py as the init file isn't meant to be heavy.
Once you've done that, run it with the development server again and test that it works as expected.
Then install a WSGI server like gunicorn into your virtual enviornment:
pip install gunicorn
And launch the app with gunicorn (this command should work, assuming you renamed your file to server.py):
gunicorn --bind '0.0.0.0:5000' server:app
Then test again.

Related

Flask MQTT high CPU usage

I'm using Flask on a project on an embedded system and I'm having performance issues. I'm running gunicorn with one eventlet worker by running:
gunicorn -b 0.0.0.0 --worker-class eventlet -w 1 'app:create_app()'
The problem I'm facing is that, when the MQTT messages start to pour with more cadence, the application starts to use almost all the CPU I have available. My initial thought was that I was handling the messages not ideally but, I even took out my handler, and just receive the messages, and the problem still persists.
I have another python application that subscribes to the same information with the paho client and this is not an issue, so I'm assuming I'm missing something on my Flask application and not the information itself.
My code is:
import eventlet
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager, current_user
from flask_socketio import SocketIO
from flask_mqtt import Mqtt
eventlet.monkey_patch()
#USERS DB
db_alchemy = SQLAlchemy()
#socketIO
socketio = SocketIO(cors_allowed_origins="*", async_mode='eventlet')
# MQTT
mqtt_client = Mqtt()
'''
APPLICATION CREATION
'''
def create_app():
app = Flask(__name__)
if app.config["ENV"] == "production":
app.config.from_object("config.ProductionConfig")
else:
app.config.from_object("config.DevelopmentConfig")
#USERS DB
db_alchemy.init_app(app)
#LoginManager
login_manager = LoginManager()
login_manager.login_view = "auth.login"
login_manager.init_app(app)
#SOCKETIO
socketio.init_app(app)
#FLASK-MQTT
app.config['MQTT_BROKER_URL'] = 'localhost' #
app.config['MQTT_BROKER_PORT'] = 1883
app.config['MQTT_KEEPALIVE'] = 20
app.config['MQTT_TLS_ENABLED'] = False
mqtt_client.init_app(app)
return app
#MQTT
#mqtt_client.on_connect()
def mqtt_on_connect():
mqtt_client.subscribe('testTopic/#', 0)
#mqtt_client.on_disconnect()
def mqtt_on_disconnect():
loggerMqtt.warning(' > Disconnected from broker')
#mqtt_client.on_subscribe()
def mqtt_on_subscribe(client, obj, mid, granted_qos):
pass
#mqtt_client.on_message()
def mqtt_on_message(client, userdata, message):
pass
#mqtt_topicSplitter(client, userdata, message)
As you can see my handler mqtt_topicSplitter is commented but I'm still having performance issues. I've tried adding an sleep command [eventlet.sleep(0.1)] on the on_message handler which solved the CPU consumption problem but resulted on my application being constantly kicked from the broker.
I also tried using other workers (gevent, asyncio, ..) without success. Using the Flask development server is not an option, since is not recommended for production.
I'm sorry if I wasn't clear, but I'm not an expert, please feel free to ask me any questions if needed.
Thanks in advance.

How to stop double message send in Flask based Slack application

I'm currently working on a custom bot for a Slack workspace to send messages when specific entries are given on a Google Sheet. I'm using a simple Flask setup with the Slack SDK and pydrive. I'm running into an issue where each message that is sent "asynchronusly" (i.e. not triggered by a specific Slack event) gets sent twice when running the Flask application. If I disable the Flask app, I only get one instance of the "Restarting Money Bot" sent. If the server is running, I get two instances, back to back. If the server is running and I respond to a slash command (test_command()), I only get the single response as required. I've attempted to alter the send method (using a raw API call rather than the SDK method) to no avail; I'm not sure where else to go with this issue.
Here's the code that's causing the issues. slack_handle just has simple wrapper functions for the API calls so I can keep Slack tokens and signatures local to one file:
from flask import Flask, request, redirect, make_response, render_template, send_from_directory
import os
import logging
import threading
import time
import json
from flask.wrappers import Response
import slack_handle
import drive_handle
logging.basicConfig(level=logging.DEBUG)
bot = Flask(__name__)
drive_access = drive_handle.gdrive()
#bot.route('/slack/test', methods=["POST"])
def test_command():
if not slack_handle.verifier.is_valid_request(request.get_data(), request.headers):
return make_response("Invalid request", 403)
td_test = threading.Thread(target = test_command_thread)
td_test.start()
return Response(status=200)
def test_command_thread():
slack_handle.sendMoneyBot("Money Bot is active :heavy_dollar_sign:")
slack_handle.sendMoneyBot("Last refreshed " + str(drive_access.access_time) + " minute(s) ago")
slack_handle.sendMoneyBot("Good connection: (" + str(drive_access.access_status) + ")")
# To run prior to server start
slack_handle.sendTreasurer("Restarting Money Bot")
td = threading.Thread(target = drive_access.mainDaemon)
td.daemon = True
td.start()
# Local server debug
if __name__ == "__main__":
port = int(os.environ.get('PORT', 5000))
bot.run(host='0.0.0.0', port=port, debug=True)
Any help would be awesome

run discord bot inside flask

I would like to have a strife bot that accesses the Flask Database (Flask SQLAlchemy). I would like to know if there is a way to make the Discord bot run and managed via the Flask Web Application, how to connect the two so that they can interact with each other
Thanks, Regard
here is my flask configuration
# bot Client Config
client = commands.Bot(command_prefix='!', case_insensitive=True)
# Load Token Config
load_dotenv('.env')
# Token Config
TOKEN = os.getenv('TOKEN')
# Picture Extension
pict_ext = ('.jpg','.png','.jpeg')
#
# Flask Configuration
def create_app(script_info=None):
# environtment configurations
# module import
from projects.dashboard.views import dashboard_blueprint
# initialize app
app = Flask(__name__, static_url_path='')
# register blueprint
app.register_blueprint(dashboard_blueprint)
return app
running discord bot in the separate thread using thread module like this
# thread function
def flask_thread(func):
thread = Thread(target=func)
print('Start Separate Thread From Bot')
thread.start()
then add it to the main file
# main file
# function run flask with separate Thread
def run():
app.run(host='0.0.0.0', port=10000, use_reloader=False)
if __name__ == '__main__':
flask_thread(func=run)
client.run(os.getenv('TOKEN'))
running with command
python main.py
and see http://localhost:10000 in your browser

Why flask doesn't work with gunicorn and telegram bot api?

I am using telegram bot api (telebot), flask and gunicorn.
When I use command python app.py everything is work fine but when I use python wsgi.py flask is stated on http://127.0.0.1:5000/ and bot doesn't answer and if I am using gunicorn --bind 0.0.0.0:8443 wsgi:app webhook is setting but telegram bot doesn't answer. I tried to add app.run from app.py to wsgi.py but it doesn't work
app.py
import logging
import time
import flask
import telebot
API_TOKEN = '111111111:token_telegram'
WEBHOOK_HOST = 'droplet ip'
WEBHOOK_PORT = 8443 # 443, 80, 88 or 8443 (port need to be 'open')
WEBHOOK_LISTEN = '0.0.0.0' # In some VPS you may need to put here the IP addr
WEBHOOK_SSL_CERT = 'webhook_cert.pem' # Path to the ssl certificate
WEBHOOK_SSL_PRIV = 'webhook_pkey.pem' # Path to the ssl private key
WEBHOOK_URL_BASE = "https://%s:%s" % (WEBHOOK_HOST, WEBHOOK_PORT)
WEBHOOK_URL_PATH = "/%s/" % (API_TOKEN)
logger = telebot.logger
telebot.logger.setLevel(logging.DEBUG)
bot = telebot.TeleBot(API_TOKEN)
app = flask.Flask(__name__)
# Empty webserver index, return nothing, just http 200
#app.route('/', methods=['GET', 'HEAD'])
def index():
return ''
# Process webhook calls
#app.route(WEBHOOK_URL_PATH, methods=['POST'])
def webhook():
if flask.request.headers.get('content-type') == 'application/json':
json_string = flask.request.get_data().decode('utf-8')
update = telebot.types.Update.de_json(json_string)
bot.process_new_updates([update])
return ''
else:
flask.abort(403)
# Handle all other messages
#bot.message_handler(func=lambda message: True, content_types=['text'])
def echo_message(message):
bot.reply_to(message, message.text)
# Remove webhook, it fails sometimes the set if there is a previous webhook
bot.remove_webhook()
#
time.sleep(1)
# Set webhook
bot.set_webhook(url=WEBHOOK_URL_BASE + WEBHOOK_URL_PATH,
certificate=open(WEBHOOK_SSL_CERT, 'r'))
if __name__ == "__main__":
# Start flask server
app.run(host=WEBHOOK_LISTEN,
port=WEBHOOK_PORT,
ssl_context=(WEBHOOK_SSL_CERT, WEBHOOK_SSL_PRIV),
debug=True)
wsgi.py
from app import app
if __name__ == "__main__":
app.run()
It's a bad practice to show python webservers to the world. It's not secure.
Good practice - using a reverse proxy, e.g. nginx
Chain
So the chain shoud be:
api.telegram.org -> your_domain -> your_nginx -> your_webserver -> your_app
SSL
Your ssl certificates shoud be checked on nginx level. On success just pass request to your webserver (flask or something else). It's also a tip about how to use infitity amount of bots on one host/port :)
How to configure nginx reverse proxy - you can find in startoverflow or google.
Telegram Webhooks
telebot is so complicated for using webhooks.
Try to use aiogram example. It's pretty simple:
from aiogram import Bot, Dispatcher, executor
from aiogram.types import Message
WEBHOOK_HOST = 'https://your.domain'
WEBHOOK_PATH = '/path/to/api'
bot = Bot('BOT:TOKEN')
dp = Dispatcher(bot)
#dp.message_handler()
async def echo(message: Message):
return message.answer(message.text)
async def on_startup(dp: Dispatcher):
await bot.set_webhook(f"{WEBHOOK_HOST}{WEBHOOK_PATH}")
if __name__ == '__main__':
executor.start_webhook(dispatcher=dp, on_startup=on_startup,
webhook_path=WEBHOOK_PATH, host='localhost', port=3000)
app.run(host=0.0.0.0, port=5000, debug=True)
Change port number and debug arguments based on your case.

Watson assistant deployment on Flask+WSGI server (gunicorn or wsgi)

I am deploying my watson assistant chatbot on Flask + Gunicorn + Nginx.
I am able to successfully dockerize and run , but something is breaking my code. Multiple watson assistant sessions are being created while I send the messages to watson services. While I try to reply for an intent I get answer for another intent or slot or does not understand message
I have reviewed all the tutorials on digital ocean and github, but I think creating chatbot session should be handled differently.
app.py
from flask import Flask, render_template,Response,make_response,jsonify
import os
from ibm_watson import AssistantV2
import random
from random import randint
import json
#import report
from io import StringIO
app = Flask(__name__)
conversation = AssistantV2(
iam_apikey = 'key',
url='https://gateway.watsonplatform.net/assistant/api',
version='2019-05-19')
session = conversation.create_session("someid").get_result()
variables = None
#context_val = {}
#app.route('/')
#app.route('/index')
def chat():
return render_template('chat.html')
#app.route('/send_message/<message>')
def send_mesage(message):
text = ''
response = conversation.message(
assistant_id = 'id',
session_id= session['session_id'],input={'text': str(message),'options': {
'return_context': True}}
).get_result()
variables = response['output'].get('user_defined')
#context = response['context']['skills']['main skill']['user_defined']
for i in response['output']['generic']:
text = text+ i['text']+'\n'
return text
if __name__ == "__main__":
app.run(host='0.0.0.0')
wsgi.py
from app import app
if __name__ == "__main__":
app.run()
Dockerfile
FROM python:3.6
WORKDIR /app
ADD . /app
RUN chgrp -R 0 /app/app.log && chmod -R g=u /app/app.log
RUN pip install -r requirements.txt
EXPOSE 8080
CMD ["gunicorn", "-b", "0.0.0.0:8080", "app", "-p 8080:8080"]
RUN chmod 770 /app
USER 1001
When working with IBM Watson Assistant with the V2 API, you need to be aware of the following objects:
First, you create an Assistant. It manages the connection to Watson Assistant.
Next, a Session is a per user interaction within the chat.
Last, a Message flows to Watson within a session with a Response coming back.
You probably have seen this simple code sample in the docs, your own code is - on a general level - similar. To make it work, you need to create Watson sessions per user sessions, then send the messages as part of the corresponding session. That way, the chat context is kept correctly. Your code currently initialize Watson and creates a session once. You need to create a session per user. Look into session management.