I'm using Flask on a project on an embedded system and I'm having performance issues. I'm running gunicorn with one eventlet worker by running:
gunicorn -b 0.0.0.0 --worker-class eventlet -w 1 'app:create_app()'
The problem I'm facing is that, when the MQTT messages start to pour with more cadence, the application starts to use almost all the CPU I have available. My initial thought was that I was handling the messages not ideally but, I even took out my handler, and just receive the messages, and the problem still persists.
I have another python application that subscribes to the same information with the paho client and this is not an issue, so I'm assuming I'm missing something on my Flask application and not the information itself.
My code is:
import eventlet
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager, current_user
from flask_socketio import SocketIO
from flask_mqtt import Mqtt
eventlet.monkey_patch()
#USERS DB
db_alchemy = SQLAlchemy()
#socketIO
socketio = SocketIO(cors_allowed_origins="*", async_mode='eventlet')
# MQTT
mqtt_client = Mqtt()
'''
APPLICATION CREATION
'''
def create_app():
app = Flask(__name__)
if app.config["ENV"] == "production":
app.config.from_object("config.ProductionConfig")
else:
app.config.from_object("config.DevelopmentConfig")
#USERS DB
db_alchemy.init_app(app)
#LoginManager
login_manager = LoginManager()
login_manager.login_view = "auth.login"
login_manager.init_app(app)
#SOCKETIO
socketio.init_app(app)
#FLASK-MQTT
app.config['MQTT_BROKER_URL'] = 'localhost' #
app.config['MQTT_BROKER_PORT'] = 1883
app.config['MQTT_KEEPALIVE'] = 20
app.config['MQTT_TLS_ENABLED'] = False
mqtt_client.init_app(app)
return app
#MQTT
#mqtt_client.on_connect()
def mqtt_on_connect():
mqtt_client.subscribe('testTopic/#', 0)
#mqtt_client.on_disconnect()
def mqtt_on_disconnect():
loggerMqtt.warning(' > Disconnected from broker')
#mqtt_client.on_subscribe()
def mqtt_on_subscribe(client, obj, mid, granted_qos):
pass
#mqtt_client.on_message()
def mqtt_on_message(client, userdata, message):
pass
#mqtt_topicSplitter(client, userdata, message)
As you can see my handler mqtt_topicSplitter is commented but I'm still having performance issues. I've tried adding an sleep command [eventlet.sleep(0.1)] on the on_message handler which solved the CPU consumption problem but resulted on my application being constantly kicked from the broker.
I also tried using other workers (gevent, asyncio, ..) without success. Using the Flask development server is not an option, since is not recommended for production.
I'm sorry if I wasn't clear, but I'm not an expert, please feel free to ask me any questions if needed.
Thanks in advance.
Related
Here's a test block of code that can be used to repeat the issue I'm experiencing
from gevent import monkey as curious_george
curious_george.patch_all(thread=True)
from flask_socketio import SocketIO
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = "my_custom_uri"
app.config['SQLALCHEMY_POOL_SIZE'] = 20
app.config['SQLALCHEMY_MAX_OVERFLOW'] = 50
db = SQLAlchemy(app)
socketio = SocketIO(app)
class test_table(db.Model):
_id = db.Column("id",db.Integer,primary_key=True)
flagged = db.Column("flagged",db.Boolean(), default=False)
def __init__(self,flagged):
self.flagged = flagged
#app.route('/test')
def test_route():
for x in range(100):
test_flag = db.session.query(test_table).filter_by(_id=1).first()
test_flag.flagged = not test_flag.flagged
db.session.commit()
# print(x)
socketio.emit('x_test',x, broadcast=True)
return("success",200)
if __name__ == '__main__':
socketio.run(app, host='0.0.0.0', port=8080)
The database commits happen first and once they finish all of the socketio emits dump (in order) at once.
Maybe I have something misconfigured? Does sqlalchemy need to be handled in a more concurrent method?
I just reinstalled my virtual environment for other reasons
Flask-Socketio 5.2.0
python-engineio 4.3.4
python-socketio 5.7.2
gevent 22.10.2
gevent-websocket 0.10.1
I don't have any related warnings on the flask server startup
I expected the socketio emits to happen in time with the database query commits
Turns out I needed to patch psycopg2 as well using the "psycogreen" library
I used this patch to fix it, and it works now.
from psycogreen import gevent
gevent.patch_psycopg()
I'm currently working on a custom bot for a Slack workspace to send messages when specific entries are given on a Google Sheet. I'm using a simple Flask setup with the Slack SDK and pydrive. I'm running into an issue where each message that is sent "asynchronusly" (i.e. not triggered by a specific Slack event) gets sent twice when running the Flask application. If I disable the Flask app, I only get one instance of the "Restarting Money Bot" sent. If the server is running, I get two instances, back to back. If the server is running and I respond to a slash command (test_command()), I only get the single response as required. I've attempted to alter the send method (using a raw API call rather than the SDK method) to no avail; I'm not sure where else to go with this issue.
Here's the code that's causing the issues. slack_handle just has simple wrapper functions for the API calls so I can keep Slack tokens and signatures local to one file:
from flask import Flask, request, redirect, make_response, render_template, send_from_directory
import os
import logging
import threading
import time
import json
from flask.wrappers import Response
import slack_handle
import drive_handle
logging.basicConfig(level=logging.DEBUG)
bot = Flask(__name__)
drive_access = drive_handle.gdrive()
#bot.route('/slack/test', methods=["POST"])
def test_command():
if not slack_handle.verifier.is_valid_request(request.get_data(), request.headers):
return make_response("Invalid request", 403)
td_test = threading.Thread(target = test_command_thread)
td_test.start()
return Response(status=200)
def test_command_thread():
slack_handle.sendMoneyBot("Money Bot is active :heavy_dollar_sign:")
slack_handle.sendMoneyBot("Last refreshed " + str(drive_access.access_time) + " minute(s) ago")
slack_handle.sendMoneyBot("Good connection: (" + str(drive_access.access_status) + ")")
# To run prior to server start
slack_handle.sendTreasurer("Restarting Money Bot")
td = threading.Thread(target = drive_access.mainDaemon)
td.daemon = True
td.start()
# Local server debug
if __name__ == "__main__":
port = int(os.environ.get('PORT', 5000))
bot.run(host='0.0.0.0', port=port, debug=True)
Any help would be awesome
I created a flask application that consists of MQTT client which simultaneously subscribe data from esp32 and store it in database.
from flask import Flask, redirect, render_template, request, session, abort,make_response,jsonify, flash, url_for
import paho.mqtt.client as mqtt
import json
import config
import db_access
#mqtt code
def on_message(client, userdata, message):
topic = message.topic
print("line 12 - topic checkpoint - ",topic)
msgDecode=str(message.payload.decode("utf-8","ignore"))
msgJson=json.loads(msgDecode) #decode json data
print("line 15 - json checkpoint - ",type(msgJson))
# deviceID = msgJson["DeviceID"]
# currentCounter = msgJson["Counter"]
# status = msgJson["Status"]
db_access.updateStatus(msgJson["DeviceID"],msgJson["Status"])
app = Flask(__name__,template_folder='templates')
'''Web portal routes'''
#app.route('/device/switch', methods=['POST'])
def switch():
#parameter parsing
deviceID = request.args.get('deviceID')
status = request.args.get('status')
statusMap = {"on":1,"off":0}
#MQTT publish
mqtt_msg = json.dumps({"deviceID":int(deviceID),"status":statusMap[status]})
client.publish(config.MQTT_STATUS_CHANGE_TOPIC,mqtt_msg)
time_over_flag = 0
loop_Counter = 0
while status != db_access.getDeviceStatus(deviceID):
time.sleep(2)
loop_Counter+=1
if loop_Counter ==2:
time_over_flag = 1
break
if time_over_flag:
return make_response(jsonify({"statusChange":False}))
else:
return make_response(jsonify({"statusChange":True}))
if __name__ == "__main__":
db_access.createUserTable()
db_access.insertUserData()
db_access.createDeviceTable()
db_access.insertDeviceData()
print("creating new instance")
client = mqtt.Client("server") #create new instance
client.on_message=on_message #attach function to callback
print("connecting to broker")
client.connect(config.MQTT_BROKER_ADDRESS)
client.loop_start()
print("Subscribing to topic","esp/#")
client.subscribe("esp/#")
app.run(debug=True, use_reloader=False)
This is code in ____init____.py
db_access.py is consist of database operations and config.py consist of configurations.
Will this work in apache?
also, I have no previous experience with WSGI
The problem with launching your code (as included) with a WSGI server, is the part in that last if block, only gets run when you execute that file directly with the python command.
To make this work, I'd try moving that block of code to the top of your file, around here:
app = Flask(__name__,template_folder='templates')
db_access.createUserTable()
db_access.insertUserData()
db_access.createDeviceTable()
db_access.insertDeviceData()
print("creating new instance")
client = mqtt.Client("server") #create new instance
client.on_message=on_message #attach function to callback
print("connecting to broker")
client.connect(config.MQTT_BROKER_ADDRESS)
client.loop_start()
print("Subscribing to topic","esp/#")
client.subscribe("esp/#")
I'd also rename that __init__.py file, to something like server.py as the init file isn't meant to be heavy.
Once you've done that, run it with the development server again and test that it works as expected.
Then install a WSGI server like gunicorn into your virtual enviornment:
pip install gunicorn
And launch the app with gunicorn (this command should work, assuming you renamed your file to server.py):
gunicorn --bind '0.0.0.0:5000' server:app
Then test again.
I have a simple flask app, say like this:
# app.py
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
I also have a slack bot reading messages
#bot.py
def serve(self):
while True:
message, channel = self.parse_slack_output(self.slack_client.rtm_read())
if message and channel:
self.handle_message(message, channel)
time.sleep(self.READ_WEBSOCKET_DELAY)
I want both the codes to run concurrently. So in app.py I do:
#app.py
if __name__ == "__main__":
import threading
import bot
flask_process = threading.Thread(target=app.run)
bot_process = threading.Thread(target=bot.serve)
bot_thread.start()
flask_thread.start()
This code works as expected with $ python app.py, But when I bring in gunicorn the bot thread doesn't seem to work.
I have tried:
gunicorn app:app
gunicorn --workers=2 app:app
gunicorn --threads=2 app:app
I also tried the multiprocessing library and got the same results.
Any idea how this issue can be tackled? Thanks.
Edit: I now understand how lame this question is. I shouldn't be writing code in if __name__ = "__main__": block. That is not what is run by gunicorn. It directly picks up the app and runs it. Still have to figure how to make it handle the bot thread.
I have made this work with the following solution:
# app.py
from flask import Flask
import threading
import bot
def create_app():
app = Flask(__name__)
bot_process = threading.Thread(target=bot.serve)
return app
app = create_app()
#app.route('/')
def hello_world():
return 'Hello, World!'
This makes sure that gunicorn --workers=1 app:app runs both the app and the bot in different threads. While this works, one drawback with this solution is I am not able to scale up the number of workers to > 1. As this would not only scale the app thread, but also the bot thread, which I don't want. The bot would then unnecessarily listen for messages in two threads.
Any better solution in your mind? please convey it. Thanks.
I'm beginner in twisted world, so first I'm trying to get my working django project configured under twisted,currently its working well on django testing server or apache via mod_wsgi.
I followed this link and this too to configure the setup, based on that I have a server.py file given bellow
So in-order to integrate django app with twisted I used the following code,
import sys
import os
from twisted.application import internet, service
from twisted.web import server, resource, wsgi, static
from twisted.python import threadpool
from twisted.internet import reactor
from django.conf import settings
import twresource # This file hold implementation of "Class Root".
class ThreadPoolService(service.Service):
def __init__(self, pool):
self.pool = pool
def startService(self):
service.Service.startService(self)
self.pool.start()
def stopService(self):
service.Service.stopService(self)
self.pool.stop()
class Root(resource.Resource):
def __init__(self, wsgi_resource):
resource.Resource.__init__(self)
self.wsgi_resource = wsgi_resource
def getChild(self, path, request):
path0 = request.prepath.pop(0)
request.postpath.insert(0, path0)
return self.wsgi_resource
PORT = 8080
# Environment setup for your Django project files:
#insert it to first so our project will get first priority.
sys.path.insert(0,"django_project")
sys.path.insert(0,".")
os.environ['DJANGO_SETTINGS_MODULE'] = 'django_project.settings'
from django.core.handlers.wsgi import WSGIHandler
def wsgi_resource():
pool = threadpool.ThreadPool()
pool.start()
# Allow Ctrl-C to get you out cleanly:
reactor.addSystemEventTrigger('after', 'shutdown', pool.stop)
wsgi_resource = wsgi.WSGIResource(reactor, pool, WSGIHandler())
return wsgi_resource
# Twisted Application Framework setup:
application = service.Application('twisted-django')
# WSGI container for Django, combine it with twisted.web.Resource:
# XXX this is the only 'ugly' part: see the 'getChild' method in twresource.Root
wsgi_root = wsgi_resource()
root = Root(wsgi_root)
#multi = service.MultiService()
#pool = threadpool.ThreadPool()
#tps = ThreadPoolService(pool)
#tps.setServiceParent(multi)
#resource = wsgi.WSGIResource(reactor, tps.pool, WSGIHandler())
#root = twresource.Root(resource)
#Admin Site media files
#staticrsrc = static.File(os.path.join(os.path.abspath("."), "/usr/haridas/eclipse_workplace/skgargpms/django/contrib/admin/media/"))
#root.putChild("admin/media", staticrsrc)
# Serve it up:
main_site = server.Site(root)
#internet.TCPServer(PORT, main_site).setServiceParent(multi)
internet.TCPServer(PORT, main_site).setServiceParent(application)
#EOF.
Using above code It worked well from command line using "twisted -ny server.py", but when we run it as daemon "twisted -y server.py" it will hang, but the app is listening to the port 8080. I can access it using telnet.
I found some fixes for this hanging issue from stackoverflow itself. It helped me to use the code sections given below, which is commented in the above server.py file.
multi = service.MultiService()
pool = threadpool.ThreadPool()
tps = ThreadPoolService(pool)
tps.setServiceParent(multi)
resource = wsgi.WSGIResource(reactor, tps.pool, WSGIHandler())
root = twresource.Root(resource)
and :-
internet.TCPServer(PORT, main_site).setServiceParent(multi)
instead of using the:-
wsgi_root = wsgi_resource()
root = Root(wsgi_root)
and :-
internet.TCPServer(PORT, main_site).setServiceParent(application)
The modified method also didn't helped me to avoid the hanging issue.Is any body out there who successfully run the django apps under twisted daemon mode?.
I maid any mistakes while combining these codes?, Currently I'm only started to learn the twisted architectures in detail. Please help me to solve this problem
I'm looking for the Twisted Application configuration (TAC) file, which integrate django app with twisted and run with out any problem in the daemon mode also.
Thanks and Regards,
Haridas N.
I think you are almost there. Just add one more line at the very end:
multi.setServiceParent(application)