I'm currently working on a custom bot for a Slack workspace to send messages when specific entries are given on a Google Sheet. I'm using a simple Flask setup with the Slack SDK and pydrive. I'm running into an issue where each message that is sent "asynchronusly" (i.e. not triggered by a specific Slack event) gets sent twice when running the Flask application. If I disable the Flask app, I only get one instance of the "Restarting Money Bot" sent. If the server is running, I get two instances, back to back. If the server is running and I respond to a slash command (test_command()), I only get the single response as required. I've attempted to alter the send method (using a raw API call rather than the SDK method) to no avail; I'm not sure where else to go with this issue.
Here's the code that's causing the issues. slack_handle just has simple wrapper functions for the API calls so I can keep Slack tokens and signatures local to one file:
from flask import Flask, request, redirect, make_response, render_template, send_from_directory
import os
import logging
import threading
import time
import json
from flask.wrappers import Response
import slack_handle
import drive_handle
logging.basicConfig(level=logging.DEBUG)
bot = Flask(__name__)
drive_access = drive_handle.gdrive()
#bot.route('/slack/test', methods=["POST"])
def test_command():
if not slack_handle.verifier.is_valid_request(request.get_data(), request.headers):
return make_response("Invalid request", 403)
td_test = threading.Thread(target = test_command_thread)
td_test.start()
return Response(status=200)
def test_command_thread():
slack_handle.sendMoneyBot("Money Bot is active :heavy_dollar_sign:")
slack_handle.sendMoneyBot("Last refreshed " + str(drive_access.access_time) + " minute(s) ago")
slack_handle.sendMoneyBot("Good connection: (" + str(drive_access.access_status) + ")")
# To run prior to server start
slack_handle.sendTreasurer("Restarting Money Bot")
td = threading.Thread(target = drive_access.mainDaemon)
td.daemon = True
td.start()
# Local server debug
if __name__ == "__main__":
port = int(os.environ.get('PORT', 5000))
bot.run(host='0.0.0.0', port=port, debug=True)
Any help would be awesome
Related
I'm using Flask on a project on an embedded system and I'm having performance issues. I'm running gunicorn with one eventlet worker by running:
gunicorn -b 0.0.0.0 --worker-class eventlet -w 1 'app:create_app()'
The problem I'm facing is that, when the MQTT messages start to pour with more cadence, the application starts to use almost all the CPU I have available. My initial thought was that I was handling the messages not ideally but, I even took out my handler, and just receive the messages, and the problem still persists.
I have another python application that subscribes to the same information with the paho client and this is not an issue, so I'm assuming I'm missing something on my Flask application and not the information itself.
My code is:
import eventlet
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager, current_user
from flask_socketio import SocketIO
from flask_mqtt import Mqtt
eventlet.monkey_patch()
#USERS DB
db_alchemy = SQLAlchemy()
#socketIO
socketio = SocketIO(cors_allowed_origins="*", async_mode='eventlet')
# MQTT
mqtt_client = Mqtt()
'''
APPLICATION CREATION
'''
def create_app():
app = Flask(__name__)
if app.config["ENV"] == "production":
app.config.from_object("config.ProductionConfig")
else:
app.config.from_object("config.DevelopmentConfig")
#USERS DB
db_alchemy.init_app(app)
#LoginManager
login_manager = LoginManager()
login_manager.login_view = "auth.login"
login_manager.init_app(app)
#SOCKETIO
socketio.init_app(app)
#FLASK-MQTT
app.config['MQTT_BROKER_URL'] = 'localhost' #
app.config['MQTT_BROKER_PORT'] = 1883
app.config['MQTT_KEEPALIVE'] = 20
app.config['MQTT_TLS_ENABLED'] = False
mqtt_client.init_app(app)
return app
#MQTT
#mqtt_client.on_connect()
def mqtt_on_connect():
mqtt_client.subscribe('testTopic/#', 0)
#mqtt_client.on_disconnect()
def mqtt_on_disconnect():
loggerMqtt.warning(' > Disconnected from broker')
#mqtt_client.on_subscribe()
def mqtt_on_subscribe(client, obj, mid, granted_qos):
pass
#mqtt_client.on_message()
def mqtt_on_message(client, userdata, message):
pass
#mqtt_topicSplitter(client, userdata, message)
As you can see my handler mqtt_topicSplitter is commented but I'm still having performance issues. I've tried adding an sleep command [eventlet.sleep(0.1)] on the on_message handler which solved the CPU consumption problem but resulted on my application being constantly kicked from the broker.
I also tried using other workers (gevent, asyncio, ..) without success. Using the Flask development server is not an option, since is not recommended for production.
I'm sorry if I wasn't clear, but I'm not an expert, please feel free to ask me any questions if needed.
Thanks in advance.
No code here, just a question. I have tried various means to get a streamlit app to run within a flask app. Main reason? Using Flask for user authentication into the streamlit app. Cannot get it to work out. Is it not possible perhaps?
Streamlit uses Tornado to serve HTTP and WebSocket data to its frontend. That is, it’s already its own web server, and is written in an existing web framework; it wouldn’t be trivial to wrap it inside another web framework.
Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user.
Flask is a synchronous web framework and not ideal for WebSockets etc.
Serving an interactive Streamlit app via flask.render_template isn’t feasible, because Streamlit apps are not static; when you interact with your Streamlit app, it is re-running your Python code to generate new results dynamically
Follow these discussions for more info
Integration with flask app
Serve streamlit within flask
import asyncio
import subprocess
from mlflow.server import app as mlflow_app
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.middleware.wsgi import WSGIMiddleware
import uvicorn
from fastapi.logger import logger
import uuid
from config import *
streamlit_app_process = None
streamlit_app_stdout = None
streamlit_app_stderr = None
async def registry_subprocess() -> None:
logger.debug("registry distance_matrix")
global streamlit_app_process
global streamlit_app_stdout
global streamlit_app_stderr
id = str(uuid.uuid1())
streamlit_app_stdout = open(f"/tmp/subprocess_stdout_{''.join(id.split('-'))}", 'w+b')
streamlit_app_stderr = open(f"/tmp/subprocess_stderr_{''.join(id.split('-'))}", 'w+b')
cmd = ['streamlit', 'run', f'{app_dir}/Home.py', f'--server.port={streamlit_app_port}', f'--server.address={streamlit_app_host}']
logger.info(f"subprocess start cmd {cmd}")
streamlit_app_process = subprocess.Popen(cmd, stdout=streamlit_app_stdout.fileno(), stderr=streamlit_app_stderr.fileno())
logger.info(f"subprocess start success {streamlit_app_process.pid} uid:{id}")
await asyncio.sleep(1)
streamlit_app_stdout.flush()
streamlit_app_stderr.flush()
[logger.info(i) for i in streamlit_app_stdout.readlines()]
[logger.info(i) for i in streamlit_app_stderr.readlines()]
async def close_subprocess() -> None:
logger.debug("close subprocess")
try:
streamlit_app_process.kill()
streamlit_app_stdout.flush()
streamlit_app_stderr.flush()
streamlit_app_stdout.close()
streamlit_app_stderr.close()
except Exception as error:
logger.error(error)
application = FastAPI()
application.add_event_handler("startup", registry_subprocess)
application.add_event_handler("shutdown", close_subprocess)
application.add_middleware(
CORSMiddleware,
allow_origins='*',
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
application.mount(f"/{mlflow_app_prefix.strip('/')}", WSGIMiddleware(mlflow_app))
if __name__ == "__main__":
uvicorn.run(application, host=mlflow_app_host, port=int(mlflow_app_port))
I'm writing an automated test in Django to check that a webhook is working on the application. The test sends a bunch of JSON to the webhook and will check that the call has been logged in the database. The problem I'm hitting however is that the test calls the http://localhost URL and the data is thus saved in my local dev database and not in the temporary database created by the test. So I now have no way to check the call has been received.
Whats the right solution for this?
from django.test import TestCase
import requests
from monzo.models import Transaction, RequestLog
class WebhookChecks(TestCase):
fixtures = ['db.json', ]
def test_simple_expense(self):
my_json = '{"type": "transaction.created", REMOVED FOR SECURITY }'
url = 'http://localhost/some_url/webhook/'
headers = {'Content-Type': 'application/json'}
r = requests.post(url, data=my_json, headers=headers)
if not "200" in str(r):
print("Something didn't work out. Error: "+str(r))
self.assertTrue("200" in str(r))
Use Djangos Client with which you can perform requests in your tests.
Example:
from django.test import Client
c = Client()
c.get('/some_url/..')
Another way is to use Djangos LiveServerTestCase.
You can use self.live_server_url instead of directly writing http://localhost.
This testcase sets up a live server which listens to localhost.
I created a flask application that consists of MQTT client which simultaneously subscribe data from esp32 and store it in database.
from flask import Flask, redirect, render_template, request, session, abort,make_response,jsonify, flash, url_for
import paho.mqtt.client as mqtt
import json
import config
import db_access
#mqtt code
def on_message(client, userdata, message):
topic = message.topic
print("line 12 - topic checkpoint - ",topic)
msgDecode=str(message.payload.decode("utf-8","ignore"))
msgJson=json.loads(msgDecode) #decode json data
print("line 15 - json checkpoint - ",type(msgJson))
# deviceID = msgJson["DeviceID"]
# currentCounter = msgJson["Counter"]
# status = msgJson["Status"]
db_access.updateStatus(msgJson["DeviceID"],msgJson["Status"])
app = Flask(__name__,template_folder='templates')
'''Web portal routes'''
#app.route('/device/switch', methods=['POST'])
def switch():
#parameter parsing
deviceID = request.args.get('deviceID')
status = request.args.get('status')
statusMap = {"on":1,"off":0}
#MQTT publish
mqtt_msg = json.dumps({"deviceID":int(deviceID),"status":statusMap[status]})
client.publish(config.MQTT_STATUS_CHANGE_TOPIC,mqtt_msg)
time_over_flag = 0
loop_Counter = 0
while status != db_access.getDeviceStatus(deviceID):
time.sleep(2)
loop_Counter+=1
if loop_Counter ==2:
time_over_flag = 1
break
if time_over_flag:
return make_response(jsonify({"statusChange":False}))
else:
return make_response(jsonify({"statusChange":True}))
if __name__ == "__main__":
db_access.createUserTable()
db_access.insertUserData()
db_access.createDeviceTable()
db_access.insertDeviceData()
print("creating new instance")
client = mqtt.Client("server") #create new instance
client.on_message=on_message #attach function to callback
print("connecting to broker")
client.connect(config.MQTT_BROKER_ADDRESS)
client.loop_start()
print("Subscribing to topic","esp/#")
client.subscribe("esp/#")
app.run(debug=True, use_reloader=False)
This is code in ____init____.py
db_access.py is consist of database operations and config.py consist of configurations.
Will this work in apache?
also, I have no previous experience with WSGI
The problem with launching your code (as included) with a WSGI server, is the part in that last if block, only gets run when you execute that file directly with the python command.
To make this work, I'd try moving that block of code to the top of your file, around here:
app = Flask(__name__,template_folder='templates')
db_access.createUserTable()
db_access.insertUserData()
db_access.createDeviceTable()
db_access.insertDeviceData()
print("creating new instance")
client = mqtt.Client("server") #create new instance
client.on_message=on_message #attach function to callback
print("connecting to broker")
client.connect(config.MQTT_BROKER_ADDRESS)
client.loop_start()
print("Subscribing to topic","esp/#")
client.subscribe("esp/#")
I'd also rename that __init__.py file, to something like server.py as the init file isn't meant to be heavy.
Once you've done that, run it with the development server again and test that it works as expected.
Then install a WSGI server like gunicorn into your virtual enviornment:
pip install gunicorn
And launch the app with gunicorn (this command should work, assuming you renamed your file to server.py):
gunicorn --bind '0.0.0.0:5000' server:app
Then test again.
I am developing a fb chatbot where for specific intents, webhooks are been fired and process via python. The python app is hosted in Heroku cloud. I'm facing a typical problem, whenever any webhook is been fired, it keeps continued to be fired in an infinite loop until the next query from chat is been triggered.
#!/usr/bin/env python
from __future__ import print_function
from future import standard_library
standard_library.install_aliases()
import urllib.request, urllib.parse, urllib.error
import json
import os
import psycopg2
import urlparse
from flask import Flask
from flask import request, render_template
from flask import make_response
# Flask should start in global layout
context = Flask(__name__)
# Webhook requests are coming to this method
#context.route('/webhook', methods=['POST'])
def webhook():
reqContext = request.get_json(silent=True, force=True)
if reqContext.get("result").get("action") == "input.welcome":
return welcome()
elif reqContext.get("result").get("action") == "yahooWeatherForecast":
return weatherhook(reqContext)
elif reqContext.get("result").get("action") == "GoogleSearch":
return searchhook()
else:
print("Good Bye")
I have enabled webhook for 3 intents only. Other intents in api.ai does not have fulfillment (Webhook or Webhook slot filling) enabled.
Can anybody help me in this.
Two things to look for in such case:
We need to send response 200 to facebook to end the response
Need to check whether the message delivery response is enabled on the fb subscription or not. If it is enabled, fb will send delivery response as well, which should be caught on webhook.