Python Flask App with MQTT - flask

I'm trying to use my Flask app as a subscriber, but it does not call the on_message callback when receiving a message. Instead, I get about like the following:
Connected with result code 0
Closing data file...
Connected with result code 0
Closing data file...
This is how I am running the Flask app:
main.py:
from flask import Flask, render_template, redirect, url_for
from flask_bootstrap import Bootstrap
from flask_nav import Nav
from flask_nav.elements import *
import paho.mqtt.client as mqtt
import time
broker_address = <broker_ip>
port = 1883
timeout = 60
username = "first"
password = "last"
uuid = "1234"
topic = "mytopic"
qos = 0
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
client.subscribe(topic)
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload) + '\n')
def on_disconnect(client, userdata, rc):
print("Closing data file...")
client = mqtt.Client(client_id=uuid)
client.on_connect = on_connect
client.on_message = on_message
client.on_disconnect = on_disconnect
client.username_pw_set(username, password)
client.connect(broker_address, port, 60)
client.loop_start()
<other Flask code>
if __name__ == "__main__":
app.run(debug=True)
I've tried using a another Python script to generate some fake data to publish to the topic, but only when that script is running to I get the output above. If that script is not running, then the main.py seems to wait for messages. This is the other script:
fake_data.py:
import paho.mqtt.client as mqtt
import time
broker_address = <broker_ip>
port = 1883
timeout = 60
username = "first"
password = "last"
uuid = "1234"
topic = "mytopic"
qos = 0
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
client = mqtt.Client(client_id=uuid, clean_session=True)
client.on_connect = on_connect
client.username_pw_set(username, password)
client.connect(broker_address, port, 60)
client.loop_start()
while True:
count = 0
while count != 30:
print("Publishing {0}".format(count))
client.publish(topic, count, qos=0)
count += 1
time.sleep(1)
My question is why the Flask app keeps connecting and disconnecting endlessly without actually processing a message.

The client id needs to be different for all clients connected to a broker. In the code you've posted both the subscriber and the publisher are using the same client id (uuid=1234).
When 2 clients with the same client id connect to the broker, the broker will kick the oldest off. If this is then set to reconnect it will then kick the second one off.
Set uuid to different values.

I know this is from a long time ago...but I believe I know the answer! In case anyone else is having the same problem (disconnecting and reconnecting): your app has to be set to keep looping the client. "client.loop_start()" is actually behaving as designed in your case. If you want the connection to stay open for a program that runs indefinitely, replace that with "client.loop_forever()".
My First Answer!
Cheers!

Related

Unable to send emails through Flask-mail

I have been trying to create a web app which takes email address as an input through HTML form and sends a one time pin for further access to website.
I have 2 html files in my template folder (one for taking user's email address and other for OTP entering)
i have config.json file which stores my accountid and password through which i intend to send the OTP.
.py file
from flask import Flask, render_template, request
from random import randint
import json
from flask_mail import *
with open('config.json','r') as f:
params = json.load(f)['params']
mail = Mail(app)
otp = randint(100,999) #otp production
app.config['MAIL_SERVER'] = 'smtp.gmail.com'
app.config['MAIL_PORT'] = 465
app.config['MAIL_USE_TLS'] = False
app.config['MAIL_USE_SSL'] = True
app.config['MAIL_USERNAME'] = params['gmail-user']
app.config['MAIL_PASSWORD'] = params['gmail-password']
#app.route('/')
def home():
return(render_template('otpgive.html'))
#app.route('/getOTP', methods = ['POST'])
def getOTP(): #OTP send and Verify here
email_id = request.form['usermail']
msg = Message('OTP for the APPLICATION', sender = 'my_email', recipients = [email_id])
#my_email is the email through which i plan to send messages.
msg.body = "Greetings! Your email has been verified successfully. Kindly note your OTP is " + str(otp)
mail.send(msg)
return render_template('otpconfirm.html')
#app.route('/validateOTP', methods = ['POST'])
def validateOTP():
userotp = request.form['otp']
if (otp == int(userotp)):
return ("Email Verified Succcessfully ")
else:
return render_template('otpgive.html',msg = 'incorrect otp')
if __name__ == '__main__':
app.run(debug = False)
#app.run(host='0.0.0.0',port=5000, debug = True)
Solutions I tried but failed:
Tried disabling the firewall.
Tried setting the port number for 0.0.0.0
Tried debug = False
I was expecting it to work. i.e send emails to the users but it shows ConnectionError or InternalServerError
Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
ConnectionRefusedError:
[WinError 10061] No connection could be made because the target machine actively refused it
I finally got the solution.
Since I was using G-Mail, I had to enable 2FA (2-Factor Auth) first on my profile, and then generate a new password for my app.
The password thus obtained was pasted in the config.json file instead of my
main profile password.
Used directions from this thread Less Secure option in gmail unavailable
Now Changes I made in my code:
Reverted back to ip: host='0.0.0.0', port=5000, debug = True
I kept firewall disabled as a precaution.
I repositioned mail = Mail(app) line to after all app.configs

How to send socket messages via Django views when socket server and views.py are split into two files?

Env: Python 3.6, and Django 2.1
I have created a Django website and a socket server, and files are organized like this:
web
...
user (a Django app)
__init__.py
views.py
...
server.py
Actually I want to build a umbrella rental system by using django, and server connects to umbrella shelf via multi-thread socket (sending some messages). Like I press the borrow button, and views.py can call the server test_function and send some messages to the connected umbrella shelf.
I can import server variables or functions in views.py, but I cannot get the right answer while server.py is running. So I want to ask you if you could give me some advice. Thanks a lot!
By the way, I tried to import the global variable clients directly in views.py, but still got [].
server.py defines a multi-thread server, which is basically as below:
clients = []
class StuckThread(threading.Thread):
def __init__(self, **kwargs):
self.name = kwargs.get('name', '')
def run(self):
while True:
# do something
def func1(self):
# do something
def test_function(thread_name):
# if the function is called by `views.py`, then `clients = []` and return 'nothing', but if I call this function in `server.py`, then I can get a wanted result, which is `got the thread`
for client in clients:
if client['thread'].name == thread_name:
return 'got the thread'
return 'nothing'
if __name__ == '__main__':
ip_port = ('0.0.0.0', 65432)
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(ip_port)
server.listen(max_listen_num)
while True:
client, address = socket.accept()
param = {'name': 'test name'}
stuck_thread = StuckThread(**param)
clients.append({"client": client, "address": address, "thread": stuck_thread})
stuck_thread.start()
and I have a Django views.py like this
def view_function(request):
from server import clients
print(clients) # got []
form server import test_function
print(test_function('test name')) # got 'nothing'
return render(request, 'something.html')
I have solve this problem by socket communication between django views.py and server.py. I open another port to receive messages from views.py. Once the borrow button is pressed, a socket client in views.py will build up and send arguments and other messages to the server.

PYMQI return data type issue

I am trying the get IBM MQ Messages out of Queue Manager by reading the queue "SYSTEM.ADMIN.COMMAND.EVENT".
I am getting the following response when I print the message.
How do I convert the following message into a user friendly format after reading the queue:
b'\x00\x00\x00\x07\x00\x00\x00$\x00\x00\x00\x03\x00\x00\x00c\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\tm\x00\x00\x00\x02\x00\x00\x00\x14\x00\x00\x00\x10\x00\x00\x1fA\x00\x00\x00\t\x00\x00\x00\x04\x00\x00\x00
\x00\x00\x0b\xe5\x00\x00\x033\x00\x00\x00\x0cnb153796
\x00\x00\x00\x03\x00\x00\x00\x10\x00\x00\x03\xf3\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00D\x00\x00\x0b\xe7\x00\x00\x033\x00\x00\x000ZANC000.YODA
\x00\x00\x00\t\x00\x00\x000\x00\x00\x1bY\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x004\x00\x00\x0b\xe9\x00\x00\x033\x00\x00\x00
\x00\x00\x00\x03\x00\x00\x00\x10\x00\x00\x03\xf2\x00\x00\x00\x1c\x00\x00\x00\x04\x00\x00\x000\x00\x00\x0b\xea\x00\x00\x033\x00\x00\x00\x1cMQ
Explorer 9.0.0
\x00\x00\x00\x04\x00\x00\x00\x18\x00\x00\x0b\xeb\x00\x00\x033\x00\x00\x00\x04
\x00\x00\x00\x03\x00\x00\x00\x10\x00\x00\x03\xfd\x00\x00\x00\xa1\x00\x00\x00\x14\x00\x00\x00\x10\x00\x00\x1fB\x00\x00\x00\x01\x00\x00\x00\x05\x00\x00\x00\x14\x00\x00\x04\xcd\x00\x00\x00\x01\x00\x00\x03\xf1'
My code is as follows, I am using Python 3.6 and Django 2.1 everything is working fine except the format or the returned message:
Please assist in figuring out how to convert the returned message from PYMQI Queue Get into a user readable format:
#
from django.shortcuts import render
from django.http import HttpResponse
import pymqi
#
queue_manager = "QueueManager"
channel = "Channel"
host = "hostname"
port = "port"
queue_name = "SYSTEM.ADMIN.COMMAND.EVENT"
user = "username"
password = "password"
connection_info = "%s(%s)" % (host, port)
#
#(option, args) = OptionParser
# Create your views here.
def index(request):
#
qmgr = pymqi.connect(queue_manager, channel, connection_info, user, password)
#
queue = pymqi.Queue(qmgr, queue_name)
print(queue.get())
messages = queue.get()
#for message in messages:
# print(type(message))
queue.close()
qmgr.disconnect()
return HttpResponse("MQ Hello Tests %s" % messages.hex())

xlwings + Django: how to not loose a connection

I am trying to deploy a spreadsheet model with a web page front end using Django. The web "app" flow is simple:
User enters data in a web form
Send form data to a Django backend view function "run_model(request)"
Parse request object to get user inputs and then populate named ranges in the excel model's input sheet using xlwings to interact with a spreadsheet model (sheet.range function is used)
Run "calculate()" on the spreadsheet
Read outputs from another tab in the spreadsheet using xlwings and named ranges (again, using sheet.range function).
The problem is that the connection to the Excel process keeps getting killed by Django (I believe it handles each request as a separate process), so I can get at most one request to work (by importing xlwings inside the view function) but when I send a second request, the connection is dead and it won't reactivate.
Basically, how can I keep the connection to the workbook alive between requests or at least re-open a connection for each request?
Ok, ended up implementing a simple "spreadsheet server" to address the issue with Django killing the connection.
First wrote code for the server (server.py), then some code to start it up from command line args (start_server.py), then had my view open a connection to this model when it needs to use it (in views.py).
So, I had to separate my (Excel + xlwings) and Django into independent processes to keep the interfaces clean and control how much access Django has to my speadsheet model. Works fine now.
start_server.py
"""
Starts spreadsheet server on specified port
Usage: python start_server.py port_number logging_filepath
port_number: sets server listening to localhost:<port_number>
logging_filepath: full path to logging file (all messages directed to this file)
"""
import argparse
import os
import subprocess
_file_path = os.path.dirname(os.path.abspath(__file__))
#command line interface
parser = argparse.ArgumentParser()
parser.add_argument('port_number',
help='sets server listening to localhost:<port_number>')
parser.add_argument('logging_filepath',help='full path to logging file (all messages directed to this file)')
args = parser.parse_args()
#set up logging
_logging_path = args.logging_filepath
print("logging output to " + _logging_path)
_log = open(_logging_path,'wb')
#set up and start server
_port = args.port_number
print('starting Excel server...')
subprocess.Popen(['python',_file_path +
'\\server.py',str(_port)],stdin=_log, stdout=_log, stderr=_log)
print("".join(['server listening on localhost:',str(_port)]))
server.py
"""
Imports package that starts Excel process (using xlwings), gets interface
to the object wrapper for the Excel model, and then serves requests to that model.
"""
import os
import sys
from multiprocessing.connection import Listener
_file_path = os.path.dirname(os.path.abspath(__file__))
sys.path.append(_file_path)
import excel_object_wrapper
_MODEL_FILENAME = 'excel_model.xls'
model_interface = excel_object_wrapper.get_model_interface(_file_path+"\\"+_MODEL_FILENAME)
model_connection = model_interface['run_location']
close_model = model_interface['close_model']
_port = sys.argv[1]
address = ('localhost', int(_port))
listener = Listener(address)
_alive = True
print('starting server on ' + str(address))
while _alive:
print("listening for connections")
conn = listener.accept()
print 'connection accepted from', listener.last_accepted
while True:
try:
input = conn.recv()
print(input)
if not input or input=='close':
print('closing connection to ' + str(conn))
conn.close()
break
if input == 'kill':
_alive = False
print('stopping server')
close_model()
conn.send('model closed')
conn.close()
listener.close()
break
except EOFError:
print('closing connection to ' + str(conn))
conn.close()
break
conn.send(model_connection(*input))
views.py (from within Django)
from __future__ import unicode_literals
import os
from multiprocessing.connection import Client
from django.shortcuts import render
from django.http import HttpResponse
def run(request):
model_connection = Client(('localhost',6000)) #we started excel server on localhost:6000 before starting Django
params = request.POST
param_a = float(params['a'])
param_b = float(params['b'])
model_connection.send((param_a ,param_b ))
results = model_connection.recv()
return render(request,'model_app/show_results.html',context={'results':results})

Emit/Broadcast Messages on REST Call in Python With Flask and Socket.IO

Background
The purpose of this project is to create a SMS based kill switch for a program I have running locally. The plan is to create web socket connection between the local program and an app hosted on Heroku. Using Twilio, receiving and SMS will trigger a POST request to this app. If it comes from a number on my whitelist, the application should send a command to the local program to shut down.
Problem
What can I do to find a reference to the namespace so that I can broadcast a message to all connected clients from a POST request?
Right now I am simply creating a new web socket client, connecting it and sending the message, because I can't seem to figure out how to get access to the namespace object in a way that I can call an emit or broadcast.
Server Code
from gevent import monkey
from flask import Flask, Response, render_template, request
from socketio import socketio_manage
from socketio.namespace import BaseNamespace
from socketio.mixins import BroadcastMixin
from time import time
import twilio.twiml
from socketIO_client import SocketIO #only necessary because of the hack solution
import socketIO_client
monkey.patch_all()
application = Flask(__name__)
application.debug = True
application.config['PORT'] = 5000
# White list
callers = {
"+15555555555": "John Smith"
}
# Part of 'hack' solution
stop_namespace = None
socketIO = None
# Part of 'hack' solution
def on_connect(*args):
global stop_namespace
stop_namespace = socketIO.define(StopNamespace, '/chat')
# Part of 'hack' solution
class StopNamespace(socketIO_client.BaseNamespace):
def on_connect(self):
self.emit("join", 'server#email.com')
print '[Connected]'
class ChatNamespace(BaseNamespace, BroadcastMixin):
stats = {
"people" : []
}
def initialize(self):
self.logger = application.logger
self.log("Socketio session started")
def log(self, message):
self.logger.info("[{0}] {1}".format(self.socket.sessid, message))
def report_stats(self):
self.broadcast_event("stats",self.stats)
def recv_connect(self):
self.log("New connection")
def recv_disconnect(self):
self.log("Client disconnected")
if self.session.has_key("email"):
email = self.session['email']
self.broadcast_event_not_me("debug", "%s left" % email)
self.stats["people"] = filter(lambda e : e != email, self.stats["people"])
self.report_stats()
def on_join(self, email):
self.log("%s joined chat" % email)
self.session['email'] = email
if not email in self.stats["people"]:
self.stats["people"].append(email)
self.report_stats()
return True, email
def on_message(self, message):
message_data = {
"sender" : self.session["email"],
"content" : message,
"sent" : time()*1000 #ms
}
self.broadcast_event_not_me("message",{ "sender" : self.session["email"], "content" : message})
return True, message_data
#application.route('/stop', methods=['GET', 'POST'])
def stop():
'''Right here SHOULD simply be Namespace.broadcast("stop") or something.'''
global socketIO
if socketIO == None or not socketIO.connected:
socketIO = SocketIO('http://0.0.0.0:5000')
socketIO.on('connect', on_connect)
global stop_namespace
if stop_namespace == None:
stop_namespace = socketIO.define(StopNamespace, '/chat')
stop_namespace.emit("join", 'server#bayhill.com')
stop_namespace.emit('message', 'STOP')
return "Stop being processed."
#application.route('/', methods=['GET'])
def landing():
return "This is Stop App"
#application.route('/socket.io/<path:remaining>')
def socketio(remaining):
try:
socketio_manage(request.environ, {'/chat': ChatNamespace}, request)
except:
application.logger.error("Exception while handling socketio connection",
exc_info=True)
return Response()
I borrowed code heavily from this project chatzilla which is admittedly pretty different because I am not really working with a browser.
Perhaps Socketio was a bad choice for web sockets and I should have used Tornado, but this seemed like it would work well and this set up helped me easily separate the REST and web socket pieces
I just use Flask-SocketIO for that.
from gevent import monkey
monkey.patch_all()
from flask import Flask
from flask.ext.socketio import SocketIO
app = Flask(__name__)
socketio = SocketIO(app)
#app.route('/trigger')
def trigger():
socketio.emit('response',
{'data': 'someone triggered me'},
namespace='/global')
return 'message sent via websocket'
if __name__ == '__main__':
socketio.run(app)