I have a flask app that processes a web socket stream of audio from Twilio.
The app works fine without gunicorn but when I start it with gunicorn I get only the first message of the socket (connect) and an unsuccessful handshake. Here is how the app looks:
from flask import Flask
from flask_sockets import Sockets
from geventwebsocket.handler import WebSocketHandler
from gevent import pywsgi
...
app = Flask(__name__)
sockets = Sockets(app)
...
#sockets.route('/media')
def media(ws):
...
if __name__ == '__main__':
server = pywsgi.WSGIServer(('', HTTP_SERVER_PORT), app, handler_class=WebSocketHandler)
server.serve_forever()
When I start the app directly using python flaskapp.py it works ok.
When I start it using gunicorn by writing:
gunicorn -k flask_sockets.worker --bind 0.0.0.0:5055 --log-level=bug flaskapp:app
this is where the connection "hangs" and carries no further than the initial connection, apparently due to the handshake failing.
It's important to note that I haven't "gevent monkey patched" the code, but I'm not sure if it has anything to do with the problem.
Any idea will much be appreciated!
Don't have the ability to test this right now, but perhaps try with:
from flask import Flask
from flask_sockets import Sockets
from geventwebsocket.handler import WebSocketHandler
from gevent import pywsgi
...
app = Flask(__name__)
sockets = Sockets(app)
...
#sockets.route('/media')
def media(ws):
...
server = pywsgi.WSGIServer(('', HTTP_SERVER_PORT), app, handler_class=WebSocketHandler)
if __name__ == '__main__':
server.serve_forever()
Then change the launch command to:
gunicorn -k flask_sockets.worker --bind 0.0.0.0:5055 --log-level=bug flaskapp:server
(Gunicorn should be importing the server object, which can't live within that final if statement, as that code only runs when launched with python directly).
Related
I'm running into a problem when using Flask with a gremlin database (it's an Amazon Neptune database) and using uWSGI. Everything works fine in my unit tests which use the test_client provided by Flask. However, in production we use uWSGI and there I get the following error:
There is no current event loop in thread 'uWSGIWorker4Core1'.
My app code is creating a connection to the database before a request and assigning it to the Flask g object. During teardown, the database connection is removed. The error happens when the app is trying to close the connection.
from flask import Flask, g
from gremlin_python.structure.graph import Graph
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal
app = Flask(__name__, instance_relative_config=True)
#app.before_request
def _db_connect():
if not hasattr(g, 'graph_conn'):
g.graph_conn = DriverRemoteConnection(app.config['DATABASE_HOST'],'g')
g.gg = traversal().withRemote(g.graph_conn)
# This hook ensures that the connection is closed when we've finished
# processing the request.
#app.teardown_appcontext
def _db_close(exc):
if hasattr(g, 'graph_conn'):
g.graph_conn.close(). # <- ERROR THROWN AT THIS LINE
del g.graph_conn
the uWSGI config does use multiple threads:
[uwsgi]
http = 0.0.0.0:3031
manage-script-name = true
module = dogmaserver:app
processes = 4
threads = 2
offload-threads = 2
stats = 0.0.0.0:9191
But my understanding of how Flask's g object worked would be that it is all on the same thread. Can anyone let me know what I'm missing?
I'm using Flask 1.0.2, gremlinpython 3.4.11 and uWSGI 2.0.17.1.
I used a workaround by removing the threads configuration option in uWSGI which makes there only be a single thread per process.
I am testing my Flask app on AWS EC2 instance (Ubuntu).
Main app:
from sonip.api.factory import create_app
app = create_app()
def main():
app.run(debug=True, threaded=True)
if __name__ == "__main__":
main()
The actual set up of the Flask app is done in a factory including registering blueprint, etc.
def create_app():
app = Flask(__name__)
app.config['SERVER_NAME'] = settings.FLASK_SERVER_NAME
app.config['SWAGGER_UI_DOC_EXPANSION'] = settings.RESTPLUS_SWAGGER_UI_DOC_EXPANSION
app.config['RESTPLUS_VALIDATE'] = settings.RESTPLUS_VALIDATE
app.config['RESTPLUS_MASK_SWAGGER'] = settings.RESTPLUS_MASK_SWAGGER
app.config['ERROR_404_HELP'] = settings.RESTPLUS_ERROR_404_HELP
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config["SQLALCHEMY_DATABASE_URI"] = "postgresql://{}:{}#{}:{}/{}".format(
settings.DB_USER,
settings.DB_PASS,
settings.DB_HOST,
settings.DB_PORT,
settings.DB_NAME)
db.init_app(app)
blueprint = Blueprint('api', __name__, url_prefix='/api')
app.register_blueprint(blueprint)
return app
When I run python application.py and use curl -X GET http://localhost:5000/api, it returns the correct Swagger page. However, if I tried to run the app by specifying host=0.0.0.0 for external traffic, I got 404 for the same request.
(env) ubuntu#ip-172-31-18-136:~/aae$ python application.py
* Serving Flask app "sonip.api.factory" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://localhost:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 948-062-124
127.0.0.1 - - [16/May/2018 18:07:24] "GET /api/ HTTP/1.1" 200 -
♥(env) ubuntu#ip-172-31-18-136:~/aae$ vi application.py
(env) ubuntu#ip-172-31-18-136:~/aae$ python application.py
* Serving Flask app "sonip.api.factory" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 948-062-124
165.225.34.185 - - [16/May/2018 18:08:28] "GET /api/ HTTP/1.1" 404 -
Port 5000 is open to allow all inbound traffic in the security group. I tried a vanilla Flask app with just a few line of code, it worked just fine.
app = Flask(__name__)
if __name__ == '__main__':
app.run(debug=True, port=8080, host='0.0.0.0')
This at least means 5000 is fine. Could it be the Blueprint or Swagger?
Setting SERVER_NAME and changing the host/port to something different in app.run() used to be a recipe for problems. At best, it's under-documented.
Try changing settings.FLASK_SERVER_NAME to 0.0.0.0:5000. Or if your app wants to be using cookies, try the trick of using something.dev:5000 and adding an entry for something.dev to your local /etc/hosts.
I have a problem with django channels.
My Django app was running perfectly with WSGI for HTTP requests.
I tried to migrate to channels in order to allow websocket requests, and it turns out that after installing channels and running ASGI (daphne) and a worker, the server answers error 503 and the browser displays error 504 (time out) for the http requests that were previously working (admin page for example).
I read all the tutorial I could find and I do not see what the problem can be. Moreover, if I run with "runserver", it works fine.
I have an Nginx in front of the app (on a separate server), working as proxy and loadbalancer.
I use Django 1.9.5 with asgi-redis>=0.10.0, channels>=0.17.0 and daphne>=0.15.0. The wsgi.py and asgi.py files are in the same folder. Redis is working.
The command I was previously using with WSGI (and which still works if I switch back to it) is:
uwsgi --http :8000 --master --enable-threads --module Cats.wsgi
The command that works using runserver is:
python manage.py runserver 0.0.0.0:8000
The commands that fail for the requests that work with the 2 other commands are:
daphne -b 0.0.0.0 -p 8000 Cats.asgi:channel_layer
python manage.py runworker
Other info:
I added 'channels' in the installed apps (in settings.py)
other settings.py relevant info
CHANNEL_LAYERS = {
"default": {
"BACKEND": "asgi_redis.RedisChannelLayer",
"ROUTING": "Cats.routing.app_routing",
"CONFIG": {
"hosts": [(os.environ['REDIS_HOST'], 6379)],
},
},
}
Cats/routing.py
from channels.routing import route, include
from main.routing import routing as main_routing
app_routing = [
include(main_routing, path=r"^/ws/main"),
]
main/routing.py
from channels.routing import route, include
http_routing = [
]
stream_routing = [
route('websocket.receive', 'main.consumers.ws_echo'), #just for test once it will work
]
routing = [
include(stream_routing),
include(http_routing),
]
main/consumers.py
def ws_echo(message):
message.reply_channel.send({
'text': message.content['text'],
})
#this consumer is just for test once it will work
Any idea what could be wrong? All help much appreciated! Ty
EDIT:
I tried a new thing:
python manage.py runserver 0.0.0.0:8000 --noworker
python manage.py runworker
And this does not work, while python manage.py runserver 0.0.0.0:8000 was working...
Any idea that could help?
channels will use default views for un-routed requests.
assuming you use the javascripts right, I suggest you use only your default Cats/routing.py file as following:
from channels.routing import route
from main.consumers import *
app_routing = [
route('websocket.connect', ws_echo, path="/ws/main")
]
or with reverse to help with your path
from django.urls import reverse
from channels.routing import route
from main.consumers import *
app_routing = [
route('websocket.connect', ws_echo, path=reverse('main view name'))
]
I think also your consumer should be changed. when browser connects using websockets the server should first handle adding message reply channel. something like:
def ws_echo(message):
Group("notifications").add(message.reply_channel)
Group("notifications").send({
"text": json.dumps({'testkey':'testvalue'})
})
the send function should probably be called up on different event and the "notifications" Group should probably changed to have a channel dedicated to the user. something like
from channels.auth import channel_session_user_from_http
#channel_session_user_from_http
def ws_echo(message):
Group("notify-private-%s" % message.user.id).add(message.reply_channel)
Group("notify-private-%s" % message.user.id).send({
"text": json.dumps({'testkey':'testvalue'})
})
If you're using heroku or dokku make sure you've properly set the "scale" to include the worker process. By default they will only run the web instance and not the worker!
For heroku
heroku ps:scale web=1:free worker=1:free
For dokku create a file named DOKKU_SCALE and add in:
web=1
worker=1
See:
http://blog.codelv.com/2017/10/timouts-django-channels-on-dokku.html
https://blog.heroku.com/in_deep_with_django_channels_the_future_of_real_time_apps_in_django
I'm trying create a simple background periodic task using Django-Celery-RabbitMQ combination. I installed Django 1.3.1, I downloaded and setup djcelery. Here is how my settings.py file looks like:
BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672
BROKER_VHOST = "/"
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"
....
import djcelery
djcelery.setup_loader()
...
INSTALLED_APPS = (
'djcelery',
)
And I put a 'tasks.py' file in my application folder with the following contents:
from celery.task import PeriodicTask
from celery.registry import tasks
from datetime import timedelta
from datetime import datetime
class MyTask(PeriodicTask):
run_every = timedelta(minutes=1)
def run(self, **kwargs):
self.get_logger().info("Time now: " + datetime.now())
print("Time now: " + datetime.now())
tasks.register(MyTask)
And then I start up my django server (local development instance):
python manage.py runserver
Then I start up the celerybeat process:
python manage.py celerybeat --logfile=<path_to_log_file> -l DEBUG
I can see entries like this in the log:
[2012-04-29 07:50:54,671: DEBUG/MainProcess] tasks.MyTask sent. id->72a5963c-6e15-4fc5-a078-dd26da663323
And I also can see the corresponding entries getting created in database, but I can't find where it is logging the text I specified in the actual run function in MyTask class.
I tried fiddling with the logging settings, tried using the django logger instead of celery logger, but of no use. I'm not even sure, my task is getting executed. If I print any debug information in the task, where does it go?
Also, this is first time I'm working with any type of message queuing system. It looks like the task will get executed as part of the celerybeat process - outside the django web framework. Will I still be able to access all the django models I created.
Thanks,
Venkat.
Celerybeat it stuff, which pushes task when it need, but not executing them. You tasks instances stored in RabbitMq server. You need to execute celeryd daemon for executing your tasks.
python manage.py celeryd --logfile=<path_to_log_file> -l DEBUG
Also if you using RabbitMq, I recommend to you to install special rabbitmq management plugins:
rabbitmq-plugins list
rabbitmq-enable rabbitmq_management
service rabbitmq-server restart
It will be available at http://:55672/ login: guest pass: guest. Here you can check how many tasks in your rabbit instance online.
You should check the RabbitMQ logs, since celery sends the tasks to RabbitMQ and it should execute them. So all the prints of the tasks should be in RabbitMQ logs.
When i run the server it shows me a message http://127.0.0.1:8000/. I want to get the url in code. I dont have request object there.
request.build_absolute_uri('/')
or you could try
import socket
socket.gethostbyname(socket.gethostname())
With Django docs here, You can use:
domain = request.get_host()
# domain = 'localhost:8000'
If you want to know exactly the IP address that the development server is started with, you can use sys.argv for this. The Django development server uses the same trick internally to restart the development server with the same arguments
Start development server:
manage.py runserver 127.0.0.1:8000
Get address in code:
if settings.DEBUG:
import sys
print sys.argv[-1]
This prints 127.0.0.1:8000
import socket
host = socket.gethostname()
import re
import subprocess
def get_ip_machine():
process = subprocess.Popen(['ifconfig'], stdout=subprocess.PIPE)
ip_regex = re.compile('(((1?[0-9]{1,2})|(2[0-4]\d)|(25[0-5]))\.){3}((1?[0-9]{1,2})|(2[0-4]\d)|(25[0-5]))')
return ip_regex.search(process.stdout.read(), re.MULTILINE).group()