Hello I have a django app. My whole system configuration is the following: python 3, django 1.11, eventlet 0.21.0.
1) Nginx as an upstream server:
upstream proj_server {
server unix:///tmp/proj1.sock fail_timeout=0;
server unix:///tmp/proj2.sock fail_timeout=0;
}
2) Supervisor that controls workers. There is a gunicorn worker:
[program:proj]
command=/home/vagrant/.virtualenvs/proj/bin/gunicorn -c /vagrant/proj/proj/proj/deploy/gunicorn.small.conf.py proj.wsgi:application
directory=/vagrant/proj/proj/proj/deploy
user=www-data
autostart=true
autorestart=true
stdout_logfile=/var/log/supervisor/proj.log
3) This is a gunicorn.small.conf content:
bind = ["unix:///tmp/proj1.sock", "unix:///tmp/proj2.sock"]
pythonpath = "/vagrant/proj/proj/proj/deploy"
workers = 2
worker_class = "eventlet"
worker_connections = 10
timeout = 60
graceful_timeout = 60
4) And this is proj.wsgi content:
"""
WSGI config for proj project.
This module contains the WSGI application used by Django's development server
and any production WSGI deployments. It should expose a module-level variable
named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover
this application via the ``WSGI_APPLICATION`` setting.
Usually you will have the standard Django WSGI application here, but it also
might make sense to replace the whole Django WSGI application with a custom one
that later delegates to the Django one. For example, you could introduce WSGI
middleware here, or combine a Django application with an application of another
framework.
"""
import eventlet
eventlet.monkey_patch()
from eventlet import wsgi
import django.core.handlers.wsgi
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")
# This application object is used by any WSGI server configured to use this
# file. This includes Django's development server, if the WSGI_APPLICATION
# setting points here.
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
# Apply WSGI middleware here.
# from helloworld.wsgi import HelloWorldApplication
# application = HelloWorldApplication(application)
So, as you can see there is a chain: nginx as an upstream server calls one of the gunicorn eventlet workers using two sockets proj1.sock or proj2.sock.
Note that according with the eventlet documentation I try to use eventlet.monkey_patch() as early as possible. The most appropriate place for that is proj.wsgi that is called by gunicorn in the first place.
However it seems that library isn't monkey patched.
To check this I added to the proj/proj/proj/__init__.py (the first module that called by the django application) the following code:
import eventlet
import os
print("monkey patched os is: " + str(eventlet.patcher.is_monkey_patched('os')))
print("monkey patched select is: " + str(eventlet.patcher.is_monkey_patched('select')))
print("monkey patched socket is: " + str(eventlet.patcher.is_monkey_patched('socket')))
print("monkey patched time is: " + str(eventlet.patcher.is_monkey_patched('time')))
print("monkey patched subprocess is: " + str(eventlet.patcher.is_monkey_patched('subprocess')))
then i issued **./manage.py check** and got that answer:
monkey patched os is: false
monkey patched select is: false
monkey patched socket is: false
monkey patched time is: false
monkey patched subprocess is: false
What am I doing wrong?
What if you change proj.wsgi file content to one line raise Exception? That should eliminate eventlet from suspects.
I'm not good with Django, here's a pure speculation:
based on name, proj.wsgi is executed when WSGI server is about to start
manage.py check doesn't seem to be related to remote network service (WSGI), seems to be a general management command, so it shouldn't execute WSGI related code
One possible solution, taken from your question text:
proj/proj/proj/init.py (the first module that called by the django application
Try to put monkey_patch call in there.
P.S.: you don't need supervisor for gunicorn, its master process (arbiter) is designed to run forever in spite of problems with workers.
Related
I am using Gunicorn + NGINX to deploy a multi-endpoint Django REST application. The application runs with
pipenv shell
foreman start
which runs 5 different websites on different ports on localhost. I am trying to deploy just a single one of these websites that runs on port 8000 using Gunicorn. However, I am facing some issues while binding shown down below. I am quite sure how to exactly bind this application using the following line:
gunicorn --bind 0.0.0.0:8000 myproject.wsgi
Here is the WSGI file that I am using:
import os
from django.core.wsgi import get_wsgi_application
if not "DJANGO_SETTINGS_MODULE" in os.environ:
raise Exception("DJANGO_SETTINGS_MODULE not set")
os.environ.setdefault("PERFORM_TIME_CONSUMING_INITIALIZATIONS", "True")
application = get_wsgi_application()
And this is how the websites were exported:
export default {
website1: "http://localhost:8007",
website2: "http://localhost:8000",
website3: "http://localhost:8005",
website4: "http://localhost:8010",
website5: "http://localhost:8009",
};
I develop on serveral projects at once.
If I run the runserver twice, I get this error:
System check identified no issues (0 silenced).
September 10, 2021 - 10:44:26
Django version 3.1.4, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Error: That port is already in use.
I know that I can supply a different port manually, but a more automatic approach would be nice.
How could I solve this clash without manually giving each project a port by hand?
I have more than 50 systems on my laptop, and I don't want to give them each a port number by hand.
You can implement something like following.
Add RUN_SERVER_PORT in settings.py
RUN_SERVER_PORT = 8080
Implement following code in manage.py
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
from django.core.management.commands.runserver import Command as runserver
from django.conf import settings
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'accountant.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
runserver.default_port = settings.RUN_SERVER_PORT
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
Proof of Work:
python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
September 19, 2021 - 12:47:59
Django version 3.2.6, using settings 'accountant.settings'
Starting development server at http://127.0.0.1:8080/
Quit the server with CONTROL-C.
You can use any available port in RUN_SERVER_PORT
You can create your own manage command runsever_foo.
Then you can fetch the default value for the port from settings.py or from os.environ.
Reference to this post: https://stackoverflow.com/a/38319452/16936785
I would recommend to amend the server port under manage.py into the following for a random port selection:
import random
from django.core.management.commands.runserver import Command
port_number = random.randint(8000, 8888)
Command.default_port = str(port_number)
put this in your .bashrc .zshrc or .bash_profile depends on your environment.
#!/usr/bin/env bash
function runserver()
{
local PORTNUM=8000
for (( ; ; )); do
nc -z 127.0.0.1 $PORTNUM
[ $? -ne 0 ] && break
PORTNUM=$(( $RANDOM % 1000 + 8000 ))
done
python manage.py runserver $PORTNUM $#
}
Then just run runserver and it should bring up a test server with a port in the range of [8000, 9000)
This function uses netcat to test if the port is available, you might need to install netcat.
To run Django on different port
python manage.py runserver <your IP>:<port> :
python manage.py runserver 0.0.0.0:5000
Or
python manage.py runserver <port> : python manage.py runserver 5000
For automation
Create a bash script like django_5000 and django_7000 where you execute differents command according to your desired port.
i am using django channels in my project using using official django channels v2, my simple channels app is completed and working fine if run python manage.py runserver
but
i want to run django channels in different port so i am now using daphne
using daphne my_project.asgi:application --port 8001 it working fine in 8001 port
INFO Starting server at tcp:port=8001:interface=127.0.0.1
INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
and i also run python manage.py runserver in another terminal parallely working fine. now my both channels in 8001 and django in 8000 port working correctly but my runserver command running ASGI/Channels instead of wsgi development server,
Starting ASGI/Channels version 2.2.0 development server at http://127.0.0.1:8000/
instead of
Starting development server at http://127.0.0.1:8000/
settings.py
ASGI_APPLICATION = 'my_project.routing.application'
WSGI_APPLICATION = 'my_project.wsgi.application'
if i debug any function in views.py request, it is ASGI request instead of django wsgi request
asgi.py
import os
import django
from channels.routing import get_default_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings")
django.setup()
application = get_default_application()
my question is:
how to get django request instead of ASGI request in our normal function view request(like def index(request)) or if we install django channels every request become ASGI request?
what is the use of the python mange.py runworker command
Like you can read here: https://asgi.readthedocs.io/en/latest/
ASGI (Asynchronous Server Gateway Interface) is a spiritual successor
to WSGI, intended to provide a standard interface between
async-capable Python web servers, frameworks, and applications.
Where WSGI provided a standard for synchronous Python apps, ASGI
provides one for both asynchronous and synchronous apps, with a WSGI
backwards-compatibility implementation and multiple servers and
application frameworks.
so answer for your question nr 1 is: Yes, all requests will be ASGI.
Question nr 2 - it's a command to run multiple workers to process your channel requests in asynchronous way https://channels.readthedocs.io/en/1.x/deploying.html#run-worker-servers
Auto reloader (on code change) works fine using app.run/flask run --host=0.0.0.0 --port 8080.
However as soon as I add flask_socketio to the mix it stops working.
E.g. bringing in:
from flask_socketio import SocketIO, send, emit
socketio = SocketIO(app, path='/kpi/socket.io')
socketio.run(app, host='0.0.0.0', port=8080, debug=True, use_reloader=True, use_debugger=True)
requirements.txt
Flask==1.0.2
Flask-Bootstrap==3.3.7.1
Gunicorn==19.8.1
inotify==0.2.9
pymysql==0.7.2
flask-socketio==3.0.1
eventlet==0.23.0
socketIO-client==0.7.2
I do get "* Restarting with stat" in the logs, same as I would if I was typically running flask.
Extraneous info: this is running in a docker container where I have a compose file for dev where I run the dev mode, and have an env variable for FLASK_DEBUG=1. Using nginx to proxy the rest to port 80 then I server that up to another local port. This all works fine and dandy until I add socketio.
Ending up doing this:
Running this in dev (this brings in the werkzeug debugger for wsgi)
# TODO run this only in dev
from werkzeug.debug import DebuggedApplication
app.debug = True
app.wsgi_app = DebuggedApplication(app.wsgi_app, evalex=True)
Then use supervisord and run uwsgi instead including the flag
--py-autoreload 1
Those both return the functionality I want in development. Just need to make them only run in dev mode now and I'm good.
I'm working on a Django app that's using the django-websocket-redis app to publish events from the server side.
I've successfully configured a test server using nginx and uWSGI to route both HTTP and WS requests correctly to my Django app, but on my local development environment I cannot get this working.
According to the django-websocket-redis documentation, it's enough to start the Django development server and everything should work fine, but it seems like this is far away from reality.
Here is what I've checked:
redis is running on localhost:6379 and is responding to PING requests
tried to run the server on different ports (80, 8080, 8000) to check if the django-websocket-redis makes any assumption about the development server's port, but nothing changed
searched for solution online, but there is nothing about this topic
On my local environment on the client side I see a 404 error when my app tries to connect to the local WebSocket. My settings.py sets the WEBSOCKET_URL to the correct URL (on test server it's working, but locally isn't).
Anyone has an idea what am I doing wrong? Thanks!
I've found a workaround to this.
I've modified my wsgi.py to this and it works now:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
# This application object is used by any WSGI server configured to use this
# file. This includes Django's development server, if the WSGI_APPLICATION
# setting points here.
from django.core.wsgi import get_wsgi_application
from django.conf import settings
from ws4redis.django_runserver import WebsocketRunServer
if settings.DEBUG:
_django_app = get_wsgi_application()
_websocket_app = WebsocketRunServer()
def application(environ, start_response):
if environ.get('PATH_INFO').startswith(settings.WEBSOCKET_URL):
return _websocket_app(environ, start_response)
return _django_app(environ, start_response)
else:
application = get_wsgi_application()
UPDATE: it's not needed to create a custom management command as I mentioned previously, so I've deleted that part of the answer.