Im running my django project using apache + mod_wsgi in daemon mode. When I have to make the server notice changes in the source code, I touch the wsgi.py file, but I have an issue with this approach.
Some tasks that are triggered from the front-end take 10 minutes to complete. If I touch the wsgi file while one of this long tasks are running, they get killed by the restart.
Is there any way to make the server to refresh the code, but keeping the previous unfinished tasks running until the are done?
Thanks!
Don't run long-running tasks in web processes. Use an offline task manager like Celery.
Related
In my django web app I am running a calculation which takes 2-10 minutes, and the AWS server times out with a 504 server error.
It seems that the best way to fix this problem is to implement a worker, and offload the calculations. Through some research, it seems that Celery with a Redis server (maybe AWS SQS instead?) are best for Django.
However, I only see tutorials for instances that are run locally. The Redis server is hosted by Railway and Celery is run in a separate terminal than Django (on the same local machine). I'm curious if/where Celery and Redis should be running on AWS.
This answer says you should try to run celery as a deamon in the background. AWS Elastic Beanstalk uses supervisord already to run some deamon processes. So you can leverage that to run celeryd and avoid creating a custom AMI for this. It works nicely for me. but don't the Celery and Redis servers still need to run somehow?
Where does the Celery server run?
How do I start it?
How can I leverage supervisord to run daemon processes? The documentation isn't helping me very much with AWS integration
You can configure Procfile to run multiple processes like main django app, celery and celery-beat in parallel as documented here:
web: <command to start your django app>
celery: celery -A <path_to_celery_app> worker
celery_beat: celery -A <path_to_celery_app> beat
I have a Django web application that executes tasks via celery. It is run with a combination of Apache, uWSGI, and Redis. For some reason, one of the tasks is being executed by the uWSGI server and the other is executed by the Python interpreter. This is causing permissions issues as uWSGI doesn't run as the same user as Python does.
What could cause the tasks to be run by different programs? Is there a setting somewhere?
Turns out, I needed to call the task with .delay() to get the Celery deamon to execute the task instead of uWSGI.
Supervisord cant controll my celery services(workers, flower, django and celerybeat).
When system startup, I can see all processes running successfully, if 1 of processes fail and supervisor does not restart it, when I reload or restart supervisord, everything under supervisord fails and cant be re-launched.
I have tried to move my celery services under monit.
I created celery services in /etc/init.d/ and services run perfectly, I can start/stop them without problems, then I set up monit scripts for all of them.
The problem is that monit is unable to start my celery-workers services, it is able to start/stop "django", "flower" and "celerybeat" without problems, but workers services under init.d are unable to be started by monit.
Could you please sugest me any idea how to fix this?
Why other services can be managed by monit without problems, but workers not?
Thank You in advance.
Django can be run either in Nginx (or some other server, but we are currently going to use Nginx) or with manage.py runserver Django own's server. In both cases I need data integrity and recovery.
For data integrity I need not to terminate (some of) Django requests until they finish. They should not terminate because they should finish started data modification in the DB to preserve data integrity (and no, using SQL transactions is not a solution). They should not terminate as soon as Nginx receives service nginx stop (on Debian Linux) or some other similar command (on other OSes), but finish processing before terminating. How to do this?
For data recovery I want:
create an empty file when the server starts, remove it when the server stops (how to do it both with Nginx and with manage.py runserver?)
When the server starts, if the file is found (indicating a crash of our software), before server starting we need to run my "data recovery" script. How to do this?
A couple things here. First, you should definitely never use runserver in production. Secondly, you don't run really Django in nginx—you run it in a WSGI server, such as uWSGI or Gunicorn. Often, these are run behind nginx, but they don't start and stop when it does.
If you're just trying to make sure that the server doesn't stop while views are still processing, you could definitely do that with uWSGI. There's a pretty good writeup on this subject in the uWSGI documentation called "The Art of Graceful Reloading".
As long as the server is gracefully reloaded, I don't think you need to worry as much about recovery. However, if you still wanted to try that, you'd need to do the empty file deletion in a bash script wrapper—there's no Python code that runs when the server stops.
I have a Django app that talks to a remote Raspberry Pi to acquire imagery through the Pi's camera. Whenever I test the "get a new image" button in the app, the browser gets hung up for about 60 seconds and the image never arrives.
The Raspberry Pi is trying to POST the image to the Django app which is then supposed to save it to persistent storage.
The logs of Nginx show a 504 "gateway timeout" at the 60 second mark. However, this worked smoothly when I was using the Django development server and took about a second to POST the image. What's going wrong now?
Make sure you're running uWSGI with multiple processes and threads.
You can test this on the uWSGI command line by adding:
--processes 4 --threads 2
or in a uWSGI ini file:
processes = 4
threads = 2
If the Pi is POSTing the image back to the app while waiting to display the result to the user, then uWSGI needs to be able to handle both things concurrently.
Another possibility is that your django app uses threads itself, and without the --threads N or --enable-threads option to uWSGI, the GIL isn't enabled when your app runs. Adding --enable-threads (or enable-threads = true to the ini file) will enable the GIL without starting multiple app threads.
See a note on python threads in the uWSGI docs if you suspect that your app's threading may be the problem.
In any case, make sure you've provided for enough concurrency if you're seeing gateway timeouts.