I have a Djano 1.8 application running on 6 Heroku dynos. Every once in a while, a request that is normally very fast (200ms) comes in and hits the Heroku router 30 second timeout. Then that dyno times out other follow-on requests, sometimes taking 25 minutes to stabilize and go back to serving requests without timing out.
I'm using gunicorn with default web_concurrency (2 workers per dyno). I read Heroku's article on timeout behavior here, and am considering adding something like --timeout 20 to my gunicorn startup in the Procfile. That seems to be what they are recommending. Do you think that will solve the problem? From looking at the gunicorn documentation on the timeout setting it says:
Workers silent for more than this many seconds are killed and restarted.
But my workers aren't silent are they? They are trying to process requests but are hung and unable to for some reason. Also, when I run gunicorn -h I can see that the default setting for timeout is 30. So if that shouldn't the long running task on my app be killed well before 5, 10 minutes, etc. Do I understand that correctly?
Related
I have a consumers in which there is tracking of model changes using #model_observer.
I subscribe to the event via #action and track its changes, everything works perfectly locally, the model has changed, I immediately received a change. But as soon as I try to put it on aws elastic beanstalk, I can subscribe/unsubscribe, but I don't get changes about events, although I see that the session is not broken.
I thought that the problem was in queues or sessions, but I checked this together with technical support, they told me that everything was working correctly and the connection was made correctly.
Maybe you know at least in which direction I should look and dig?
Just in case, I summarize: everything works correctly locally, when uploading to the server, only subscribe to the event / unsubscribe from the event works, but for some reason the changes do not work
my consumer.py with #model_observer
my django.conf
my Procfile:
web: gunicorn --bind :8000 --workers 3 --threads 2 settings.wsgi:application websocket: daphne -b 0.0.0.0 -p 5000 settings.asgi:application
I watched redis, I watched DB, cleaned up sessions
We are using celery 3.1.17 with redis backend in our flask 0.10.1 application. On our server every celery task created from some celery task is getting created twice. For example,
#celery.task(name='send_some_xyz_users_alerts')
def send_some_xyz_users_alerts():
list_of_users = find_some_list_of_users()
for user in list_of_users:
send_user_alert.delay(user)
#celery.task(name='send_user_alert')
def send_user_alert(user):
data = get_data_for_user(user)
send_mail_to_user(data)
If we start send_some_xyz_users_alerts from our application it runs once. I then see 2 send_user_alert tasks running in celery for each user. Both these tasks have different task_ids. We have 2 workers running on server. Some times these duplicate tasks run on same worker. Sometimes on different workers. I have tried lot to find the problem without any luck. Would really appreciate if someone knows why this could happen. Things were running fine for months on these versions of celery and flask and suddenly we are seeing this problem on our servers. Tasks run fine on local env.
I am running Django using Gunicorn behind Nginx. In one of my installation, when I run the gunicorn process, I keep getting debug output, it's like workers are being created in every second (I assume this because django is loading very slow and note the message "[20205] [DEBUG] 3 workers"). You can check the detail output at this gist
In similar setup, I am running 3 more installations without any such issues and respective site loads almost instantly.
Any idea why this is happening? Thanks.
The polling of the workers every second on --log-level debug was introduced in gunicorn==19.2.
Change the log level to info.
I have a Django 1.6 app running on the Heroku cedar-14 platform. I send tasks to be performed by a worker via Celery 3.1.18 with the redis bundle and the Redis To Go (Micro) add-on. I have very few tasks and they complete in 3-5 seconds.
I send my tasks to the worker via Celery's delay() method. Unfortunately, the worker doesn't log reception of the task for 10-15 minutes later -- after which the task completes in the expected 3-5 seconds. What might be causing this delay?
I don't see this delay on my development server and I have been using this app for a couple of years now -- this delay is a recent development. I did recently upgrade to Heroku's cedar-14 from cedar and to a larger Redis To Go plan, but I am uncertain if this delay is associated with these changes.
Update
It looks as if task has to wait for a worker dyno to run the task after some delay. In the past, a worker started running the task immediately when a task was submitted.
This is the Procfile I use:
web: newrelic-admin run-program python manage.py run_gunicorn -b "0.0.0.0:$PORT" -t 30 -w 3
worker: newrelic-admin run-program celery -A myapp.celery:CELERY_APP -E -B --loglevel=INFO worker
So the question becomes how do I return to the behavior where submittal of a celery task causes a task to run immediately?
I supsect this issue applies: https://github.com/celery/celery/issues/2296
I'm developing a web application with Django which uses Celery to process asynchronous tasks, especially for transactional emails.
One on my email task is scheduled with the ETA option but it's executed multiple times in parallel resulting in mail chain, very anoying. I can't figure out exactly why.
I checked twice my Django code and I'm sure that it is publish only one time.
I'm using Redis as a broker/backend result.
My Celery daemon is hosted on Heroku and launched via this command:
python manage.py celeryd -E -B --loglevel=INFO
Thanks for your help.
EDIT: I find a valid solution here thanks to a guy on the #celery IRC channel: http://loose-bits.com/2010/10/distributed-task-locking-in-celery.html
Have you checked the Ensuring a task is only executed one at a time docs?