When I inspect celery -A proj inspect active_queues I see two servers showing their queues they are listening to and they are pointing to same default queue name celery. Still the task issued by django app gets executed twice by both servers(Once by each celery server - so two times).
I can see the transport type is also direct - the default one.
On my local task gets executed once so I am sure that the task is called only once by my django app.
What can I be missing here?
Ok, i looked up the docs, i think you need to set celerybeat-scheduler in your settings.py which makes sure tasks are being scheduled by a single scheduler.
http://celery.readthedocs.org/en/latest/configuration.html#celerybeat-scheduler
On Redis you can set the current database for the application you are running, setting the database will separate the information to use different apps.
If you are using Django the configuration is
CELERY_BROKER_VHOST = {number of the database}
If you are not using Django i beleive the configuration is CELERY_REDIS_DB or redis_db depending on your celery version
For instance for your first application could be CELERY_BROKER_VHOST = 1
For the second application could be CELERY_BROKER_VHOST = 2
and for your local development could be CELERY_BROKER_VHOST = 99
http://docs.celeryproject.org/en/latest/userguide/configuration.html#id8
Related
I have a flask application. This application mimics routing of vehicles in a city and when a vehicle reaches a designated point, I have to wait for 30-180 seconds before starting it again. I am trying to use apscheduler for this.
When the vehicle arrives I start an apscheduler job (with the 'date' trigger for X seconds). When the job fires, I do my processing.
This works well on my dev machine when I am running the flask app standalone. But when I try to run it on my production server (where the app is running in uwsgi mode) the job never fires. I have already set --enable-threads=true for the app, so that doesn't seem to be the problem.
My relevant code is like this.
At my app initialization.
scheduler = BackgroundScheduler()
scheduler.start()
Whenever the trigger happens.
scheduler.add_job(func=myfunc, trigger='date', run_date=datetime.datetime.now() + datetime.timedelta(seconds=value)).
Anything I am missing in using apscheduler in uwsgi mode? Or any other options in flask to achieve what I want to?
I'm using uwsgi-emperor on Debian 8 in my production systems. For a specific Django project, I need to run a computing intensive setup task only once at the launch of the vassal. The vassal can have multiple workers/threads, but the task must be executed only one time, no matter how many workers/threads are spawned.
Currently, I'm executing this setup task every time a new worker is launched, but this is clearly suboptimal. The setup task is an invocation of a method from the same Django project, but I think that doesn't change the problem.
Is there any way of doing this from uWSGI?
You could try to use a singletone approach, this code at settings.py would call the startup_only_once() function only once:
from tendo.singleton import SingleInstance
def startup_only_once():
print("One time only")
try:
FIRST_START = SingleInstance()
startup_only_once()
except:
pass
I just installed Celery and I want to create a simple status page that shows the current number of workers and their status.
Is this possible? From web searches the best I found was celery.current_app.control.inspect()
But as far as I can see, it doesn't mention anything about workers. (I'm using Kombu with SQS for the backend if that matters)
In the documentation of celery workers it is explained the output of the inspect commands.
By default using celery.current_app.control.inspect() returns an "inspector object" that allows you to ask for the state of all the running workers. For example if you execute this code with two running workers named 'adder' and 'sleeper':
i = celery.current_app.control.inspect()
i.registered()
the call to i.registered() could return something like:
{
'adder#example.com': ['tasks.add'],
'sleeper#example.com': ['tasks.sleeptask'],
}
In conclusion, the "inspector" methods registered, active, scheduled, etc. return a dictionary with the results classified by the workers chosen when celery.current_app.control.inspect() was called (if no workers are passed as arguments, all workers are implicitly selected).
I'm using django, celery and rabbitmq to process tasks, in let's call it APP1. In another host i have APP2, that needs to get results from tasks processed in APP1.
Both APP/hosts have access to rabbitmq, and my first approach was to simple try to share a queue from both APPs without success.
What is the best approach to achieve this?
One possible approach is to have the task run on APP1, and when it is done processing the task, post another task to celery. Call this new task ProcessResults. The data for this task will be the result of the original task. The worker for this new task would be located on APP2.
Just use the same result backend that you used in APP1, for example in APP2:
from celery import Celery
from celery.result import AsyncResult
# set the backend URL that APP1 is using
app = Celery(backend='backend_url')
# The task ID that was queued in APP1
task = AsyncResult('task_id')
# get the task result
task.result
You need to store the task ID from APP1 to be able to get its result in APP2, or maybe use a custom task ID if that can help without storing it, but you need to use Task.apply_async() to set a custom ID:
task.apply_async(args, kwargs, task_id='custom_id')
I have scheduled a task using django celery (djcelery) by defining it as so:
#periodic_task(run_every=timedelta(minutes=1))
def mytask():
# Do something
I decided to remove this task from the codebase.
However, even after restarting the celery server, this task continues to be scheduled every 1 minute, although it reports an error message since this task no longer exists. Do I have to do something to clear old periodic tasks from the djcelery database, in addition to restarting the server?
You might need to remove your task from the table djcelery_periodictask as well.