I need to run celery task only when django request finished.
Is it possible?
I've found that the best way to make sure your task happens after the request is finished is to write a custom middleware. In the process_response method, you can handle any quick actions that don't impact page load time or performance too much. Anything else, you can hand off to Celery. Any saving or database transactions are completed by the time process_response is called (AFAICT).
Try something like this:
Django sends request_finished at the end of every request.
You can access request object through sender argument,
from django.dispatch import receiver
from django.core.signals import request_finished
from app.tasks import my_task
#receiver(request_finished)
def add_celery_task(sender):
if sender.__name__ != 'StaticFilesHandler':
my_task.delay()
If you are running server in development environment it's good to check sender's name to avoid adding too many celery task for every static file you are serving.
You can run the task in the background, using delay method of celery. I mean just before returning the response you can call the delay method to put the task in the background.
Some thing like this:
task_name.delay(arg1, arg2, ...)
By doing this your task will be put into background and run asynchronously, this is not going to block the request response cycle .
Related
Everything is working except task autodiscovery under Django. (Celery 5.2.6, Django 3.0.6)
If I run celery worker, the worker triggers autodiscovery, finds all my tasks, and displays them in a list as part of it's startup process. However, if I run my Django app or Django shell, this never happens.
Additionally, even though the docs promise that accessing the task registry will trigger autodiscovery, it does not. I print app.tasks, I call app.tasks.keys(), and still no autodiscovery - it only shows the tasks that are built-in or were registered when the module containing them happened to be imported for other reasons.
What do I need to do to trigger task autodiscovery?
PS - If I try adding force=True to app.autodiscover_tasks(), it fails because the Django app registry hasn't finished loading at that time.
I dug into the code and tried a bunch of things. Here's what I learned.
How Autodiscovery is Performed
Autodiscovery is actually performed by app._autodiscover_tasks() (src), which is invoked by the celery.signals.import_modules signal (src), which is sent by app.loader.import_default_modules() (src). The signal is connected to app._autodiscover_tasks() in app.autodiscover_tasks (src).
In other words, the only way to get autodiscovery to trigger is to:
call app.autodiscover_tasks() to register app._autodiscover_tasks as a receiver for signal celery.signals.import_modules
call app.loader.import_default_modules() to send the signal
The docs instruct me to do the first, but not the second. Thus, in my Django app, autodiscovery does not happen.
How Autodiscovery is Performed in celery
If you run celery with the commands beat, report, shell, or worker, the resulting code path eventually calls app.loader.import_default_modules() directly. Thus, autodiscovery happens under those conditions - but not when you run your Django app, or open a Django shell, or any other context but those four commands, unless you explicitly call app.loader.import_default_modules().
On Finalize
The docs promise that accessing app.tasks will cause the app to finalize, which I assumed triggered autodiscovery, but it has nothing to do with that. All that finalize does is send an app.on_after_finalize signal, which does nothing.
My Solution
After much frustrating and experimentation, I settled on this snippet from my celery.py:
app = Celery("myproj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
#app.on_after_configure.connect()
def trigger_autodiscovery(sender, **kwargs):
app.loader.import_default_modules()
I am trying to create an asynchronous task using celery, but I am not achieving success.
I have a task that sends emails:
#shared_task()
def send_email_example(email_id):
...
I call it using the delay() method:
class SomeModelExample:
...
def example(self):
...
send_email_example.delay(self.id)
Locally, I run the celery and can use it. However, in my server, when I use the method that calls the function it takes more than 30 seconds and I receive a status code 502.
I hope that my celery setup is ok, because my periodc tasks works.
Solution
Testing it myself I saw that the celery is really ok. The problem was with my tests on the server.
At this stage I would try:
Use decorator without brackets: #shared_task
Increase harakiri time to >60 seconds on your server to see if it is a task problem or a server problem.
This is a generic question that I seek answer to because of a celery task I saw in my company's codebase from a previous employee.
It's a shared task that calls an endpoint like
#shared_task(time_limit=60*60)
def celery_task_here(some_args):
data = get_data(user, url, server_name)
# some other logic to build csv and stuff
def get_data(user, url, server_name):
client = APIClient()
client.force_authenticate(user=user)
response = client.get(some_url, format='json', SERVER_NAME=server_name)
and all the logic resides in that endpoint.
Now what I understand is that this will make the server do all the work and do not utilize celery's advantage, but I do see celery log producing queries when I run this locally. I'd like to know who's actually doing the job in this case, celery or the django server?
If the task is called via celery_task_here.delay, the task will be pushed to a queue, then the worker process that is responsible for handling the queue will actually execute the task, which is not the "Django server". The worker process could potentially be on the same machine as your Django instance, it depends on your environment.
If you were to call the task via celery_task_here.s (or as a normal function) the task would be executed by the Django server.
It depends of how the task is called
If it is meant to be called as celery task with apply_async or delay than it is executed as celery task by celery worker process
You still can call it as normal function without sending it to celery if you just call it as function
I read http://bottlepy.org/docs/dev/tutorial_app.html#server-setup
and running Apache + Bottle + Python
and Bottle + Apache + WSGI + Sessions
and I would like to know if one can run asynchronous rest api calls to bottle on mod_wsgi server to a py function that does not return anything(its a backend logic) and is non blocking - so I looked up gevent but i am haven't found a solution where you can run mod_wsgi with gevents.
Is there any solution to async calls to run on apache server using mod_wsgi or any other alternative?
UPDATE
as per andreans' answer below;
I ran a simple myip address return with bottle + celery. so one has to run a celery as #celery.task and then run(host='localhost', port=8080, debug=True)? does it require to start celery worker on terminal as well? never used celery before [runnin locally] also running bottle with decorator #route(/something) works but app.route doesnt where app = Bottle() possibly due to some .wsgi file error?
Sorry, can't fit into the comment box. Every request must get a response eventually (or fail/time out). If you really don't need to return any data to the client, send back just an empty response with a status code. If the processing of the request takes time, it should run asynchronously, and that's where celery comes in. So a blocking implementation of your request handler:
def request_processor_long_running_blocking_func(request_data):
# process request data, which takes a lot of time
# result is probably written into db
pass
def request_handler_func(request):
request_processor_long_running_blocking_func(request.data)
return HttpResponse(status=200)
If I understood correctly this is what you're trying to avoid, by making the request_processor_long_running_blocking_func run asynchronously, so the request_handler_func won't block. This would be solved with celery like this:
from celery.task import task
#task
def request_processor_long_running_blocking_func(request_data):
# the task decorator wraps your blocking function, with celery's Task class
# which has a delay method available for you to call, which will run your function
# asynchronously on one of your celery background worker processes
pass
def request_handler_func(request):
request_processor_long_running_blocking_func.delay(request.data)
# calling the function with delay won't block, it returns immediately
# and your response is sent back instantly
return HttpResponse(status=200)
One more thing, send these task requests with ajax, so your web interface won't be reloaded or anything, so the user can continue using your app after sending the request
I need a scheduler for my next project, and since I'm coding using Django I went for Celery.
What I am looking for is a way for a task to tell Django when it is done, so I can update the database and use SSE to tell the user. All this can be done fairly simple with just putting all the logic into the task. But what do I do when I am planning to have several celery workers?
I found a bunch of info online to cover the single-worker-case, but not many covering the problem if you have more than one worker.
What I thought about was using http callbacks from the workers to the web-server to let it know that the task is done. Looking at celery.task.http looked promising, but didnt do what I needed.
Is the solution to use signals and hook up manual http calls? Or am I on the wrong path? Isn't this a common problem? How can this be solved more elegantly?
So, what are you mean when you tell tell to Django? Is I understand you right, django request which initiliazed a Celery task, is still alive a time when this task is finished? I that case you can check some storage ( database, memcached, etc ). and send your SSE.
Look, there is one way to do that.
1. You django view send task to Celery, after that it goes to infinite loop ( or loop with timeout 60sec?) and waits results in memcached.
Celery gets task executes, and pastes results to memcached.
Django view gets new results, exit the loop and sends your SSE.
Next variant is
Django view sends task to Celery, and returns
Celery execute tasks, after executing it makes simple HTTP requests to your django app.
Django receives a http request from Celery, parse params and send SSE to your user again
Here is some code that seems to do what I want:
In django settings:
CELERY_ANNOTATIONS = {
"*": {
"on_failure": celery_handlers.on_failure,
"on_success": celery_handlers.on_success
}
}
In the celery_handlers.py file included:
def on_failure(self, exc, task_id, *args, **kwargs):
# Use urllib or similar to poke eg; api-int.mysite.com/task_handler/TASK_ID
pass
def on_success(self, retval, task_id, *args, **kwargs):
# Use urllib or similar to poke eg; api-int.mysite.com/task_handler/TASK_ID
pass
And then you can just setup api-int to use something like:
from celery.result import AsyncResult
task_obj = AsyncResult(task_id)
# Logic to handle task_obj.result and related goes here....