I want to schedule the callback api according to the cron expression without beat_schedule...
My code is executing but not scheduling the callback api.
Related
This is a generic question that I seek answer to because of a celery task I saw in my company's codebase from a previous employee.
It's a shared task that calls an endpoint like
#shared_task(time_limit=60*60)
def celery_task_here(some_args):
data = get_data(user, url, server_name)
# some other logic to build csv and stuff
def get_data(user, url, server_name):
client = APIClient()
client.force_authenticate(user=user)
response = client.get(some_url, format='json', SERVER_NAME=server_name)
and all the logic resides in that endpoint.
Now what I understand is that this will make the server do all the work and do not utilize celery's advantage, but I do see celery log producing queries when I run this locally. I'd like to know who's actually doing the job in this case, celery or the django server?
If the task is called via celery_task_here.delay, the task will be pushed to a queue, then the worker process that is responsible for handling the queue will actually execute the task, which is not the "Django server". The worker process could potentially be on the same machine as your Django instance, it depends on your environment.
If you were to call the task via celery_task_here.s (or as a normal function) the task would be executed by the Django server.
It depends of how the task is called
If it is meant to be called as celery task with apply_async or delay than it is executed as celery task by celery worker process
You still can call it as normal function without sending it to celery if you just call it as function
Is it possible to achieve interoperability between a scheduler and a pub/sub in the Google Cloud, so that a task is triggered after a specific time every day, but only if a message arrives?
UPDATED:
Example would be a task scheduled for 10:00 am waits for a msg (a pre-requisite).
At 10:00 the msg has not arrived. The job is not triggered. The msg arrives at 11:00. The job is triggered. (It can then send a msg to start the task to be executed)
At 09:00 the msg arrives. The job is not executed. At 10:00 the job is triggered.
The msg never arrives. The job is never executed.
Your puzzle seems to be an excellent match for using Cloud Tasks. At a high level, I would imagine you writing a Cloud Function that subscribes to the topic that is being published upon. The Cloud Function would contain your processing logic:
Received after 10:00am, run your job immediately.
Received before 10:00am, use Cloud Tasks to post a a task to run your job at 10:00am.
... and that's it.
Google's recommended practise is to use Google Cloud Composer such tasks.
You can use cloud composers for variety of use cases including batch processing, real-time / stream processing and cron job / scheduled task style processing.
https://cloud.google.com/composer/
Under the hood Composer is running Apache Airflow over managed GKE cluster. So it's not only orchestration tool but it also gives ability to run code using DAGs (which is essentially a cloud function). Have a look at some example DAG triggers below:
https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf
So essentially if you create a conditional DAG trigger then it should do the trick.
Hope this helps.
I have a code base with several apps each with tasks.py, and have a total of 100 of these functions
#periodic_task(run_every=crontab(minute='20'))
def sync_calendar_availability_and_prices(listing_id_list=None, reapply_rules_after_sync=False):
Its in the old format of celery periodic task definition but works fine on celery==4.1.
These get executed every so many hours or mins via beat and also I call them ad-hoc in the codebase by using .delay(). I want all the .delay() calls to go into a certain celery queue manual_call_queue and periodic beat fired calls for same function to go to periodic_beat_fired_queue -- is this an easy 1-2 line config change somewhere at a global level to do this?
I use rabbitmq, celery, django and django-celery-beat
To send periodic tasks to a specific queue, send queue/options arg.
#periodic_task(run_every=crontab(minute='20'), queue='manual_call_queue', options={'queue': 'periodic_beat_fired_queue'})
def sync_calendar_availability_and_prices(listing_id_list=None, reapply_rules_after_sync=False):
queue='manual_call_queue' is used when task is invoked with .delay or .apply_async
options={'queue': 'periodic_beat_fired_queue'} is used when celery beat invokes task.
This question already has answers here:
zappa scheduling with Python
(2 answers)
Closed 5 years ago.
I have been experimenting with deploying Django apps to AWS Lambda with Zappa.
In some of my other (EC2/EBS hosted) Django projects, if there is a need to perform some heavier calculation that can take some time (such as sending a lot of emails, or just some processing that takes over a minute), Celery is used. It is a task queue system where the tasks are sent to a queue, a response can be returned immediately and workers can process the tasks later.
What would be the best way to implement a Celery-like task queuing system for a Zappa-Django app running in Lambda?
Zappa/Lambda supports scheduled tasks, and the models of the app could be designed in such a way that the processing could be done by scheduled functions later and the results could be saved to DB. But I do not think polling for tasks once a minute is robust enough, there is a oftena need to start the delayed task immediately.
Is there an easy way to return a response from a Django view immediately and have a function (from inside the Django app) with arbitrary parameters queued to be executed later?
You can do it using SNS. Subscribe lambda to topic and publish messages there with json payload.
I made a db driven task queue for zappa. https://github.com/andytwoods/zappa-call-later . Early days, but we are using it in production.
Every X minutes, a Zappa event pings a function that checks for tasks. Tasks can be delayed, repeated etc.
I've implemented a small test which uses celery for message queueing and I just want to make sure I understand how it works on a basic level (Django-Celery, Using Redis as a broker).
My understanding is that when I place a call to start an asyncronous task, the task information is placed in redis and then a celeryd instance connected to the broker consumes and executes the task. Is this essentially what is happening?
If I setup a periodic task thats supposed to execute once every hour does that task get executed on all task consumers? If so is there a way to limit it so that only one consumer will ever execute a periodic task?
The workers will consume as many messages as the broker contains. If you have 8 workers, but only 1 message, 1 of the 8 workers will consume the message, executing the task.