Is it possible to get seconds remaining to start scheduled celery task
. eg: my_task.apply_async(countdown=100)
You could see the dump of scheduled tasks http://docs.celeryproject.org/en/latest/userguide/workers.html#dump-of-scheduled-eta-tasks
Every task has eta param, so you can calculate remaining time.
Related
I want to know if we can schedule a DAG to run continuously after 2 minutes of completion of the same DAG in Airflow.
Edit:
My DAG should run in such a way that every time it completes its run, it has to wait for 2 minutes and start running again. I don't want to schedule my DAG to run for every 2 minutes instead it should continuously run right after 2 minutes of completion of the same DAG.
You could schedule your dag at an arbitrary time in a day and use the TriggerDagRunOperator to trigger itself again. To wait for 2 minutes before triggering itself, you could simply introduce a sleep task.
DAG:
Task 1 >> Task 2 >> Task 3 BashOperator(bash_command="sleep 120") >> Task 4 TriggerDagRunOperator(trigger_dag_id="this-dag-id")
Yes, you can schedule DAG to run every 2 Minutes.
Set schedule_interval='*/2 * * * *'
Schedule_interval accepts a CRON expression:
https://en.wikipedia.org/wiki/Cron#CRON_expression
Structuring dag
If you want to rerun continuously after 2 mins try configuring TriggerDagRunOperator
Everyday I have a set of tasks (A few hundred) that needs to performed at random time. So a periodic task is probably not what I want. It seems like I cannot dynamically change crontab
Should I:
Dynamically schedule a task whenever it's received a user, and let celery "wake up" at the scheduled time to perform the task? If so, how is it done?
OR
Create a celery tasks that wakes up every 60 seconds to look into the database for tasks scheduled current time. So the database acts as a queue. I am wondering if this would put too much load on the server?
I have this task which is set to crontab(day_of_month=1). But then when it perform the tasks in continues to send task minutely which is supposed to perform once.
from my tasks.py
from celery.task.schedules import crontab
#periodic_task(run_every=crontab(day_of_month=1))
def Sample():
...
Am I missing something?
By default crontab will run every minute so you need to specify minutes and hours.
Change #periodic_task(run_every=crontab(day_of_month=1)) to #periodic_task(run_every=crontab(minute=0, hour=0, day_of_month=1))
This would run the task only at midnight on the first day of the month.
We set up a schedule to execute a command.
It is scheduled to run every 5 minutes as follows: 20090201T235900|20190201T235900|127|00:05:00
However, from the logs we see it runs only every hour.
Is there a reason for this?
check scheduling frequency in your sitecore.config file
<sitecore>
<scheduling>
<!-- Time between checking for scheduled tasks waiting to execute -->
<frequency>00:05:00</frequency>
</scheduling>
</sitecore>
The scheduling interval is based on the the scheduler interval and the job interval. Every scheduler interval period, all the configured jobs are evaluated. This is logged. During that evaluation, each job checked against the last time it ran, if that interval is greater that the configured job interval, the job is started.
It's fairly simple, but it's important to understand the mechanism. You can also see how it allows no way of inherently running jobs at a specific time, only at approximate intervals.
You can also see that jobs can never run more frequently than the scheduler interval regardless of the job interval. It is not unreasonable to set the scheduler to one-minute intervals to reduce the inaccuracy of job timings to no more than a minute.
In a worse case, with a 5 minute sheduler interval and a 5 minute job interval. The delay to job starting could be up to 9 minutes 59 seconds.
Does anyone know that when a scheduled task in Coldfusion runs it resets the interval timer or does the task run at the set interval time no matter how long that task run for?
For example, i create a task to run every 10 minutes that takes 5 minutes to run starting from 12pm. Will the task run at 12:00, then 12:10, then 12:20 etc etc.
Or would it run at 12:00 which takes 5 minutes, then at 12:15 ten minutes after the task has finished, then another 5 minutes to run so the next one would run at 12:30 etc etc.
Hope that makes sense.
Chris
For example, i create a task to run
every 10 minutes that takes 5 minutes
to run starting from 12pm. Will the
task run at 12:00, then 12:10, then
12:20 etc etc.
Yes.
The task will always run on the interval. So if you set it every 10 minutes it will run every 10 minutes after the first run.
Note: If the task runs over time (i.e. longer than the interval) than it will NOT queue. That particular run will just be skipped and the task will run at the next interval as per usual.
Hope that helps!