i need to implement a comprehensive retry schedule in Airflow, say 3x with 5min interval and then 4x with 2h interval. what is the best way to do so? i am thinking to run the same task with the second retry schedule from on failure callback of the task with the first retry schedule, but the problem is that in that case i cannot (or don't know how) to make downstream task(s) depend on the task executed from callback only if it was started in the first place. maybe my idea is totally wrong. any advice?
We are using AWS SWF for our workflows where we need to schedule an activity based on cron expression. We are evaluating the Cron and using WorkflowClock for creating the timer.
Wnated to get answers for the following questions
Can workflow clock sometimes fail to deliver the control back to workflow. Does SWF guarantee the timer will definitely go off and schedule the next decision
By what deviation can timer delivered be off by original cron, meaning if we start a timer of 3600 seconds, can timer get delayed and go off by let's say after 3700 seconds . Any p100 data for this ?
Are these timers on exactly-once or atleast-once delivery model
I believe timer delivery is guaranteed.
I don't have data, but I think it is delivered within a second unless there is a major issue with the service going on. Note that delivering timer means adding TimerFired event into the history and scheduling a decision task. If a workflow worker is down then actual timer processing can be postponed for a long time.
They are delivered to a workflow exactly once.
Is there a way to set a maximum running time for AWS Batch jobs (or queues)? This is a standard setting in most batch managers, which avoids wasting resources when a job hangs for whatever reason.
As of April, 2018, AWS Batch now supports setting a Job Timeout when submitting a Job, or in the job definition.
https://aws.amazon.com/about-aws/whats-new/2018/04/aws-batch-adds-support-for-automatic-termination-with-job-execution-timeout/
You specify an attemptDurationSeconds parameter, which must be at least 60 seconds, either in your job definition, or when you submit the job. When this number of seconds has passed following the job attempt's startedAt timestamp, AWS Batch terminates the job. On the compute resource, your job's container receives a SIGTERM signal to give your application a chance to shut down gracefully; if the container is still running after 30 seconds, a SIGKILL signal is sent to forcefully shut down the container.
Source: https://docs.aws.amazon.com/batch/latest/userguide/job_timeouts.html
POST /v1/submitjob HTTP/1.1
Content-type: application/json
{
...
"timeout": {
"attemptDurationSeconds": number
}
}
AFAIK there is no feature to do this. However, a workaround was suggested in the forum for a similar question.
One idea is to call Batch as an Activity from Step Functions, pingback
back on a schedule (e.g. every minute) from that job. If it stops
responding then you can detect that situation as a Timeout in the
activity and act accordingly (terminate the job etc.). Not an ideal
solution (especially if the job continues to ping back as a "zombie"),
but it's a start. You'd also likely have to store activity tokens in a
database to trace them to Batch job id.
Alternatively, you split that setup into 2 steps, and schedule a Batch
job from a Lambda in the first state, then pass the Batch job id to
the second step which then polls Batch (from another Lambda) for its
state with Retry and IntervalSeconds (e.g. once every minute, or even
with exponential backoff), and MaxAttempts calculated based on your
timeout. This way, you don't need any external state storage
mechanism, long polling or even a "ping back" from the job (it CAN be
a zombie), but the downside is more steps.
There is no option to set timeout on batch job but you can setup a lambda function that triggers every 1 hour or so and deletes jobs created before say 24 hours.
working with aws for some time now and could not find a way to set a maximum running time for batch jobs.
However there are some alternative way which you could utilize.
AWS Forum
Sadly there is no way to set the limit execution time on AWS Batch.
One solution may be to edit the docker's entry point to schedule the execution time limit.
Is it possible to get seconds remaining to start scheduled celery task
. eg: my_task.apply_async(countdown=100)
You could see the dump of scheduled tasks http://docs.celeryproject.org/en/latest/userguide/workers.html#dump-of-scheduled-eta-tasks
Every task has eta param, so you can calculate remaining time.
Does anyone know that when a scheduled task in Coldfusion runs it resets the interval timer or does the task run at the set interval time no matter how long that task run for?
For example, i create a task to run every 10 minutes that takes 5 minutes to run starting from 12pm. Will the task run at 12:00, then 12:10, then 12:20 etc etc.
Or would it run at 12:00 which takes 5 minutes, then at 12:15 ten minutes after the task has finished, then another 5 minutes to run so the next one would run at 12:30 etc etc.
Hope that makes sense.
Chris
For example, i create a task to run
every 10 minutes that takes 5 minutes
to run starting from 12pm. Will the
task run at 12:00, then 12:10, then
12:20 etc etc.
Yes.
The task will always run on the interval. So if you set it every 10 minutes it will run every 10 minutes after the first run.
Note: If the task runs over time (i.e. longer than the interval) than it will NOT queue. That particular run will just be skipped and the task will run at the next interval as per usual.
Hope that helps!