I have tried viewing similar answers on stackoverflow to this problem, however my case is slightly different.
I am executing backfill jobs via Airflow CLI, and the backfilled dag runs get stuck in a running state, with the first task in the dag in a queued (grey) state.
The scheduler doesn't seem to ever kick off the first task.
I do not have depends_on_past=True set as dag_defaults
dag_defaults = {
"start_date": datetime.today() - timedelta(days=2),
"on_failure_callback": on_failure_callback,
"provide_context": True
}
I am forced to Run every task manually. :( Rather than just letting the scheduler take its course and run them automatically.
Note: I am executing the backfill cli commands via Airflow worker pods on a K8S cluster.
Has anyone else faced a similar issue using the backfill cli commands?
UPDATE:
I realised my backfill runs fall outside the total dag interval. I.e before the dag start_date causing a blocking schedule dependancy.
While you can still create the run, it will not run automatically, but you can manually run each task.
As a workaround would need to change the start_date to be before or on my oldest backfill date.
Would be nice if there was a way to override the backfill cmd or provide a --force option that could mock the start_date in for that specific dag_run, rather than being bound to the total interval.
UPDATE: I realised my backfill runs fall outside the total dag
interval. I.e before the dag start_date causing a blocking schedule
dependancy.
While you can still create the run, it will not run automatically, but you can manually run each task.
As a workaround would need to change the start_date to be before or on my oldest backfill date.
Would be nice if there was a way to override the backfill cmd or provide a --force option that could mock the start_date in for that specific dag_run, rather than being bound to the total interval.
Related
I want to trigger airflow dags in postman.
recently I am hitting
http://localhost:8080/api/v1/dags/a_pipline/dagRuns/run_id
It's asking for run_id.
How can I specify run id in dags
I have found answers after a lot of struggle. Its working on postman restAPI
POST: http://localhost:8080/api/v1/dags/a_pipline/dagRuns
body: select json formate
{
"dag_run_id": "manual__2022-10-22T11:05:59.461142+00:00"
}
Coming to Airflow from 15 years of using LSF, the terminology trips me up.
A dag_id is its "name"
The above is a "dag_run_id", which is the id of an instance of a dag having run, or in the process of running. What LSF would call it's flow_id.
In LSF these are purely numerical, issued sequentially across all flow instances, 1,2,3 and so on.
In Airflow, these are datetimestamps, which theoretically are not unique across the whole system, but should be within a dag.
In the airflow CLI - you trigger a dag as follows:
airflow dags trigger name_of_the_dag
(glossing over authentication)
I'm just getting to grips with scripts doing this via the REST API...
I have submitted 3 jobs in parallel in AWS Batch and I wanted to create a trigger when all these 3 jobs are completed.
Something like I should be able to specify 3 job ids and can update DB once all 3 jobs are done.
I can do this task easily by having long pooling but wanted to do something based on events.
I need your help with this.
The easiest option would be to create a fourth Batch job that specifies the other three jobs as dependencies. This job will sit in the PENDING state until the other three jobs have succeeded, and then it will run. Inside that job, you could update the DB or do whatever other actions you wanted.
One downside to this approach is that if one of the jobs fails, the pending job will automatically go into a FAILED state without running.
I want to know the status of dag whether it is running or failure or success. I am triggering dag through CL argument airflow trigger and after the execution of job, I want to know the status of the run. I couldn't find any way.
I tried airflow dag_state but it is giving none. What should I do if there are more than one runs in a day to get status of latest run through command line argument or through python code.
You can use list_dag_runs command with the CLI to list the dag runs for a given dag ID. The information returned includes the state of each run.
You can also retrieve the information via python code a few different ways. One such way that I've used in the past is the 'find' method in airflow.models.dagrun.DagRun
An example with python3 on how to get the state of dag_runs via DagRun.find():
dag_id = 'fake_dag_id'
dag_runs = DagRun.find(dag_id=dag_id)
for dag_run in dag_runs:
print(dag_run.state)
You can use the following CL
airflow dag_state dag_id execution_date
Example:
airflow dag_state test_dag_id 2019-11-08T18:36:39.628099+00:00
In the above example:
test_dag_id is the actual dag
2019-11-08T18:36:39.628099+00:00 is the execution date. You can get this from airflow UI for your run.
Other option is to use airflow rest api plugin. This is better option. You can trigger a DAG and also check the status of the dag.
https://github.com/teamclairvoyant/airflow-rest-api-plugin
can I reset start_date of an existing airflow dag without changing its name? I have dag running currently in airflow .
You should never change start_date without changing the dag name. Doing so will lead to unpredictable results. And changing dag name is also safe for your operations as when the dag name changed, the dag under old name will immediately stop running and new dag will initiate in seconds. As it should never happen that an old name dag existing while the new name dag exists simultaneously if you simply change the dag name then save. Especially, it will be highly safe if you are using git to sync your code and always change the dag name when start_date and interval changed.
My tasks are added to Task Queue, but nothing executed automatically. I need to click the button "Run now" to run tasks, tasks are executed without problem. Have I missed some configurations ?
I use default queue configuration, standard App Engine with python 27.
from google.appengine.api import taskqueue
taskqueue.add(
url='/inserturl',
params={'name': 'tablename'})
This documentation is for the API you are now mentioning. The idea would be the same: you need to specify the parameter for when you want the task to be executed. In this case, you have different options, such as countdown or eta. Here is the specific documentation for the method you are using to add a task to the queue (taskqueue.add)
ORIGINAL ANSWER
If you follow this tutorial to create queues and tasks, you will see it is based on the following github repo. If you go to the file where the tasks are created (create_app_engine_queue_task.py). There is where you should specify the time when the task must be executed. In this tutorial, to finally create the task, they use the following command:
python create_app_engine_queue_task.py --project=$PROJECT_ID --location=$LOCATION_ID --queue=$QUEUE_ID --payload=hello
However, it is missing the time when you want to execute it, it should look like this
python create_app_engine_queue_task.py --project=$PROJECT_ID --location=$LOCATION_ID --queue=$QUEUE_ID --payload=hello --in_seconds=["countdown" for when the task will be executed, in seconds]
Basically, the key is in this part of the code in create_app_engine_queue_task.py:
if in_seconds is not None:
# Convert "seconds from now" into an rfc3339 datetime string.
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=in_seconds)
# Create Timestamp protobuf.
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(d)
# Add the timestamp to the tasks.
task['schedule_time'] = timestamp
If you create the task now and you go to your console, you will see you task will execute and disappear from the queue in the amount of seconds you specified.