Evaluating value of run_dag.conf - google-cloud-platform

Question

I'm very new to airflow, and have been playing with it on GCP.
I'm modifying the example at https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf that shows how a DAG can be invoked by a cloud function.
That simple DAG just prints the content of run_dag.conf using the bash operator.
I'm now trying to get the value of run_dag.conf['bucket'] and run_dag.conf['name'] in order to create an example where I use the CloudSqlImportOperator.
My problem is that it seems that I can't find a way to get those values to be passed as part of the body on the operator.
My understanding is that jinja templates get evaluated at the operators. My first attempt was to do:
import_body = {
"importContext": {
"fileType": "csv",
"database": "dw",
"uri": "gs://{{ dag_run.conf['bucket'] }}/{{ dag_run.conf['name'] }}",
"csvImportOptions": {
"table": "SHAKESPEARE",
"columns": ["word", "word_count", "corpus", "corpus_date"]
}
}
}
And that fails because the jinja template section never gets evaluated, and the operator receives a literal "gs://{{ dag_run.conf['bucket'] }}/{{ dag_run.conf['name'] }}" instead.
I tried to pass a string instead:
import_body = """{
"importContext": {
"fileType": "csv",
"database": "dw",
"uri": "gs://{{ dag_run.conf['bucket'] }}/{{ dag_run.conf['name'] }}",
"csvImportOptions": {
"table": "SHAKESPEARE",
"columns": ["word", "word_count", "corpus", "corpus_date"]
}
}
}"""
And still I'm getting an error now : 'str' object has no attribute 'get'
I've seen examples using the PythonOperator and kwargs to fetch the contents, but so far no example of reading the contents of that dag_run.conf object inside the code.
What would be a proper way of doing that?
Cheers

Answer 1

In the example you mentioned, the jinja template is passed to the parameter bash_command which is a templated field. If you look at the python operator source code you can see that the only templated parameter is templated_dict so you to make airflow evaluate the {{ dag_run.conf['bucket'] }} you need to pass it through this variable. I am shooting into the dark here because you did not post the full code but the solution should be something like the following:
Inside the python code that your python operator call (works with python3):
import_body = f'''{
"importContext": {
"fileType": "csv",
"database": "dw",
"uri": "gs://{templated_dict['bucket']}/{templated_dict['name']}",
"csvImportOptions": {
"table": "SHAKESPEARE",
"columns": ["word", "word_count", "corpus", "corpus_date"]
}
}
}'''
when you define the python operator in the DAG:
python_operator.PythonOperator(
task_id=f'task_id',
python_callable=my_func,
provide_context=True,
templated_dict={
"bucket": "{{ dag_run.conf['bucket'] }}",
"name": "{{ dag_run.conf['name'] }}"
},
dag=dag
)
Note that I referenced to the airflow version 1.10.2, which I assume you are running because you tagged google cloud composer and this is the latest version supported.
If you look at 1.10.3 you can see that op_args and op_kwargs are added to the templated fields of the python operator. So in the next version update, you will be able to pass it as ousing also those.