Importing airflow variables in a json file using gitlab ci/cd - google-cloud-platform

Question

How to import variables using gitlab ci/cd yml file.
I have found Importing airflow variables in a json file using the command line but not helping out

Answer 1

You can import a Json file as Airflow variables.
variables.json file :
{
"feature": {
"param1": "param1",
"param2": "param2",
...
}
}
For example, this file can be put in the following structure :
my-project
config
dags
variables
dev
variables.json
prd
variables.json
You can then create a Shell script to deploy these variables and file to Cloud Composer, deploy_dags_config.sh file :
#!/usr/bin/env bash
set -e
set -o pipefail
set -u
export FEATURE_NAME=my_feature
export ENV=dev
export COMPOSER_ENVIRONMENT=my-composer-env
export ENVIRONMENT_LOCATION=europe-west1
export GCP_PROJECT_ID=my-gcp-project
echo "### Deploying the data config variables of module ${FEATURE_NAME} to composer"
# deploy variables
gcloud composer environments storage data import \
--source config/dags/variables/${ENV}/variables.json \
--destination "${FEATURE_NAME}"/config \
--environment ${COMPOSER_ENVIRONMENT} \
--location ${ENVIRONMENT_LOCATION} \
--project ${GCP_PROJECT_ID}
gcloud beta composer environments run ${COMPOSER_ENVIRONMENT} \
--project ${GCP_PROJECT_ID} \
--location ${ENVIRONMENT_LOCATION} \
variables import \
-- /home/airflow/gcs/data/"${FEATURE_NAME}"/config/variables.json
echo "Variables of ${FEATURE_NAME} are well imported in environment ${COMPOSER_ENVIRONMENT} for project ${GCP_PROJECT_ID}"
This Shell script is the used in Gitlab CI yaml file :
deploy_conf:
image: google/cloud-sdk:416.0.0
script:
- . ./authentication.sh
- . ./deploy_dags_config.sh
Your Gitlab have to be authenticated to GCP.
In the Airflow DAG code, the variables can be then retrieved in a Dict as follow :
from typing import Dict
from airflow.models import Variable
variables:Dict = Variable.get("feature", deserialize_json=True)
Because the root node of variables.json file and object is feature (this name should be unique) :
{
"feature": {
"param1": "param1",
"param2": "param2",
...
}
}