I am trying to do a backfill in Google Composer using a gcloud composer command but I am struggling to pass a -y or --yes for a corresponding --reset_dagruns argument.
The response I get is airflow: error: unrecognized arguments: -y.
Command:
gcloud composer environments run my_env --project my_project --location us-east1 backfill -- --reset_dagruns -y -s 2020-01-01 -e 2020-01-31 my_dag
How can I supply this argument?
Before answering your specific question I have a suggestion:
If your DAG (my_dag) already specifies the start & end date over which you need backfill, then just clear the status of runs you need backfilled/redone. The scheduler then picks them up again.
For your question specifically:
The AirFlow CLI documentation has unfortunately not proved directly usable with Google Cloud Composer, this is what works on Composer version 1.10.2
gcloud --project=my_project composer environments run my_env --location us-east1 backfill -- my_dag --reset_dagruns -s 2020-01-01 -e 2020-01-31
Note that this will hold on to your CLI session and will stop if you ctrl-c out, if your backfill is long you are better off defining a start date on the dag and setting catchup=True
In case anyone else has this issue, I was not able to find a way to pass in the argument using gcloud composer.
Instead I used gcloud container clusters get-credentials to do a kubectl exec to get a bash prompt on the airflow scheduler. From there I was able to use the native airflow command to move things along.
Related
Is there a GCP command for creating OR replacing a cloud run job? I'm using github-actions to create cloud run and scheduler jobs, and need to keep switching the commands between:
gcloud alpha run jobs create
and
gcloud alpha run jobs update
Is there a way to create the job and overwrite it if it already exists?
gcloud beta run jobs deploy was recently added to gcloud which does what you're looking for. Documentation is here
To create a Cloud Run new job :
gcloud beta run jobs create JOB_NAME --image IMAGE_URL OPTIONS
To update existing job :
gcloud beta run jobs update JOB_NAME
If you want a single command to handle the creation and update at the same time, you can develop your own Shell script, example :
#!/usr/bin/env bash
set -e
set -o pipefail
set -u
export JOB_NAME=my_job
res=$(gcloud beta run jobs describe $JOB_NAME --region=europe-west1 || echo "NOT_EXIST")
echo "#######Result : $res"
if [ "$res" = "NOT_EXIST" ]; then
echo "Creating your job..."
gcloud beta run jobs create $JOB_NAME
else
echo "Updating your job..."
gcloud beta run jobs update $JOB_NAME
fi
This is a strange one.
A Google Cloud Run deployment run from gcloud commandline on my OSx Mac works — while the identical command run from the identical gcloud version, using a Service Account user within our Alpine based Ci/Cd Gitlab runner container / executor crashes and complains about un-recognized arguments.
With the arguments copied and pasted why is the gcloud (within Alpine gitlab runner / executor container) failing due to not recognizing the arguments where my local install works fine?
As background:
We run Ci/Cd within a Gitlab Ci Runner where the docker executor that deploys our final container previously needed to use Kubectl to push that container to a GCP Managed K8s Cluster — which was expensive. So we moved the production container to Cloud Run — and it was cheaper.
Now I am working on resetting our Ci/CD deployments and ran into the above issue while attempting to deploy a container from within our GitLab Ci pipeline.
The gcloud command that works looks like this (on my local Mac)
gcloud run deploy site-production \
--platform=managed \
--allow-unauthenticated \
--image=us.gcr.io/some-site-333333/site:master \
--region=us-east1
That same (EXACT) command on the GitLab runner gets me:
ERROR: (gcloud.run.deploy) unrecognized arguments:
--platform=managed
--allow-unauthenticated
--image=us.gcr.io/some-site-333333/site:master
--region=us-east1
To search the help text of gcloud commands, run:
gcloud help -- SEARCH_TERMS
Seems super weird — and I was pretty sure I must have had a typo or something — but the command itself was copied (and modified) from Google's own Cloud Run docs.
If I am missing something dumb let me know — until then my plan is to start shaving off optional flags to try to see which one of those parameters it's complaining about. Ideas are appreciated!
Try to make a one liner command like:
gcloud run deploy site-production --platform=managed --allow-unauthenticated --image=us.gcr.io/some-site-333333/site:master --region=us-east1
I'm aware that many of the common Airflow management commands are made available through the gcloud CLI. However, I'm troubleshooting some DAG scheduling and would like to use the schedule and next_execution commands directly on the cluster.
Is there an easy way to do this?
It's possible to access the full Airflow CLI by using kubectl exec to SSH into Composer pods. To do so, obtain the name of the GKE cluster associated with your environment, and get cluster credentials for it:
gcloud container clusters get-credentials $CLUSTER_NAME --zone=$ZONE
Then, use kubectl to check for the Composer namespace, and then find a pod and SSH to it:
kubectl get namespaces | grep composer
kubectl get pods --namespace=$NAMESPACE | grep airflow
kubectl exec -it --namespace=$NAMESPACE $POD_NAME -- bash
From within a pod, you can use airflow with any command supported by that version of Airflow. However, it should also be noted that this also provides full access to commands that can make your environment permanently unusable (such as resetdb), so they should be used with care.
I'm working on Cloud Run, which seems to be beta yet, preventing from redeploying as shown below. It works if I delete the service from GCP console, then deploy the same Docker as a new service. I could not find a way to to set revisionTemplate.
I run this command to deploy a Cloud Run service using gcloud.
gcloud beta run deploy v2-cms --image gcr.io/my-project/v2-cms --quiet
Then, it fails saying like this.
X Deploying...
. Creating Revision...
. Routing traffic...
Deployment failed
ERROR: gcloud crashed (AttributeError): 'NoneType' object has no attribute 'revisionTemplate'
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
To fix this issue, please update gcloud to ite latest version with gcloud components update
Make sure that your local Tensorflow version is still supported by GCloud https://cloud.google.com/ai-platform/training/docs/runtime-version-list
I tried to run airflow test cli in the Google Cloud Composer environment but it does not work.
Basically, I want to run airflow test to test a task in the airflow environment. I am following the instruction here: https://cloud.google.com/composer/docs/how-to/accessing/airflow-cli
This is the command I run:
gcloud beta composer environments run ENVIRONMENT_NAME test MY_DAG FIRST_TASK 2018-05-05
Output:
ERROR: (gcloud.beta.composer.environments.run) unrecognized arguments:
You need to include two hyphens between the Airflow sub-command ("test") and its args. The hyphens direct gcloud to ignore the args that follow and pass them through to the Airflow sub-command.
gcloud beta composer environments run ENVIRONMENT_NAME test -- MY_DAG FIRST_TASK 2018-05-05
Reference: https://cloud.google.com/sdk/gcloud/reference/beta/composer/environments/run