I tried to run airflow test cli in the Google Cloud Composer environment but it does not work.
Basically, I want to run airflow test to test a task in the airflow environment. I am following the instruction here: https://cloud.google.com/composer/docs/how-to/accessing/airflow-cli
This is the command I run:
gcloud beta composer environments run ENVIRONMENT_NAME test MY_DAG FIRST_TASK 2018-05-05
Output:
ERROR: (gcloud.beta.composer.environments.run) unrecognized arguments:
You need to include two hyphens between the Airflow sub-command ("test") and its args. The hyphens direct gcloud to ignore the args that follow and pass them through to the Airflow sub-command.
gcloud beta composer environments run ENVIRONMENT_NAME test -- MY_DAG FIRST_TASK 2018-05-05
Reference: https://cloud.google.com/sdk/gcloud/reference/beta/composer/environments/run
Related
This is a strange one.
A Google Cloud Run deployment run from gcloud commandline on my OSx Mac works — while the identical command run from the identical gcloud version, using a Service Account user within our Alpine based Ci/Cd Gitlab runner container / executor crashes and complains about un-recognized arguments.
With the arguments copied and pasted why is the gcloud (within Alpine gitlab runner / executor container) failing due to not recognizing the arguments where my local install works fine?
As background:
We run Ci/Cd within a Gitlab Ci Runner where the docker executor that deploys our final container previously needed to use Kubectl to push that container to a GCP Managed K8s Cluster — which was expensive. So we moved the production container to Cloud Run — and it was cheaper.
Now I am working on resetting our Ci/CD deployments and ran into the above issue while attempting to deploy a container from within our GitLab Ci pipeline.
The gcloud command that works looks like this (on my local Mac)
gcloud run deploy site-production \
--platform=managed \
--allow-unauthenticated \
--image=us.gcr.io/some-site-333333/site:master \
--region=us-east1
That same (EXACT) command on the GitLab runner gets me:
ERROR: (gcloud.run.deploy) unrecognized arguments:
--platform=managed
--allow-unauthenticated
--image=us.gcr.io/some-site-333333/site:master
--region=us-east1
To search the help text of gcloud commands, run:
gcloud help -- SEARCH_TERMS
Seems super weird — and I was pretty sure I must have had a typo or something — but the command itself was copied (and modified) from Google's own Cloud Run docs.
If I am missing something dumb let me know — until then my plan is to start shaving off optional flags to try to see which one of those parameters it's complaining about. Ideas are appreciated!
Try to make a one liner command like:
gcloud run deploy site-production --platform=managed --allow-unauthenticated --image=us.gcr.io/some-site-333333/site:master --region=us-east1
I am using the new way to deploy google cloud run (and loving!) but how can I pass the service name on the command?
I saw the docs but nothing related.
gcloud beta run deploy --source .
Service name: my-cloud-run-name
How can I pass the service name by the command line? Something like:
gcloud beta run deploy --name my-cloud-run-name --source .
Use a positional argument: gcloud beta run deploy SERVICE-NAME --source .
I am trying to do a backfill in Google Composer using a gcloud composer command but I am struggling to pass a -y or --yes for a corresponding --reset_dagruns argument.
The response I get is airflow: error: unrecognized arguments: -y.
Command:
gcloud composer environments run my_env --project my_project --location us-east1 backfill -- --reset_dagruns -y -s 2020-01-01 -e 2020-01-31 my_dag
How can I supply this argument?
Before answering your specific question I have a suggestion:
If your DAG (my_dag) already specifies the start & end date over which you need backfill, then just clear the status of runs you need backfilled/redone. The scheduler then picks them up again.
For your question specifically:
The AirFlow CLI documentation has unfortunately not proved directly usable with Google Cloud Composer, this is what works on Composer version 1.10.2
gcloud --project=my_project composer environments run my_env --location us-east1 backfill -- my_dag --reset_dagruns -s 2020-01-01 -e 2020-01-31
Note that this will hold on to your CLI session and will stop if you ctrl-c out, if your backfill is long you are better off defining a start date on the dag and setting catchup=True
In case anyone else has this issue, I was not able to find a way to pass in the argument using gcloud composer.
Instead I used gcloud container clusters get-credentials to do a kubectl exec to get a bash prompt on the airflow scheduler. From there I was able to use the native airflow command to move things along.
My Azure DevOps pipeline runs on hosted ubuntu 16.04 agent.
The first bash script installs Google SDK. Then:
gcloud auth activate-service-account
Activated service account credentials for:
gcloud config set project $1
Updated property [core/project].
gcloud info --run-diagnostics
Property diagnostic passed (1/1 checks passed).
bq version
This is BigQuery CLI 2.0.47
All works fine until:
bq ls
and script stalls and does nothing
Permissions are fine as same script runs with no problem on VM.
Same with commands like bq rm,bq mk - the pipeline just stucks.
Any ideas what can be wrong or how to troubleshoot the issue?
I'm working on Cloud Run, which seems to be beta yet, preventing from redeploying as shown below. It works if I delete the service from GCP console, then deploy the same Docker as a new service. I could not find a way to to set revisionTemplate.
I run this command to deploy a Cloud Run service using gcloud.
gcloud beta run deploy v2-cms --image gcr.io/my-project/v2-cms --quiet
Then, it fails saying like this.
X Deploying...
. Creating Revision...
. Routing traffic...
Deployment failed
ERROR: gcloud crashed (AttributeError): 'NoneType' object has no attribute 'revisionTemplate'
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
To fix this issue, please update gcloud to ite latest version with gcloud components update
Make sure that your local Tensorflow version is still supported by GCloud https://cloud.google.com/ai-platform/training/docs/runtime-version-list