Is there a GCP command for creating OR replacing a cloud run job? I'm using github-actions to create cloud run and scheduler jobs, and need to keep switching the commands between:
gcloud alpha run jobs create
and
gcloud alpha run jobs update
Is there a way to create the job and overwrite it if it already exists?
gcloud beta run jobs deploy was recently added to gcloud which does what you're looking for. Documentation is here
To create a Cloud Run new job :
gcloud beta run jobs create JOB_NAME --image IMAGE_URL OPTIONS
To update existing job :
gcloud beta run jobs update JOB_NAME
If you want a single command to handle the creation and update at the same time, you can develop your own Shell script, example :
#!/usr/bin/env bash
set -e
set -o pipefail
set -u
export JOB_NAME=my_job
res=$(gcloud beta run jobs describe $JOB_NAME --region=europe-west1 || echo "NOT_EXIST")
echo "#######Result : $res"
if [ "$res" = "NOT_EXIST" ]; then
echo "Creating your job..."
gcloud beta run jobs create $JOB_NAME
else
echo "Updating your job..."
gcloud beta run jobs update $JOB_NAME
fi
Related
I want to implement an automated code to shut down VM Instances in my Google Cloud account via Crontab. I use Ubuntu 18.
I tried this code:
#!/bin/bash
source/etc/profile
echo Y | gcloud compute instances stop my-vm-name --zone us-central1-a
sleep 120s
echo "completed"
It's work mannually but when i add this code in cron
20 17 * * * /bin/sh /home/user/Desktop/script.sh >>/home/user/cron.log
The VM doesn't stop in my GC compute engine. I made the answers to this question but, the rest does not work.
Google cloud SDK code to execute via cron
Can anyone help?
I am using the new way to deploy google cloud run (and loving!) but how can I pass the service name on the command?
I saw the docs but nothing related.
gcloud beta run deploy --source .
Service name: my-cloud-run-name
How can I pass the service name by the command line? Something like:
gcloud beta run deploy --name my-cloud-run-name --source .
Use a positional argument: gcloud beta run deploy SERVICE-NAME --source .
I am trying to do a backfill in Google Composer using a gcloud composer command but I am struggling to pass a -y or --yes for a corresponding --reset_dagruns argument.
The response I get is airflow: error: unrecognized arguments: -y.
Command:
gcloud composer environments run my_env --project my_project --location us-east1 backfill -- --reset_dagruns -y -s 2020-01-01 -e 2020-01-31 my_dag
How can I supply this argument?
Before answering your specific question I have a suggestion:
If your DAG (my_dag) already specifies the start & end date over which you need backfill, then just clear the status of runs you need backfilled/redone. The scheduler then picks them up again.
For your question specifically:
The AirFlow CLI documentation has unfortunately not proved directly usable with Google Cloud Composer, this is what works on Composer version 1.10.2
gcloud --project=my_project composer environments run my_env --location us-east1 backfill -- my_dag --reset_dagruns -s 2020-01-01 -e 2020-01-31
Note that this will hold on to your CLI session and will stop if you ctrl-c out, if your backfill is long you are better off defining a start date on the dag and setting catchup=True
In case anyone else has this issue, I was not able to find a way to pass in the argument using gcloud composer.
Instead I used gcloud container clusters get-credentials to do a kubectl exec to get a bash prompt on the airflow scheduler. From there I was able to use the native airflow command to move things along.
When using gcloud run deploy, how can I specify the service name with command-line args? I'm hoping to prevent the need for interactivity at deploy-time.
I'm currently deploying my service like so:
gcloud run deploy --image gcr.io/<PROJECT>/<TAG> --platform managed
There is a service parameter, which is positional as opposed to named.
From the docs:
gcloud run deploy [[SERVICE] --namespace=NAMESPACE] etc...
So you could do this:
gcloud run deploy <SERVICE_NAME> --image gcr.io/<PROJECT>/<TAG> --platform managed
https://cloud.google.com/sdk/gcloud/reference/run/deploy#POSITIONAL-ARGUMENTS
I tried to run airflow test cli in the Google Cloud Composer environment but it does not work.
Basically, I want to run airflow test to test a task in the airflow environment. I am following the instruction here: https://cloud.google.com/composer/docs/how-to/accessing/airflow-cli
This is the command I run:
gcloud beta composer environments run ENVIRONMENT_NAME test MY_DAG FIRST_TASK 2018-05-05
Output:
ERROR: (gcloud.beta.composer.environments.run) unrecognized arguments:
You need to include two hyphens between the Airflow sub-command ("test") and its args. The hyphens direct gcloud to ignore the args that follow and pass them through to the Airflow sub-command.
gcloud beta composer environments run ENVIRONMENT_NAME test -- MY_DAG FIRST_TASK 2018-05-05
Reference: https://cloud.google.com/sdk/gcloud/reference/beta/composer/environments/run