How to extract actual timestamp in Cloud Build CI/CD pipeline yaml script or Cloud Build Triggers page - google-cloud-platform

I have a cloud_build.yaml script for my CI/CD pipeline on GP using Cloud Build. In command line I can pass a subtitution variable which will include the actual timestamp: "notebook-instance-$(date +%Y-%m-%d-%H-%M)-v05". This is working fine.
When I add github trigger on the Cloud Build webpage, then I didn't find the way to get the timestamp extracted in the same way that I was using in cli $(date +%Y-%m-%d-%H-%M)-v05:
Any idea idea how to do that on the Triggers Cloud Build page ?
I aslo tried to do it inside the cloud_build.yaml script but without success for now.
- name: 'gcr.io/cloud-builders/gcloud'
id: Deploy the AI Platform Notebook instance
args:
- 'deployment-manager'
- 'deployments'
- 'create'
- '$(date -u +%Y-%m-%d-%H-%M)-${_NAME_INSTANCE}'
Any idea how to extract and create a variable using the actual timestamp in the .yaml CloudBuild script ?
A third option is to extract the timestamp in my .jinja deployment script. Here I get the same issue as well that I don't find the way to to extract the actual timestampt to build my variable name.

One of the solution is to do the following:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: sh
args:
- '-c'
- |
gcloud \
deployment-manager \
deployments \
create \
xxxx
The issue is that you cannot use it in another step later. Another option is is to write te variable in a file on the workspace. This can be access later during the build stackoverflow

Related

How to deploy Cloud Functions with secrets from Secret Manager using Cloud Build?

I have a Cloud Function that I want deployed in my CD pipeline using Cloud Build. The function needs a couple of secrets stored in Secret Manager that I want to pull in as environment variables using the --set-secrets flag.
When I deploy manually with the CLI I have no issue:
gcloud beta functions deploy myfunction \
--source src \
--trigger-topic mytopic \
--region europe-west1 \
--runtime python39 \
--set-secrets 'env_1=secret_1:latest','env_2=secret_2:latest'
However, when I try to deploy using Cloud Build with this configuration:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args:
- beta
- functions
- deploy
- myfunction
- --source=src
- --trigger-topic=mytopic
- --region=europe-west1
- --runtime=python39
- --set-secrets='env_1=secret_1:latest','env_2=secret_2:latest'
I get an error that the --set-secrets argument must match the pattern 'SECRET:VERSION' or 'projects/{PROJECT}/secrets/{SECRET}:{VERSION}' or 'projects/{PROJECT}/secrets/{SECRET}/versions/{VERSION}' where VERSION is a number or the label 'latest'. I don't understand why I get this error as I think my argument comforms to said pattern.
Is there something I am missing?
First, follow Guillaume's suggestion to remove the quotation marks around each pair. Afterwards, it should look like this:
--set-secrets=env_1=secret_1:latest,env_2=secret_2:latest
Or alternatively, my suggestion is to enclose all your arguments as a list like the example below. I tested the config below and it worked on my end.
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
args: ['gcloud', 'beta','functions', 'deploy', 'myfunction', '--region=europe-west1', '--source=src', '--trigger-topic=mytopic', '--runtime=python39', '--set-secrets=env_1=secret_1:latest,env_2=secret_2:latest']
Note: Do not put spaces in --set-secrets value if you have multiple secrets
To learn more, check out this documentation.
Here is some documentation: https://cloud.google.com/build/docs/securing-builds/use-secrets
You need to use the secretEnv key as well as the availableSecrets declaration in your cloudbuild.yaml
In addition to Cadet's answer as this was what I needed, and I believe is best practice. https://cloud.google.com/build/docs/securing-builds/use-secrets
For those looking to utilise secrets manager, this is how to write your script
--set-secrets=app_user=projects/369177123123/secrets/app_user:1,app_pass=projects/369177123123/secrets/app_pass:1
Make sure that the secret names "app_user" and "app_pass" correspond with what you have named in your secrets manager.
Also, take note of the project number "369177123123" as it must match your own project number. You can get your project number by visiting your dashboard. or see this link https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects
Lastly, make sure you give the service account access to the secrets. See https://cloud.google.com/build/docs/securing-builds/use-secrets#grant_permissions

Is there a way to inspect the process.env variables on a cloud run service?

After deployment, is there a way to inspect the process.env variables on a running cloud run service?
I thought they would be available in the following page:
https://console.cloud.google.com/run/detail
Is there a way to make them available here? Or to inspect it in some other way?
PS: This is a Docker container.
I have the following ENV on my Dockerfile. And I know they are present, because everything is working as it should. But I cannot see them in the service details:
Dockerfile
ENV NODE_ENV=production
ENV PROJECT_ID=$PROJECT_ID
ENV SERVER_ENV=$SERVER_ENV
I'm using a cloudbuild.yaml file. The ENV directives are present in my Dockerfile, and they are being passed to my container. Maybe I should add env to my cloudbuild.yaml file? Because I'm using --substitutions on my gcloub builds sumbmit call and they are passed as --build-arg to my Docker build step. But I'm not declaring them as env in my cloudbuild.yaml.
I followed the official documentation and set the environment variables on a Cloud Run service using the console.Then I was able to list them on the Google Cloud Console.
You can set environment variables using the Cloud Console, the gcloud
command line, or a YAML file when you create a new service or deploy a
new revision:
With the help of #marian.vladoi's answer. This what I've ended up doing
In my deploy step from cloudbuild.yaml file:
I added the --set-env-vars parameter
steps:
# DEPLOY CONTAINER WITH GCLOUD
- name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
entrypoint: gcloud
args:
- "beta"
- "run"
- "deploy"
- "SERVICE_NAME"
- "--image=gcr.io/$PROJECT_ID/SERVICE_NAME:$_TAG_NAME"
- "--platform=managed"
- "--region=us-central1"
- "--min-instances=$_MIN_INSTANCES"
- "--max-instances=3"
- "--set-env-vars=PROJECT_ID=$PROJECT_ID,SERVER_ENV=$_SERVER_ENV,NODE_ENV=production"
- "--port=8080"
- "--allow-unauthenticated"
timeout: 180s

How can I call gcloud commands from a shell script during a build step?

I have automatic builds set up in Google Cloud, so that each time I push to the master branch of my repository, a new image is built and pushed to Google Container Registry.
These images pile up quickly, and I don't need all the old ones. So I would like to add a build step that runs a bash script which calls gcloud container images list-tags, loops the results, and deletes the old ones with gcloud container images delete.
I have the script written and it works locally. I am having trouble figuring out how to run it as a step in Cloud Builder.
It seems there are 2 options:
- name: 'ubuntu'
args: ['bash', './container-registry-cleanup.sh']
In the above step in cloudbuild.yml I try to run the bash command in the ubuntu image. This doesn't work because the gcloud command does not exist in this image.
- name: 'gcr.io/cloud-builders/gcloud'
args: [what goes here???]
In the above step in cloudbuild.yml I try to use the gcloud image, but since "Arguments passed to this builder will be passed to gcloud directly", I don't know how to call my bash script here.
What can I do?
You can customize the entry point of your build step. If you need gcloud installed, use the gcloud cloud builder and do this
step:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: "bash"
args:
- "-c"
- |
echo "enter 1 bash command per line"
ls -la
gcloud version
...
As per the official documentation Creating custom build steps indicates, you need a custom build step to execute a shell script from your source, the step's container image must contain a tool capable of running the script.
The below example, shows how to configure your args, for the execution to perform correctly.
steps:
- name: 'ubuntu'
args: ['bash', './myscript.bash']
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/custom-script-test', '.']
images: ['gcr.io/$PROJECT_ID/custom-script-test']
I would recommend you to take a look at the above documentation and the example as well, to test and confirm if it will help you achieve the execution of the script.
For your case, specifically, there is this other answer here, where is indicated that you will need to override the endpoint of the build to bash, so the script runs. It's indicated as follow:
- name: gcr.io/cloud-builders/gcloud
entrypoint: /bin/bash
args: ['-c', 'gcloud compute instances list > gce-list.txt']
Besides that, these two below articles, include more information and examples on how to configure customized scripts to run in your Cloud Build, that I would recommend you to take a look.
CI/CD: Google Cloud Build — Custom Scripts
Mastering Google Cloud Build Config Syntax
Let me know if the information helped you!

Google cloud build - custom machine type

I'm using the Google Cloud Build service to create images of my application. I created a build trigger that looks for a git tag in a specific format. Each time that Cloud Build detects a new tag, a new build is performed.
Since the build time is pretty long, I am trying to make it faster.
I found that it's possible to ask Google to build the application on a faster machine (Source).
gcloud builds submit --config=cloudbuild.yaml --machine-type=n1-highcpu-8 .
This code works if you choose the manual build option. Since I created the build trigger from the GCP user interface, I can't find any place to define the machine-type argument.
How can I choose the machine-type on automatic build triggers?
UPDATE:
In the Trigger window, I chose Build Configuration=Docker File and this is my docker file preview:
docker build \
-t gcr.io/PROJ_NAME/APP_NAME/$TAG_NAME:$COMMIT_SHA \
-f deployments/docker/APPNAME.docker \
.
How should my buildconfig.yaml file look like?
You need to change to Build Configuration=Cloud Build configuration file, and commit the cloudbuild.yaml to git.
Then use the machineType field in the options property of your cloudbuild.yaml file.
E.g
steps:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/PROJ_NAME/APP_NAME/$TAG_NAME:$COMMIT_SHA', '-f', 'deployments/docker/APPNAME.docker', '.']
options:
machineType: 'N1_HIGHCPU_8'

How can I save google cloud build step text output to file

I'm trying to use google cloud build. At one step, I need to get a list of all running compute instances.
- name: gcr.io/cloud-builders/gcloud
args: ['compute', 'instances', 'list']
and it works fine. Problem starts when I tried to save the output to a file
Trial 1: failed
- name: gcr.io/cloud-builders/gcloud
args: ['compute', 'instances', 'list', '> gce-list.txt']
Trial 2: failed
- name: gcr.io/cloud-builders/gcloud
args: ['compute', 'instances', 'list', '>', 'gce-list.txt']
Trial 3: failed
- name: gcr.io/cloud-builders/gcloud
args: >
compute instances list > gce-list.txt
Trial 4: failed
- name: gcr.io/cloud-builders/gcloud
args: |
compute instances list > gce-list.txt
UPDATE: 2018-09-04 17:50
Trial 5: failed
Build an gcloud image based on ubuntu
Used that image to run custom script file 'list-gce.sh'
list-gce.sh calls gcloud compute instances list
For more details you can check this gist:
https://gist.github.com/mahmoud-samy/e67f141e8b5d553de68a58a30a432ed2
Unfortunately I got this strange error:
rev 1
ERROR: (gcloud) unrecognized arguments: list (did you mean 'list'?)
rev 2
ERROR: (gcloud) unrecognized arguments: --version (did you mean '--version'?)
Any suggestions, or references?
In addition to other answers, to do cmd > foo.txt, you need to override the build entrypoint to bash (or sh):
- name: gcr.io/cloud-builders/gcloud
entrypoint: /bin/bash
args: ['-c', 'gcloud compute instances list > gce-list.txt']
Those commands are not executed in a shell, so shell operations such as pipes (|) and redirections (>) are not available.
Workaround
Use a gcloud container which does have a shell. The gcr.io/cloud-builders/gcloud container should have bash, as it is ultimately derived from an Ubuntu 16.04 image.
In your Cloud Build task sequence, execute a shell script which performs the gcloud calls for you and redirects the output to a file. This has some observations:
You'll need to store the shell script somewhere sensible; probably in your source repository so it becomes available to the build.
The gcloud container can still be used, as this will ensure the Google Cloud SDK tools are available to your script. You will need to override the entrypoint in the Cloud Build manifest to be /bin/bash, or some other shell, and pass the path to your script as an argument.
As DazWilkin identifies in a comment, the Cloud Build service account will also require the compute.instances.list permission to list instances.
The /workspace directory is mounted into all Cloud Build containers and its contents will be persisted between and accessible from subsequent build steps. If the output of the gcloud command, or a post-processed version, is require by subsequent build steps, you can write it out here.
Relevant Google documentation.