How can I add a custom message to Cloud Build logs?
I've tried using the bash entrypoint with the Docker builder (for example) and echoing some strings, but they don't appear in the build logs. Is there a way to achieve this?
Make sure that the builder image you're using has bash in it. I tested this code and replaced the gcloud builder with docker and it is working fine. Here's an example code:
steps:
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-eEuo'
- 'pipefail'
- '-c'
- |-
if (( $(date '+%-e') % 2 )); then
echo "today is an odd day"
else
echo "today is an odd day, with an even number"
fi
And here's the Logs:
Related
I have the below steps
steps:
# This step show the version of Gradle
- id: Gradle Install
name: gradle:7.4.2-jdk17-alpine
entrypoint: gradle
args: ["--version"]
# This step build the gradle application
- id: Build
name: gradle:7.4.2-jdk17-alpine
entrypoint: gradle
args: ["build"]
# This step run test
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: gradle
args: ["publish"]
The last step I want to do only on MASTER branch
Found one link related to this https://github.com/GoogleCloudPlatform/cloud-builders/issues/138
Its using a bash command, how can I put the gradle command inside the bash.
Update
After the suggestion answer I have updated the steps as
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: "bash"
args:
- "-c"
- |
[[ "$BRANCH_NAME" == "develop" ]] && gradle publish
The build pipeline failed with below exception
Starting Step #2 - "Publish"
Step #2 - "Publish": Already have image: gradle:7.4.2-jdk17-alpine
Finished Step #2 - "Publish"
ERROR
ERROR: build step 2 "gradle:7.4.2-jdk17-alpine" failed: starting step container failed: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
The suggested solution didn't work for me, I have to write the below code
# This step run test
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: "sh"
args:
- -c
- |
if [ $BRANCH_NAME == 'master' ]
then
echo "Branch is = $BRANCH_NAME"
gradle publish
fi
Current workarounds are mentioned as following :
Using different cloudbuild.yaml files for each branch
Overriding entrypoint and injecting bash as mentioned in the link:
steps:
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
echo "Here's a convenient pattern to use for embedding shell scripts in cloudbuild.yaml."
echo "This step only pushes an image if this build was triggered by a push to master."
[[ "$BRANCH_NAME" == "master" ]] && docker push gcr.io/$PROJECT_ID/image
This tutorial outlines an alternative where you check in different cloudbuild.yaml files on different development branches.
You can try the following for Gradle command as mentioned by bhito:
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: sh
args: - c
- |
[[ "$BRANCH_NAME" == "master" ]] && gradle publish
Cloud build provides configuring triggers by branch, tag, and pr . This lets you define different build configs to use for different repo events, e.g. one for prs, another for deploying to prod, etc.you can refer to the documentation on how to create and manage triggers.
you can check this blog for more updates on additional features and can go through the release notes for more Cloud build updates.
To gain some more insights on Gradle, you can refer to the link
I have shell script that I use in order to be able to create my resources on Google Cloud Platform.
It looks something like this:
REGION=us-east1
# Create buckets
FILES_SOURCE=${DEVSHELL_PROJECT_ID}-source-$(date +%s)
gsutil mb -c regional -l ${REGION} gs://${FILES_SOURCE}
FUNCTIONS_BUCKET=${DEVSHELL_PROJECT_ID}-functions-$(date +%s)
gsutil mb -c regional -l ${REGION} gs://${FUNCTIONS_BUCKET}
I also have a Cloud Build enabled for my project with a trigger defined inside of it. Some of the values for my substitution variables should be equal to FILES_SOURCE and FUNCTIONS_BUCKET from the script above. If I have my Cloud Build enabled prior to the execution of my shell script, is it possible to somehow assign those values (and their keys) from the shell script?
I can see that we have gcloud builds interface but it doesn't seem to have such options.
You must be referring to user-defined substitution variables because default substitutions are automatically defined to you by Cloud Build. With regards to gcloud builds interface, you can set --substitutions flag to specify your user-defined variables but looking at your example, it seems that those aren't fixed.
Unfortunately you won't be able to specify user-defined substitution variables if the values came from a shell script. However, there's a workaround so that your shell script variables will persist the entire build steps by saving the values on a file and then read it as you require.
You've not specified how you intend to use the variables but here's an example:
build.sh
REGION=us-east1
DEVSHELL_PROJECT_ID=sample-proj
FUNCTIONS_BUCKET=${DEVSHELL_PROJECT_ID}-functions-$(date +%s)
FILES_SOURCE=${DEVSHELL_PROJECT_ID}-source-$(date +%s)
# Store variables on a file
echo $FUNCTIONS_BUCKET > /workspace/functions-bucket &&
echo $FILES_SOURCE > /workspace/files-source
echo "Saved values."
cloudbuild.yaml
steps:
- id: "Read script and store values"
name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args: ['./build.sh']
- id: "Read Values"
name: gcr.io/cloud-builders/gcloud
entrypoint: 'bash'
args:
- -c
- |
# Read from "/workspace"
echo "First we saved " $(cat /workspace/functions-bucket) &&
echo "Then we saved " $(cat /workspace/files-source)
Note: We used /workspace because Cloud Build uses it as a working directory by default.
Reference: https://medium.com/google-cloud/how-to-pass-data-between-cloud-build-steps-de5c9ebc4cdd
You can't override the substitution variables during the Cloud Build process. So, you have 2 solutions
Either you work with "Linux" variable, and the answer of Donnald is the right solution (you have to read the value from file in each step and then use it)
Or you can call a Cloud Build in Cloud Build. Like this
Create the Cloud Build file for your core build, with substitution variables
steps:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- -c
- |
echo $_FUNCTIONS_BUCKET
echo $_FILES_SOURCE
...
substitutions:
_FUNCTIONS_BUCKET:
_FILES_SOURCE:
Then, Create a the file for initialization: cloudbuild-init.yaml
steps:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- -c
- |
REGION=us-east1
DEVSHELL_PROJECT_ID=sample-proj
FUNCTIONS_BUCKET=$${DEVSHELL_PROJECT_ID}-functions-$$(date +%s)
FILES_SOURCE=$${DEVSHELL_PROJECT_ID}-source-$$(date +%s)
gcloud builds submit --async --substitutions=_FUNCTIONS_BUCKET=$${FUNCTIONS_BUCKET},_FILES_SOURCE=$${FILES_SOURCE}
You can note the async to not wait the end of the underlying Cloud Build before finishing the init one. Else you will pay twice the cost of the Build. On the other hand, you won't know in the trigger if your job worked or not.
It's matter of tradeoffs here.
I have automatic builds set up in Google Cloud, so that each time I push to the master branch of my repository, a new image is built and pushed to Google Container Registry.
These images pile up quickly, and I don't need all the old ones. So I would like to add a build step that runs a bash script which calls gcloud container images list-tags, loops the results, and deletes the old ones with gcloud container images delete.
I have the script written and it works locally. I am having trouble figuring out how to run it as a step in Cloud Builder.
It seems there are 2 options:
- name: 'ubuntu'
args: ['bash', './container-registry-cleanup.sh']
In the above step in cloudbuild.yml I try to run the bash command in the ubuntu image. This doesn't work because the gcloud command does not exist in this image.
- name: 'gcr.io/cloud-builders/gcloud'
args: [what goes here???]
In the above step in cloudbuild.yml I try to use the gcloud image, but since "Arguments passed to this builder will be passed to gcloud directly", I don't know how to call my bash script here.
What can I do?
You can customize the entry point of your build step. If you need gcloud installed, use the gcloud cloud builder and do this
step:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: "bash"
args:
- "-c"
- |
echo "enter 1 bash command per line"
ls -la
gcloud version
...
As per the official documentation Creating custom build steps indicates, you need a custom build step to execute a shell script from your source, the step's container image must contain a tool capable of running the script.
The below example, shows how to configure your args, for the execution to perform correctly.
steps:
- name: 'ubuntu'
args: ['bash', './myscript.bash']
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/custom-script-test', '.']
images: ['gcr.io/$PROJECT_ID/custom-script-test']
I would recommend you to take a look at the above documentation and the example as well, to test and confirm if it will help you achieve the execution of the script.
For your case, specifically, there is this other answer here, where is indicated that you will need to override the endpoint of the build to bash, so the script runs. It's indicated as follow:
- name: gcr.io/cloud-builders/gcloud
entrypoint: /bin/bash
args: ['-c', 'gcloud compute instances list > gce-list.txt']
Besides that, these two below articles, include more information and examples on how to configure customized scripts to run in your Cloud Build, that I would recommend you to take a look.
CI/CD: Google Cloud Build — Custom Scripts
Mastering Google Cloud Build Config Syntax
Let me know if the information helped you!
I have the following cloudbuild.yaml file:
substitutions:
_CLOUDSDK_COMPUTE_ZONE: us-central1-a
_CLOUDSDK_CONTAINER_CLUSTER: $_CLOUDSDK_CONTAINER_CLUSTER
steps:
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
args:
- '-Dsonar.host.url=https://sonar.test.io'
- '-Dsonar.login=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
- '-Dsonar.projectKey=test-service'
- '-Dsonar.sources=.'
- id: 'build test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA', '.']
- id: 'push test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
- id: 'set test-service image in yamls'
name: 'ubuntu'
args: ['bash','-c','sed -i "s,TEST_SERVICE,gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA," k8s/*.yaml']
- id: kubectl-apply
name: 'gcr.io/cloud-builders/kubectl'
args: ['apply', '-f', 'k8s/']
env:
- 'CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}'
images: ['gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
I would like to make the sonar-scanner step conditional (if we are on the production branch, I want to skip the sonar step; other branches should run that step). I would also like to use the same cloudbuild.yaml across all branches.
Is it possible to do this?
You have 2 solutions
Make 2 triggers, each one with their own configuration. 1 on Prod, 1 on UAT/DEV.
You can script your execution. It's dirty but you keep only 1 CI/CD config file
steps:
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
entrypoint: 'bash'
args:
- '-c'
- 'if [ $BRANCH_NAME != 'prod' ]; then sonar-scanner -Dsonar.host.url=https://sonar.test.io -Dsonar.login=XXXX -Dsonar.projectKey=test-service -Dsonar.sources=. ; fi'
It is not (yet) possible to create conditional steps in cloud build, as is possible with gitlab-ci for example. What we did is to create multiple projects within GCP. You could create a project for development, staging and production. They are all sourced from the same git repository to keep environments identical to each other. This means they have the same cloudbuild.yaml file.
If you would somehow need to run a particular script only in the development environment, for example, an end-to-end test, you would specify a condition on $BRANCH_NAME or $PROJECT_ID within the build step itself. However, making too much of these conditionals will harm maintainability and your environments won't be an exact mirror of eachother. Nevertheless, here is a simple example:
---
timeout: 300s
steps:
# Branch name conditional
- name: gcr.io/google.com/cloudsdktool/cloud-sdk
entrypoint: bash
args:
- -c
- |
if [[ "$BRANCH_NAME" == "develop" ]]
then
echo "Development stuff only"
elif [[ "$BRANCH_NAME" == "release" ]]
then
echo "Acceptance stuff only"
elif [[ "$BRANCH_NAME" == "main" ]]
then
echo "Production stuff only"
fi
Besides building different projects per environment, I would also recommend building a project per domain or application. This means you have a logical separation between the data stored in the projects. You can then group all the development projects under a folder called development etc. Those folders are part of an organization or even another folder.
This logical grouping is one of the real benefits of using GCP, I find it very convenient. Azure has a somewhat similar structure with resource groups and subscriptions. AWS also has a resource group structure.
I'm using Cloud Build with the gcloud builder. I override the entrypoint to be bq so I can run some BigQuery SQL in my build step. Previously, I had the SQL embedded directly in the YAML config for Cloud Build. This works fine:
steps:
- name: gcr.io/cloud-builders/gcloud
entrypoint: 'bq'
args: ['query', '--use_legacy_sql=false', 'SELECT 1']
Now I'd like to refactor the SQL out of the YAML and into a file instead. According to here, you can cat the file or pipe it to bq. This works on the command line without any problems.
But, I can't get it to work with Cloud Build. I've tried lots of different combinations, and escaping chars etc. but no matter what I try the shell doesn't evaluate/execute the cat my_query.sl backticks, and instead thinks that it's the query itself:
Works fine:
Build in Cloud Build it won't work:
steps:
- name: gcr.io/cloud-builders/gcloud
entrypoint: 'bq'
args: ['query', '--use_legacy_sql=false', '`cat my_query.sql`']
I also tried piping it instead of using cat, but I get the same error.
I must be missing something obvious here, but I can't see it. I could build a custom docker image, and wrap everything in a shell script, but I'd rather not have to do that if possible.
How do you use Cloud Build with shell evaluation inside a build step?
You can create a custom Bash script, e.g.:
#!/bin/bash
if [ $# -eq 0 ]; then
echo "No arguments supplied"
fi
bq query --use_legacy_sql=false < $1
Name this run_query.sh, then define your steps as:
steps:
- name: gcr.io/cloud-builders/gcloud
entrypoint: 'bash'
args: ['run_query.sh', 'my_query.sql']
Disclaimer: this is based on reading the docs, but I haven't actually used Cloud Build.
I have done this:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
dir: 'my/directory'
args: ['-c', 'bq --project_id=my-project-name query --use_legacy_sql=false < ./my_query.sql']
Which works with gcloud builds submit ... and eliminates one file if you prefer.