Actually I am working on a pipeline, all good until there because it has worked. But when I want to explain it is not clear to me what each step represents physically, for example a step "could" be a node within a cluster.Please, if someone has a clear explanation of it, explain it to us.
Example 1 of a step
File config cloud build:
steps:
- name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
args: ["bin/deploy-dags-composer.sh"]
env:
- 'COMPOSER_BUCKET=${_COMPOSER_BUCKET}'
- 'ARTIFACTS_BUCKET=${_ARTIFACTS_BUCKET}'
id: 'DEPLOY-DAGS-PLUGINS-DEPENDS-STEP'
Bash File
#! /bin/bash
gsutil cp bin/plugins/* gs://$COMPOSER_BUCKET/plugins/
gsutil cp bin/dependencies/* gs://$ARTIFACTS_BUCKET/dags/dependencies/
gsutil cp bin/dags/* gs://$COMPOSER_BUCKET/dags/
gsutil cp bin/sql-scripts/* gs://$ARTIFACTS_BUCKET/path/bin/sql-scripts/composer/
Example 2 several steps
File config cloud build
steps:
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_COMPOSER_BUCKET}/plugins/']
dir: 'bin/plugins/'
id: 'deploy-plugins-step'
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_ARTIFACTS_BUCKET}/dags/dependencies/']
dir: 'bin/dependencies/'
id: 'deploy-dependencies-step'
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_COMPOSER_BUCKET}/dags/']
dir: 'bin/dags/'
id: 'deploy-dags-step'
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_ARTIFACTS_BUCKET}/projects/path/bin/sql-scripts/composer/']
dir: 'bin/sql-scripts/'
id: 'deploy-scripts-step'
In Cloud Build, a step is a stage of the processing. This stage is described by a container to load, containing the required binaries for the processing to perform in the stage.
To this container, you can define an entrypoint, the binary to run in side the container, and args to pass to it.
You have also several option that you can see here.
An important concept to understand is that ONLY the /workspace directory is kept from one step to another one. At the end of each step, the container is offloaded and you lost all the data in memory (like environment variable) and the data stored outside of the /workspace directory (such as system dependency installation). Keep this in mind, many of issues come from there.
EDIT 1:
In a step, you can, out of the box, run 1 command on one container. gsutil, gcloud, mvn, node,.... All depends on your container and your entrypoint.
But there is a useful advance capacity, when you need to run many commands on the same container. It can occur for many reason. You can do such like this
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- -c
- |
MY_ENV_VAR=$(curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance)
echo $${MY_ENV_VAR} > env-var-save.file
# Continue the commands that you need/want ....
Related
I have a pretty simple cloudbuild.yaml which, on git push, runs some npm scripts, rsyncs to a bucket, and then invalidates the CDN cache in the load balancer.
It runs fairly quickly, a couple of minutes tops, however the final step causes it to hang for about 12 minutes whilst it waits for confirmation that the CDN has invalidated. Bearing in mind that we're talking about 10 files in the CDN, not the entirety of wikipedia or anything.
# when pushed to git create a google cloud bucket with the files in served statically
steps:
# build the static files using npm install and npm run build and npm run postcss
- name: 'node:16'
args: ['npm', 'install']
- name: 'node:16'
args: ['npm', 'run', 'build']
- name: 'node:16'
args: ['npm', 'run', 'postcss']
# push to google cloud storage
- name: 'gcr.io/cloud-builders/gsutil'
args: ['-m', 'rsync', '-r', '-c', '-d', './static', 'gs://foo.com']
# invalidate cdn cache for load balancer foo-load-balancer
- name: 'gcr.io/cloud-builders/gcloud'
args: ['compute', 'url-maps', 'invalidate-cdn-cache', 'foo-load-balancer', '--path', '/*']
timeout: 1200s
The method I can think of is to create a cloud function and call it at the end (probably via pubsub, so instant confirmation) - but it seems like a fair bit of effort. Is there an easy in-the-box way of doing this in cloudbuild? It of course still needs to be run in series.
Just add --async to your command to return without waiting for the operation to complete:
- name: 'gcr.io/cloud-builders/gcloud'
args: ['compute', 'url-maps', 'invalidate-cdn-cache', 'foo-load-balancer', '--path','/*', '--async']
All of the documentation is here, at the gcloud reference documentation.
I’m trying to copy a folder (/data/out) from Cloud Build to Google Cloud Storage:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
volumes:
- name: 'vol1'
path: '/data'
entrypoint: 'gsutil'
args:
- -m
- cp
- -r
- /data/out
- gs://mybucket/files
When running this the first time, all the content from /data/out is copied to the bucket, which is what I wanted.
When I run it a second time to overwrite/update the files, there’s a folder /files/out instead, so the new files are in mybucket/files/out/* instead of mybucket/files/*.
How can I fix this?
I tried to add another step to remove the folder before copying it again (rm -r gs://mybucket/files), but this step fails the first time, as the folder does not exist yet, causing the build to fail/stop entirely.
Finally got this to work with rsync instead of cp thanks to Cloud Build -> Google Cloud Storage: Question about downtimes at deployment time
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
id: copy
volumes:
- name: 'vol1'
path: '/data'
entrypoint: 'gsutil'
args:
- -m
- rsync
- -r
- -d
- /data/out/
- gs://mybucket/folder
I copied a local folder generated in a previous step to a GS bucket. subsequently builds ran and overwrite the files.
I used:
- name: 'gcr.io/cloud-builders/gsutil'
args: [ 'cp', '-r',
'./public',
'gs://{BUCKET_NAME}/test' ]
What is the best way to access a file in one bucket whose path is stored in another file in different bucket?
Most multi fold steps suggest using Cloud Storage for persisting values between steps and writing values of variables into files, but after many attempts, we are unable to achieve this even for a simple use case in Cloud Build setup.
Few of the many things we have tried:
Concatenating steps in Cloud Shell which works in Cloud Shell, but fails in Cloud Build steps:
gsutil cp gs://bucket_1/filepath.txt filepath.txt && filepath=$(<filepath.txt) && gsutil cp gs://bucket_2/filepath local_dest.txt
Cloud Build fails as it doesn't recognize the command "filepath=$(<filepath.txt)".
Cloudbuild.yaml steps (simplified to test). Step1 succeeds, but Step2 fails
- name: gcr.io/cloud-builders/gsutil
id: 'step1'
entrypoint: 'gsutil'
args: ['cp', 'gs://bucket_1/filepath.txt', 'filepath.txt']
volumes:
- name: data
path: /persistent_volume
- name: gcr.io/cloud-builders/gsutil
id: 'step2'
entrypoint: 'gsutil'
args: ['filepath=$(<filepath.txt)', 'gsutil cp gs://anc-android-builds/$BRANCH_NAME/filepath local_dest.txt']
volumes:
- name: data
path: /persistent_volume
Error:
CommandException: Invalid command "filepath=$(<filepath.txt)".
We've tried different ways of pushing this, and breaking it down into multiple steps as well, but nothing seems to work.
This must be a simple answer, but we can't seem to figure this out. Please help and advise.
In order to achieve what you're looking for, you need to modify your 2nd step, as for now the entrypoint is expecting a gsutil command but not receiving it straight away. Therefore you need to change to something like:
- name: gcr.io/cloud-builders/gsutil
id: 'step2'
entrypoint: 'bash'
args:
- '-c'
- |
- filepath=$(<filepath.txt) && gsutil cp gs://anc-android-builds/$BRANCH_NAME/filepath local_dest.txt
volumes:
- name: data
path: /persistent_volume
You may need to tweak it a bit more, depending on you exact scenario but this should be the correct path to achieve what you are intending to.
I'm currently facing an issue with my Google Cloud Build for CI/CD.
First, I build new docker images of multiple microservices and use Terraform to create the GCP infrastructure for the containers that they will also live in production.
Then I perform some Integration / System Tests and if everything is fine I push new versions of the microservice images to the container registry for later deployment.
My problem is, that the Terraformed infrastructure doesn't get destroyed if the cloud build fails.
Is there a way to always execute a cloud build step even if some previous steps have failed, here I would want to always execute "terraform destroy"?
Or specifically for Terraform, is there a way to define a self-destructive Terraform environment?
cloudbuild.yaml example with just one docker container
steps:
# build fresh ...
- id: build
name: 'gcr.io/cloud-builders/docker'
dir: '...'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/staging/...:latest', '-t', 'gcr.io/$PROJECT_ID/staging/...:$BUILD_ID', '.', '--file', 'production.dockerfile']
# push
- id: push
name: 'gcr.io/cloud-builders/docker'
dir: '...'
args: ['push', 'gcr.io/$PROJECT_ID/staging/...']
waitFor: [build]
# setup terraform
- id: terraform-init
name: 'hashicorp/terraform:0.12.28'
dir: '...'
args: ['init']
waitFor: [push]
# deploy GCP resources
- id: terraform-apply
name: 'hashicorp/terraform:0.12.28'
dir: '...'
args: ['apply', '-auto-approve']
waitFor: [terraform-init]
# tests
- id: tests
name: 'python:3.7-slim'
dir: '...'
waitFor: [terraform-apply]
entrypoint: /bin/sh
args:
- -c
- 'pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate'
# remove GCP resources
- id: terraform-destroy
name: 'hashicorp/terraform:0.12.28'
dir: '...'
args: ['destroy', '-auto-approve']
waitFor: [tests]
Google Cloud Build doesn't yet support allow_failure or some similar mechanism as mentioned in this unsolved but closed issue.
What you can do, and as mentioned in the linked issue, is to chain shell conditional operators.
If you want to run a command on failure then you can do something like this:
- id: tests
name: 'python:3.7-slim'
dir: '...'
waitFor: [terraform-apply]
entrypoint: /bin/sh
args:
- -c
- pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate || echo "This failed!"
This would run your test as normal and then echo This failed! to the logs if the tests fail. If you want to run terraform destroy -auto-approve on the failure then you would need to replace the echo "This failed!" with terraform destroy -auto-approve. Of course you will also need the Terraform binaries in the Docker image you are using so will need to use a custom image that has both Python and Terraform in it for that to work.
- id: tests
name: 'example-customer-python-and-terraform-image:3.7-slim-0.12.28'
dir: '...'
waitFor: [terraform-apply]
entrypoint: /bin/sh
args:
- -c
- pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate || terraform destroy -auto-approve ; false"
The above job also runs false at the end of the command so that it will return a non 0 exit code and mark the job as failed still instead of only failing if terraform destroy failed as well.
An alternative to this would be to use something like Test Kitchen which will automatically stand up infrastructure, run the necessary verifiers and then destroy it at the end all in a single kitchen test command.
It's probably also worth mentioning that your pipeline is entirely serial so you don't need to use waitFor. This is mentioned in the Google Cloud Build documentation:
A build step specifies an action that you want Cloud Build to perform.
For each build step, Cloud Build executes a docker container as an
instance of docker run. Build steps are analogous to commands in a
script and provide you with the flexibility of executing arbitrary
instructions in your build. If you can package a build tool into a
container, Cloud Build can execute it as part of your build. By
default, Cloud Build executes all steps of a build serially on the
same machine. If you have steps that can run concurrently, use the
waitFor option.
and
Use the waitFor field in a build step to specify which steps must run
before the build step is run. If no values are provided for waitFor,
the build step waits for all prior build steps in the build request to
complete successfully before running. For instructions on using
waitFor and id, see Configuring build step order.
I have the following cloudbuild.yaml file:
substitutions:
_CLOUDSDK_COMPUTE_ZONE: us-central1-a
_CLOUDSDK_CONTAINER_CLUSTER: $_CLOUDSDK_CONTAINER_CLUSTER
steps:
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
args:
- '-Dsonar.host.url=https://sonar.test.io'
- '-Dsonar.login=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
- '-Dsonar.projectKey=test-service'
- '-Dsonar.sources=.'
- id: 'build test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA', '.']
- id: 'push test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
- id: 'set test-service image in yamls'
name: 'ubuntu'
args: ['bash','-c','sed -i "s,TEST_SERVICE,gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA," k8s/*.yaml']
- id: kubectl-apply
name: 'gcr.io/cloud-builders/kubectl'
args: ['apply', '-f', 'k8s/']
env:
- 'CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}'
images: ['gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
I would like to make the sonar-scanner step conditional (if we are on the production branch, I want to skip the sonar step; other branches should run that step). I would also like to use the same cloudbuild.yaml across all branches.
Is it possible to do this?
You have 2 solutions
Make 2 triggers, each one with their own configuration. 1 on Prod, 1 on UAT/DEV.
You can script your execution. It's dirty but you keep only 1 CI/CD config file
steps:
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
entrypoint: 'bash'
args:
- '-c'
- 'if [ $BRANCH_NAME != 'prod' ]; then sonar-scanner -Dsonar.host.url=https://sonar.test.io -Dsonar.login=XXXX -Dsonar.projectKey=test-service -Dsonar.sources=. ; fi'
It is not (yet) possible to create conditional steps in cloud build, as is possible with gitlab-ci for example. What we did is to create multiple projects within GCP. You could create a project for development, staging and production. They are all sourced from the same git repository to keep environments identical to each other. This means they have the same cloudbuild.yaml file.
If you would somehow need to run a particular script only in the development environment, for example, an end-to-end test, you would specify a condition on $BRANCH_NAME or $PROJECT_ID within the build step itself. However, making too much of these conditionals will harm maintainability and your environments won't be an exact mirror of eachother. Nevertheless, here is a simple example:
---
timeout: 300s
steps:
# Branch name conditional
- name: gcr.io/google.com/cloudsdktool/cloud-sdk
entrypoint: bash
args:
- -c
- |
if [[ "$BRANCH_NAME" == "develop" ]]
then
echo "Development stuff only"
elif [[ "$BRANCH_NAME" == "release" ]]
then
echo "Acceptance stuff only"
elif [[ "$BRANCH_NAME" == "main" ]]
then
echo "Production stuff only"
fi
Besides building different projects per environment, I would also recommend building a project per domain or application. This means you have a logical separation between the data stored in the projects. You can then group all the development projects under a folder called development etc. Those folders are part of an organization or even another folder.
This logical grouping is one of the real benefits of using GCP, I find it very convenient. Azure has a somewhat similar structure with resource groups and subscriptions. AWS also has a resource group structure.