I want to run some jobs automatically on merge request and otherwise as manual in GitLab CI. Is there any way of accessing artefacts from previous jobs from inside detached job created on merge request? Here is a sample pipeline for reference,
branch_job1:
stage: stage1
script:
- touch log.txt
artifacts:
paths:
- log.txt
detached_job2:
stage: stage2
dependencies: [branch_job1]
needs:
- job: branch_job1
artifacts: true
script:
- echo "hello"
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: always
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
- if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
when: manual
allow_failure: true
Here branch_job1 runs on pushing code to branch, detached_job2 runs if its pushed to default branch or as a detached job on merge request. job2 needs job1 but as jobs2 is detached, its not finding job1. How to solve this issue?
Related
I have a pretty simple cloudbuild.yaml which, on git push, runs some npm scripts, rsyncs to a bucket, and then invalidates the CDN cache in the load balancer.
It runs fairly quickly, a couple of minutes tops, however the final step causes it to hang for about 12 minutes whilst it waits for confirmation that the CDN has invalidated. Bearing in mind that we're talking about 10 files in the CDN, not the entirety of wikipedia or anything.
# when pushed to git create a google cloud bucket with the files in served statically
steps:
# build the static files using npm install and npm run build and npm run postcss
- name: 'node:16'
args: ['npm', 'install']
- name: 'node:16'
args: ['npm', 'run', 'build']
- name: 'node:16'
args: ['npm', 'run', 'postcss']
# push to google cloud storage
- name: 'gcr.io/cloud-builders/gsutil'
args: ['-m', 'rsync', '-r', '-c', '-d', './static', 'gs://foo.com']
# invalidate cdn cache for load balancer foo-load-balancer
- name: 'gcr.io/cloud-builders/gcloud'
args: ['compute', 'url-maps', 'invalidate-cdn-cache', 'foo-load-balancer', '--path', '/*']
timeout: 1200s
The method I can think of is to create a cloud function and call it at the end (probably via pubsub, so instant confirmation) - but it seems like a fair bit of effort. Is there an easy in-the-box way of doing this in cloudbuild? It of course still needs to be run in series.
Just add --async to your command to return without waiting for the operation to complete:
- name: 'gcr.io/cloud-builders/gcloud'
args: ['compute', 'url-maps', 'invalidate-cdn-cache', 'foo-load-balancer', '--path','/*', '--async']
All of the documentation is here, at the gcloud reference documentation.
I have the below steps
steps:
# This step show the version of Gradle
- id: Gradle Install
name: gradle:7.4.2-jdk17-alpine
entrypoint: gradle
args: ["--version"]
# This step build the gradle application
- id: Build
name: gradle:7.4.2-jdk17-alpine
entrypoint: gradle
args: ["build"]
# This step run test
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: gradle
args: ["publish"]
The last step I want to do only on MASTER branch
Found one link related to this https://github.com/GoogleCloudPlatform/cloud-builders/issues/138
Its using a bash command, how can I put the gradle command inside the bash.
Update
After the suggestion answer I have updated the steps as
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: "bash"
args:
- "-c"
- |
[[ "$BRANCH_NAME" == "develop" ]] && gradle publish
The build pipeline failed with below exception
Starting Step #2 - "Publish"
Step #2 - "Publish": Already have image: gradle:7.4.2-jdk17-alpine
Finished Step #2 - "Publish"
ERROR
ERROR: build step 2 "gradle:7.4.2-jdk17-alpine" failed: starting step container failed: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
The suggested solution didn't work for me, I have to write the below code
# This step run test
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: "sh"
args:
- -c
- |
if [ $BRANCH_NAME == 'master' ]
then
echo "Branch is = $BRANCH_NAME"
gradle publish
fi
Current workarounds are mentioned as following :
Using different cloudbuild.yaml files for each branch
Overriding entrypoint and injecting bash as mentioned in the link:
steps:
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
echo "Here's a convenient pattern to use for embedding shell scripts in cloudbuild.yaml."
echo "This step only pushes an image if this build was triggered by a push to master."
[[ "$BRANCH_NAME" == "master" ]] && docker push gcr.io/$PROJECT_ID/image
This tutorial outlines an alternative where you check in different cloudbuild.yaml files on different development branches.
You can try the following for Gradle command as mentioned by bhito:
- id: Publish
name: gradle:7.4.2-jdk17-alpine
entrypoint: sh
args: - c
- |
[[ "$BRANCH_NAME" == "master" ]] && gradle publish
Cloud build provides configuring triggers by branch, tag, and pr . This lets you define different build configs to use for different repo events, e.g. one for prs, another for deploying to prod, etc.you can refer to the documentation on how to create and manage triggers.
you can check this blog for more updates on additional features and can go through the release notes for more Cloud build updates.
To gain some more insights on Gradle, you can refer to the link
I am trying to deploy my AWS infrastructure using Terraform from GitLab CI CD Pipeline.
I am using the GitLab managed image and it's default Terraform template.
I have configured S3 backend and it's pointing to the S3 bucket used to store the tf state file.
I had stored CI CD variables in GitLab for: AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY' and 'S3_BUCKET'.
Everything was working fine, until I changed the 'AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY' and 'S3_BUCKET' which was pointing to a different AWS account.
Now I am getting the following error:
$ terraform init
Initializing the backend...
Backend configuration changed!
Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.
Error: Error loading state:
AccessDenied: Access Denied
status code: 403, request id: XXXXXXXXXXXX,
host id: XXXXXXXXXXXXXXXXXXXXX
Terraform failed to load the default state from the "s3" backend.
State migration cannot occur unless the state can be loaded. Backend
modification and state migration has been aborted. The state in both the
source and the destination remain unmodified. Please resolve the
above error and try again.
Cleaning up file based variables 00:00
ERROR: Job failed: exit code 1
Since this issue happened because I changed the access_key and secret_key (It was working fine from my local VS Code), I commented out the 'cache:' block in the .gitlab-ci.yml file and it worked!
The following is my .gitlab-ci.yml file:
.gitlab-ci.yml
stages:
- validate
- plan
- apply
- destroy
image:
name: registry.gitlab.com/gitlab-org/gitlab-build-images:terraform
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
# Default output file for Terraform plan
variables:
PLAN: plan.tfplan
JSON_PLAN_FILE: tfplan.json
STATE: dbrest.tfstate
cache:
paths:
- .terraform
before_script:
- alias convert_report="jq -r '([.resource_changes[]?.change.actions?]|flatten)|{\"create \":(map(select(.==\"create\"))|length),\"update\":(map(select(.==\"update\"))|length),\"delete\":(map(select(.==\"delete\"))|length)}'"
- terraform --version
- terraform init
validate:
stage: validate
script:
- terraform validate
only:
- tags
plan:
stage: plan
script:
- terraform plan -out=plan_file
- terraform show --json plan_file > plan.json
artifacts:
paths:
- plan.json
expire_in: 2 weeks
when: on_success
reports:
terraform: plan.json
only:
- tags
allow_failure: true
apply:
stage: apply
extends: plan
environment:
name: production
script:
- terraform apply --auto-approve
dependencies:
- plan
only:
- tags
when: manual
terraform destroy:
extends: apply
stage: destroy
script:
- terraform destroy --auto-approve
needs: ["plan","apply"]
when: manual
only:
- tags
The issue clearly happens if I don't comment out the below block. However it used to work before I made changes to the AWS access_key and secret_key.
#cache:
# paths:
# - .terraform
When the cache was not commented, the following was the result in the CI CI Pipeline:
Is cache being stored anywhere? And How do I clear it?
Think it's related to GitLab.
It seems that runner cache can be cleared off from GitLab from the UI itself.
Go to GitLab -> CI CD -> Pipelines and hit the 'Clear Runner Cache' button to clear the cache.
It actually works!
I'm currently facing an issue with my Google Cloud Build for CI/CD.
First, I build new docker images of multiple microservices and use Terraform to create the GCP infrastructure for the containers that they will also live in production.
Then I perform some Integration / System Tests and if everything is fine I push new versions of the microservice images to the container registry for later deployment.
My problem is, that the Terraformed infrastructure doesn't get destroyed if the cloud build fails.
Is there a way to always execute a cloud build step even if some previous steps have failed, here I would want to always execute "terraform destroy"?
Or specifically for Terraform, is there a way to define a self-destructive Terraform environment?
cloudbuild.yaml example with just one docker container
steps:
# build fresh ...
- id: build
name: 'gcr.io/cloud-builders/docker'
dir: '...'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/staging/...:latest', '-t', 'gcr.io/$PROJECT_ID/staging/...:$BUILD_ID', '.', '--file', 'production.dockerfile']
# push
- id: push
name: 'gcr.io/cloud-builders/docker'
dir: '...'
args: ['push', 'gcr.io/$PROJECT_ID/staging/...']
waitFor: [build]
# setup terraform
- id: terraform-init
name: 'hashicorp/terraform:0.12.28'
dir: '...'
args: ['init']
waitFor: [push]
# deploy GCP resources
- id: terraform-apply
name: 'hashicorp/terraform:0.12.28'
dir: '...'
args: ['apply', '-auto-approve']
waitFor: [terraform-init]
# tests
- id: tests
name: 'python:3.7-slim'
dir: '...'
waitFor: [terraform-apply]
entrypoint: /bin/sh
args:
- -c
- 'pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate'
# remove GCP resources
- id: terraform-destroy
name: 'hashicorp/terraform:0.12.28'
dir: '...'
args: ['destroy', '-auto-approve']
waitFor: [tests]
Google Cloud Build doesn't yet support allow_failure or some similar mechanism as mentioned in this unsolved but closed issue.
What you can do, and as mentioned in the linked issue, is to chain shell conditional operators.
If you want to run a command on failure then you can do something like this:
- id: tests
name: 'python:3.7-slim'
dir: '...'
waitFor: [terraform-apply]
entrypoint: /bin/sh
args:
- -c
- pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate || echo "This failed!"
This would run your test as normal and then echo This failed! to the logs if the tests fail. If you want to run terraform destroy -auto-approve on the failure then you would need to replace the echo "This failed!" with terraform destroy -auto-approve. Of course you will also need the Terraform binaries in the Docker image you are using so will need to use a custom image that has both Python and Terraform in it for that to work.
- id: tests
name: 'example-customer-python-and-terraform-image:3.7-slim-0.12.28'
dir: '...'
waitFor: [terraform-apply]
entrypoint: /bin/sh
args:
- -c
- pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate || terraform destroy -auto-approve ; false"
The above job also runs false at the end of the command so that it will return a non 0 exit code and mark the job as failed still instead of only failing if terraform destroy failed as well.
An alternative to this would be to use something like Test Kitchen which will automatically stand up infrastructure, run the necessary verifiers and then destroy it at the end all in a single kitchen test command.
It's probably also worth mentioning that your pipeline is entirely serial so you don't need to use waitFor. This is mentioned in the Google Cloud Build documentation:
A build step specifies an action that you want Cloud Build to perform.
For each build step, Cloud Build executes a docker container as an
instance of docker run. Build steps are analogous to commands in a
script and provide you with the flexibility of executing arbitrary
instructions in your build. If you can package a build tool into a
container, Cloud Build can execute it as part of your build. By
default, Cloud Build executes all steps of a build serially on the
same machine. If you have steps that can run concurrently, use the
waitFor option.
and
Use the waitFor field in a build step to specify which steps must run
before the build step is run. If no values are provided for waitFor,
the build step waits for all prior build steps in the build request to
complete successfully before running. For instructions on using
waitFor and id, see Configuring build step order.
I have the following cloudbuild.yaml file:
substitutions:
_CLOUDSDK_COMPUTE_ZONE: us-central1-a
_CLOUDSDK_CONTAINER_CLUSTER: $_CLOUDSDK_CONTAINER_CLUSTER
steps:
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
args:
- '-Dsonar.host.url=https://sonar.test.io'
- '-Dsonar.login=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
- '-Dsonar.projectKey=test-service'
- '-Dsonar.sources=.'
- id: 'build test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA', '.']
- id: 'push test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
- id: 'set test-service image in yamls'
name: 'ubuntu'
args: ['bash','-c','sed -i "s,TEST_SERVICE,gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA," k8s/*.yaml']
- id: kubectl-apply
name: 'gcr.io/cloud-builders/kubectl'
args: ['apply', '-f', 'k8s/']
env:
- 'CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}'
images: ['gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
I would like to make the sonar-scanner step conditional (if we are on the production branch, I want to skip the sonar step; other branches should run that step). I would also like to use the same cloudbuild.yaml across all branches.
Is it possible to do this?
You have 2 solutions
Make 2 triggers, each one with their own configuration. 1 on Prod, 1 on UAT/DEV.
You can script your execution. It's dirty but you keep only 1 CI/CD config file
steps:
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
entrypoint: 'bash'
args:
- '-c'
- 'if [ $BRANCH_NAME != 'prod' ]; then sonar-scanner -Dsonar.host.url=https://sonar.test.io -Dsonar.login=XXXX -Dsonar.projectKey=test-service -Dsonar.sources=. ; fi'
It is not (yet) possible to create conditional steps in cloud build, as is possible with gitlab-ci for example. What we did is to create multiple projects within GCP. You could create a project for development, staging and production. They are all sourced from the same git repository to keep environments identical to each other. This means they have the same cloudbuild.yaml file.
If you would somehow need to run a particular script only in the development environment, for example, an end-to-end test, you would specify a condition on $BRANCH_NAME or $PROJECT_ID within the build step itself. However, making too much of these conditionals will harm maintainability and your environments won't be an exact mirror of eachother. Nevertheless, here is a simple example:
---
timeout: 300s
steps:
# Branch name conditional
- name: gcr.io/google.com/cloudsdktool/cloud-sdk
entrypoint: bash
args:
- -c
- |
if [[ "$BRANCH_NAME" == "develop" ]]
then
echo "Development stuff only"
elif [[ "$BRANCH_NAME" == "release" ]]
then
echo "Acceptance stuff only"
elif [[ "$BRANCH_NAME" == "main" ]]
then
echo "Production stuff only"
fi
Besides building different projects per environment, I would also recommend building a project per domain or application. This means you have a logical separation between the data stored in the projects. You can then group all the development projects under a folder called development etc. Those folders are part of an organization or even another folder.
This logical grouping is one of the real benefits of using GCP, I find it very convenient. Azure has a somewhat similar structure with resource groups and subscriptions. AWS also has a resource group structure.