GCP cloudbuild optional step - google-cloud-platform

Having this config on cloudbuild.yaml (there are other similar fragments on the file):
- name: 'gcr.io/cloud-builders/gcloud'
id: 'step_1'
args: ['builds',
'submit',
'--config=path_to_sub_app_1/app_1_build.yaml',
'--substitutions=VAR_1=${ENV_VAR_1}']
waitFor: ['Docker push']
- name: 'gcr.io/cloud-builders/gcloud'
id: 'step_2'
args: ['builds',
'submit',
'--config=path_to_sub_app_2/app_2_build.yaml',
'--substitutions=VAR_1=${ENV_VAR_1}']
waitFor: ['Docker push']
Is it possible to skip step_1 and continue the execution normally (step_2)?

Use entrypoint: 'bash':
- name: 'gcr.io/cloud-builders/gcloud'
id: 'step_1'
entrypoint: 'bash'
args:
- '-c'
- |
if [ "$_SKIP_STEP" != "true" ]
then
gcloud builds submit --config=path_to_sub_app_1/app_1_build.yaml --substitutions=VAR_1=${ENV_VAR_1}
fi
waitFor: ['Docker push']
Define this var: _SKIP_STEP="false"
And now we can run the build and skip step_1:
gcloud builds submit --config=cloudbuild.yaml --substitutions=_SKIP_STEP=true

Related

Error: Deployment failed Resource is not in the state deploymentSuccessful Finishing: CodeDeployDeployApplication

I am quite new to azure devops. I am trying to deploy my app from azure to aws ec2 using codedeploy. The build pipeline is giving "Error: Deployment failed Dev-TitasEcom-Frontend Resource is not in the state deploymentSuccessful" error. I have searched a lot but could not find any proper resource to resolve the issue. Below given the code of azure-pipelines.yaml
pr:
branches:
include:
- master
- dev
paths:
include:
- ./*
jobs:
- job: Build_Job
displayName: Build
pool:
vmImage: 'ubuntu-latest'
demands:
- npm
steps:
- checkout: self
clean: false
# - powershell: 'npm cache clean --force'
# displayName: 'PowerShell Script'
# env:
# APPDATA: npm-cache
# - task: CodeDeployDeployApplication#1
# inputs:
# awsCredentials: 'AWS Service Con'
# regionName: 'us-east-1'
# applicationName: 'Dev-TitasEcom-Frontend'
# deploymentGroupName: 'Dev-erp-Frontend'
# #deploymentRevisionSource: 'workspace'
#revisionBundle: 'Dev-erp-Frontend-Rev'
# bucketName: 'Dev-erp-Frontend'
# fileExistsBehavior: 'OVERWRITE'
# batch: "true"
- task: CodeDeployDeployApplication#1
inputs:
awsCredentials: 'AWS Service Con 1'
regionName: 'us-east-1'
applicationName: 'Dev-TitasEcom-Frontend-11'
deploymentGroupName: 'Dev-erp-Frontend'
environmentName: 'DevTitasecomFrontend-env-12'
applicationType: 'version'
bucketName: 'elasticbeanstalk-us-east-1-3067722226831765'
fileExistsBehavior: 'OVERWRITE'
#versionLabel: 'v.0.01'
batch: "true"
- task: Npm#1
displayName: 'Npm Install'
inputs:
workingDir: "./"
command: "ci"
- task: Npm#1
displayName: 'Lint Client App'
inputs:
workingDir: "./"
command: "custom"
customCommand: "run lint"
continueOnError: true
- task: Npm#1
displayName: 'Copy Assets'
inputs:
workingDir: "./"
command: "custom"
customCommand: "run copy-files"
continueOnError: false
- task: Npm#1
displayName: 'Build Client App'
inputs:
workingDir: "./"
command: "custom"
customCommand: "run build:prod"
# Archive files
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.BinariesDirectory)'
includeRootFolder: true
archiveType: 'zip' # Options: zip, 7z, tar, wim
archiveFile: '$(Build.ArtifactStagingDirectory)/www.zip'
replaceExistingArchive: true
- task: CopyFiles#2
inputs:
Contents: 'www/**'
TargetFolder: '$(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact'
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
ArtifactName: 'titas-ecom-erp'

How to access GSM secrets through Cloud Build and pass to Cloud Function

How does one pass a secret from Google Secrets Manager (GSM) to a Cloud Function when using Cloud Build? The below cloudbuild.yaml has three steps. Further, I'm using volumes to create permanent storage between build steps. I can confirm GSM retrieval by Cloud Build. However, when I attempt to pass a secret in yaml format using --env-vars-file I encounter the following error ...
Already have image (with digest): gcr.io/cloud-builders/gcloud
ERROR: gcloud crashed (AttributeError): 'str' object has no attribute 'items'
cloudbuild.yaml:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
volumes:
- name: 'secrets'
path: '/secrets'
entrypoint: "bash"
args:
- "-c"
- |
echo -n 'gsm_secret:' > /secrets/my-secret-file.txt
- name: 'gcr.io/cloud-builders/gcloud'
volumes:
- name: 'secrets'
path: '/secrets'
entrypoint: "bash"
args:
- "-c"
- |
gcloud components update
gcloud beta secrets versions access --secret=MySecret latest >> /secrets/my-secret-file.txt
cat /secrets/my-secret-file.txt
- name: 'gcr.io/cloud-builders/gcloud'
volumes:
- name: 'secrets'
path: '/secrets'
args: [
'functions', 'deploy', 'gsm-foobar',
'--project=[...]',
'--trigger-http',
'--runtime=go111',
'--region=us-central1',
'--memory=256MB',
'--timeout=540',
'--entry-point=GSM',
'--allow-unauthenticated',
'--source=https://source.developers.google.com/[...]',
'--service-account', '[...]#appspot.gserviceaccount.com',
'--env-vars-file', '/secrets/my-secret-file.txt'
]
Update:
Usage of volumes is not required as /workspace is permanent storage between steps in Cloud Build. Also, gcloud components update is no longer necessary as the default Cloud SDK version, as of today, is 279.0.0
A Solution:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: "bash"
args:
- "-c"
- |
echo "gsm_secret: $(gcloud beta secrets versions access --secret=MySecret latest)" > /workspace/my-secret-file.txt
cat /workspace/my-secret-file.txt
- name: 'gcr.io/cloud-builders/gcloud'
args: [
'functions', 'deploy', 'gsm-foobar',
[...]
'--entry-point=GSM',
'--allow-unauthenticated',
'--source=https://source.developers.google.com/[...]',
'--service-account', '[...]#appspot.gserviceaccount.com',
'--env-vars-file=/workspace/my-secret-file.txt'
]
On second read, I realize your 2nd step puts the secret value in the file. I think you're missing the newline.
NB I've not tried this for myself!
Ensure you have a newline at the end of your secrets file.
See: https://cloud.google.com/functions/docs/env-var
Update: tried it ;-)
I think your issue was the final newline.
Using the following in a step prior to the deployment, works:
echo "gsm_secret: $(gcloud beta secrets versions access --secret=MySecret latest)" > /secrets/my-secret-file.txt
Or, more simply, perhaps:
steps:
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: /bin/bash
args:
- "-c"
- |
gcloud functions deploy ... \
--set-env-vars=NAME=$(gcloud beta secrets versions access --secret=name latest)
Also, see secretEnv. This is a more elegant mechanism..This functionality should perhaps be augmented by Google to support secret manager (in addition to KMS).
As of 2021 February 10, you can access Secret Manager secrets directly from Cloud Build using the availableSecrets field:
steps:
- id: 'deploy'
name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- 'gcloud functions deploy --set-env-vars=SECRET=$$MY_SECRET'
secretEnv: ['MY_SECRET']
availableSecrets:
secretManager:
- versionName: 'projects/my-project/secrets/my-secret/versions/latest'
env: 'MY_SECRET'
Documentation

GCP cloudbuild.yaml conditional step error

This is my cloud build file
substitutions:
_CLOUDSDK_COMPUTE_ZONE: us-central1-a
_CLOUDSDK_CONTAINER_CLUSTER: $_CLOUDSDK_CONTAINER_CLUSTER
steps:
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
entrypoint: 'bash'
args:
- '-c'
- 'if [ $BRANCH_NAME != 'production' ]; then sonar-scanner -Dsonar.host.url=https://sonar.test.io -Dsonar.login=${_SONAR_LOGIN} -Dsonar.projectKey=service -Dsonar.sources=. ; fi'
- id: 'build test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA', '.']
- id: 'push test-service image'
name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
- id: 'set test-service image in yamls'
name: 'ubuntu'
args: ['bash','-c','sed -i "s,TEST_SERVICE,gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA," k8s/*.yaml']
- id: kubectl-apply
name: 'gcr.io/cloud-builders/kubectl'
args: ['apply', '-f', 'k8s/']
env:
- 'CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}'
images: ['gcr.io/$PROJECT_ID/$REPO_NAME/$BRANCH_NAME:$SHORT_SHA']
I am getting error
failed unmarshalling build config cloudbuild.yaml: yaml: line 17: did
not find expected key
Update 1
As per #cloudomation suggestion updated if condition
- 'if [ $BRANCH_NAME != "production" ]; then sonar-scanner -Dsonar.host.url=https://sonar.test.io -Dsonar.login=${_SONAR_LOGIN} -Dsonar.projectKey=service -Dsonar.sources=. ; fi'
Condition is working but when it's true getting this error
Step #1: Digest: sha256:ef0de1c8e48544b9693b9aab2222bf849028bb66881762bf77e055b0abbf7f2b Step #1: Status: Downloaded newer image for gcr.io/wotnot-235414/sonar-scanner:latest Step #1: gcr.io/project-235414/sonar-scanner:latest Step #1: /opt/sonar-scanner-3.2.0.1227-linux/bin/sonar-scanner: exec: line 59: /opt/sonar-scanner-3.2.0.1227-linux/jre/bin/java: not found Finished Step #1 ERROR ERROR: build step 1 "gcr.io/project-235414/sonar-scanner:latest" failed: exit status 127
but when my step is like this way it's working perfectly
- name: gcr.io/$PROJECT_ID/sonar-scanner:latest
args:
- '-Dsonar.host.url=https://sonar.test.io'
- '-Dsonar.login=XXXXXXXXXXXXXX'
- '-Dsonar.projectKey=service-name'
- '-Dsonar.sources=.'
also this is running that mean there no issue in builder image it's just issue of passing parameters
docker run gcr.io/$PROJECT_ID/sonar-scanner:latest bash -Dsonar.host.url=https://sonar.test.io -Dsonar.login=${_SONAR_LOGIN} -Dsonar.projectKey=service -Dsonar.sources=.
if condition working but after that then condition having some issue
You should escape the quotes:
- 'if [ $BRANCH_NAME != \'production\' ]; then sonar-scanner -Dsonar.host.url=https://sonar.test.io -Dsonar.login=${_SONAR_LOGIN} -Dsonar.projectKey=service -Dsonar.sources=. ; fi'
or use double quotes:
- 'if [ $BRANCH_NAME != "production" ]; then sonar-scanner -Dsonar.host.url=https://sonar.test.io -Dsonar.login=${_SONAR_LOGIN} -Dsonar.projectKey=service -Dsonar.sources=. ; fi'

Google Cloud Build sub builds

Is possible have multiple cloudbuild.yaml files per subdirectory?
For example:
my-app:
- service1
- cloudbuild.yaml
- service2
- cloudbuild.yaml
cloudbuild.yaml
The answer is almost correct. This will not work, because you forgot to include ".", which tells to upload and build the current directory. The correct way to include a sub/child cloudbuild.yaml would then be:
# Include cloudbuild sub step
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'builds'
- 'submit'
- '.'
- '--config'
- 'cloudbuild.yaml'
Yes, definitely! Are you trying to initialize the builds of service1 and service2 from my-app/cloudbuild.yaml?
Example of using a meta config to initialize other builds: https://github.com/GoogleCloudPlatform/cloudbuild-integration-testing/blob/master/cloudbuild.meta.yaml
Here is a cloudbuild.meta.yaml building off of your example:
steps:
- id: 'build service1'
name: 'gcr.io/cloud-builders/gcloud'
args: ['builds', 'submit', '--config service1/cloudbuild.yaml']
waitFor: ['-'] #start in parallel
- id: 'build service2'
name: 'gcr.io/cloud-builders/gcloud'
args: ['builds', 'submit', '--config service2/cloudbuild.yaml']
waitFor: ['-'] # start in parallel

How do I get Google Cloud Build to properly substitute values when it responds to a GitHub trigger?

The Problem
A GitHub trigger set up in Google Cloud Build doesn't actually substitute the configured values while running the Build (cloudbuild.yaml)
This is the Google Cloud Build config
cloudbuild.yaml
steps:
- name: 'gcr.io/cloud-builders/gcloud'
id: 'decrypt .npmrc'
args:
- kms
- decrypt
- --ciphertext-file=npmrc.enc
- --plaintext-file=/root/.npmrc
- --location=global
- --keyring=my-keyring
- --key=npm-key
- --project=${_CLOUD_KMS_PROJECT}
volumes:
- name: 'home'
path: /root/
- name: 'gcr.io/cloud-builders/npm'
id: 'install'
args: ['install']
env:
- HOME=/root/
volumes:
- name: 'home'
path: /root/
- name: 'gcr.io/cloud-builders/npm'
id: 'test'
args: ['run', 'test']
- name: gcr.io/$PROJECT_ID/skaffold:alpha
id: 'deploy'
args: ['run', '-f=${_SKAFFOLD_FILE}']
env:
- CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}
- CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}
substitutions:
_SKAFFOLD_FILE: dummy.yaml
_CLOUDSDK_COMPUTE_ZONE: us-west1-a
_CLOUDSDK_CONTAINER_CLUSTER: dummy
timeout: 1000s
Curiously, when triggering the build via a gcloud SDK call it works ✅ ex:
gcloud builds submit --config=cloudbuild.yaml --substitutions=_SKAFFOLD_FILE=skaffold.yaml,_CLOUDSDK_COMPUTE_ZONE=us-west1-a,_CLOUDSDK_CONTAINER_CLUSTER=skaffold .
Some more context
Cloud Build Trigger dashboard (img)
Trigger configuration with substitutions (img)
Missing substitutions in Build history from triggered build (img)
In contrast: Correctly substituted values when running gcloud builds submit (img)