I’m trying to copy a folder (/data/out) from Cloud Build to Google Cloud Storage:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
volumes:
- name: 'vol1'
path: '/data'
entrypoint: 'gsutil'
args:
- -m
- cp
- -r
- /data/out
- gs://mybucket/files
When running this the first time, all the content from /data/out is copied to the bucket, which is what I wanted.
When I run it a second time to overwrite/update the files, there’s a folder /files/out instead, so the new files are in mybucket/files/out/* instead of mybucket/files/*.
How can I fix this?
I tried to add another step to remove the folder before copying it again (rm -r gs://mybucket/files), but this step fails the first time, as the folder does not exist yet, causing the build to fail/stop entirely.
Finally got this to work with rsync instead of cp thanks to Cloud Build -> Google Cloud Storage: Question about downtimes at deployment time
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
id: copy
volumes:
- name: 'vol1'
path: '/data'
entrypoint: 'gsutil'
args:
- -m
- rsync
- -r
- -d
- /data/out/
- gs://mybucket/folder
I copied a local folder generated in a previous step to a GS bucket. subsequently builds ran and overwrite the files.
I used:
- name: 'gcr.io/cloud-builders/gsutil'
args: [ 'cp', '-r',
'./public',
'gs://{BUCKET_NAME}/test' ]
Related
I have a React application that is Dockerized and hosted on Google Cloud Build. I have set environment variables on the Cloud Build, but I am unable to access them within my React application. What am I doing wrong and how can I access these environment variables in my React application?
steps:
name: gcr.io/cloud-builders/docker
env:
-"VITE_PUBLIC_KEY={$_VITE_PUBLIC_KEY}",
-"VITE_SERVICE_ID={$_VITE_SERVICE_ID}",
-"VITE_TEMPLATE_ID={$_VITE_TEMPLATE_ID}"
args:
build
'--no-cache'
'-t'
'image_name'
.
'-f'
Dockerfile.prod
name: gcr.io/cloud-builders/docker
args:
push
'image_name'
name: gcr.io/cloud-builders/gcloud
args:
run
deploy
bob
'--image'
'image_name'
'--region'
$_DEPLOY_REGION
'--allow-unauthenticated'
'--platform'
$_PLATFORM
timeout: 600s
this is the yaml file:
I dont have a backend solution, I just want to be able to access 3 enviromnent variables within my application on client side. without declaring a .env file.
Tried declaring the enviroments in Cloud Run as well as declaring in the cloudbuild.yaml file. It works on aws but a different problem arises on aws.
One solution could be to hardcode the environment variables directly into your React code. This is not recommended as it could lead to security vulnerabilities and make it difficult to change the values in the future without re-deploying the entire application. Additionally, it would not be a true solution to accessing the environment variables as they would not be dynamically set at runtime.
There are different options, one and the best (in my opinion) is to store your env variables using the secret manager and then access them through your code (check this GitHub Repo).
. . .
The other option is to access the same secrets that you created before when your pipeline is running, the downside is that you always have to redeploy your pipeline to update the env variables.
This is an example:
steps:
# STEP 0 - BUILD CONTAINER 1
- id: Build-container-image-container-one
name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
docker build -t gcr.io/$PROJECT_ID/container_one -f 'build/container_one.Dockerfile' .
# STEP 2 - PUSH CONTAINER 1
- id: Push-to-Container-Registry-container-one
name: 'gcr.io/cloud-builders/docker'
args:
- push
- gcr.io/$PROJECT_ID/container_one
waitFor: ["Build-container-image-container-one"]
# STEP 3 - DEPLOY CONTAINER 1
- id: Deploy-Cloud-Run-container-one
name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args:
- run
- deploy
- container_one
- --image=gcr.io/$PROJECT_ID/container_one
- --region={YOUR REGION}
- --port={YOUR PORT}
- --memory=3Gi
- --cpu=1
waitFor: ["Build-container-image-container-one", "Push-to-Container-Registry-container-one"]
# STEP 4 - ENV VARIABLES
- id: Accessing-secrets-for-env-variables
name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- |
gcloud secrets versions access latest --secret=ENV_VARIABLE_ONE > key1.txt
gcloud secrets versions access latest --secret=ENV_VARIABLE_TWO > key2.txt
gcloud secrets versions access latest --secret=ENV_VARIABLE_THREE > key3.txt
waitFor: ["Push-to-Container-Registry-container-one", "Build-container-image-container-one"]
# STEP 5 - SETTING KEYS
- id: Setting-keys
name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args: ['-c', 'gcloud run services update container_one --region={YOUR REGION} --set-env-vars="ENV_VARIABLE_ONE=$(cat key1.txt), ENV_VARIABLE_TWO=$(cat key2.txt), ENV_VARIABLE_THREE=$(cat key3.txt)"']
images:
- gcr.io/$PROJECT_ID/container_one
When uploading an *.tar.gz build artifact to Cloud Storage bucket, it wrongfully applies MIME type application/tar, while it would have to apply MIME type application/tar+gzip (or the official MIME type application/gzip), in order to be able to download and extract the uploaded *.tar.gz archive then again. This only works, when I manually set the MIME type afterwards (in the object details), but I'd be looking for a way, to define the proper MIME type right away. How this can be done?
The cloudbuild.yaml which produces the issue roughly looks alike this:
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:latest'
entrypoint: 'bash'
args:
- '-c'
- |-
- tar -zcf ./test_${SHORT_SHA}.tar.gz ./$_UPLOAD_DIRNAME
env:
- '_UPLOAD_DIRNAME=$_UPLOAD_DIRNAME'
- 'SHORT_SHA=$SHORT_SHA'
artifacts:
objects:
location: 'gs://some-bucket/'
paths: ['*.tar.gz']
After reading: Working With Object Metadata, it turned out that the artifacts node won't suffice.
gsutil seems to be the only way to explicitly pass the desired MIME type application/tar+gzip:
steps:
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:latest'
entrypoint: 'bash'
args:
- '-c'
- |-
tar -zcf ./test_${SHORT_SHA}.tar.gz ./$_UPLOAD_DIRNAME
gsutil -h "Content-Type:application/tar+gzip" cp ./test_${SHORT_SHA}.tar.gz ${_GOOGLE_STORAGE_BUCKET}
gsutil ls -L ${_GOOGLE_STORAGE_BUCKET}test_${SHORT_SHA}.tar.gz
env:
- '_UPLOAD_DIRNAME=$_UPLOAD_DIRNAME'
- '_GOOGLE_STORAGE_BUCKET=$_GOOGLE_STORAGE_BUCKET'
- 'SHORT_SHA=$SHORT_SHA'
Actually I am working on a pipeline, all good until there because it has worked. But when I want to explain it is not clear to me what each step represents physically, for example a step "could" be a node within a cluster.Please, if someone has a clear explanation of it, explain it to us.
Example 1 of a step
File config cloud build:
steps:
- name: "gcr.io/google.com/cloudsdktool/cloud-sdk"
args: ["bin/deploy-dags-composer.sh"]
env:
- 'COMPOSER_BUCKET=${_COMPOSER_BUCKET}'
- 'ARTIFACTS_BUCKET=${_ARTIFACTS_BUCKET}'
id: 'DEPLOY-DAGS-PLUGINS-DEPENDS-STEP'
Bash File
#! /bin/bash
gsutil cp bin/plugins/* gs://$COMPOSER_BUCKET/plugins/
gsutil cp bin/dependencies/* gs://$ARTIFACTS_BUCKET/dags/dependencies/
gsutil cp bin/dags/* gs://$COMPOSER_BUCKET/dags/
gsutil cp bin/sql-scripts/* gs://$ARTIFACTS_BUCKET/path/bin/sql-scripts/composer/
Example 2 several steps
File config cloud build
steps:
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_COMPOSER_BUCKET}/plugins/']
dir: 'bin/plugins/'
id: 'deploy-plugins-step'
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_ARTIFACTS_BUCKET}/dags/dependencies/']
dir: 'bin/dependencies/'
id: 'deploy-dependencies-step'
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_COMPOSER_BUCKET}/dags/']
dir: 'bin/dags/'
id: 'deploy-dags-step'
- name: gcr.io/cloud-builders/gsutil
args: ['cp', '*', 'gs://${_ARTIFACTS_BUCKET}/projects/path/bin/sql-scripts/composer/']
dir: 'bin/sql-scripts/'
id: 'deploy-scripts-step'
In Cloud Build, a step is a stage of the processing. This stage is described by a container to load, containing the required binaries for the processing to perform in the stage.
To this container, you can define an entrypoint, the binary to run in side the container, and args to pass to it.
You have also several option that you can see here.
An important concept to understand is that ONLY the /workspace directory is kept from one step to another one. At the end of each step, the container is offloaded and you lost all the data in memory (like environment variable) and the data stored outside of the /workspace directory (such as system dependency installation). Keep this in mind, many of issues come from there.
EDIT 1:
In a step, you can, out of the box, run 1 command on one container. gsutil, gcloud, mvn, node,.... All depends on your container and your entrypoint.
But there is a useful advance capacity, when you need to run many commands on the same container. It can occur for many reason. You can do such like this
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- -c
- |
MY_ENV_VAR=$(curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance)
echo $${MY_ENV_VAR} > env-var-save.file
# Continue the commands that you need/want ....
What is the best way to access a file in one bucket whose path is stored in another file in different bucket?
Most multi fold steps suggest using Cloud Storage for persisting values between steps and writing values of variables into files, but after many attempts, we are unable to achieve this even for a simple use case in Cloud Build setup.
Few of the many things we have tried:
Concatenating steps in Cloud Shell which works in Cloud Shell, but fails in Cloud Build steps:
gsutil cp gs://bucket_1/filepath.txt filepath.txt && filepath=$(<filepath.txt) && gsutil cp gs://bucket_2/filepath local_dest.txt
Cloud Build fails as it doesn't recognize the command "filepath=$(<filepath.txt)".
Cloudbuild.yaml steps (simplified to test). Step1 succeeds, but Step2 fails
- name: gcr.io/cloud-builders/gsutil
id: 'step1'
entrypoint: 'gsutil'
args: ['cp', 'gs://bucket_1/filepath.txt', 'filepath.txt']
volumes:
- name: data
path: /persistent_volume
- name: gcr.io/cloud-builders/gsutil
id: 'step2'
entrypoint: 'gsutil'
args: ['filepath=$(<filepath.txt)', 'gsutil cp gs://anc-android-builds/$BRANCH_NAME/filepath local_dest.txt']
volumes:
- name: data
path: /persistent_volume
Error:
CommandException: Invalid command "filepath=$(<filepath.txt)".
We've tried different ways of pushing this, and breaking it down into multiple steps as well, but nothing seems to work.
This must be a simple answer, but we can't seem to figure this out. Please help and advise.
In order to achieve what you're looking for, you need to modify your 2nd step, as for now the entrypoint is expecting a gsutil command but not receiving it straight away. Therefore you need to change to something like:
- name: gcr.io/cloud-builders/gsutil
id: 'step2'
entrypoint: 'bash'
args:
- '-c'
- |
- filepath=$(<filepath.txt) && gsutil cp gs://anc-android-builds/$BRANCH_NAME/filepath local_dest.txt
volumes:
- name: data
path: /persistent_volume
You may need to tweak it a bit more, depending on you exact scenario but this should be the correct path to achieve what you are intending to.
I have website, which stored on AWS EC2 servers.
We have 2 servers, one for production environment and another one for development and staging environments.
Development and staging environments located in different folders. For example development env stored in /var/www/development, while staging stored in /var/www/staging.
I'd like to use AWS CodeDeploy to upload files directly from bitbucket. I put AppSpec file, which copy source code to /var/www/html folder and install all dependencies and configurations. But I want my AppSpec file to copy source code to /var/www/development or to /var/www/staging depending on Development group, that was selected.
Is there is any way to do it or, maybe, there are some better approach in my situation?
The appspec.yml is a bit inflexible so use the following to deploy code in to different folders on the same instance.
version: 0.0
os: linux
files:
- source: /
destination: /var/www/my-temp-dir
permissions:
- object: /var/www/my-temp-dir
owner: ec2-user
group: ec2-user
hooks:
BeforeInstall:
- location: ci/integrations-deploy-pre.sh
runas: root
AfterInstall:
- location: ci/integrations-deploy-post.sh
runas: root
Inside of my integrations-deploy-post.sh file, I then use the CodeDeploy environment variables to move the files in to the place I need them to be;
#!/bin/bash
if [ "$DEPLOYMENT_GROUP_NAME" == "Staging" ]
then
cp -R /var/www/my-temp-dir /var/www/my-staging-dir
chown -R ec2-user:ec2-user /var/www/my-staging-dir
# Insert other commands that need to run...
fi
if [ "$DEPLOYMENT_GROUP_NAME" == "UAT" ]
then
cp -R /var/www/my-temp-dir /var/www/my-uat-dir
chown -R ec2-user:ec2-user /var/www/my-uat-dir
# Insert other commands that need to run...
fi
NOTE: In my integrations-deploy-post.sh You'll also need the commands you want to run on production. Removed for simplicity.
The recommended way to change AppSpec or custom scripts behavior is to utilize environment variables provided by the CodeDeploy agent. You have access to the deployment group name and the application name.
if [ "$DEPLOYMENT_GROUP_NAME" == "Staging" ]; then
# Copy to /var/www/staging
elif [ "$DEPLOYMENT_GROUP_NAME" == "Development" ]; then
# Copy to /var/www/development
elif [ "$DEPLOYMENT_GROUP_NAME" == "Production" ]; then
# Copy to /var/www/html
else
# Fail the deployment
fi
I had the same problem, but I used source control as a solution to this.
My workflow is using Gitlab CI > AWS Code Pipeline (S3 Source and CodeDeploy).
So in my development branch, my AppSpec file would look like this:-
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/my-project-dev
hooks:
AfterInstall:
- location: scripts/after_install.sh
timeout: 400
runas: root
in my staging branch:-
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/my-project-staging
hooks:
AfterInstall:
- location: scripts/after_install.sh
timeout: 400
runas: root
My Gitlab-CI just uses a shell executor to my EC2 instance and it basically compresses my project folder and uploads to S3.
.gitlab-ci.yml
stages:
- deploy
setup dependencies:
stage: .pre
script:
- echo "Setup Dependencies"
- pip install awscli
deploy to s3:
stage: deploy
script:
- tar -cvzf /tmp/artifact_$CI_COMMIT_REF_NAME.tar ./*
- echo "Copy artifact to S3"
- aws s3 cp /tmp/artifact_$CI_COMMIT_REF_NAME.tar s3://project-artifacts/
clean up:
stage: .post
script:
- echo "Removing generated artifact"
- rm /tmp/artifact_$CI_COMMIT_REF_NAME.tar
Note that $CI_COMMIT_REF_NAME is used to differentiate the artifact file being generated. In development branch it would be artifact_development.tar, in staging branch artifact_staging.tar.
Then, I have 2 pipelines listening to the two respective artifacts which deploys to 2 different CodeDeploy Application.
Not sure if this is the best way, surely welcome any suggestions that is better