I have my current Cloud Build working, I connect my github repo to trigger the Cloud Build when I push to the main branch which then creates my Cloud Function, but I am confused about the the --source flag. I have read the google cloud function docs. They state that the
minimal source repository URL is: https://source.developers.google.com/projects/${PROJECT}/repos/${REPO}. If I were to input this into my cloudbuild.yaml file, does this mean that I am mimicking the complete path of my github url? I am currently just using . which I believe is just the entire root directory.
my cloudbuild.yaml file:
steps:
- name: "gcr.io/cloud-builders/gcloud"
id: "deploypokedex"
args:
- functions
- deploy
- my_pokedex_function
- --source=.
- --entry-point=get_pokemon
- --trigger-topic=pokedex
- --timeout=540s
- --runtime=python39
- --region=us-central1
Yes you are mimicking the complete path of the Github URL. --source=. means that you are calling the source code in your current working directory. You can check this link on how to configure the Cloud Build deployment.
Also based on the documentation you provided,
If you do not specify the --source flag:
The current directory will be used for new function deployments.
If the function was previously deployed using a local filesystem path, then the function's source code will be updated using the current directory.
If the function was previously deployed using a Google Cloud Storage location or a source repository, then the function's source code will not be updated.
Let me know if you have questions or clarifications.
Related
In GCP cloud build, I have a Bitbucket Data Center repository connected that uses a proxy; https://bitbucket-sv3.mydomain.com. However, the repo lives at https://bitbucket.mydomain.com
When I push to the repo, my trigger, which uses the connected repo starts a build by trying to clone the second domain, which the workerpool can't reach and I get a connection error. I assume that the webhook created by gcloud is just sending the server's address and the cloud build machine is using it from the payload. Is there some way I can force the build to use the sv3 address when it clones the source repo during the Fetch Source step?
Here's an example of my cloudbuild.yaml showing some of the things I've tried:
steps:
- name: gcr.io/cloud-builders/git
env:
- "_URL='https://bitbucket-sv3.mydomain.com/reponame.git'"
- "_HEAD_REPO_URL='https://bitbucket-sv3.mydomain.com/reponame.git'"
args: ['clone', 'https://bitbucket-sv3.mydowmain.com/reponame.git']
An alternative would be to stop the FETCHSOURCE step, but I haven't found a way to do that.
How do I use a custom builder image in Cloud Build which is stored in a repository in Artifact Registry (instead of Container Registry?)
I have set up a pipeline in Cloud Build where some python code is executed using official python images. As I want to cache my python dependencies, I wanted to create a custom Cloud Builder as shown in the official documentation here.
GCP clearly indicates to switch to Artifact Registry as Container Registry will be replaced by the former. Consequently, I have pushed my docker image to Artifact Registry. I also gave my Cloud Builder Service Account the reader permissions to Artifact Registry.
Using the image in a Cloud Build step like this
steps:
- name: 'europe-west3-docker.pkg.dev/xxxx/yyyy:latest'
id: install_dependencies
entrypoint: pip
args: ["install", "-r", "requirements.txt", "--user"]
throws the following error
Step #0 - "install_dependencies": Pulling image: europe-west3-docker.pkg.dev/xxxx/yyyy:latest
Step #0 - "install_dependencies": Error response from daemon: manifest for europe-west3-docker.pkg.dev/xxxx/yyyy:latest not found: manifest unknown: Requested entity was not found.
"xxxx" is the repository name and "yyyy" the name of my image. The tag "latest" exists.
I can pull the image locally and access the repository.
I could not find any documentation on how to integrate these images from Artifact Registry. There is only this official guide, where the image is built using the Docker image from Container Registry – however this should not be future proof.
It looks like you need to add your Project ID to your image name.
You can use the "$PROJECT_ID" Cloud Build default substitution variable.
So your updated image name would look something like this:
steps:
- name: 'europe-west3-docker.pkg.dev/$PROJECT_ID/xxxx/yyyy:latest'
For more details about substituting variable values in Cloud Build see:
https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values
I have a fairly simply Google Cloud Function that I'm deploying from Cloud Source Repositories.
I'm using the Google Cloud Shell as my development machine.
When I make updates to the function as I'm developing, I use the CLI to push updates to my Source Repository. However, running the gcloud functions deploy ... command from the command line doesn't seem to force GCF to pull in the latest source.
Occasionally, the deploy command after pushing new source code will simply state "Nothing to update." (which is incorrect.)
More often, it will go through the deployment process but the function will still run the previous version of the code.
When this happens the only way I can get the function to update is to use the dashboard, "Edit" the function, and then hit the Deploy button (even though I didn't change anything.)
Am I forgetting to do some kind of versioning or tagging that is required? Is there a way to force the CLI to pull the most current commit from the source repo?
I think you're looking for the --source=SOURCE gcloud functions deploy option to point to a source repository instead of the current directory (the default):
--source=SOURCE
Location of source code to deploy. Location of the source can be one
of the following three options:
Source code in Google Cloud Storage (must be a .zip archive),
Reference to source repository or,
Local filesystem path (root directory of function source).
Note that if you do not specify the --source flag:
Current directory will be used for new function deployments.
If the function is previously deployed using a local filesystem path, then function's source code will be updated using the current
directory.
If the function is previously deployed using a Google Cloud Storage location or a source repository, then the function's source code will
not be updated.
The value of the flag will be interpreted as a Cloud Storage location,
if it starts with gs://.
The value will be interpreted as a reference to a source repository,
if it starts with https://.
Otherwise, it will be interpreted as the local filesystem path. When
deploying source from the local filesystem, this command skips files
specified in the .gcloudignore file (see gcloud topic
gcloudignore for more information). If the .gcloudignore file
doesn't exist, the command will try to create it.
The minimal source repository URL is:
https://source.developers.google.com/projects/${PROJECT}/repos/${REPO}
By using the URL above, sources from the root directory of the
repository on the revision tagged master will be used.
If you want to deploy from a revision different from master, append
one of the following three sources to the URL:
/revisions/${REVISION},
/moveable-aliases/${MOVEABLE_ALIAS},
/fixed-aliases/${FIXED_ALIAS}.
If you'd like to deploy sources from a directory different from the
root, you must specify a revision, a moveable alias, or a fixed alias,
as above, and append /paths/${PATH_TO_SOURCES_DIRECTORY} to the URL.
Overall, the URL should match the following regular expression:
^https://source\.developers\.google\.com/projects/
(?<accountId>[^/]+)/repos/(?<repoName>[^/]+)
(((/revisions/(?<commit>[^/]+))|(/moveable-aliases/(?<branch>[^/]+))|
(/fixed-aliases/(?<tag>[^/]+)))(/paths/(?<path>.*))?)?$
An example of a validly formatted source repository URL is:
https://source.developers.google.com/projects/123456789/repos/testrepo/
moveable-aliases/alternate-branch/paths/path-to=source
I have project A and project B.
I use a GCP Cloud Source Repository on project A as my 'origin' remote.
I use Cloud Build with a trigger on changes to the 'develop' branch of the repo to trigger builds. As part of the build I deploy some stuff with the gcloud builder, to project A.
Now, I want to run the same build on project B. Maybe the same branch, maybe a different branch (i.e. 'release-*'). In the end want to deploy some stuff with the gcloud builder to project B.
The problem is, when I'm on project B (in Google Cloud Console), I can't even see the repo in project A. It asks me to "connect repository", but I can only select GitHub or Bitbucket repos for mirroring. The option "Cloud Source Repositories" is greyed out, telling me that they "are already connected". Just evidently not one from another project.
I could set up a new repo on project B, and push to both repos, but that seems inefficient (and likely not sustainable long term). The curious thing is, that such a setup could easily be achieved using an external Bitbucket/GitHub repo as origin and mirrored in both projects.
Is anything like this at all possible in Google Cloud Platform without external dependencies?
I also tried running all my builds in project A and have a separate trigger that deploys to project B (I use substitutions to manage that), but it fails with permission issues. Cloud Builds seem to always run with a Cloud Build service account, of which you can manage the roles, but I can't see how I could give it access to another project. Also in this case both builds would appear indistinguishable in a single build history, which is not ideal.
I faced a similar problem and I solved it by having multiple Cloud Build files.
A Cloud Build file (which got triggered when codes were pushed to a certain branch) was dedicated to copying all of my source codes into the new project source repo, of which it also has it's own Cloud Build file for deployment to that project.
Here is a sample of the Cloud Build file that copies sources to another project:
steps:
- name: gcr.io/cloud-builders/git
args: ['checkout', '--orphan', 'temp']
- name: gcr.io/cloud-builders/git
args: ['add', '-A']
- name: gcr.io/cloud-builders/git
args: ['config', '--global', 'user.name', 'Your Name']
- name: gcr.io/cloud-builders/git
args: ['config', '--global', 'user.email', 'Your Email']
- name: gcr.io/cloud-builders/git
args: ['commit', '-am', 'latest production commit']
- name: gcr.io/cloud-builders/git
args: ['branch', '-D', 'master']
- name: gcr.io/cloud-builders/git
args: ['branch', '-m', 'master']
- name: gcr.io/cloud-builders/git
args: ['push', '-f', 'https://source.developers.google.com/p/project-prod/r/project-repo', 'master']
This pushed all of the source codes into the new project.
Note that: You need to give your Cloud Build service account permissions to push source codes into the other project source repositories.
As you have already said, you can host your repos outside in BitBucket/Github and sync them to each project, but you need to pay an extra for each build.
You could use third party services otherwise to build your repos outside and deploy the result wherever you want for ex. look into CircleCI or similar service.
You could give permissions to build that it could refer to resources from another project, but I would keep them separated to minimize complexity.
My solution:
From service A, create new Cloud Build on branch release-* with Build Configuration specify $_PROJECT_ID is project B id
On GCP Cloud Build definition, add new Variable name _PROJECT_ID is project B id
NOTE: Remember grant permissons for your service account of project A(#cloudbuild.gserviceaccount.com) on project B
cloudbuild.yaml
- name: gcr.io/cloud-builders/docker
args:
- build
- '--no-cache'
- '-t'
- '$_GCR_HOSTNAME/$_PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
- .
- '-f'
- Dockerfile
id: Build
- name: gcr.io/cloud-builders/docker
args:
- push
- '$_GCR_HOSTNAME/$_PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
id: Push
- name: gcr.io/cloud-builders/gcloud
args:
- beta
- run
- deploy
- $_SERVICE_NAME
- '--platform=managed'
- '--image=$_GCR_HOSTNAME/$_PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
- >-
--labels=managed-by=gcp-cloud-build-deploy-cloud-run,commit-sha=$COMMIT_SHA,gcb-build-id=$BUILD_ID,gcb-trigger-id=$_TRIGGER_ID,$_LABELS
- '--region=$_DEPLOY_REGION'
- '--quiet'
- '--project=$_PROJECT_ID'
id: Deploy
entrypoint: gcloud
images:
- '$_GCR_HOSTNAME/$_PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
options:
substitutionOption: ALLOW_LOOSE
timeout: '20m'
tags:
- gcp-cloud-build-deploy-cloud-run
- gcp-cloud-build-deploy-cloud-run-managed
- driveit-hp-agreement-mngt-api```
[1]: https://i.stack.imgur.com/XhRJ4.png
Unfortunately Google doesn't seem to provide that functionality within Source Repositories (would rock if you could).
An alternative option you could consider (though involves external dependencies) is to mirror your Source Repositories first to GitHub or Bitbucket, then mirror back again into Source Repositories. That way, any changes made to any mirror of the repository will sync. (i.e. a change pushed in Project B will sync with Bitbucket, and likewise in Project A)
EDIT
To illustrate my alternative solution, here is a simple diagram
I'm using "Google App Engine" from GCP to host a static website. I already created the website files (HTML, JS) and yaml using Visual Studio Code. I have the folder with those files stored locally in my local computer.
I downloaded the Cloud SDK Shell for Windows. I logged in to my account, and selected the project. According to videos and tutorials, I need to deploy the app using "gcloud app deploy".
However I got an error saying that an "app.yaml" file is required to deploy this directory...
I'm trying to follow this tutorial:
https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website#before_you_begin
I'm also trying to follow the steps contained in this video:
https://www.youtube.com/watch?v=mlcO7nfQzSg
How do I specify the root folder where I have my "app.yaml" file?
Thanks in advance!!
I already tried with many commands and unfortunately none of them have worked
The particular case in which gcloud app deploy works without additional arguments is for single-service applications only and only if the command is executed in the directory in which the service's app.yaml configuration file exists (and has that exact name, can't use a different name).
For other cases deployables can/must be specified. From gcloud app deploy:
SYNOPSIS
gcloud app deploy [DEPLOYABLES …] [--bucket=BUCKET] [--image-url=IMAGE_URL] [--no-promote] [--no-stop-previous-version]
[--version=VERSION, -v VERSION] [GCLOUD_WIDE_FLAG …]
DESCRIPTION
This command is used to deploy both code and configuration to the App
Engine server. As an input it takes one or more DEPLOYABLES that
should be uploaded. A DEPLOYABLE can be a service's .yaml file or a
configuration's .yaml file (for more information about configuration
files specific to your App Engine environment, refer to
https://cloud.google.com/appengine/docs/standard/python/configuration-files
or
https://cloud.google.com/appengine/docs/flexible/python/configuration-files).
Note, for Java Standard apps, you must add the path to the
appengine-web.xml file inside the WEB-INF directory. gcloud app
deploy skips files specified in the .gcloudignore file (see gcloud
topic gcloudignore for more information).
So apart from running the command with no arguments in the directory in which your app.yaml exists is to specify the app.yaml (with a full or relative path if needed) as a deployable:
gcloud app deploy path/to/your/app.yaml
IMHO doing this is a good habit - specifying deployables is more reliable and is the only way to deploy apps with multiple services or using routing via a dispatch.yaml file.
gcloud app deploy will look at the current directory first for app.yaml. Generally you will change to the directory with app.yaml and your other files before deploying