Google Cloud Source Build Trigger [duplicate] - google-cloud-platform

I am trying to deploy code from this repo:
https://github.com/anishkny/puppeteer-on-cloud-functions
in Google Cloud Build. My cloudbuild.yaml file contents are:
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args: ['beta', 'functions', 'deploy', 'screenshot', '--trigger-http', '--runtime', 'nodejs8', '--memory', '1024MB']
I have given the following roles to my Cloud Build Service account (****#cloudbuild.gserviceaccount.com):
Cloud Build Service Account
Cloud Functions Developer
Yet, in my Cloud Build log I see the following error:
starting build "1f04522c-fe60-4a25-a4a8-d70e496e2821"
FETCHSOURCE
Fetching storage object: gs://628906418368.cloudbuild-source.googleusercontent.com/94762cc396ed1bb46e8c5dbfa3fa42550140c2eb-b3cfa476-cb21-45ba-849c-c28423982a0f.tar.gz#1534532794239047
Copying gs://628906418368.cloudbuild-source.googleusercontent.com/94762cc396ed1bb46e8c5dbfa3fa42550140c2eb-b3cfa476-cb21-45ba-849c-c28423982a0f.tar.gz#1534532794239047...
/ [0 files][ 0.0 B/ 835.0 B]
/ [1 files][ 835.0 B/ 835.0 B]
Operation completed over 1 objects/835.0 B.
tar: Substituting `.' for empty member name
BUILD
Already have image (with digest): gcr.io/cloud-builders/gcloud
ERROR: (gcloud.beta.functions.deploy) ResponseError: status=[403], code=[Forbidden], message=[The caller does not have permission]
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcloud" failed: exit status 1
What am I missing?

It would appear that the permissions changed when (perhaps) Cloud Functions went GA. Another customer raised this issue today and I recalled your question.
The Cloud Build robot (${NUM}#cloudbuild.gserviceaccount.com) additionally needs to be a serviceAccountUser of the ${PROJECT-ID}#appspot.gserviceaccount.com account:
NB While the Cloud Build robot local part is the project number (${NUM}), the appspot robot local part is the project ID (${PROJECT})
Please try:
PROJECT=[[YOUR-PROJECT-ID]]
NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)')
gcloud iam service-accounts add-iam-policy-binding \
${PROJECT}#appspot.gserviceaccount.com \
--member=serviceAccount:${NUM}#cloudbuild.gserviceaccount.com \
--role=roles/iam.serviceAccountUser \
--project=${PROJECT}
Let me know!

I struggled with this too after reading quite a bit of documentation. A combination of the above answers got me on the right track. Basically, something like the following is needed:
PROJECT=[PROJECT-NAME]
NUM=$(gcloud projects describe $PROJECT --format='value(projectNumber)')
gcloud iam service-accounts add-iam-policy-binding \
${PROJECT}#appspot.gserviceaccount.com \
--member=serviceAccount:${NUM}#cloudbuild.gserviceaccount.com \
--role=roles/iam.serviceAccountUser \
--project=${PROJECT}
gcloud iam service-accounts add-iam-policy-binding \
${PROJECT}#[INSERT_YOUR_IAM_OWNER_SERVICE_ACCOUNT_NAME].iam.gserviceaccount.com \
--member='serviceAccount:service-${NUM}#gcf-admin-robot.iam.gserviceaccount.com' \
--role='roles/iam.serviceAccountUser'
Also, I added the "Cloud Functions Developer" role to my #cloudbuild.gserviceaccount.com account via the IAM Console.

According to Cloud Build documentation, for Cloud Functions you have to grant the "Project Editor" role to your service account.
But, Cloud Functions documentation states that alternatively to using the Project Editor role, you can use "the Cloud Functions Developer role [but you have to] ensure that you have granted the Service Account User role". Regarding Service Accounts, it indicates to have "the CloudFunctions.ServiceAgent role on your project" and to "have permissions for trigger sources, such as Pub/Sub or the Cloud Storage bucket triggering your function".
Due to those considerations, my understanding is that the documentation omitted to specify all the roles your service account would need and went directly to indicate to grant the Project Editor role.

You have to update Service Account permissions on Cloud Build settings page.
Here is instructions https://cloud.google.com/cloud-build/docs/deploying-builds/deploy-cloud-run#fully-managed
You just have to set the status of the Cloud Run Admin role to ENABLED on that page:

start your cloud build with auth
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args: ['auth', 'activate-service-account', 'xoxox#xoxo-dev.iam.gserviceaccount.com', '--key-file=account.json', '--project=rabbito-dev']
and then simply your code deployment on cloud function
- name: 'gcr.io/cloud-builders/gcloud'
args: ['beta', 'functions', 'deploy', 'screenshot', '--trigger-http', '--runtime', 'nodejs8', '--memory', '1024MB']

Please add 'Cloud Functions Service Agent' role to your service account alongside 'Cloud Functions Developer'.

Related

How do I deploy a Google Cloud Function (2nd generation)?

I've previously only used Cloud Functions of gen. 1 but now plan to move to 2nd generation and is just trying to deploy/test a first basic function. I'm just taking the Google sample for a storage triggered function and try to deploy it, but it keeps failing.
This is what it looks like:
> gcloud functions deploy nodejs-finalize-function --gen2 --runtime=nodejs16 --project myproject --region=europe-west3 --source=. --entry-point=handleImage --trigger-event-filters='type=google.cloud.storage.object.v1.finalized' --trigger-event-filters='bucket=se_my_images'
Preparing function...done.
X Deploying function...
✓ [Build] Logs are available at [https://console.cloud.google.com/cloud-build/builds;region=europe-west3/a8355043-adf0-4485-a510-1d54b7e11111?project=123445666123]
✓ [Service]
✓ [Trigger]
- [ArtifactRegistry] Deleting function artifacts in Artifact Registry...
. [Healthcheck]
. [Triggercheck]
Failed.
ERROR: (gcloud.functions.deploy) OperationError: code=7, message=Creating trigger failed for projects/myproject/locations/europe-west3/triggers/nodejs-finalize-function-898863: The Cloud Storage service account for your bucket is unable to publish to Cloud Pub/Sub topics in the specified project.
To use GCS CloudEvent triggers, the GCS service account requires the Pub/Sub Publisher (roles/pubsub.publisher) IAM role in the specified project. (See https://cloud.google.com/eventarc/docs/run/quickstart-storage#before-you-begin)
The error looks easy to understand, but I have aded the Pub/Sub Publisher role to all my service accounts now (the ones listed below) and I still keep getting the same error.
>gcloud iam service-accounts list --project myproject
DISPLAY NAME EMAIL DISABLED
firebase-adminsdk firebase-adminsdk-u2x33#myproject.iam.gserviceaccount.com False
Default compute service account 930445666575-compute#developer.gserviceaccount.com False
backend-dev backend-dev#myproject.iam.gserviceaccount.com False
App Engine default service account myproject#appspot.gserviceaccount.com False
I don't know how to move forward from here so I hope someone can help.
*** EDIT ***.
I added the role to the listed service accounts in the GCP console, IAM > Permissions > View By Principal page/view where I used the Edit Principal button to assign an additional role (Pub/Sub Publisher) to the service accounts (note that I added the role to all my listed service accounts since I'm not 100% sure which one is used by GCP for cloud deployment).
since you are already using gcloud cli, i suggest you follow the step 2 which says:
PROJECT_ID=$(gcloud config get-value project)
PROJECT_NUMBER=$(gcloud projects list --filter="project_id:$PROJECT_ID" --format='value(project_number)')
SERVICE_ACCOUNT=$(gsutil kms serviceaccount -p $PROJECT_NUMBER)
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:$SERVICE_ACCOUNT \
--role roles/pubsub.publisher
After these 4 cmd's, you should have no problems.. I don't use gcp interface for iam purposes, since all my iam policies are uploaded by terraform/terragrunt.

GCP workload identity federation - Github provider - 'Unable to acquire impersonated credentials'

I've followed these instructions to the letter to allow me to use the short lived token authentication method to access gcloud resources from our github actions workflow.
I've created the service account, workload identity pool and github provider pool using the exact instructions above, but it doesn't appear that the auth step is getting the correct token (or any token at all). The GCP service account has the correct IAM permissions.
On the gcloud compute instances list step, I'm receiving the error:
ERROR: (gcloud.compute.instances.list) There was a problem refreshing your current auth tokens: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n "error": {\n "code": 403,\n "message": "The caller does not have permission",\n "status": "PERMISSION_DENIED"\n }\n}\n')
Please run:
$ gcloud auth login
to obtain new credentials.
My github actions file is as follows:
jobs:
backup:
permissions:
contents: 'read'
id-token: 'write'
runs-on: ubuntu-latest
steps:
- name: 'Checkout code'
uses: actions/checkout#v3
- id: 'auth'
name: 'Authenticate to Google Cloud'
uses: 'google-github-actions/auth#v0'
with:
workload_identity_provider: 'projects/*******/locations/global/workloadIdentityPools/**REDACTED**/providers/github-provider'
service_account: '**REDACTED**#**REDACTED**.iam.gserviceaccount.com'
# Install gcloud, `setup-gcloud` automatically picks up authentication from `auth`.
- name: 'Set up Cloud SDK'
uses: 'google-github-actions/setup-gcloud#v0'
- name: 'Use gcloud CLI'
run: gcloud compute instances list
I enabled logging for the token exchange and I can see it occurring (with no obvious errors) in GCP logs either. So I'm completely stumped.
Any ideas?
So I later found out what this was. Despite running:
gcloud iam service-accounts add-iam-policy-binding SERVICE_ACCOUNT_EMAIL \
--role=roles/iam.workloadIdentityUser \
--member="MEMBER_EXPRESSION"
As per the docs, it had not granted permission - I went into the console and checked the workload identity pool under "connected service accounts" menu (to the left) and the service account wasn't in there, so I added it manually.
In addition to the OP's own answer of the service account not being connected (bound) at all, this can result from the service account binding being constrained using attribute mappings.
In the default setup for GitHub Actions discussed here on the google blog for WIF, the provider is set up with a set of attribute mappings:
--attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository"
Those attributes can be used to restrict access when you connect (bind) the service account... in the following, the member uses the repository attribute to constrain it so that only actions executing in my-org/my-repo on GitHub will be permitted.
gcloud iam service-accounts add-iam-policy-binding "my-service-account#${PROJECT_ID}.iam.gserviceaccount.com" \
--project="${PROJECT_ID}" \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/1234567890/locations/global/workloadIdentityPools/my-pool/attribute.repository/my-org/my-repo"
A cross-repository provider
Of course, you want to use strong restrictions here. Restricting to a repository, or even a particular branch (so only main branch actions have privilege to deploy to production, for example). No restrictions allows anything, which is absolutely not what you want!!!
In my case, I set up my WIF provider, then tried to reuse it from another repository, resulting in the error experienced by the OP.
I chose to add the repository_owner attribute mapping from the list of all possible attributes that are in GitHub's OIDC token (the attribute mappings are editable in the google cloud console), then bind my service account to that rather than the repository-specific principal:
--attribute-mapping="google.subject=assertion.sub,attribute.repository_owner=assertion.repository_owner"
and
gcloud iam service-accounts add-iam-policy-binding "my-service-account#${PROJECT_ID}.iam.gserviceaccount.com" \
--project="${PROJECT_ID}" \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/1234567890/locations/global/workloadIdentityPools/my-pool/attribute.repository_owner/my-org"
Bingo, it works a charm now.
Take care to think about your attack surface though, loosening this constraint too widely creates real vulnerability.

What roles do my Cloud Build service account need to deploy an http triggered unauthenticated Cloud Function?

I was trying to deploy an http triggered Cloud Function with Cloud Build using this configuration.
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args:
- beta
- functions
- deploy
- myfunction
- --source=start_shopify_installation
- --trigger-http
- --region=europe-west1
- --runtime=nodejs14
- --allow-unauthenticated
- --ingress-settings=all
- --security-level=secure-always
- --set-secrets=env_1=secret_1:latest
When I got an error saying Cloud Build could't set an IAM policy.
WARNING: Setting IAM policy failed, try:
gcloud alpha functions add-iam-policy-binding myfunction\
--region=europe-west1 \
--member=allUsers \
--role=roles/cloudfunctions.invoker
The function gets deployed and when I check in the GCP console it looks like the allUsers member has the role Cloud Functions Invoker, but it doesn't have allow unauthenticated in the Authentication column. When I go to invoke the function I get a 'missing permissions' error.
When I execute the suggested command from my Cloud Shell it works just fine. However if I fill it in as an extra step in my deployment configuration that step fails.
I think that my Cloud Build service account must be missing a role in order to make the function accessible without authetication? Currently it has these roles: Cloud Build Service Account, Cloud Functions Developer and Service Account User.
EDIT
I added the Project IAM Admin role to the Cloud Build service account and tried again.
Unfortunately, it didn't change anything.
I reproduced your error (warning) on my side and fixed it: I can see allUsers having Cloud Functions Invoker role in the function's PERMISSIONS tab.
In fact your cloud build service account needs the cloudfunctions.functions.setIamPolicy permission. So the solution is replace Cloud Functions Developer role with Cloud Functions Admin role.
Use of the --allow-unauthenticated flag modifies IAM permissions. To ensure that unauthorized developers cannot modify function permissions, the user or service that is deploying the function must have the cloudfunctions.functions.setIamPolicy permission. This permission is included in both the Owner and Cloud Functions Admin roles.
Ref: https://cloud.google.com/functions/docs/securing/managing-access-iam#at_deployment

Google Cloud Platform Service Account is Unable to Access Project

I encounter the following warning:
WARNING: You do not appear to have access to project [$PROJECT] or it does not exist.
after running the following commands locally:
Activate and set a service account:
gcloud auth activate-service-account \
$SERVICE_ACCOUNT \
--key-file=key.json
#=>
Activated service account credentials for: [$SERVICE_ACCOUNT]
Select $PROJECT as the above service account:
gcloud config set project $PROJECT
#=>
Updated property [core/project].
WARNING: You do not appear to have access to project [$PROJECT] or it does not exist.
My own GCP account is associated with the following roles:
App Engine Admin
Cloud Build Editor
Cloud Scheduler Admin
Storage Object Creator
Storage Object Viewer
Why is this service account unable to set $PROJECT? Is there a role or permission I am missing?
The solution to this issue might be to enable the Cloud Resource Manager API in your Google Cloud Console here by clicking enable.
I believe this is an erroneous warning message. I see the same warning message on my service account despite the fact that the account has permissions on my GCP project and can successfully perform necessary actions.
You might be seeing this error due to an unrelated problem. In my case, I was trying to deploy to AppEngine from a continuous integration environment (Circle CI), but I hadn't enabled the App Engine Admin API. Once I enabled the API, I was able to deploy successfully.
I encountered this error when I started out with Google CLoud Platform.
The issue was that I configured/set a non-existing project (my-kube-project)
as my default project using the command below:
gcloud config set project my-kube-project
Here's how I solved it:
I had to list my existing projects first:
gcloud projects list
And then I copied the ID of the project that I wanted, and rannthe command again this time:
gcloud config set project gold-magpie-258213
And it worked fine.
Note: You cannot change the ID of a project's ID or Number,you can only change the Name.
That's all.
I hope this helps
I was encountering the same error when trying to deploy an app to Google App Engine via a service account configured in CircleCI and resolved it by having the following roles (permissions) attached to my service role:
App Engine Deployer
App Engine Service Admin
Cloud Build Editor
Storage Object Creator
Storage Object Viewer
I also had the App Engine Admin API enabled, but not the Cloud Resource Manager API.
The
WARNING: You do not appear to have access to project [$PROJECT_ID] or it does not exist.
warning will appear if there isn't at least one role granted to the service account that contains the resourcemanager.projects.get permission.
In other words, the warning will appear if the result of the following commands is blank:
Gather all roles for a given $SERVICE_ACCOUNT (this works for any account, not just service accounts):
gcloud projects get-iam-policy $PROJECT_ID \
--flatten='bindings[].members' \
--format='table(bindings.role)' \
--filter="bindings.members:${SERVICE_ACCOUNT}"
#=>
ROLE
. . .
For each $ROLE gathered above, either:
gcloud iam roles describe $ROLE \
--flatten='includedPermissions' \
--format='value(includedPermissions)' \
--project=$PROJECT_ID | grep \
--regexp '^resourcemanager.projects.get$'
if the $ROLE is a custom (projects/$PROJECT_ID/roles/$ROLE), or:
gcloud iam roles describe roles/$ROLE \
--flatten='includedPermissions' \
--format='value(includedPermissions)' | grep \
--regexp '^resourcemanager.projects.get$'
if the $ROLE is a curated (roles/$ROLE).
Note: the difference between gcloud command formatting for custom and curated roles is what makes listing all permissions associated with all roles associated with a single account difficult.
If you have confirmed that none of the roles associated with a service account contain the resourcemanager.projects.get permission, then either:
Update at least one of the custom roles associated with the service account with the resourcemanager.projects.get permission:
gcloud iam roles update $ROLE \
--add-permissions=resourcemanager.projects.get \
--project=$PROJECT_ID
#=>
description: $ROLE_DESCRIPTION
etag: . . .
includedPermissions:
. . .
- resourcemanager.projects.get
. . .
name: projects/$PROJECT_ID/roles/$ROLE
stage: . . .
title: $ROLE_TITLE
Warning: make sure to use the --add-permissions flag here when updating, as the --permissions flag will remove any other permissions the custom role used to have.
Create a custom role:
gcloud iam roles create $ROLE \
--description="$ROLE_DESCRIPTION" \
--permissions=resourcemanager.projects.get \
--project=$PROJECT_ID \
--title='$ROLE_TITLE'
#=>
Created role [$ROLE].
description: $ROLE_DESCRIPTION
etag: . . .
includedPermissions:
- resourcemanager.projects.get
name: projects/$PROJECT_ID/roles/$ROLE
stage: . . .
title: $ROLE_TITLE
and associate it with the service account:
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:$SERVICE_ACCOUNT \
--role=projects/$PROJECT_ID/roles/$ROLE
#=>
Updated IAM policy for project [$PROJECT_ID].
auditConfigs:
. . .
Associate the service account with a curated role that already contains the resourcemanager.projects.get permission, which has been discussed above.
If you want to know which curated roles already contain the resourcemanager.projects.get permission and don't want to craft a complex shell loop, it might be easier to go here and filter all roles by Permission:resourcemanager.projects.get.
Note: if you are running into issues, be sure to read the requirements for granting access to resources here.

Unable to create google project with Terraform

I'm following the Google GKE and SQL with terraform tutorial
But I'm not able to create a google_project.project.
I have tried both as the owner of the project and as the service described in the tutorial. Both attempts end with this error:
Error: Error applying plan:
1 error(s) occurred:
* google_project.project: 1 error(s) occurred:
* google_project.project: error creating project terraform-dev-357aa670
(terraform-dev): googleapi: Error 403: User is not authorized., forbidden.
If you received a 403 error, make sure
you have the `roles/resourcemanager.projectCreator` permission
I would think that I had the correct permissions as the project owner, but apparently not.
Here's how I created the service account:
$ gcloud organizations add-iam-policy-binding ${TF_VAR_org_id} \ (gke_my-domain-218910_europe-west1-b_my-domain-vpc-native/default)
> --member serviceAccount:terraform#${TF_ADMIN}.iam.gserviceaccount.com \
> --role roles/resourcemanager.projectCreator
Updated IAM policy for organization [00000].
bindings:
- members:
- domain:my-domain.no
role: roles/billing.creator
- members:
- serviceAccount:terraform#my-domain-terraform-admin-3.iam.gserviceaccount.com
- serviceAccount:terraform#my-domain-terraform-admin.iam.gserviceaccount.com
role: roles/billing.user
- members:
- domain:min-familie.no
- serviceAccount:terraform#my-domain-terraform-admin-3.iam.gserviceaccount.com
- serviceAccount:terraform#my-domain-terraform-admin.iam.gserviceaccount.com
role: roles/resourcemanager.projectCreator
etag: BwWJxJTDnQs=
version: 19d
Creating a project "manually" works.
$ gcloud projects create ${TF_ADMIN}.
Any ideas what might be wrong?
In order to create folders and projects, your account need to have the respective permissions and, of course you need to make sure that you are using the right account.
First make sure the user has the right permissions:
gcloud organizations add-iam-policy-binding YOUR_ORGANIZATION_ID --member=user:your#email.com --role=roles/billing.admin
gcloud organizations add-iam-policy-binding YOUR_ORGANIZATION_ID --member=user:your#email.com --role=roles/resourcemanager.organizationAdmin
gcloud organizations add-iam-policy-binding YOUR_ORGANIZATION_ID --member=user:your#email.com --role=roles/resourcemanager.folderCreator
gcloud organizations add-iam-policy-binding YOUR_ORGANIZATION_ID --member=user:your#email.com --role=roles/resourcemanager.projectCreator
Then make sure you set the application defaults and login to exactly this account:
gcloud auth application-default login
The set a project that the API calls will be billed to by default. Read more about this here. If you don't set this, you might get a quota error when you run terraform apply.
gcloud auth application-default set-quota-project SOME_BILLING_PROJECT
I had exact same problem!
Steps that solved this problem for me:
Downloaded the key for that Service Account (Using GCP Console) to : /Users/johndoe/sa.json
export GOOGLE_APPLICATION_CREDENTIALS=/Users/johndoe/factory.json
terraform apply
Hope this works for you.
Found the solution from Seth Fargo here:
https://github.com/sethvargo/vault-on-gke/issues/16