I'm trying to deploy a new revision of a container running on managed Google Cloud Run.
When I try to update the container I get the following message:
Cloud Run error: Internal system error. Missing necessary permission for xxxxxxxxxx#serverless-robot-prod.iam.gserviceaccount.com on resource <resource-id>
For others cloud run services everything works fine, I'm able to release an update.
What can be happening in this specific case?
Related
I cannot seem to deploy/run any Google Cloud Run services.
I have attempted this from multiple accounts (with billing on all accounts and projects), created fresh projects in each account, added every permission I could find to try to get around this.
I've built my own container based on the Hello World example from here: https://cloud.google.com/run/docs/quickstarts/build-and-deploy
Trying to deploy:
helloworld-csharp>gcloud run deploy --image gcr.io/[Project ID]/helloworld --platform managed
Service name (helloworld):
Deploying container to Cloud Run service [helloworld] in project [Project ID] region [us-west1]
Deploying...
Creating Revision... Cloud Run error: Internal system error. Missing necessary permission for service-[ID]#serverless-robot-prod.iam.gserviceaccount.com on resource [Project ID]. Please visit https://cloud.google.com/run
/docs/troubleshooting for in-depth troubleshooting documentation....failed
Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Internal system error. Missing necessary permission for service-[ID]#serverless-robot-prod.iam.gserviceaccount.com on resource [Project ID]. Please visit https://cloud.google.com/run/docs/troubleshooting for in-depth troubleshooting documentation.
Trying to deploy the simple 'hello' example here from the web console leaves me with the same error:
Cloud Run error: Internal system error. Missing necessary permission for service-[ID]#serverless-robot-prod.iam.gserviceaccount.com on resource [Project ID]. Please visit https(...)cloud.google.com/run/docs/troubleshooting for in-depth troubleshooting documentation.
I have the following users in the project, as they were auto-setup and configured when I enabled the API:
[ID]-compute#developer.gserviceaccount.com Compute Engine default service account
[ID]#cloudbuild.gserviceaccount.com Cloud Build Service Account
[ID]#cloudservices.gserviceaccount.com Google APIs Service Agent
service-[ID]#compute-system.iam.gserviceaccount.com Compute Engine Service Agent
service-[ID]#gcp-sa-cloudbuild.iam.gserviceaccount.com Cloud Build Service Account
service-[ID]#serverless-robot-prod.iam.gserviceaccount.com Google Cloud Run Service Agent
Yes, it seems it was indeed a Google issue, I didn't change anything, just went back to the console and I can start all my test containers without any issue now...
EDIT: The so called duplicate question was way off since 1. I could push another image and 2. I could not push a build image. Finally, point #3 is the solution was totally different and ONLY related to pushing build images via cloudbuild. ie. I beg to differ that this question WAS different.
Running into some more google cloud security stuff. We currently deploy to cloud run like so
docker build . --tag gcr.io/myproject/authservice
docker push gcr.io/myproject/authservice
gcloud run deploy staging-admin --region us-west1 --image gcr.io/myproject/authservice --platform managed
I did the quick start for google builds but I am getting permission errors. I did this command
https://cloud.google.com/cloud-build/docs/quickstart-build
The command I ran was
gcloud builds submit --tag gcr.io/myproject/quickstart-image
This is all the same project but submitting builds gets this same error over and over and over(I am not sure why it doesn't just exit on first error.
The push refers to repository [gcr.io/myproject/quickstart-image]
e3831abe9997: Preparing
60664c29ef5a: Preparing
denied: Token exchange failed for project 'myproject'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control
Any ideas how to fix so I can use google cloud build?
Complementing the previous answer, as is mentioned in this document to perform actions in Container Registry the role "sotrage admin" is necessary
Do you have "roles/storage.admin" role? If not, add it and try.
The Could build service account has this format [project_number]#cloudbuild.gserviceaccount.com please add the role "roles/storage.admin" by following this steps
Open the Cloud IAM page
Select your Cloud project.
In the permissions table, locate the row with the email address
ending with #cloudbuild.gserviceaccount.com. This is your Cloud
Build service account.
Click on the pencil icon.
Select the role you wish to grant to the Cloud Build service
account.
Click Save.
BE WARNED: I read the duplicate question post but in my case
I can push items
only the build one is failing AND the solution I found is different than any of the other question answers
This was a VERY weird issue. The storage permission MUST be a red herring because these permissions fixed the issue
I found some documentation somewhere that I can't seem to find on a google github repo about adding these permissions AND a document on the TWO #cloudbuild.gserviceaccount.com accouts AND you must add the permissions to the correct one!!!! One is owned by google and you should not touch.
In my case, the permission / token exchange failed error was caused by having the storage bucket used by Google Container Registry inside a VPC Service Perimeter.
This can be checked / confirmed via the VPC Service Controls logs - accessible easily from the troubleshooting page.
There is a (very clunky) way to get Cloud Build working to push images to a registry inside a VPC perimeter. It involves running a build worker pool and applying appropriate config + permissions to the perimeter etc.
I was happily deploying to Kubernetes Engine for a while, but while working on an integrated cloud container builder pipeline, I started getting into trouble.
I don't know what changed. I can not deploy to kubernetes anymore, even in ways I did before without cloud builder.
The pods rollout process gives an error indicating that it is unable to pull from the registry. Which seems weird because the images exist (I can pull them using cli) and I granted all possibly related permissions to my user and the cloud builder service account.
I get the error ImagePullBackOff and see this in the pod events:
Failed to pull image
"gcr.io/my-project/backend:f4711979-eaab-4de1-afd8-d2e37eaeb988":
rpc error: code = Unknown desc = unauthorized: authentication required
What's going on? Who needs authorization, and for what?
In my case, my cluster didn't have the Storage read permission, which is necessary for GKE to pull an image from GCR.
My cluster didn't have proper permissions because I created the cluster through terraform and didn't include the node_config.oauth_scopes block. When creating a cluster through the console, the Storage read permission is added by default.
The credentials in my project somehow got messed up. I solved the problem by re-initializing a few APIs including Kubernetes Engine, Deployment Manager and Container Builder.
First time I tried this I didn't succeed, because to disable something you have to disable first all the APIs that depend on it. If you do this via the GCloud web UI then you'll likely see a list of services that are not all available for disabling in the UI.
I learned that using the gcloud CLI you can list all APIs of your project and disable everything properly.
Things worked after that.
The reason I knew things were messed up, is because I had a copy of the same things as a production environment, and there these problems did not exist. The development environment had a lot of iterations and messing around with credentials, so somewhere things got corrupted.
These are some examples of useful commands:
gcloud projects get-iam-policy $PROJECT_ID
gcloud services disable container.googleapis.com --verbosity=debug
gcloud services enable container.googleapis.com
More info here, including how to restore service account credentials.
I've currently got an opsworks stack using Chef 12 but I seem unable to deploy to it, even with a publicly available git repo that Amazon provide by themselves.
The deploy says it completed successfully but the app code does not exist on the server. The only error I see in the logs (even though it's not labelled as an error) is a 404 page not found which I can only assume is when it tries to grab the repo.
According to this question Opsworks deploy to custom layer do I need to use https://github.com/aws/opsworks-cookbooks/tree/release-chef-11.4/deploy as well in my custom cookbook for the deploy to work? Is it just a case of referencing say the php deploy recipe as part of my deploy phase in my layer?
Thanks
Can anybody please guide me how to easily deploy code from github to aws using aws cpde deploy ? I gave tried my best to deploy my code and it is not working as it gives error every time it launch deploy revision.
You mentioned deployment is failing. So I'm assuming github automatically kicking off the deployment part is working. Now for the deployment failure, do you see the instance being deployed marked failed? Can you also look into the instance and see if the codedeploy agent is running fine and paste the log here if possible?
-Surya