It seems like using service accounts with Dataproc is not possible because when using a service account with Cloud Dataproc I'm getting permission errors.
For example, running the command gcloud beta dataproc clusters list yeilds an error which says (gcloud.beta.dataproc.clusters.list) Request had insufficient authentication scopes
Does Cloud Dataproc allow for the use of service accounts?
Yes, you can use service accounts with Cloud Dataproc.
This error is due to the requirement that the cloud-platform scope be enabled. The long name for this scope is https://www.googleapis.com/auth/cloud-platform.
When creating a Google Compute Engine instance it means skipping the dropdown scopes and just clicking the top-level Allow API access to all Google Cloud services in the same project.
Related
I am trying to deploy my AutoML trained model using Cloud Run, but am having difficulties with IAM permissions. I'm using this as a guide, and keep getting the following error at the build step. I'm new to Cloud Run/GCP but as far as I can see, I have granted the right roles to the right accounts. I appreciate any assistance you can provide as I'm really stumped.
Error message:
Step #3: ERROR: (gcloud.run.deploy) User [REDACTED#cloudbuild.gserviceaccount.com] does not have permission to access namespaces instance [REDACTED] (or it may not exist):
Google Cloud Run Service Agent does not have permission to get access tokens for the service account REDACTED#cloudbuild.gserviceaccount.com.
Please give service-REDACTED#serverless-robot-prod.iam.gserviceaccount.com permission iam.serviceAccounts.getAccessToken on the service account.
Alternatively, if the service account is unspecified or in the same project you are deploying in, ensure that the Service Agent is assigned the Google Cloud Run Service Agent role roles/run.serviceAgent.
Here are the roles assigned to the Cloud Run Service Agent
Default service account has the Cloud Run Admin Role
Here are the accounts to which I've given access to the Default compute service account, the Cloud Build Service Agent and Cloud Run Service Agent are added:
New to GCP and use IAC for our Terraform. I've managed to build most of the initial organisation config in Terraform no problem with the exception of Cloud Identity and Organisation Policies. I'm using gcloud provided login credentials. Whenever I try to build for those two services I get this:
Error creating Group: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the cloudidentity.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
So in this case i'm using the Google Cloud SDK, so the error makes sense. However, the two options it presents don't work:
Setting a quota project makes no difference
I can't create a service account at the organisational level (and when I create one within a project it can't configure these organisational level constructs)
So how do I go about Terraforming these services?
Thanks.
Service account "abcdefc-compute#developer.gserviceaccount.com" does not exist.
I am trying to create a kubernetes cluster but GCP gives me the error above.
I checked for the account name in service account but could not find it, rather I have
'ayushaccount#abcdef.iam.gserviceaccount.com'.
I tried to create another service account with this email "abcdefc-compute#developer.gserviceaccount.com" but it does not allow me to create.
I am new to GCP and I do not know how to solve this problem. All I am looking for to create a kubernetes cluster in GCP.
Looks like you are missing the default service account for your GCP project.
You have two options:
(re)create the default service account
when creating your GKE cluster, under NODE POOLS, go to default-pool->Security and for Service account, select one the one which exists.
If you want to (re)create the default service account, you can disable/enable the Google Compute Engine API via the console or run gcloud services enable compute.googleapis.com from Cloud Shell or from the command line on your workstation.
Say I have this case where
I have to run some test with dataflow
inside this dataflow job I need to access a gcs bucket and save my output there.
I will need to run the dataflow job with my own SA instead of the default SA.
I created a Google Service Account to run my dataflow job. But after I enabled the dataflow API. I end up having 2 SA in front of me.
the service account agent --> 123456789#dataflow.gserviceaccount.com
the dataflow job runner service account --> dataflow-job-runner#MY-PROJECT-ID.iam.gserviceaccount.com
It got me really confused to see what the official document says
Some Google Cloud services have Google-managed service accounts that allow the services to access your resources. These service accounts are sometimes known as service agents.
If I create a dataflow job to run with the dataflow-job-runner#MY-PROJECT-ID.iam.gserviceaccount.com SA, I suppose I'd need to grant the roles/storage.objectAdmin for it.
The question is
Do I need to grant any permission to the service account agent?
What does the service account agent actually do, what does it has to access any resource?
Several Google Cloud services such as Cloud Dataflow require two sets of permissions.
The program that you write uses a service account. You grant this service account IAM roles to access resources that require authorization that your program requires. For example, reading data from Cloud Storage or issuing queries to BigQuery.
The service agent applies to the service's runtime. For example when you launch a job on Cloud Dataflow, Cloud Dataflow needs to launch VMs to run your program on. Your program is not launching the VMs, the service is. Therefore the service requires its own set of permissions. This is what the service agent is for.
By using two different service accounts, separation of privilege is achieved.
I need to execute a dataflow job using a service account , I'm following a very simple and basic example wordcount offered within the same platform itself.
Which is weird is the error I'm getting:
According to this, GCP requires the service account having permissions as Dataflow worker in order to execute my job. The weir part comes over when the error kept on showing up even though I have already set the required permissions:
Can someone explain this strange behavior? thanks so much
To run a Dataflow job, a project must enable billing and the following Google Cloud Platform APIs:
Google Cloud Dataflow API
Compute Engine API (Google Compute Engine)
Google Cloud Logging API
Google Cloud Storage
Google Cloud Storage JSON API
BigQuery API
Google Cloud Pub/Sub
Google Cloud Datastore API
You should also have enough quota in the project for any one of the APIs you are using in the Dataflow job.
I would suggest you to create a fresh service account which its name has not been used before and then granting roles/dataflow.worker to this new fresh service account. Remember, that Cloud IAM propagation takes fewer than 60 seconds, up to 7 minutes, so please have a couple of minutes between an IAM change and Dataflow job creation.
Another possible workaround is to delete the Dataflow worker permission and add it again. The permission remains after the removal of the account, pointing to its old ID. This ID must not be refreshed until explicitly deleting the role.
I encourage you to visit Dataflow IAM roles with role descriptions and permissions documentation.