Say I have this case where
I have to run some test with dataflow
inside this dataflow job I need to access a gcs bucket and save my output there.
I will need to run the dataflow job with my own SA instead of the default SA.
I created a Google Service Account to run my dataflow job. But after I enabled the dataflow API. I end up having 2 SA in front of me.
the service account agent --> 123456789#dataflow.gserviceaccount.com
the dataflow job runner service account --> dataflow-job-runner#MY-PROJECT-ID.iam.gserviceaccount.com
It got me really confused to see what the official document says
Some Google Cloud services have Google-managed service accounts that allow the services to access your resources. These service accounts are sometimes known as service agents.
If I create a dataflow job to run with the dataflow-job-runner#MY-PROJECT-ID.iam.gserviceaccount.com SA, I suppose I'd need to grant the roles/storage.objectAdmin for it.
The question is
Do I need to grant any permission to the service account agent?
What does the service account agent actually do, what does it has to access any resource?
Several Google Cloud services such as Cloud Dataflow require two sets of permissions.
The program that you write uses a service account. You grant this service account IAM roles to access resources that require authorization that your program requires. For example, reading data from Cloud Storage or issuing queries to BigQuery.
The service agent applies to the service's runtime. For example when you launch a job on Cloud Dataflow, Cloud Dataflow needs to launch VMs to run your program on. Your program is not launching the VMs, the service is. Therefore the service requires its own set of permissions. This is what the service agent is for.
By using two different service accounts, separation of privilege is achieved.
Related
I am trying to deploy my AutoML trained model using Cloud Run, but am having difficulties with IAM permissions. I'm using this as a guide, and keep getting the following error at the build step. I'm new to Cloud Run/GCP but as far as I can see, I have granted the right roles to the right accounts. I appreciate any assistance you can provide as I'm really stumped.
Error message:
Step #3: ERROR: (gcloud.run.deploy) User [REDACTED#cloudbuild.gserviceaccount.com] does not have permission to access namespaces instance [REDACTED] (or it may not exist):
Google Cloud Run Service Agent does not have permission to get access tokens for the service account REDACTED#cloudbuild.gserviceaccount.com.
Please give service-REDACTED#serverless-robot-prod.iam.gserviceaccount.com permission iam.serviceAccounts.getAccessToken on the service account.
Alternatively, if the service account is unspecified or in the same project you are deploying in, ensure that the Service Agent is assigned the Google Cloud Run Service Agent role roles/run.serviceAgent.
Here are the roles assigned to the Cloud Run Service Agent
Default service account has the Cloud Run Admin Role
Here are the accounts to which I've given access to the Default compute service account, the Cloud Build Service Agent and Cloud Run Service Agent are added:
I am trying to create a Cron job programmatically in the CloudScheduler Google Cloud Platform using the following API explorer.
Reference: Cloud Scheduler Documentation
Even though I have given the user Owner permission and verified it in Policy Troubleshooter that it has cloudscheduler.jobs.create, I am still getting the following error.
{
"error": {
"code": 403,
"message": "The principal (user or service account) lacks IAM permission \"cloudscheduler.jobs.create\" for the resource \"projects/cloud-monitoring-saurav/locations/us-central\" (or the resource may not exist).",
"status": "PERMISSION_DENIED"
}
}
I had the same issue. The problem was that the region i specified did not support the cloud scheduler. You seem to have the same issue: "us-central" is not suppported. Try "us-central1"
The error is caused by using a service account that does not have an IAM role that includes the permission cloudscheduler.jobs.create. An example role is roles/cloudscheduler.admin aka Cloud Scheduler Admin. I have the feeling that you have mixed the permission of the service account that you use with Cloud Scheduler (at runtime, when a job triggers something) and the permission of the account currently creating the job (aka your account for example).
You actually need two service accounts for the job to get created. You need one that you set up yourself (can be whatever name you like and doesn't require any special permissions) and you also need the one for the default Cloud Scheduler itself ( which is managed by Google)
Use an existing service account to be used for the call from Cloud Scheduler to your HTTP target or you can create a new service account for this purpose. The service account must belong to the same project as the one in which the Cloud Scheduler jobs are created. This is the client service account. Use this one when specifying the service account to generate the OAuth / OICD tokens.
If your target is part of Google Cloud, like Cloud Functions/Cloud Run update your client service account by granting it the necessary IAM role (Cloud function invoker for cloud functions and Cloud Run Invoker for Cloud Run).The receiving service automatically verifies the generated token. If your target is outside of Google Cloud, the receiving service must manually verify the token.
The other service account is the default Cloud Scheduler service account which must also be present in your project and have the Cloud Scheduler Service Agent role granted to it. This is so it can generate header tokens on behalf of your client service account to authenticate to your target. The Cloud Scheduler service account with this role granted is automatically set up when you enable the Cloud Scheduler API, unless you enabled it prior to March 19, 2019, in which case you must add the role manually.
Note : Do not remove the service-YOUR_PROJECT_NUMBER#gcp-sa-cloudscheduler.iam.gserviceaccount.com service account from your project, or its Cloud Scheduler Service Agent role. Doing so will result in 403 responses to endpoints requiring authentication, even if your job's service account has the appropriate role.
In my case it required the permission: cloudscheduler.jobs.delete.
I found the role the by permission name: https://cloud.google.com/iam/docs/permissions-reference
It was Cloud Scheduler Admin (roles/cloudscheduler.admin)
Then I added it to my service account roles.
We have two projects in our GCP account; one for our Dev environment and one for our Test environment at the moment. Terraform manages most of our infrastructure, so we have minimal clicking around the GUI, or CLI commands.
I have assumed we enabled the Pub/Sub API by deploying to it with Terraform in both of our environments, although we may have needed to do this manually. We noticed that Google created a default Pub/Sub service account for us in our Dev environment, but not in our Test environment. This docs page suggests it should make this service account.
Additionally, we have noticed multiple Pub/Sub subscriptions working, apparently without any service account. We believe that the service account is only needed for this particular Subscription because it is a push to an e-mail server. Therefore, it needs a service account with the 'Service Account Token Creator' role.
We've attempted to redeploy the whole infrastructure and disable/re-enable the Pub/Sub API. Neither seemed to kick GCP into creating the Service Account. Further to this, we attempted to make the default service account manually. Still, GCP constrains the name a user can give a service account themselves, so we're unable to create a service account with the name that the Pub/Sub service would expect.
We wonder if there is some configuration of the project we may have missed or if anyone has seen this previously?
Does it not exist or does you not see it?
I'm pretty sure that it exists but without any role granted on it and you don't see it in the UI. Try to grant a role on this default service account, and it will appear in the IAM page!
I need to execute a dataflow job using a service account , I'm following a very simple and basic example wordcount offered within the same platform itself.
Which is weird is the error I'm getting:
According to this, GCP requires the service account having permissions as Dataflow worker in order to execute my job. The weir part comes over when the error kept on showing up even though I have already set the required permissions:
Can someone explain this strange behavior? thanks so much
To run a Dataflow job, a project must enable billing and the following Google Cloud Platform APIs:
Google Cloud Dataflow API
Compute Engine API (Google Compute Engine)
Google Cloud Logging API
Google Cloud Storage
Google Cloud Storage JSON API
BigQuery API
Google Cloud Pub/Sub
Google Cloud Datastore API
You should also have enough quota in the project for any one of the APIs you are using in the Dataflow job.
I would suggest you to create a fresh service account which its name has not been used before and then granting roles/dataflow.worker to this new fresh service account. Remember, that Cloud IAM propagation takes fewer than 60 seconds, up to 7 minutes, so please have a couple of minutes between an IAM change and Dataflow job creation.
Another possible workaround is to delete the Dataflow worker permission and add it again. The permission remains after the removal of the account, pointing to its old ID. This ID must not be refreshed until explicitly deleting the role.
I encourage you to visit Dataflow IAM roles with role descriptions and permissions documentation.
Enabling the Cloud Run API (dev console→Cloud Run→Enable) creates five service accounts. I want to understand their purpose. I need to know if it's my responsibility to configure them for least privileged access.
The Default compute service account has the Editor role. This is the Cloud Run runtime service account. Its purpose is clear, and I know it's my responsibility to configure it for least privileged access.
The App Engine default service account has the Editor role. This matches the description of the Cloud Functions runtime service account. Its purpose is unclear, given the existence of the Cloud Run runtime service account. I don't know if it's my responsibility to configure it for least privileged access.
The Google Container Registry Service Agent (Editor role) and Google Cloud Run Service Agent (Cloud Run Service Agent role) are both Google-managed service accounts "used to access the APIs of Google Cloud Platform services":
I'd like to see Google-managed service accounts configured for least privileged access. I'd also like to be able to filter the Google-managed service accounts in the IAM section of the GCP console. That said, I know I should ignore them.
The unnamed {project-number}{at}cloudbuild.gserviceaccount.com service account has the Cloud Build Service Account role. This service account "can perform builds" but does not appear in the Cloud Run Building Containers docs. It's used for Continuous Deployment—but can't do that without additional user configuration. It's not a Google-managed service account, but it does not appear in the Service Accounts section of the GCP console like the runtime service accounts. Its purpose is unclear. I don't know if it's my responsibility to configure it for least privileged access.
Cloud Run PM:
Yep, exactly right.
We should probably not create this if you're only using Run (and likely not enable the App Engine APIs, which is what created this). During Alpha, this was the runtime service account, and it's likely that it wasn't cleaned up.
I have a feeling it's stuck as Editor because it accesses Cloud Storage, which is oddly broken for "non Editor access" (I'm still trying to track down the exact issue, but it looks like there's a connection to the legacy Editor role that requires it).
Is already "least privileged" from it's perspective, as it only has the permissions to do the things that Run needs to do in order to set up resources on your behalf.
This is the runtime service account equivalent for Cloud Build, and falls into the same category as 1,2. If you need a build to deploy to Cloud Run, you have to grant this account something like Cloud Run Deployer (plus to the additional step of allowing the build service account to act as your runtime service account, to prevent [or at least acknowledge] privilege escalation).
I too want better filtering of "Google created" and "Google managed" and have been talking with the Cloud IAM team about this.