We have a cloud composer instance set up with the default compute engine service account as the environment's service account (set up by default). We have created a new service account to use instead in the cloud composer instance but we can't find an option to change the environment's service account in the Environment Configuration page. Other fields have "Edit" options at the end of their rows but we can't find the option to edit the service account field.
I found an old Google Group conversation from 2018 that said it's technically possible by creating a new node pool in GKE but this isn't recommended due to the likelihood that it will break the environment.
How do we swap out the old, default compute engine service account to our new service account?
Cloud Composer 2 uses two types of IAM service accounts:
Cloud Composer Service Agent account
Environment's service account
Normally the Cloud Composer default SA can't be changed in the configuration and it will be used to create the Composer instances and machines.
But you can set the environment's service account. This account will be used at runtime with Ariflow scheduler and workers, to allows access to GCP resources from DAGs.
Unfortunately if you created a Composer instance with an environment's service account, you can't change it after.
If you need to change this SA, you have to destroy and recreate the Composer Cluster with this new Service Account.
If you have a tool like Terraform, it's easier to automate this process and recreate the Cluster with your configuration.
In the case of Composer cluster recreation, you can copy all your DAG folders from the previous bucket to the new with gsutil.
IMO it's impossible to change Service Account for running Cloud Composer. I'd setup new environment and just copy DAGs from the old bucket to the new one. And you can test your code with new service account (if you have sufficient permissions).
Related
We have GCP account credentials(username/password). We have installed gcloud CLI on the Amazon Linux EC2 machine. We would like to create a script that would auto-login to the GCP account and do the below things sequentially using gcloud CLI.
Login to the GCP account.
Create Project and specify a meaningful project-id.
Create a service account with a meaningful ID.
Assign the owner role to the service account.
Create and download a new JSON key.
Please help us to achieve this
You should use a Service Account not a User (username|password) for automation. The Service Account should be suitably permissioned so that it can create Projects and Service Accounts.
I was unable to find a source for this (but it used to be that?) Google monitors User Accounts for apparent use of automation (e.g. for bots) and these accounts may be disabled.
I've deleted the default service account and it has been longer than 30 days. I don't know if it applies to all marketplace solutions, but the one that I want to use can't be launched without the compute engine default service account.
What are the IAM permissions I need to set to create a service account that has the same permissions as the compute engine default service account to launch VM from marketplace?
I tried Compute Admin, compute.imageUser and Compute Instance Admin, but to no avail.
In addition to that, why does the marketplace solution require the default service account when it is recommended to disable/remove the default compute engine service account because of the editor role?
What is the compute engine default service account?
By default, the account is automatically granted the project editor role on the project and is listed in the IAM section of Cloud Console. This service account is only deleted when the project is deleted. However, you can change the roles granted to this account, including revoking all access to your project.
Documentation
You can undelete a service account only if it is deleted fewer than 30 days ago.
Instead of that, we can create a new service account and grant an ‘Editor’ role to it, as a default compute engine service account has the same role by default. Refer to Compute Engine default service account for more information.
To set the service account as the compute engine default service account on the project, we can use the following command:
gcloud alpha compute project-info set-default-service-account
But since the command is in the ‘alpha’ launch stage, it is not available for everyone.
I could suggest the following options:
Create a new project.
Request an Alpha feature that allows setting a new service account as the compute engine default service account.
If you have questions regarding an Alpha release or participation in an Alpha program, please reach out to sales. In this case a sales team needs to approve it.
Service account "abcdefc-compute#developer.gserviceaccount.com" does not exist.
I am trying to create a kubernetes cluster but GCP gives me the error above.
I checked for the account name in service account but could not find it, rather I have
'ayushaccount#abcdef.iam.gserviceaccount.com'.
I tried to create another service account with this email "abcdefc-compute#developer.gserviceaccount.com" but it does not allow me to create.
I am new to GCP and I do not know how to solve this problem. All I am looking for to create a kubernetes cluster in GCP.
Looks like you are missing the default service account for your GCP project.
You have two options:
(re)create the default service account
when creating your GKE cluster, under NODE POOLS, go to default-pool->Security and for Service account, select one the one which exists.
If you want to (re)create the default service account, you can disable/enable the Google Compute Engine API via the console or run gcloud services enable compute.googleapis.com from Cloud Shell or from the command line on your workstation.
Say I have this case where
I have to run some test with dataflow
inside this dataflow job I need to access a gcs bucket and save my output there.
I will need to run the dataflow job with my own SA instead of the default SA.
I created a Google Service Account to run my dataflow job. But after I enabled the dataflow API. I end up having 2 SA in front of me.
the service account agent --> 123456789#dataflow.gserviceaccount.com
the dataflow job runner service account --> dataflow-job-runner#MY-PROJECT-ID.iam.gserviceaccount.com
It got me really confused to see what the official document says
Some Google Cloud services have Google-managed service accounts that allow the services to access your resources. These service accounts are sometimes known as service agents.
If I create a dataflow job to run with the dataflow-job-runner#MY-PROJECT-ID.iam.gserviceaccount.com SA, I suppose I'd need to grant the roles/storage.objectAdmin for it.
The question is
Do I need to grant any permission to the service account agent?
What does the service account agent actually do, what does it has to access any resource?
Several Google Cloud services such as Cloud Dataflow require two sets of permissions.
The program that you write uses a service account. You grant this service account IAM roles to access resources that require authorization that your program requires. For example, reading data from Cloud Storage or issuing queries to BigQuery.
The service agent applies to the service's runtime. For example when you launch a job on Cloud Dataflow, Cloud Dataflow needs to launch VMs to run your program on. Your program is not launching the VMs, the service is. Therefore the service requires its own set of permissions. This is what the service agent is for.
By using two different service accounts, separation of privilege is achieved.
I am trying to access some credentials stored in google Secret Manager. To access this its required to have credentials setup in the Cluster machine where the jar is running.
I have SSH into the master instance, and seen there is nothing configured for GOOGLE_APPLICATION_CREDENTIALS.
I am curious to know how to assign GOOGLE_APPLICATION_CREDENTIALS or any other alternative that allows to use GCP APIs that require credentials.
If you are running on Dataproc clusters, default GCE service account should be already configured for you. Assuming your clusters are running outside GCP environment, in that case you want to follow this instruction to manually set up a service account that has editor/owner role for Google Secret Manager, and download the credential key file and point GOOGLE_APPLICATION_CREDENTIALS to it.