I want to copy an object from a Google Compute Engine instance to a Google Storage Bucket using gsutil cp. Both belong to the same owner (me) and to the same project. I want to automate the whole machine so authenticating manually is not what I want.
I have activated the necessary permissions to use a service account on a Compute instance (details below) but when I try to gsutil cp a file to the bucket, I get an AccessDeniedException.
The error message complains about missing storage.object.create or storage.object.list permissions depending on if my bucket target path ends in a folder (gs://<bucket>/test/) or file (gs://<bucket>/test.txt).
What I did to get permissions (I have already tried a lot, including creating redundant custom roles which I also assigned to the service account):
Start the instance:
gcloud instances create <instance> [...] \
--service--account <name>#<project>.iam.gserviceaccount.com \
--scopes cloud-platform,storage-full
Give the service account permissions on creation.
Give the service account permissions afterwards as well (just to be safe):
gcloud projects add-iam-policy-binding <project> \
--member serviceAccount:<name>#<project>.iam.gserviceaccount.com \
--role roles/storage.objectAdmin
Edit Storage bucket permissions for the service account:
gsutil iam ch \
serviceAccount:<name>#<project>.iam.gserviceaccount.com:roles/storage.objectAdmin \
gs://<bucket>
Edit Storage bucket access control list (owner permission):
gsutil acl ch -u <name>#<project>.iam.gserviceaccount.com:O gs://<bucket>
At some point enabled bucket-level IAM policies instead of per-object policies (just to be safe).
On the instance, use
gcloud auth activate-service-account --key-file <key>.json to authenticate the account.
However, no matter, what I do, the error does not change and I am not able to write to the bucket. I can, however, read files from the bucket. I also get the same error on a local machine using the service account, so the problem is not related to instances.
As well as ensuring that the service account that you're using has appropriate permissions, you also need to ensure that the instance you're using has the appropriate scopes to access Google Cloud Storage, which they usually don't be default. You can either set the scopes to Allow full access to all Cloud APIs or set them individually if you'd prefer. You can find instructions on how to do so here.
I created a new service account and executed the exact same steps... Now it works.
Related
I have saved BI tool setup files in a folder on google cloud storage . we have windows VM created on GCP where i want to move this folder containing all the setup files ( around 60 gb) from google cloud storage by using gsutil command but it is throwing error
I am using below command
gsutil cp -r gs://bucket-name/folder-name C:\Users\user-name\
getting error as AccessDeniedException: 403 sa-d-edw-ce-cognosserver#prj-edw-d-edw-7f58.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.
can someone please help me to understand where I am making mistake ?
There are two likely problems:
The CLI is using an identity that does not possess the required permissions.
The Compute Engine instance has restricted the permissions via scopes or has disabled scopes preventing all API access.
To modify IAM permissions/roles requires permissions as well on your account. Otherwise, you will need to contact an administrator for the ORG or project.
The CLI gsutil is using an identity (either a user or service account). That identity does not have an IAM role attached that contains the IAM permission storage.objects.list.
There are a number of IAM roles that have that permission. If you only need to list and read Cloud Storage objects, use the role Storage Legacy Bucket Reader aka roles/storage.legacyBucketReader. The following link provides details on the available roles:
IAM roles for Cloud Storage
Your Google Compute Engine Windows VM instance has a service account attached to it. The Google Cloud CLI tools can use that service account or the credentials from gcloud auth login. There are a few more methods.
To complicate this a bit more, each Compute Engine has scopes assigned which limit a service accounts permissions. The default scopes allow Cloud Storage object read. In the Google Cloud Console GUI lookup or modify the assigned scopes. The following command will output details on the VM which will include the key serviceAccounts.scope.
gcloud compute instances describe INSTANCE_NAME --project PROJECT_ID --zone ZONE
Figure out which identity your VM is using
gcloud auth list
Add an IAM role to that identity
Windows command syntax.
For a service account:
gcloud projects add-iam-policy-binding PROJECT_ID ^
--member="serviceAccount:REPLACE_WITH_SERVICE_ACCOUNT_EMAIL_ADDRESS" ^
--role="roles/storage.legacyBucketReader"
For a user account:
gcloud projects add-iam-policy-binding PROJECT_ID ^
--member="user:REPLACE_WITH_USER_EMAIL_ADDRESS" ^
--role="roles/storage.legacyBucketReader"
I'm trying to build a flex-template image using a service account:
gcloud dataflow flex-template build "$TEMPLATE_PATH" \
--image-gcr-path "$TEMPLATE_IMAGE" \
--sdk-language "JAVA" \
--flex-template-base-image JAVA11 \
--metadata-file "metadata.json" \
--jar "target/XXX.jar" \
--env FLEX_TEMPLATE_JAVA_MAIN_CLASS="XXX"
The service account has the following roles:
"roles/appengine.appAdmin",
"roles/bigquery.admin",
"roles/cloudfunctions.admin",
"roles/cloudtasks.admin",
"roles/compute.viewer",
"roles/container.admin",
"roles/dataproc.admin",
"roles/iam.securityAdmin",
"roles/iam.serviceAccountAdmin",
"roles/iam.serviceAccountUser",
"roles/iam.roleAdmin",
"roles/resourcemanager.projectIamAdmin",
"roles/pubsub.admin",
"roles/serviceusage.serviceUsageAdmin",
"roles/servicemanagement.admin",
"roles/spanner.admin",
"roles/storage.admin",
"roles/storage.objectAdmin",
"roles/firebase.admin",
"roles/cloudconfig.admin",
"roles/vpcaccess.admin",
"roles/compute.instanceAdmin.v1",
"roles/dataflow.admin",
"roles/dataflow.serviceAgent"
However, even with the dataflow.admin and dataflow.serviceAgent roles, my service account is still unable to perform this task.
The documentation https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates advises to grant the roles/owner role to the service account, but I'm hesitant to do that as this is meant to be part of a CI/CD pipeline and giving a service account an owner role doesn't really make sense to me unless I'm completely wrong.
Is there any way to circumvent this issue without granting the owner role to the service account?
I just ran into the exact same issue and spent a few hours figuring this out. We use terraform service account as well. As you mentioned there are 2 main issues: service account access and the build logs access.
By default, cloud build will use a default service account of form [project_number]#cloudbuild.gserviceaccount.com so you need to grant permissions to this service account to write to your gcs bucket backing the gcr container registry. I granted roles/storage.admin to my service account.
Like you mentioned, by default again, cloud build saves the logs at gs://[project_number].cloudbuild-logs.googleusercontent.com. This seems to be a hidden bucket in the project, at least I could not see it. In adddition, can't configure google_storage_bucket_iam_member for it, instead the recommendation as per this doc is to give roles/viewer at the project level to the service account running the gcloud dataflow ... command.
I was able to run the command successfully after the above changes.
I know that I can do it via the UI (Cloud Console), and that I can also assign a role. Although, how do I grant a single permission easily?
For example, I was pushing an image to Google Container Registry with a newly created service account, and I got an error saying that this service account doesn't have the storage.buckets.get permission. What is the easiest way to grant this specific permission using the CLI?
You can't directly grant a permission to a service account, that's simply not how Google Cloud IAM works. Only roles are assigned to service accounts, users or groups which in turn usually contain a set of permissions.
If you want a role to only contain a single permission, or only permissions you're interested in, you can look into creating a custom role, which allows you to specify which permission(s) you want to give to a role of your definition in order to restrict the access on a more granular level. And then, assign that custom role to the service account:
Using the gcloud CLI you can create a custom role with
gcloud iam roles create, i.e:
gcloud iam roles create bucketViewer \
--project example-project-id-1 \
--title "Bucket viewer" \
--description "This role has only the storage.buckets.get permission" \
--permissions storage.buckets.get
This will create a custom role with the ID bucketViewer, for the
project ID example-project-id-1, containing only the permission
storage.buckets.get. Replace these values as desired and
accordingly.
Once done, you can assign this custom role also with a single gcloud
command by using gcloud projects add-iam-policy-binding:
gcloud projects add-iam-policy-binding example-project-id-1 \
--member='serviceAccount:test-proj1#example.domain.com' \
--role='projects/example-project-id-1/roles/bucketViewer'
Replace example-project-id-1 with your project ID, and
test-proj1#example.domain.com with the actual name of the service
account you want to assign the role to.
You most likely don't want to assign single permission. It usually requires more permissions to achieve what you want.
Those permissions are organized into roles - you either pick existing one, or create own, like described in this answer https://stackoverflow.com/a/59757152.
But typically there are some existing predefined roles. You need to find them in Google Cloud documentation - e.g. for container registry https://cloud.google.com/container-registry/docs/access-control - your choice could be Storage Object Admin (roles/storage.objectAdmin).
Those roles are actually Cloud Storage roles which are described in https://cloud.google.com/storage/docs/access-control/iam-roles.
I have an storage bucket that I created on GCP. I created the bucket following the instructions described here (https://cloud.google.com/storage/docs/creating-buckets). Additionally, I created it using uniform bucket-level access control.
However, I want the objects in the bucket to be accessible by instances running under a certain service account. Although, I do not see how to do that. In the permissions settings, I do not see how I can specify a service account for read-write access.
To create a service account, run the following command in Cloud Shell:
gcloud iam service-accounts create storage-sa --display-name "storage service account"
You can grant roles to a service account so that the service account can perform specific actions on the resources in your GCP project. For example, you might grant the storage.admin role to a service account so that it has control over objects and buckets in Google Cloud Storage.
gcloud projects add-iam-policy-binding <Your Project ID> --member <Service Account ID> --role <Role You want to Grant>
Once role is granted you can select this service account while creating the instance.
Alternatively, to do this via Google Cloud Console see Creating and enabling service accounts for instances
Once you have created your service account, you can then change/set the access control list (ACL) permissions on your bucket or objects using ths gsutil command.
Specifically:
gsutil acl set [-f] [-r] [-a] file-or-canned_acl_name url...
gsutil acl get url
gsutil acl ch [-f] [-r] <grant>... url...
where each <grant> is one of the following forms:
-u <id|email>:<perm>
-g <id|email|domain|All|AllAuth>:<perm>
-p <viewers|editors|owners>-<project number>:<perm>
-d <id|email|domain|All|AllAuth|<viewers|editors|owners>-<project number>>:<perm>
Please review the following article for more depth and description:
acl - Get, set, or change bucket and/or object ACLs
You can also set/change acls through the Cloud Console web interface and through GCS API.
You have to create a service account Creating a new service account.
Set up a new instance to run as a service account Set instance.
In the Google Cloud Console go to Storage/bucket/right_corner dots/Edit bucket permissions
Add Member/servive account/
Role/Storage Admin
I created a service user:
gcloud iam service-accounts create test01 --display-name "test01"
And I gave him full access to Cloud Storage:
gcloud projects add-iam-policy-binding project-name \
--member serviceAccount:test01#project-name.iam.gserviceaccount.com \
--role roles/storage.admin
This code works:
from google.cloud import storage
client = storage.Client()
buckets = list(client.list_buckets())
print(buckets)
bucket = client.get_bucket('bucket-name')
print list(bucket.list_blobs())
But my project has multiple buckets for different environments, and for security reasons I want to add access for only one bucket per user.
In the documentation I found this text:
When applied to an individual bucket, control applies only to the specified bucket and objects within the bucket.
How to apply roles/storage.admin to an individual bucket?
Update:
I tried ACL, and there is a problem: I add access to user:
gsutil iam ch \
serviceAccount:test01#project-name.iam.gserviceaccount.com:legacyBucketOwner \
gs://bucket-name
User can list all files, add files, create files, view his own files.
But user can't view files of other users.
Update 2:
I updated default ACL:
gsutil defacl ch -u \
test01#project-name.iam.gserviceaccount.com:OWNER gs://bucket-name
I waited a lot of time, created another file by another user, and it's still inaccessible by test01.
Solution:
I made it from scratch, and it works:
gsutil mb -p example-logs -c regional -l EUROPE-WEST2 gs://example-dev
gcloud iam service-accounts create test-dev --display-name "test-dev"
gcloud iam service-accounts create test-second --display-name "test-second"
# download 2 json keys from https://console.cloud.google.com/iam-admin/serviceaccounts
gsutil iam ch serviceAccount:test-dev#example-logs.iam.gserviceaccount.com:legacyBucketOwner gs://example-dev
gsutil iam ch serviceAccount:test-second#example-logs.iam.gserviceaccount.com:legacyBucketOwner gs://example-dev
gsutil defacl ch -u test-dev#example-logs.iam.gserviceaccount.com:OWNER gs://example-dev
In order for a user to work with a bucket, that user must be granted authority to work with that bucket. This is achieved with permissions. Permissions can be bundled into roles and we can give a user a role which means that the user will have that role.
For example, a user can be given the role "Storage Admin" and will then be able to perform work against all buckets in your project.
If that is too much, then you can choose NOT to give the user "Storage Admin" and then it will not be allowed to access any bucket. Obviously that is too restrictive. What you can then do is pick the individual buckets that you wish the user to access and, for each of those buckets, change the permissions of THOSE buckets. Within the permissions of a bucket you can name users and roles. For just THAT bucket, the named user will have the named role.
For more details see Creating and Managing Access Control Lists (ACLs).
You can apply storageAdmin to individual bucket like below:
gsutil iam ch serviceAccount:service_account_email:admin gs://bucket_name