I am trying to figure out a way to allow a GCP user to list buckets but only those where the user has permissions (through ACL). The reason is because it can be overwhelming the number of buckets and the user experience would not be the best. Any ideas ?
Thanks!
I am trying to figure out a way to allow a GCP user to list buckets
but only those where the user has permissions (through ACL).
You cannot accomplish your goal using either Bucket ACLs or IAM permissions.
To list Google Cloud Storage buckets, you need the IAM permission storage.buckets.list.
This permission grants the IAM member account permission to list all buckets in the project. You cannot restrict this permission further to list only specific bucket names. This permission does not allow listing the objects in a bucket.
For a future design decision, you can use different projects and organize your buckets under projects. This will limit access to only IAM members of that project.
When you create a bucket you permanently define its name, location and the project it is part of. These characteristics cannot be changed later.
If you're using the CLI, you can write a script that gets the permissions for each listed bucket, and only displays it if the user account is in the permission list:
for bucket in $(gsutil ls); do
if gsutil acl get $bucket|grep -q $(gcloud config get-value account) ; then
echo $bucket;
fi;
done
Note that inherited permissions (e.g. at the project level) will not appear with this script.
This can't be accomplished with the console, but if you need a web interface listing only certain buckets, then you can build it yourself by calling the API and doing the same thing that the CLI script does.
Related
I have a centralized CloudTrail bucket which contains the CloudTrail logs of multiple accounts. Is it possible to write a bucket policy which allows that account 123456789112 can only download logs from Awslogs/123456789112 and that account 456789012345can only download logs from Awslogs/456789012345etc ? I don't want to hardcode this for each account since I have a lot of accounts. Is there a way to do this?
AWS IAM policies (and bucket policies) support a few policy variables that you can use as dynamic values such as aws:SourceIp, however account ID is not one of them. There is a aws:userid variable but it's the account ID only for the root user, for other principals like IAM user/role it is the user/role name. Technically if you used the AWS root user to access this bucket, you could use the userid variable in the Resource element to achieve what you want but it is strongly recommended not to use the root user for such everyday tasks (AWS recommendation).
There are also policy condition keys like aws:PrincipalAccount but without a relevant policy variable these cannot be used to dynamically compare the requesting account ID with the resource. There are no other IAM feature that could be used to achieve this.
I don't know your exact environment but a few things to consider:
I'd recommend to explicitly list the allowed principal ARNs anyway because even if you have many accounts, you should allow only specific IAM users/roles to read the bucket to follow the least privilege principle. Granting access based on account ID would allow all users/roles in that account to read these files and not just specific services. (unless this is the objective)
since this is a cross-account access (principal in account A wants to read from the bucket located in account B), you will need to allow this access on both sides, both in the requester's IAM policy and the target bucket's policy. Just a heads up. More info on AWS.
I would consider using Terraform to simplify the management of these resources
Hope this helps, let me know if you have more questions!
Usually there is Compute Engine default service account that is created automatically by GCP, this account is used for example by VM agents to access different resources across GCP and by default has role/editor permissions.
Suppose I want to create GCS bucket that can only be accessed by this default service account and no one else. I've looked into ACLs and tried to add an ACL to the bucket with this default service account email but it didn't really work.
I realized that I can still access bucket and objects in this bucket from other accounts that have for example storage bucket read and storage object read permissions and I'm not sure what I did wrong (maybe some default ACLs are present?).
My questions are:
Is it possible to limit access to just that default account? In that case who will not be able to access it?
What would be the best way to do it? (would appreciate a lot an example using Storage API)
There are still roles such as role/StorageAdmin, and actually no matter what ACLs will be put on the bucket I could still access it if I had this role (or higher role such as owner) right?
Thanks!
I recommend you not to use ACL (and Google also). It's better to switch the bucket in uniform IAM policy.
There are 2 bad side of ACL:
New created files aren't ACL and you need to set it everytime that you create a ne file
It's difficult to know who has and who hasn't access with ACL. IAM service is better for auditing.
When you switch to Uniform IAM access, Owner, Viewer, and Editor role no longer have access to buckets (the role/storage.admin isn't included in this primitive role). It could solve in one click all the unwanted access. Else, as John said, remove all the IAM permission on the bucket and the project that have access to the bucket except your service account.
You can control access to buckets and objects using Cloud IAM and ACLs.
For example grant the service account WRITE (R: READ,W: WRITE,O: OWNER) access to the bucket using ACLs:
gsutil acl ch -u service-account#project.iam.gserviceaccount.com:W gs://my-bucket
To remove access of service account from the bucket:
gsutil acl ch -d service-account#project.iam.gserviceaccount.com gs://my-bucket
If There are roles such as role/StorageAdmin in the IAM identities (project level), they will have access to all the GCS resources of the project. You might have to change the permission to avoid them having access.
I'm trying to do gsutil ls however that results in:
ubuntu#ip:~$ gsutil ls
AccessDeniedException: 403 xxxxxxxxxxxx#xxxxxxxxxx.iam.gserviceaccount.com does not have storage.buckets.list access to project xxxxxxxxxxxxxxx.
Can I give this permission with only read / viewer access IAM roles?
You certainly can. At a minimum, you can always create a custom role with exactly the permissions you want. You do this by clicking the Create Role button at the top of the roles tab. Then, once it is created, apply that role to your service account on the IAM page, like any other role.
Alternatively, you can use the same roles tab in the cloud console to search for that permission explicitly to see which roles contain it and see if any would work for you.
In this case, I don't see an obvious default one that is limited, however. That said,you could look at Storage Legacy Bucket Reader (roles/storage.legacyBucketReader) as a starting point for a custom role in this case -- if you select this role on the roles tab, you can 'Create Role from Selection' to use it as a starting point).
The command gsutil ls lists the buckets in your project.
To list buckets you need the permission storage.buckets.list.
To list the objects in a bucket you need the permission storage.objects.list.
Neither of those permissions allows you to read an object. To read an object you need the permission storage.objects.get.
To only read an object, you do not need the list permissions. However, since you are using the gsutil command, you do.
There are several predefined roles that you can attach to your service account to grant the necessary permissions for gsutil.
Recommended:
roles/storage.objectViewer
Or the following two roles:
roles/storage.legacyObjectReader
roles/storage.legacyBucketReader
If you ONLY want to assign a role to read an object but not list them:
roles/storage.legacyObjectReader
Read only permissions for a GCP GCS bucket as of January 2022:
storage.buckets.get
storage.buckets.list
storage.objects.get
storage.objects.list
The "Viewer" role for the project is probably what you are looking for to view all the buckets in a project.
Otherwise, by giving only the "storage object viewer" role, you can only view the content INSIDE the bucket, by going to the correct URL of the bucket.
Need to create S3 bucket and few objects which are intranet facing. But users may not be having AWS access.
How to restrict S3 access to users of my companies Active Directory.
You need to do two things,
Connective Active Directory to AWS IAM
Create Roles for users to whichever the way you want it to S3
AD to IAM:
https://aws.amazon.com/blogs/security/how-to-connect-your-on-premises-active-directory-to-aws-using-ad-connector/
S3 IAM Role based Access:
https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/
Hope it helps.
I have tried to access files in a bucket and I keep getting access denied on the files. I can see them in the GCS console but can access them through that and cannot access them through gsutil either running the command below.
gsutil cp gs://my-bucket/folder-a/folder-b/mypdf.pdf files/
But all this returns is AccessDeniedException: 403 Forbidden
I can list all the files and such but not actually access them. I've tried adding my user to the acl but that still had no effect. All the files were uploaded from a VM through a fuse mount which worked perfectly and just lost all access.
I've checked these posts but none seem to have a solution thats helped me
Can't access resource as OWNER despite the fact I'm the owner
gsutil copy returning "AccessDeniedException: 403 Insufficient Permission" from GCE
gsutil cors set command returns 403 AccessDeniedException
Although, quite an old question. But I had a similar issue recently. After trying many options suggested here without success, I carefully re-examined my script and discovered I was getting the error as a result of a mistake in my bucket address gs://my-bucket. I fixed it and it worked perfectly!
This is quite possible. Owning a bucket grants FULL_CONTROL permission to that bucket, which includes the ability to list objects within that bucket. However, bucket permissions do not automatically imply any sort of object permissions, which means that if some other account is uploading objects and sets ACLs to be something like "private," the owner of the bucket won't have access to it (although the bucket owner can delete the object, even if they can't read it, as deleting objects is a bucket permission).
I'm not familiar with the default FUSE settings, but if I had to guess, you're using your project's system account to upload the objects, and they're set to private. That's fine. The easiest way to test that would be to run gsutil from a GCE host, where the default credentials will be the system account. If that works, you could use gsutil to switch the ACLs to something more permissive, like "project-private."
The command to do that would be:
gsutil acl set -R project-private gs://muBucketName/
tl;dr The Owner (basic) role has only a subset of the GCS permissions present in the Storage Admin (predefined) role—notably, Owners cannot access bucket metadata, list/read objects, etc. You would need to grant the Storage Admin (or another, less privileged) role to provide the needed permissions.
NOTE: This explanation applies to GCS buckets using uniform bucket-level access.
In my case, I had enabled uniform bucket-level access on an existing bucket, and found I could no longer list objects, despite being an Owner of its GCP project.
This seemed to contradict how GCP IAM permissions are inherited— organization → folder → project → resource / GCS bucket—since I expected to have Owner access at the bucket level as well.
But as it turns out, the Owner permissions were being inherited as expected, rather, they were insufficient for listing GCS objects.
The Storage Admin role has the following permissions which are not present in the Owner role: [1]
storage.buckets.get
storage.buckets.getIamPolicy
storage.buckets.setIamPolicy
storage.buckets.update
storage.multipartUploads.abort
storage.multipartUploads.create
storage.multipartUploads.list
storage.multipartUploads.listParts
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.getIamPolicy
storage.objects.list
storage.objects.setIamPolicy
storage.objects.update
This explained the seemingly strange behavior. And indeed, after granting the Storage Admin role (whereby my user was both Owner and Storage Admin), I was able to access the GCS bucket.
Footnotes
Though the documentation page Understanding roles omits the list of permissions for Owner (and other basic roles), it's possible to see this information in the GCP console:
Go to "IAM & Admin"
Go to "Roles"
Filter for "Owner"
Go to "Owner"
(See list of permissions)