We wanted to copy a file from one project's storage to another.
I have credentials for project A and project B in separate service accounts.
The only way we knew how to copy files was to add service key credential permissions to the bucket's access control list.
Is there some other way to run commands across accounts using multiple service keys?
You can use Cloud Storage Transfer Service to accomplish this.
The docs should guide you to setup the permissions for buckets in both projects and do the transfers programmatically or on the console.
You need to get the service account email associated to the Storage Transfer Service by entering your project ID in the Try this API page. You then need to give this service account email the required roles to access the data from the source. Storage Object Viewer should be enough permissions.
At the data destination, you need get the service account email for the second project ID, then give it the Storage Legacy Bucket Writer role.
You can then do the transfer using the snippets in the docs.
Related
I am writing an application where I have to upload media files to GCS. So, I created a storage bucket and also created a service account which is being used by the application to put and get images from the bucket. To access this service account from the application I had to generate a private key as a JSON file.
I am tested my code and it is working fine. Now, I want to push this code to my Github repository but I don't want this service account key to be in Github.
How do I manage to keep this service account key secret, yet all my fellow colleagues should be able to use it.
I am going to put my application on GCP Container Instance and I want it to work there as well.
As I understand, if your application works from inside the GCP and use some custom service account, you might not need any private keys (as json files) at all.
The custom service account, which is used by your application, should get relevant IAM roles/permissions on the correspondent GCS bucket. And that's all you might need to do.
You can assign those IAM roles/permissions either manually (through UI console), or using CLI commands, or as part of your deployment CI/CD pipeline.
I have 1 s3 bucket per customer. Customers are external entities and they dont share data with anyone else. I write to S3 and customer reads from S3. As per this architecture, I can only scale to 1000 buckets as there is a limit to s3 buckets per account. I was hoping to use APs to create 1 AP per customer and put data in one bucket. The customer can then read the files from the bucket using AP.
Bucket000001/prefix01 . -> customeraccount1
Bucket000001/prefix02 . -> customeraccount2
...
S3 access points require you to set policy for a IAM user in access point as well as the bucket level. If I have 1000s of IAM users, do I need to set policy for each of them in the bucket? This would result in one giant policy. there is a max policy size in the bucket, so I may not be able to do that.
Is this the right use case where access points can help?
The recommended approach would be:
Do NOT assign IAM Users to your customers. These types of AWS credentials should only be used by your internal staff and your own applications.
You should provide a web application (or an API) where customers can authenticate against your own user database (or you could use Amazon Cognito to manage authentication).
Once authenticated, the application should grant access either to a web interface to access Amazon S3, or the application should provide temporary credentials for accessing Amazon S3 (more details below).
Do not use one bucket per customer. This is not scalable. Instead, store all customer data in ONE bucket, with each user having their own folder. There is no limit on the amount of data you can store in Amazon S3. This also makes it easier for you to manage and maintain, since it is easier to perform functions across all content rather than having to go into separate buckets. (An exception might be if you wish to segment buckets by customer location (region) or customer type. But do not use one bucket per customer. There is no reason to do this.)
When granting access to Amazon S3, assign permissions at the folder-level to ensure customers only see their own data.
Option 1: Access via Web Application
If your customers access Amazon S3 via a web application, then you can code that application to enforce security at the folder level. For example, when they request a list of files, only display files within their folder.
This security can be managed totally within your own code.
Option 2: Access via Temporary Credentials
If your customers use programmatic access (eg using the AWS CLI or a custom app running on their systems), then:
The customer should authenticate to your application (how this is done will vary depending upon how you are authenticating users)
Once authenticated, the application should generate temporary credentials using the AWS Security Token Service (STS). While generating the credentials, grant access to Amazon S3 but specify the customer's folder in the ARN (eg arn:aws:s3:::storage-bucket/customer1/*) so that they can only access content within their folder.
Return these temporary credentials to the customer. They can then use these credentials to make API calls directly to Amazon S3 (eg from the AWS Command-Line Interface (CLI) or a custom app). They will be limited to their own folder.
This approach is commonly done with mobile applications. The mobile app authenticates against the backend, receives temporary credentials, then uses those credentials to interact directly against S3. Thus, the back-end app is only used for authentication.
Examples on YouTube:
5 Minutes to Amazon Cognito: Federated Identity and Mobile App Demo
Overview Security Token Service STS
AWS: Use the Session Token Service to Securely Upload Files to S3
We have some way to achieve your goal.
use IAM group to grant access to a folder. Create a group, add a user to a group, and assign a role to the group to access the folder.
Another way is to use bucket policy (${aws:username} in Condition) to grant Access to User-Specific Folders. Refer to this link https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/
I'd like to configure our company container registry on GCP to:
Allow staff to push new images with new tags
Not allow existing tags to be replaced
The goal is to avoid using latest tag - or any other mutable tag - and consistently use new, immutable tags for new images.
Is there a set of IAM roles or permissions that can achieve this behaviour?
You don't have to use IAM roles. It should be Service Account.
You need to create a service account, set GCR Editor role for it, download JSON file then send it to your staffs.
A service account JSON key file is a long-lived credential that is scoped to a specific GCP Console project and its resources.
The service account you use to push and pull images must be correctly configured with the required permissions and access scope for interaction with Container Registry.
Service accounts automatically created by GCP, such as the Container Registry service account, are granted the read-write Editor role for the parent project. The Compute Engine default service account is configured with read-only access to storage within the same project. You may wish to grant other service accounts more specific permissions. Pushing and pulling images across projects requires proper configuration of both permissions and access scopes on the service account that interacts with Container Registry.
For more information about the required service account permissions and scopes to push and pull images, refer to the requirements for using Container Registry with Google Cloud Platform.
i am trying to give permissions for google cloud storage buckets with service account json file using django-storages.But the items in buckets are getting accessible only when i gave access to allUsers with Objects View Permission.How can i restrict public access for the bucket.
You can take a look to this link that contains a useful guide about the process required to connect Django to GCS by using Service account JSON files; in this way, you can implement this authentication method to access to your buckets instead of making your data public. Adittionally, please keep in mind it is required to assign the Cloud Storage IAM Roles to the service account, by using the IAM console or by creating ACLs, in order to grant the access permissions.
Finally, once you have your Service account key file ready, you can authenticate your application by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable with the path of the json file or passing the path directly into your code, as explained in the GCS official documentation.
I'm attempting to shuttle data between buckets in different projects using Google Cloud's Storage Transfer Service based on an event that's being emitted. This event points to a subdirectory of a bucket belonging to project A that I want to copy to a destination bucket in project B, and so naturally STS looked like a perfect fit.
I've successfully crafted the request, including the prefix expressions, and I believe that I'm just dealing with a permissions error now because when I use my code to copy between buckets in the same project it creates and starts the transfer as expected. When I use my service account to being the transfer across buckets, however, I get the following error:
Error creating the transfer job:
Failed to obtain the location of the destination Google Cloud Storage (GCS) bucket due to insufficient permissions.
Please verify that the necessary permissions have been granted.
So from here I've looked into the permissions of my service account. I've manually added that service account as a project editor (and then owner) of the second project, and I've added the account to the ACLs for the destination bucket I'm trying to transfer into.
Are there any other avenues that I'm missing to look down? I haven't had much success googling around for others that have hit my situation.
One problem that you may be running into is you may be granting access to the wrong service account. One service account is created for each project. The service account can be found using the get googleServiceAccounts command documented here. It should look like
storage-transfer-{UID}#partnercontent.gserviceaccount.com
The UID will be different for each project. Try verifying that the service account with the UID specific to the projectId you specify in the transferJobs create request has READ/LIST access in source bucket and READ/WRITE/LIST access in sink bucket.
If the number of transfers you need to set up is relatively small, you might also try using the console, which is a GUI designed to simplify the process of seting up a transfer. Instructions for using the console can be found here.