We have an amazon s3 account and a number of important documents are saved there in the bucket.
Is there a way we can secure those resources so that they are not deleted from the S3 account by any team member other than the primary account holder?
Also, how can we back up all the S3 resources in a google drive?
Thanks in advance.
The highest level of securing object from being delete is by using
MFA delete which can only be enabled by the root user.
The MFA delete also will not allow for disabling versioning in your bucket.
Regarding Google drive, I'm not aware of any build in AWS tool for that. I think you would have to look at some third party tools, or develop your own.
For backing up all S3 resources to Google Drive (or vice versa) - rclone running on a schedule is probably one of the simplest solutions that can achieve this nicely for you
Confidential documents
Some organizations keep confidential documents in a separate AWS Account, so that normal users do not have access to the documents.
Cross-account read permissions can be granted to appropriate users (eg for confidential HR documents).
Critical documents
If you just wish to "backup" to avoid accidental deletion, one option is to use Amazon S3 Same Region Replication to copy documents to another bucket. The destination bucket can be in a different account, so normal users do not have the ability to delete the copies.
In both cases, credentials to the secondary accounts would only be given sparingly, and they can also be protected by using Multi-Factor Authentication.
Related
Currently, we use AWS IAM User permanent credentials to transfer customers' data from our company's internal AWS S3 buckets to customers' Google BigQuery tables following BigQuery Data Transfer Service documentation.
Using permanent credentials possesses security risks related to the data stored in AWS S3.
We would like to use AWS IAM Role temporary credentials, which require the support of a session token on the BiqQuery side to get authorized on the AWS side.
Is there a way that the BigQuery Data Transfer Servce can use AWS IAM roles or temporary credentials to authorise against AWS and transfer data?
We considered Omni framework (https://cloud.google.com/bigquery/docs/omni-aws-cross-cloud-transfer) to transfer data from S3 to BQ, however, we faced several concerns/limitations:
Omni framework targets data analysis use-case rather than data transfer from external services. This concerns us that the design of Omni framework may have drawbacks in relation to data transfer at high scale
Omni framework currently supports only AWS-US-EAST-1 region (we require support at least in AWS-US-WEST-2 and AWS-EU-CENTRAL-1 and corresponding Google regions). This is not backward compatible with current customers' setup to transfer data from internal S3 to customers' BQ.
Our current customers will need to signup for Omni service to properly migrate from the current transfer solution we use
We considered a workaround with exporting data from S3 through staging in GCS (i.e. S3 -> GCS -> BQ), but this will also require a lot of effort from both customers and our company's sides to migrate to the new solution.
Is there a way that the BigQuery Data Transfer Servce can use AWS IAM roles or temporary credentials to authorise against AWS and transfer data?
No unfortunately.
The official Google BigQuery Data Transfer Service only mentions AWS access keys all throughout the documentation:
The access key ID and secret access key are used to access the Amazon S3 data on your behalf. As a best practice, create a unique access key ID and secret access key specifically for Amazon S3 transfers to give minimal access to the BigQuery Data Transfer Service. For information on managing your access keys, see the AWS general reference documentation.
The irony of the Google documentation is that while it refers to best practices and links to the official AWS docs, it actually doesn't endorse best practices and ignores what AWS mention:
We recommend that you use temporary access keys over long term access keys, as mentioned in the previous section.
Important
Unless there is no other option, we strongly recommend that you don't create long-term access keys for your (root) user. If a malicious user gains access to your (root) user access keys, they can completely take over your account.
You have a few options:
hook into both sides manually (i.e. link up various SDKs and/or APIs)
find an alternative BigQuery-compatible service, which does as such
accept the risk of long-term access keys.
In conclusion, Google is at fault here of not following security best practices and you - as a consumer - will have to bear the risk.
Suppose you have to share data with a third party over the internet and the data is stored in AWS. What would be the most secure and easy way to do this?
Since sending mail is not very secure, i thought of the solution of creating a S3 bucket and run a SFTP server (with AWS Family) on it. Is there a better solution in AWS to achieve this?
This depends on how you want to "share data" and where that data resides.
Let's say you have an object in Amazon S3 that you would like to make available. There are several options for sharing access:
You could create an Amazon S3 pre-signed URL, which provides time-limited access to a private object. This is similar to storing something in DropBox and using the "Get Link" command to obtain a special URL that provides access to the object.
If the other people have their own AWS Account, you could share a specific bucket or an object with them. This has the benefit that you could put objects in a bucket and they can retrieve any of them whenever they wish.
You could write a web application that requires users to authenticate and then gives them the ability to access objects in Amazon S3. This would be similar to a photo-sharing website, where people login and can access/share photos. You would be responsible for writing this application and managing the authentication.
Update
Based on the information you provided (S3, few users, automated), the easiest method would probably be to have the other users sign-up to AWS or provide them with IAM access credentials from your own AWS Account (not recommended if you have large numbers of such users).
You can grant permission for them to access your data and they could use the AWS Command-Line Interface (CLI) to access/download the data. This can be automated with the aws s3 cp and aws s3 sync commands.
I have 1 s3 bucket per customer. Customers are external entities and they dont share data with anyone else. I write to S3 and customer reads from S3. As per this architecture, I can only scale to 1000 buckets as there is a limit to s3 buckets per account. I was hoping to use APs to create 1 AP per customer and put data in one bucket. The customer can then read the files from the bucket using AP.
Bucket000001/prefix01 . -> customeraccount1
Bucket000001/prefix02 . -> customeraccount2
...
S3 access points require you to set policy for a IAM user in access point as well as the bucket level. If I have 1000s of IAM users, do I need to set policy for each of them in the bucket? This would result in one giant policy. there is a max policy size in the bucket, so I may not be able to do that.
Is this the right use case where access points can help?
The recommended approach would be:
Do NOT assign IAM Users to your customers. These types of AWS credentials should only be used by your internal staff and your own applications.
You should provide a web application (or an API) where customers can authenticate against your own user database (or you could use Amazon Cognito to manage authentication).
Once authenticated, the application should grant access either to a web interface to access Amazon S3, or the application should provide temporary credentials for accessing Amazon S3 (more details below).
Do not use one bucket per customer. This is not scalable. Instead, store all customer data in ONE bucket, with each user having their own folder. There is no limit on the amount of data you can store in Amazon S3. This also makes it easier for you to manage and maintain, since it is easier to perform functions across all content rather than having to go into separate buckets. (An exception might be if you wish to segment buckets by customer location (region) or customer type. But do not use one bucket per customer. There is no reason to do this.)
When granting access to Amazon S3, assign permissions at the folder-level to ensure customers only see their own data.
Option 1: Access via Web Application
If your customers access Amazon S3 via a web application, then you can code that application to enforce security at the folder level. For example, when they request a list of files, only display files within their folder.
This security can be managed totally within your own code.
Option 2: Access via Temporary Credentials
If your customers use programmatic access (eg using the AWS CLI or a custom app running on their systems), then:
The customer should authenticate to your application (how this is done will vary depending upon how you are authenticating users)
Once authenticated, the application should generate temporary credentials using the AWS Security Token Service (STS). While generating the credentials, grant access to Amazon S3 but specify the customer's folder in the ARN (eg arn:aws:s3:::storage-bucket/customer1/*) so that they can only access content within their folder.
Return these temporary credentials to the customer. They can then use these credentials to make API calls directly to Amazon S3 (eg from the AWS Command-Line Interface (CLI) or a custom app). They will be limited to their own folder.
This approach is commonly done with mobile applications. The mobile app authenticates against the backend, receives temporary credentials, then uses those credentials to interact directly against S3. Thus, the back-end app is only used for authentication.
Examples on YouTube:
5 Minutes to Amazon Cognito: Federated Identity and Mobile App Demo
Overview Security Token Service STS
AWS: Use the Session Token Service to Securely Upload Files to S3
We have some way to achieve your goal.
use IAM group to grant access to a folder. Create a group, add a user to a group, and assign a role to the group to access the folder.
Another way is to use bucket policy (${aws:username} in Condition) to grant Access to User-Specific Folders. Refer to this link https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/
I am having a hard time understanding Azure docs and terminologies. The problem is this. My customer has an azure bucket and we need to read/write to this bucket. They won't be sharing their storage account credentials either.
This can be achieved in AWS by following this:
https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/
I have just created an IAM user and asked my customers to allow the necessary permissions in the bucket policy. Thus, with one IAM user and one set of credentials, I can write to multiple buckets belonging to multiple AWS accounts.
Is something like above also possible in Azure?
they can create a Shared access signature while they can control what kind of access you need to have and also when to expire.
I want to allow a certain AWS account read permissions to one of my S3 objects (file) via a URL.
Is it possible to grant permissions to other AWS account using his AWS account ID (The user's AWS account Id is the only information I have about his account)?
Yes, you can do this. You want to use the Principal element.
You can find examples here.
(I know links are generally frowned upon, but AWS technologies change at such a rapid pace that actual examples may be obsolete within days or months)