Simplest way to setup bucket-per-client in Amazon S3 - amazon-web-services

Question

I want some clients to be able to programmatically upload files for me. Amazon S3 obviously sounds great in terms of availability and durability. But setup seems like such a complicated (and hence, error-prone) step for this: AFAIK I can't avoid creating users, groups, roles, policies...
Is there something simpler that would allow me to create a bucket with a token, so that I can simply give that token to the client w/o wasting time and w/o the risk of clicking something wrong that could lead to problems or security holes?
P.S. I won't need to do that for thousands of client, just a few.

Answer 1

You can make a automatized system for create many s3 buckets you need, you can use Terraform for interact with the API of your cloud provider and make this tasks programatically, so with this you can create many things around this (maybe a frontend/backend for create this buckets from your browser and of course you can make a function for send through a email all info around the access for the bucket by example).
In this repo you can see an example: https://github.com/terraform-aws-modules/terraform-aws-s3-bucket/tree/master/examples/s3-replication

Answer 2

Your first consideration should be how these clients interact with Amazon S3. This will then impact how you allow them to access Amazon S3.
There are several options:
Option 1: Provide IAM User credentials
Normally, IAM credentials should only be given to staff in your own company. However, if you have a small number of well-known clients, you could create an IAM User for each of them.
You can then assign permissions that allow them to access a specific bucket, or a path within a shared bucket, and they can use the AWS CLI to upload/download files, or programmatically via an AWS SDK.
You would need to give them an Access Key + Secret Key to access their S3 storage.
Rather than using separate buckets, you could grant access to a path within a shared bucket with a relatively simple Bucket Policy that grants access to a path based on their IAM Username. See:IAM policy elements: Variables and tags - AWS Identity and Access Management
Option 2: Provide temporary credentials
If they are programmatically accessing AWS, then the clients could:
Programmatically authenticate against your back-end application
The back-end application uses the AWS Security Token Service (STS) to generate temporary credentials and returns them to the client
The client then uses those credentials in the same way as Option 1
The difference with this option is that the clients authenticate to your own back-end rather than using IAM User credentials.
Option 3: Pre-signed URLs
Instead of providing credentials to your clients, your back-end app can generate Amazon S3 pre-signed URLs, which are time-limited URL that provides temporary access to upload/download private objects in Amazon S3.
This allows the back-end to totally control which objects the users can upload or download. For example, think of a photo-sharing application that keeps photos private. When a user wants to view one of their photos, the app can generate a pre-signed URL that grants access to a private object without having to provide AWS credentials.
Bottom line
The simplest option is to create an IAM user for each client (Option 1) and provide them credentials. They could then use the AWS Command-Line Interface (CLI) or a program you provide to interact with S3.
If you consider this to be too complex, then you might want to use services like box.com or even Microsoft OneDrive, which provide a more friendly interface on top of storage services.

Answer 3

Thanks to John Rotenstein's great detailed answer with various options, I think I found the simplest available option for my case. It is based on IAM policy elements: Variables and tags and involves creating a single bucket with separate "home folders" in it (one per client). No need for user groups or roles.
Here is a step-by-step guide:
Create a common bucket (let's call it bucket-shared-with-clients).
Create a single (universal) policy like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": ["s3:ListBucket"],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::bucket-shared-with-clients"],
"Condition": {"StringLike": {"s3:prefix": ["${aws:username}/*"]}}
},
{
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::bucket-shared-with-clients/${aws:username}/*"]
}
]
}
Create IAM user accounts - one per client. The users need to be with Programmatic access enabled and in the Permission view simply go for the Attach existing policies directly option.
Here is an example Python client that uploads a file:
import boto3
ACCESS_KEY = 'YourAccessKeyComesHere'
SECRET_KEY = 'YourSecterKeyComesHere'
USERNAME = 'client-a' # The username is also used as a "home folder", as can be seen below
FILE_TO_UPLOAD = 'some-file.json'
session = boto3.Session(aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
bucket = s3.Bucket('bucket-shared-with-clients')
key = f'{USERNAME}/{FILE_TO_UPLOAD}' # If USERNAME didn't match the client's IAM User, we would get an AccessDenied error
bucket.upload_file(FILE_TO_UPLOAD, key)