Let's say I have 1k users (not AWS users) and each user have their own S3 bucket. I am going to use signed URL's for them to access their data. For security reasons I want to assume a role when creating the signed URL so that it's limited to their bucket only if they start messing around with the URL.
What would be the best way to do this? Create a role and a policy for each user? Can I create a single role that I attach a policy to when creating the signed URL?
Thanks!
You should not create one bucket per user. Buckets are not a scalable resource because there are limits to the number of buckets in an AWS Account.
You could keep all user data in a bucket. If you wish to segregate it by user, use a subdirectory.
An alternate approach is to put all data intermixed, rather than dividing it by user. Your application should maintain a database of each object, together with metadata such as the 'owner', when it was created, any sharing permissions, etc. Then, your app can provide users with access to their data by using Amazon S3 pre-signed URLs. This way, your app controls access.
One benefit of this approach is that data can be shared amongst users. For example, let's say you have a photo-sharing app and a user wants to share a photo with another user. The app can maintain a list of what has been shared between users and then create pre-signed URLs to let users access data that was uploaded by a different user. It's the app that determines ownership rather than 'where' a file is placed.
You should not be afraid that a pre-signed URL can give access to other content. A pre-signed URL contains a hashed signature that verifies the object and expiry time. The signature cannot be reverse-engineered and it cannot be used to grant access to other resources. There is no need to use specific IAM Roles that have a subset of permissions.
Related
I made a dashboard where users can upload files. I want to make it so that users can only have access to S3 urls that are files that they uploaded. How can I achieve this?
The users are application based, meaning they are, in my case, Django users.
Thanks! Anything helps!!
As this is entirely application based there are a few steps you could take to try and mitigate against accidental exposure.
Firstly organise your S3 folder structure in a way that uses prefixes for usernames, this way from a hierarchical point of view you can limit the scope of where users objects are stored. By prefix I mean the key might look like this users/$USERNAME/file.txt where $USERNAME is actually the users username.
You could enhance this in in your application by expanding to use Cognito, with a seperate user for every user you have created. When the user logs into your application you could also have the login occur via the Cognito user (this can be done programmatically).
With a successful login you'll be provided temporary credentials for IAM, all users can be assigned to a Cognito group which can have an IAM role attached. When you login it will assume this role, which allows some special properties to be supported in IAM.
By using ${cognito-identity.amazonaws.com:sub} you can actually limit the IAM permissions to only access that prefix of the S3 bucket. This moves responsibility from your application to Cognito and IAM.
More information about this is available in: Amazon S3: Allows Amazon Cognito Users to Access Objects in Their Bucket.
What I am trying to achieve is the following:
Create users dynamicly through API(users might grow alot - 50-100k+ eventually)
Give those users access to a specific prefix of an AWS S3 bucket(IAM policy)
Currently my idea is to create AWS IAM Users and generate credentials for those users(The credentials should not be temporary). This works fine, but the problem is that AWS is limited to 5000 IAM users. Is there another way to avoid that limit. One way that I found out is via cognito users -> https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_cognito-bucket.html
However I do not think that there is a way to create long-term access keys(as the IAM user access keys) for those cognito users ?
Is there another way to achieve this ?
Thanks in advance!
You should not use IAM for application users. IAM is for staff within your organisation to operate your AWS infrastructure.
Your application should operate its own authentication method separate from IAM (as suggested in the above comments). An example of using AWS for this task would be to use Amazon Cognito.
Once a user has authenticated, you have a couple of options:
Option 1: Using AWS credentials
If you want to allow the authenticated users to access AWS resources (eg Amazon S3) via AWS API calls, then you can create temporary credentials that have limited permissions (eg can access any object within a given path of a given bucket). These credentials can then be provided to the users. This method is commonly used for mobile applications that are capable of making API calls directly to AWS. It requires that the users have software that can use the AWS credentials.
Option 2: Amazon S3 pre-signed URLS
If you are running a web application and you want users to be able to access private objects in Amazon S3, you can generate pre-signed URLs. For example, let's say you are running a photo-sharing website. The process would be:
Photos are kept in private S3 buckets.
Users authenticate to the application.
The application can then show them their private photos: When the application generates any links to this private content, or embeds content in the page (eg via <img> tags), it generates a pre-signed URL, which provides time-limited access to private content.
The user then accesses the URL, or their browser requests data (eg images) from that URL.
Amazon S3 verifies the signature on the URL and check the validity time. If it is correct, then S3 returns the private object.
The application uses a set of IAM credentials to sign the pre-signed URL. This can be done in a couple of lines of code and does not require an API call to AWS.
The benefit of this method is that the application is responsible for determining which objects the user may access. For example, let's say a user wants to share their photos with another user. This sharing information can be stored in a database and the application can consult the database when sharing photos. If a user is entitled to view another user's photos, the application can generate a pre-signed URL without caring in which directory the photos are stored. This is a much more flexible approach than using storage location to grant access. However, it does require additional logic within the application.
See: Amazon S3 pre-signed URLs
Is it possible to give different access to different buckets in s3? In detail, I have 10 different buckets in s3 and each of those bucket related to different people. So I want to give them access only to their particular bucket(by sharing a URL or something like that)
Is this possible?
The normal way to assign access is:
Permanent credentials (eg associate with an IAM User) are only provided to internal IT staff who are managing or using the AWS services.
End users of a web application should be authenticated by the application (eg using Amazon Cognito, LDAP, AD, Google). The application will then be responsible for generating Pre-Signed URLs for uploading and downloading files.
For mobile applications, it is quite common to create temporary credentials using the Security Token Service, which allows the mobile app to directly make AWS API calls. The credentials can be given limited permissions, such as only being able to access one S3 bucket.
So, it really comes down to 'how' the users will be accessing the bucket. If they are doing it directly, then provide temporary credentials via STS. If they are doing it via an application, then the application will be responsible for providing individual access to upload/download.
By the way, it's not necessarily a good idea to give a different bucket to every user, because there is a limit on the number of buckets you can create. Instead, you could give access to separate paths within the same bucket. Proper use of permissions will ensure they cannot see/impact other users' data.
For how this works with IAM Users, see: Variables in AWS Access Control Policies | AWS News Blog
I need to allow my users to access and upload content to AWS S3.
Now I have two decisions to make:
Create a separate bucket for each user or store data for each user in different directories in a single bucket.
Allow them access via signed url or create a separate IAM user for everyone?
The process needs to be fully automated and scalable (i.e. many users sign up or stop using the services every day).
Each user uploads files every few seconds. I therefore thought that a separate IAM user would save me the roundtrip to get a signed URL, but I am not sure if it is practical to have potentially thousands of IAM users.
You can only have 100 buckets per AWS account, total not per region so you shouldn't create a bucket per user.
Use keys (folders) to organise the data. Provide access via signed cookies (or urls, but cookies are better for per user) and do the authentication in your application (or use AWS Cognito). IAM isn't really designed for your application end users but for AWS service users.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-choosing-signed-urls-cookies.html
I have created a bucket in Amazon S3 and have uploaded 2 files in it and made them public. I have the links through which I can access them from anywhere on the Internet. I now want to put some restriction on who can download the files. Can someone please help me with that. I did try the documentation, but got confused.
I want that at the time of download using the public link it should ask for some credentials or something to authenticate the user at that time. Is this possible?
By default, all objects in Amazon S3 are private. You can then add permissions so that people can access your objects. This can be done via:
Access Control List permissions on individual objects
A Bucket Policy
IAM Users and Groups
A Pre-Signed URL
As long as at least one of these methods is granting access, your users will be able to access the objects from Amazon S3.
1. Access Control List on individual objects
The Make Public option in the Amazon S3 management console will grant Open/Download permissions to all Internet users. This can be used to grant public access to specific objects.
2. Bucket Policy
A Bucket Policy can be used to grant access to a whole bucket or a portion of a bucket. It can also be used to specify limits to access. For example, a policy could make a specific directory within a bucket public to users from a specific range of IP addresses, during particular times of the day, and only when accessing the bucket via SSL.
A bucket policy is a good way to grant public access to many objects (eg a particular directory) without having to specify permissions on each individual object. This is commonly used for static websites served out of an S3 bucket.
3. IAM Users and Groups
This is similar to defining a Bucket Policy, but permissions are assigned to specific Users or Groups of users. Thus, only those users have permission to access the objects. Users must authenticate themselves when accessing the objects, so this is most commonly used when accessing objects via the AWS API, such as using the aws s3 commands from the AWS Command-Line Interface (CLI).
Rather than being prompted to authenticate, users must provide the authentication when making the API call. A simple way of doing this is to store user credentials in a local configuration file, which the CLI will automatically use when calling the S3 API.
4. Pre-Signed URL
A Pre-Signed URL can be used to grant access to S3 objects as a way of "overriding" access controls. A normally private object can be accessed via a URL by appending an expiry time and signature. This is a great way to serve private content without requiring a web server.
Typically, an application constructs a Pre-Signed URL when it wishes to grant access to an object. For example, let's say you have a photo-sharing website and a user has authenticated to your website. You now wish to display their pictures in a web page. The pictures are normally private, but your application can generate Pre-Signed URLs that grant them temporary access to the pictures. The Pre-Signed URL will expire after a particular date/time.
Regarding the pre-signed URL, the signature is in the request headers, hence it should be within HTTPS/TLS encryption. But do check for yourself.