What I am trying to achieve is the following:
Create users dynamicly through API(users might grow alot - 50-100k+ eventually)
Give those users access to a specific prefix of an AWS S3 bucket(IAM policy)
Currently my idea is to create AWS IAM Users and generate credentials for those users(The credentials should not be temporary). This works fine, but the problem is that AWS is limited to 5000 IAM users. Is there another way to avoid that limit. One way that I found out is via cognito users -> https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_cognito-bucket.html
However I do not think that there is a way to create long-term access keys(as the IAM user access keys) for those cognito users ?
Is there another way to achieve this ?
Thanks in advance!
You should not use IAM for application users. IAM is for staff within your organisation to operate your AWS infrastructure.
Your application should operate its own authentication method separate from IAM (as suggested in the above comments). An example of using AWS for this task would be to use Amazon Cognito.
Once a user has authenticated, you have a couple of options:
Option 1: Using AWS credentials
If you want to allow the authenticated users to access AWS resources (eg Amazon S3) via AWS API calls, then you can create temporary credentials that have limited permissions (eg can access any object within a given path of a given bucket). These credentials can then be provided to the users. This method is commonly used for mobile applications that are capable of making API calls directly to AWS. It requires that the users have software that can use the AWS credentials.
Option 2: Amazon S3 pre-signed URLS
If you are running a web application and you want users to be able to access private objects in Amazon S3, you can generate pre-signed URLs. For example, let's say you are running a photo-sharing website. The process would be:
Photos are kept in private S3 buckets.
Users authenticate to the application.
The application can then show them their private photos: When the application generates any links to this private content, or embeds content in the page (eg via <img> tags), it generates a pre-signed URL, which provides time-limited access to private content.
The user then accesses the URL, or their browser requests data (eg images) from that URL.
Amazon S3 verifies the signature on the URL and check the validity time. If it is correct, then S3 returns the private object.
The application uses a set of IAM credentials to sign the pre-signed URL. This can be done in a couple of lines of code and does not require an API call to AWS.
The benefit of this method is that the application is responsible for determining which objects the user may access. For example, let's say a user wants to share their photos with another user. This sharing information can be stored in a database and the application can consult the database when sharing photos. If a user is entitled to view another user's photos, the application can generate a pre-signed URL without caring in which directory the photos are stored. This is a much more flexible approach than using storage location to grant access. However, it does require additional logic within the application.
See: Amazon S3 pre-signed URLs
Related
I am trying to understand access security as it relates to Amazon S3. I want to host some files in an S3 bucket, using CloudFront to access it via my domain. I need to limit access to certain companies/individuals. In addition I need to manage that access individually.
A second access model is project based, where I need to make a library of files available to a particular project team, and I need to be able to add and remove team members in an ad hoc manner, and then close access for the whole project at some point. The bucket in question might be the same for both scenarios.
I assume something like this is possible in AWS, but all I can find (and understand) on the AWS site involves using IAM to control access via the AWS console. I don't see any indication that I could create an IAM user, add them to an IAM group, give the group read only access to the bucket and then provide the name and password via System.Net.WebClient in PowerShell to actually download the available file. Am I missing something, and this IS possible? Or am I not correct in my assumption that this can be done with AWS?
I did find Amazon CloudFront vs. S3 --> restrict access by domain? - Stack Overflow that talks about using CloudFront to limit access by Domain, but that won't work in a WfH scenario, as those home machines won't be on the corporate domain, but the corporate BIM Manager needs to manage access to content libraries for the WfH staff. I REALLY hope I am not running into an example of AWS just not being ready for the current reality.
Content stored in Amazon S3 is private by default. There are several ways that access can be granted:
Use a bucket policy to make the entire bucket (or a directory within it) publicly accessible to everyone. This is good for websites where anyone can read the content.
Assign permission to IAM Users to grant access only to users or applications that need to access to the bucket. This is typically used within your organization. Never create an IAM User for somebody outside your organization.
Create presigned URLs to grant temporary access to private objects. This is typically used by applications to grant web-based access to content stored in Amazon S3.
To provide an example for pre-signed URLs, imagine that you have a photo-sharing website. Photos provided by users are private. The flow would be:
A user logs in. The application confirms their identity against a database or an authentication service (eg Login with Google).
When the user wants to view a photo, the application first checks whether they are entitled to view the photo (eg it is their photo). If they are entitled to view the photo, the application generates a pre-signed URL and returns it as a link, or embeds the link in an HTML page (eg in a <img> tag).
When the user accesses the link, the browser sends the URL request to Amazon S3, which verifies the encrypted signature in the signed URL. If if it is correct and the link has not yet expired, the photo is returned and is displayed in the web browser.
Users can also share photos with other users. When another user accesses a photo, the application checks the database to confirm that it was shared with the user. If so, it provides a pre-signed URL to access the photo.
This architecture has the application perform all of the logic around Access Permissions. It is very flexible since you can write whatever rules you want, and then the user is sent to Amazon S3 to obtain the file. Think of it like buying theater tickets online -- you just show the ticket and the door and you are allowed to sit in the seat. That's what Amazon S3 is doing -- it is checking the ticket (signed URL) and then giving you access to the file.
See: Amazon S3 pre-signed URLs
Mobile apps
Another common architecture is to generate temporary credentials using the AWS Security Token Service (STS). This is typically done with mobile apps. The flow is:
A user logs into a mobile app. The app sends the login details to a back-end application, which verifies the user's identity.
The back-end app then uses AWS STS to generate temporary credentials and assigns permissions to the credentials, such as being permitted to access a certain directory within an Amazon S3 bucket. (The permissions can actually be for anything in AWS, such as launching computers or creating databases.)
The back-end app sends these temporary credentials back to the mobile app.
The mobile app then uses those credentials to make calls directly to Amazon S3 to access files.
Amazon S3 checks the credentials being used and, if they have permission for the files being requests, grants access. This can be done for uploads, downloads, listing files, etc.
This architecture takes advantage of the fact that mobile apps are quite powerful and they can communicate directly with AWS services such as Amazon S3. The permissions granted are based upon the user who logs in. These permissions are determined by the back-end application, which you would code. Think of it like a temporary employee who has been granted a building access pass for the day, but they can only access certain areas.
See: IAM Role Archives - Jayendra's Blog
The above architectures are building blocks for how you wish to develop your applications. Every application is different, just like the two use-cases in your question. You can securely incorporate Amazon S3 in your applications while maintaining full control of how access is granted. Your applications can then concentrate on the business logic of controlling access, without having to actually serve the content (which is left up to Amazon S3). It's like selling the tickets without having to run the theater.
You ask whether Amazon S3 is "ready for the current reality". Many of the popular web sites you use every day run on AWS, and you probably never realize it.
If you are willing to issue IAM User credentials (max 5000 per account), the steps would be:
Create an IAM User for each user and select Programmatic access
This will provide an Access Key and Secret Key that you can provide to each user
Attach permissions to each IAM User, or put the users in an IAM Group and attach permissions to the IAM Group
Each user can run aws configure on their computer (using the AWS Command-Line Interface (CLI) to store their Access Key and Secret Key
They can then use the AWS CLI to upload/download files
If you want the users to be able to access via the Amazon S3 management console, you will need to provide some additional permissions: Grant a User Amazon S3 Console Access to Only a Certain Bucket
Alternatively, users could use a program like CyberDuck for an easy Drag & Drop interface to Amazon S3. Cyberduck will also ask for the Access Key and Secret Key.
I'm not sure if this is the appropriate use case, so please tell me what to look for if I'm incorrect in my assumption of how to do this.
What I'm trying to do:
I have an s3 bucket with different 'packs' that users can download. Upon their purchase, they are given a user role in Wordpress. I have an S3 browser set up via php that makes requests to the bucket for info.
Based on their 'role', it will only show files that match prefix (whole pack users see all, single product people only see single product prefix).
In that way, the server will be sending the files on behalf of the user, and changing IAM roles based on the user's permission level. Do I have to have it set that way? Can I just analyze the WP role and specify and endpoint or query that notes the prefixes allowed?
Pack users see /
Individual users see /--prefix/
If that makes sense
Thanks in advance! I've never used AWS, so this is all new to me. :)
This sounds too complex. It's possible to do with AWS STS but it would be extremely fragile.
I presume you're hiding the actual S3 bucket from end users and are streaming through your php application? If so, it makes more sense to do any role-based filtering in the php application as you have far more logic available to you there - IAM is granular, but restrictions to resources in S3 is going to be funky and there's always a chance you'll get something wrong and expose the incorrect downloads.
Rather do this inside your app:
establish the role you've granted
issue the S3 ls command filtered by the role - i.e. if the role permits only --prefix, issue the ls command so that it only lists files matching --prefix
don't expose files in the bucket globally - only your app should have access to the S3 bucket - that way people also can't share links once they've downloaded a pack.
this has the added benefit of not encoding your S3 bucket structure in IAM, and keeps your decision logic isolated to code.
There are basically three ways you can grant access to private content in Amazon S3.
Option 1: IAM credentials
You can add a policy to an IAM User, so that they can access private content. However, such credentials should only be used by staff in your own organization. it should not be used to grant access to application users.
Option 2: Temporary credentials via STS
Your application can generate temporary credentials via the AWS Security Token Service. These credentials can be given specific permissions and are valid for a limited time period. This is ideal for granting mobile apps access to Amazon S3 because they can communicate directly with S3 without having to go via the back-end app. The credentials would only be granted access to resources they are permitted to use.
These types of credentials can also be used by web applications, where the web apps make calls directly to AWS services (eg from Node/JavaScript in the browser). However, this doesn't seem suitable for your WordPress situation.
Option 3: Pre-Signed URLs
Imagine a photo-sharing application where users can access their private photos, and users can also share photos with other users. When a user requests access to a particular photo (or when the back-end app is creating an HTML page that uses a photo), the app can generate a pre-signed URL that grants temporary access to an Amazon S3 object.
Each pre-signed URL gives access only to a single S3 object and only for a selected time period (eg 5 minutes). This means that all the permission logic for whether a user is entitled to access a file can be performed in the back-end application. When the back-end application provides a pre-signed URL to the user's browser, the user can access the content directly from Amazon S3 without going via the back-end.
See: Amazon S3 pre-signed URLs
Your situation sounds suitable for Option #3. Once you have determined that a user is permitted to access a particular file in S3, it can generate the pre-signed URL and include it as a link (or even in <img src=...> tags). The user can then download the file. There is no need to use IAM Roles in this process.
I have 1 s3 bucket per customer. Customers are external entities and they dont share data with anyone else. I write to S3 and customer reads from S3. As per this architecture, I can only scale to 1000 buckets as there is a limit to s3 buckets per account. I was hoping to use APs to create 1 AP per customer and put data in one bucket. The customer can then read the files from the bucket using AP.
Bucket000001/prefix01 . -> customeraccount1
Bucket000001/prefix02 . -> customeraccount2
...
S3 access points require you to set policy for a IAM user in access point as well as the bucket level. If I have 1000s of IAM users, do I need to set policy for each of them in the bucket? This would result in one giant policy. there is a max policy size in the bucket, so I may not be able to do that.
Is this the right use case where access points can help?
The recommended approach would be:
Do NOT assign IAM Users to your customers. These types of AWS credentials should only be used by your internal staff and your own applications.
You should provide a web application (or an API) where customers can authenticate against your own user database (or you could use Amazon Cognito to manage authentication).
Once authenticated, the application should grant access either to a web interface to access Amazon S3, or the application should provide temporary credentials for accessing Amazon S3 (more details below).
Do not use one bucket per customer. This is not scalable. Instead, store all customer data in ONE bucket, with each user having their own folder. There is no limit on the amount of data you can store in Amazon S3. This also makes it easier for you to manage and maintain, since it is easier to perform functions across all content rather than having to go into separate buckets. (An exception might be if you wish to segment buckets by customer location (region) or customer type. But do not use one bucket per customer. There is no reason to do this.)
When granting access to Amazon S3, assign permissions at the folder-level to ensure customers only see their own data.
Option 1: Access via Web Application
If your customers access Amazon S3 via a web application, then you can code that application to enforce security at the folder level. For example, when they request a list of files, only display files within their folder.
This security can be managed totally within your own code.
Option 2: Access via Temporary Credentials
If your customers use programmatic access (eg using the AWS CLI or a custom app running on their systems), then:
The customer should authenticate to your application (how this is done will vary depending upon how you are authenticating users)
Once authenticated, the application should generate temporary credentials using the AWS Security Token Service (STS). While generating the credentials, grant access to Amazon S3 but specify the customer's folder in the ARN (eg arn:aws:s3:::storage-bucket/customer1/*) so that they can only access content within their folder.
Return these temporary credentials to the customer. They can then use these credentials to make API calls directly to Amazon S3 (eg from the AWS Command-Line Interface (CLI) or a custom app). They will be limited to their own folder.
This approach is commonly done with mobile applications. The mobile app authenticates against the backend, receives temporary credentials, then uses those credentials to interact directly against S3. Thus, the back-end app is only used for authentication.
Examples on YouTube:
5 Minutes to Amazon Cognito: Federated Identity and Mobile App Demo
Overview Security Token Service STS
AWS: Use the Session Token Service to Securely Upload Files to S3
We have some way to achieve your goal.
use IAM group to grant access to a folder. Create a group, add a user to a group, and assign a role to the group to access the folder.
Another way is to use bucket policy (${aws:username} in Condition) to grant Access to User-Specific Folders. Refer to this link https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/
How can I let another user access to my AWS S3 bucket without having to create an IAM role for it and sending the key/secret.
I want the third-party to decide the credentials for himself.
Is this even possible?
Basically I'm searching for something similar to OAuth for S3
By default, Amazon S3 buckets are private.
Access to objects can be granted in several ways:
A Bucket Policy can make a bucket, or part of a bucket, publicly accessible (not applicable for your use-case)
The Access Control List (ACL) on an object can make it publicly accessible (not applicable for your use-case)
IAM Users can be granted permissions on an Amazon S3 bucket (but IAM Users should only be used for your staff, not for application users)
IAM Roles can be temporarily assumed, but first require authentication (more on this below)
Pre-Signed URLs can be generated to provide time-limited access to Amazon S3 objects
For your use-case, the most applicable approach would be:
Users authenticate to your application. This could use Amazon Cognito, or whatever authentication method you wish to use.
When a user wishes to access a private object, your application determines whether they are entitled to such access (done via your own code).
If they are permitted access, the application should generate a pre-signed URL. This URL can be included in tags like <a> and <img>.
When the user accesses the URL, they will be able to access the object directly from Amazon S3.
Once the expiry time passes, the pre-signed URL no longer works.
So, you are welcome to use OAuth or any other authentication method, but it is the responsibility of your application to determine whether they are entitled to access individual objects and, if so, generate and return the pre-signed URL. (It's just a couple of lines of code, no API calls required to generate it.)
See: Share an Object with Others - Amazon Simple Storage Service
Amazon Cognito has the ability to issue credentials associated with an IAM Role, and users could then use these credentials to make API calls to AWS. However, it is generic role that would be shared by many users and is not a way to grant user-specific permissions.
Is it possible to give different access to different buckets in s3? In detail, I have 10 different buckets in s3 and each of those bucket related to different people. So I want to give them access only to their particular bucket(by sharing a URL or something like that)
Is this possible?
The normal way to assign access is:
Permanent credentials (eg associate with an IAM User) are only provided to internal IT staff who are managing or using the AWS services.
End users of a web application should be authenticated by the application (eg using Amazon Cognito, LDAP, AD, Google). The application will then be responsible for generating Pre-Signed URLs for uploading and downloading files.
For mobile applications, it is quite common to create temporary credentials using the Security Token Service, which allows the mobile app to directly make AWS API calls. The credentials can be given limited permissions, such as only being able to access one S3 bucket.
So, it really comes down to 'how' the users will be accessing the bucket. If they are doing it directly, then provide temporary credentials via STS. If they are doing it via an application, then the application will be responsible for providing individual access to upload/download.
By the way, it's not necessarily a good idea to give a different bucket to every user, because there is a limit on the number of buckets you can create. Instead, you could give access to separate paths within the same bucket. Proper use of permissions will ensure they cannot see/impact other users' data.
For how this works with IAM Users, see: Variables in AWS Access Control Policies | AWS News Blog