I work on a SaaS application where Creators can create Groups and invite others to their Group to share files, chat and so on. Only people within specific group should have access to this group's files.
People from other group must not have access to not their group's files.
And of course all files permission should be set to 'Private', i.e. they should not be searchable/visible/accessable by anonymous users of Internet since information in those files is for personal use only.
I am new to Amazon S3 and don't know how to achieve it... Should I create only 1 main bucket? Or create for each group a new Amazon Bucket?
It is not recommended to use AWS Identity and Access Management (IAM) for storing application users. Application users should be maintained in a separate database (or LDAP, Active directory, etc).
Therefore, creating "one bucket per group" is not feasible, since it is not possible to assign your applications users to permissions within Amazon S3.
The better method would be to manage permissions within your application. When a user requests access to a file, the application can determine whether they should be permitted access. If they are permitted, then the application can generate a Pre-Signed URL.
A Pre-Signed URL permits access to private objects stored on Amazon S3. It is a means of keeping objects secure, yet granting temporary access to a specific object.
When listing available files, your application would generate links that include the pre-signed URL. Then, when a user clicks the link, they can access the file. Then, after a certain time has expired (eg 10 minutes), the link will no longer function. So, if a user shares a link with somebody else, it will probably have timed-out.
See: Creating a pre-signed URL in Ruby
Related
Is it possible in S3 to allow dynamic groups of users access to resources in a bucket? For example, I know you can use Cognito to restrict access of users' content to the respective users. However, I don't know how to apply some dynamic rule which would require DB access. Some example scenarios I can think of:
Instagram-like functionality, users can connect with friends and upload photos. Only friends can view a user's photos.
Project-level resources. Multiple users can be added to a project, and only members of the project may view its resources. Projects can be created and managed by users and so are not pre-defined.
Users have private file storage, but can share files with other users.
Now the obvious 1st layer of protection would be the front-end simply not giving the links to these resources to unauthorized users. But suppose in the second scenario, the S3 link to SECRET_COMPANY_DATA.zip gets leaked. I would hope that when someone tries to access that link, it only succeeds if they're in the associated project and have sufficient privileges.
I think, to some degree, this can be handled with adding custom claims to the cognito token, e.g. you could probably add a project_id claim and do a similar path-based Allow on it. But if a user can be part of multiple projects, this seems to go out the window.
It seems to me like this should be a common enough requirement enough that there is a simple solution. Any advice?
The best approach would be:
Keep your bucket private, with no Bucket Policy
Users authenticate to your app
When a user requests access to a file stored in Amazon S3, the app should check if they are permitted to access the file. This could check who 'owns' the file, their list of friends, their projects, etc. You would program all this logic in your own app.
If the user is authorised to access the file, the your app should generate an Amazon S3 pre-signed URL, which is a time-limited URL that provides temporary access to a private object. This URL can be inserted into HTML, such as in <a HREF="..."> or <img src="...">.
When the user clicks the link, Amazon S3 will verify the signature and will confirm that the link has not yet expired. If everything is okay, it will return the file to the user's browser.
This approach means that your app can control all the authentication and authorization, while S3 will be responsible for serving the content to the user.
If another person got access to the pre-signed URL, then they can also download the content. Therefore, keep the expiry time to a minimum (a few minutes). After this period, the URL will no longer work.
Your app can generate the pre-signed URL in a few lines of code. It does not require a call to AWS to create the URL.
This question is in the same line of thought than Is it possible to give token access to link to amazon s3 storage?.
Basically, we are building an app where groups of users can save pictures, that should be visible only to their own group.
We are thinking of using either a folder per user group, or it could even be an independent S3 bucket per user group.
The rules are very simple:
Any member of Group A should be able to add a picture to the Group A folder (or bucket)
Any member of Group A should be able to read all pictures of the Group A folder (or bucket)
No member of Group A should not have access to any of the pictures
However, the solution used by the post mentioned above (temporary pre-signed URLs) is not usable, as we need the client to be able to write files on his bucket as well as read the files on his bucket, without having any access to any other bucket. The file write part is the difficulty here and the reason why we cannot use pre-signed URLs.
Additionally, the solution from various AWS security posts that we read (for example https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/) do not apply because they show how to control accesses for IAM groups of for other AWS accounts. In our case, a group of users does not have an IAM account...
The only solutions that we see so far are either insecure or wasteful
Open buckets to everybody and rely on obfuscating the folder / bucket names (lots of security issues, including the ability to brute force and read / overwrite anybody's files)
Have a back-end that acts as a facade between the app and S3, validating the accesses. S3 has no public access, the bucket is only opened to an IAM role that the back-end has. However this is a big waste of bandwidth, since all the data would transit on the EC2 instance(s) of that back-end
Any better solution?
Is this kind of customized access doable with S3?
The correct way to achieve your goal is to use Amazon S3 pre-signed URLs, which are time-limited URLs that provides temporary access to a private object.
You can also Upload objects using presigned URLs - Amazon Simple Storage Service.
The flow is basically:
Users authenticate to your back-end app
When a user wants to access a private object, the back-end verifies that they are permitted to access the object (using your own business logic, such as the Groups you mention). If they are allowed to access the object, the back-end generates a pre-signed URL.
The pre-signed URL is returned to the user's browser, such as putting it in a <img src="..."> tag.
When the user's browser requests the object, S3 verifies the signature in the pre-signed URL. If it is valid and the time period has not expired, S3 provides the requested object. (Otherwise, it returns Access Denied.)
A similar process is used when users upload objects:
Users authenticate to your back-end app
They request the opportunity to upload a file
Your back-end app generates an S3 Pre-signed URL that is included in the HTML page for upload
Your back-end should track the object in a database so it knows who performed the upload and keeps track of who is permitted to access the object (eg particular users or groups)
Your back-end app is fully responsible for deciding whether particular users can upload/download objects. It then hands-off the actual upload/download process to S3 via the pre-signed URLs. This reduces load on your server because all uploads/downloads go direct to/from S3.
I am trying to understand access security as it relates to Amazon S3. I want to host some files in an S3 bucket, using CloudFront to access it via my domain. I need to limit access to certain companies/individuals. In addition I need to manage that access individually.
A second access model is project based, where I need to make a library of files available to a particular project team, and I need to be able to add and remove team members in an ad hoc manner, and then close access for the whole project at some point. The bucket in question might be the same for both scenarios.
I assume something like this is possible in AWS, but all I can find (and understand) on the AWS site involves using IAM to control access via the AWS console. I don't see any indication that I could create an IAM user, add them to an IAM group, give the group read only access to the bucket and then provide the name and password via System.Net.WebClient in PowerShell to actually download the available file. Am I missing something, and this IS possible? Or am I not correct in my assumption that this can be done with AWS?
I did find Amazon CloudFront vs. S3 --> restrict access by domain? - Stack Overflow that talks about using CloudFront to limit access by Domain, but that won't work in a WfH scenario, as those home machines won't be on the corporate domain, but the corporate BIM Manager needs to manage access to content libraries for the WfH staff. I REALLY hope I am not running into an example of AWS just not being ready for the current reality.
Content stored in Amazon S3 is private by default. There are several ways that access can be granted:
Use a bucket policy to make the entire bucket (or a directory within it) publicly accessible to everyone. This is good for websites where anyone can read the content.
Assign permission to IAM Users to grant access only to users or applications that need to access to the bucket. This is typically used within your organization. Never create an IAM User for somebody outside your organization.
Create presigned URLs to grant temporary access to private objects. This is typically used by applications to grant web-based access to content stored in Amazon S3.
To provide an example for pre-signed URLs, imagine that you have a photo-sharing website. Photos provided by users are private. The flow would be:
A user logs in. The application confirms their identity against a database or an authentication service (eg Login with Google).
When the user wants to view a photo, the application first checks whether they are entitled to view the photo (eg it is their photo). If they are entitled to view the photo, the application generates a pre-signed URL and returns it as a link, or embeds the link in an HTML page (eg in a <img> tag).
When the user accesses the link, the browser sends the URL request to Amazon S3, which verifies the encrypted signature in the signed URL. If if it is correct and the link has not yet expired, the photo is returned and is displayed in the web browser.
Users can also share photos with other users. When another user accesses a photo, the application checks the database to confirm that it was shared with the user. If so, it provides a pre-signed URL to access the photo.
This architecture has the application perform all of the logic around Access Permissions. It is very flexible since you can write whatever rules you want, and then the user is sent to Amazon S3 to obtain the file. Think of it like buying theater tickets online -- you just show the ticket and the door and you are allowed to sit in the seat. That's what Amazon S3 is doing -- it is checking the ticket (signed URL) and then giving you access to the file.
See: Amazon S3 pre-signed URLs
Mobile apps
Another common architecture is to generate temporary credentials using the AWS Security Token Service (STS). This is typically done with mobile apps. The flow is:
A user logs into a mobile app. The app sends the login details to a back-end application, which verifies the user's identity.
The back-end app then uses AWS STS to generate temporary credentials and assigns permissions to the credentials, such as being permitted to access a certain directory within an Amazon S3 bucket. (The permissions can actually be for anything in AWS, such as launching computers or creating databases.)
The back-end app sends these temporary credentials back to the mobile app.
The mobile app then uses those credentials to make calls directly to Amazon S3 to access files.
Amazon S3 checks the credentials being used and, if they have permission for the files being requests, grants access. This can be done for uploads, downloads, listing files, etc.
This architecture takes advantage of the fact that mobile apps are quite powerful and they can communicate directly with AWS services such as Amazon S3. The permissions granted are based upon the user who logs in. These permissions are determined by the back-end application, which you would code. Think of it like a temporary employee who has been granted a building access pass for the day, but they can only access certain areas.
See: IAM Role Archives - Jayendra's Blog
The above architectures are building blocks for how you wish to develop your applications. Every application is different, just like the two use-cases in your question. You can securely incorporate Amazon S3 in your applications while maintaining full control of how access is granted. Your applications can then concentrate on the business logic of controlling access, without having to actually serve the content (which is left up to Amazon S3). It's like selling the tickets without having to run the theater.
You ask whether Amazon S3 is "ready for the current reality". Many of the popular web sites you use every day run on AWS, and you probably never realize it.
If you are willing to issue IAM User credentials (max 5000 per account), the steps would be:
Create an IAM User for each user and select Programmatic access
This will provide an Access Key and Secret Key that you can provide to each user
Attach permissions to each IAM User, or put the users in an IAM Group and attach permissions to the IAM Group
Each user can run aws configure on their computer (using the AWS Command-Line Interface (CLI) to store their Access Key and Secret Key
They can then use the AWS CLI to upload/download files
If you want the users to be able to access via the Amazon S3 management console, you will need to provide some additional permissions: Grant a User Amazon S3 Console Access to Only a Certain Bucket
Alternatively, users could use a program like CyberDuck for an easy Drag & Drop interface to Amazon S3. Cyberduck will also ask for the Access Key and Secret Key.
I'm new to S3 and I'm wondering how real-world web applications typically interact with it, in particular how user access permissions are handled.
Say, for instance, that I have designed a basic project management web application which, amongst other features, permits users to upload project files into a shared space which other project members can access.
So User file upload/read access would be determined by project membership but also by project roles.
Using S3, would one simply create a Bucket for the entire application with a single S3 user with all permissions and leave the handling of the user permissions to the application ? Or am I missing something ? I haven't been able to find many examples of real-world S3 usage online, in particular where access permissions are concerned.
The typical architecture is to keep the Amazon S3 buckets totally private.
When your application determines that a user is permitted to upload or download a file, it can generate a Presigned URL. This is a time-limited URL that allows an object to be uploaded or downloaded.
When uploading, it is also possible to Create a POST Policy to enforce some restrictions on the upload, such as its length, type and where it is being stored. If the upload meets the requirements, the file will be accepted.
You should maintain a database that identifies all objects that have been uploaded and maps it to the 'owner', permission groups, shares, etc. All of this is application-specific. Later, when a user requests a particular object for download, your app can generate a pre-signed URL that lets the user download the object even those it is a private object.
Always have your application determine permissions for accessing an object. Do not define application users as IAM Users.
If there is a straight-forward permission model (eg all of one user's files are in one path/folder within an S3 bucket), you can generate temporary credentials using the AWS Security Token Service that grants List and Get permissions on the given path. This can be useful for mobile applications that could then directly call the Amazon S3 API to retrieve objects. However, it is not suitable for a web-based application.
I have an application where users are part of a 'group' of users. Each group can 'upload' documents to the application. Behind the scenes I am using S3 to store these documents.
I've spent a ton of time reading the AWS documentation but still don't understand the simplest/correct way to do the following:
User 1 in group A can upload documents to application
User 2 in group A can see and access all group A documents in application
User 3 in group B can upload documents to application
User 3 in group B cannot see any documents that belong to group A (and vice-versa)
Should I be using the API to create a new bucket for each 'group'?
Or can all of this be done in a single bucket with subdirectories for each group & then set access limitations?
Should I be setting up an IAM group policy and applying it to each web app user?
I'm not sure of the best architecture for this scenario so would really appreciate a point in the right direction.
AWS credentials should be assigned to your application and to your IT staff who need to maintain the application.
Users of your application should not be given AWS credentials.
Users should interact directly with your application and your application will make calls to the AWS API from the back-end. This way, your application has full control of what data they can see and what operations they can perform.
Think of it like a database -- you never want to give users direct access to a database. Instead, they should always interact via an application, which will store and update information in a database.
There are some common exceptions to the above:
If you want users to access/download a file stored in S3, your application can generate a pre-signed URL, which is a time-limited URL that permits access to an Amazon S3 object. Your application is responsible for generating the URL when it wants to grant access and the URl can be included in an HTML page (eg show a private picture on a web page).
If you want to allow users to upload files directly to S3, you could again use a pre-signed URL or you could grant public Write access to an Amazon S3 bucket. Think of it like a modern FTP server.
Bottom line: Your application is in charge! Also, consider using pre-signed URLs to provide direct access to objects when the application permits it.