I'm working to build a web portal that displays the contents of an S3 bucket to authenticated users. Users would be allowed to download objects via presigned URLs so the content/bandwidth wouldn't need to be ushered through the web portal and credentials wouldn't need to be passed to the client. This works well for single objects. However, I'm uncertain how to leverage presigned URLs when users want to download many objects e.g. all objects with a specific prefix. It seems the issue may be more of a limitation with standard web technologies i.e. multiple downloads triggered by a single action.
I've seen some apps dynamically create a .zip containing all the objects, but I'm trying to avoid moving data through the portal. I also found AWS POST Policies leveraging condition keys like 'starts-with' but it doesn't look like a POST Policy will help with getting objects. The STS AssumeRole could be used to generate temporary/limited credentials to download the objects of a specific prefix, but the user would still need to download each object. Am I overlooking a better solution?
Related
Is it possible in S3 to allow dynamic groups of users access to resources in a bucket? For example, I know you can use Cognito to restrict access of users' content to the respective users. However, I don't know how to apply some dynamic rule which would require DB access. Some example scenarios I can think of:
Instagram-like functionality, users can connect with friends and upload photos. Only friends can view a user's photos.
Project-level resources. Multiple users can be added to a project, and only members of the project may view its resources. Projects can be created and managed by users and so are not pre-defined.
Users have private file storage, but can share files with other users.
Now the obvious 1st layer of protection would be the front-end simply not giving the links to these resources to unauthorized users. But suppose in the second scenario, the S3 link to SECRET_COMPANY_DATA.zip gets leaked. I would hope that when someone tries to access that link, it only succeeds if they're in the associated project and have sufficient privileges.
I think, to some degree, this can be handled with adding custom claims to the cognito token, e.g. you could probably add a project_id claim and do a similar path-based Allow on it. But if a user can be part of multiple projects, this seems to go out the window.
It seems to me like this should be a common enough requirement enough that there is a simple solution. Any advice?
The best approach would be:
Keep your bucket private, with no Bucket Policy
Users authenticate to your app
When a user requests access to a file stored in Amazon S3, the app should check if they are permitted to access the file. This could check who 'owns' the file, their list of friends, their projects, etc. You would program all this logic in your own app.
If the user is authorised to access the file, the your app should generate an Amazon S3 pre-signed URL, which is a time-limited URL that provides temporary access to a private object. This URL can be inserted into HTML, such as in <a HREF="..."> or <img src="...">.
When the user clicks the link, Amazon S3 will verify the signature and will confirm that the link has not yet expired. If everything is okay, it will return the file to the user's browser.
This approach means that your app can control all the authentication and authorization, while S3 will be responsible for serving the content to the user.
If another person got access to the pre-signed URL, then they can also download the content. Therefore, keep the expiry time to a minimum (a few minutes). After this period, the URL will no longer work.
Your app can generate the pre-signed URL in a few lines of code. It does not require a call to AWS to create the URL.
I'm new to S3 and I'm wondering how real-world web applications typically interact with it, in particular how user access permissions are handled.
Say, for instance, that I have designed a basic project management web application which, amongst other features, permits users to upload project files into a shared space which other project members can access.
So User file upload/read access would be determined by project membership but also by project roles.
Using S3, would one simply create a Bucket for the entire application with a single S3 user with all permissions and leave the handling of the user permissions to the application ? Or am I missing something ? I haven't been able to find many examples of real-world S3 usage online, in particular where access permissions are concerned.
The typical architecture is to keep the Amazon S3 buckets totally private.
When your application determines that a user is permitted to upload or download a file, it can generate a Presigned URL. This is a time-limited URL that allows an object to be uploaded or downloaded.
When uploading, it is also possible to Create a POST Policy to enforce some restrictions on the upload, such as its length, type and where it is being stored. If the upload meets the requirements, the file will be accepted.
You should maintain a database that identifies all objects that have been uploaded and maps it to the 'owner', permission groups, shares, etc. All of this is application-specific. Later, when a user requests a particular object for download, your app can generate a pre-signed URL that lets the user download the object even those it is a private object.
Always have your application determine permissions for accessing an object. Do not define application users as IAM Users.
If there is a straight-forward permission model (eg all of one user's files are in one path/folder within an S3 bucket), you can generate temporary credentials using the AWS Security Token Service that grants List and Get permissions on the given path. This can be useful for mobile applications that could then directly call the Amazon S3 API to retrieve objects. However, it is not suitable for a web-based application.
I am integrating the ability for users of my web app to be able to upload images to my site. I want to store these images in an AWS S3 bucket, but I need to be careful with privacy and making sure only people that should have access to these files can see them.
Users should have access to these files via <img src="s3_link"> but should not be able to access the bucket directly or list the objects within.
I can accomplish this by making the bucket public but this seems dangerous.
How do I set up a proper bucket policy to allow these images to be loaded onto a webpage in an <img> tag?
S3 supports pre-signed URLs. They can be used to restrict access to specific user.
See: Share an Object with Others
You might be able to use something like https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-4 (Restricting Access to a Specific HTTP Referrer).
It's going to be difficult to ensure complete control since S3 is a basic CND without the ability to create/grant permissions on the fly, coupled with the fact that you want your <img /> tags to grab the content directly from S3.
If you are trying to restrict downloads, you'll need to setup some basic logic to grant your users some type of access token that they will provide when requesting content to download (will require a lambda script/DB or a service that can pull the images down and then serve them if the caller is authenticated).
It sounds like you'll need authenticated users to request access to content via an API, passing in an Authorization token which the API will then verify if they have access to pull down the requested content.
Can I allow a 3rd party file upload to an S3 bucket without using IAM? I would like to avoid the hassle of sending them credentials for an AWS account, but still take advantage of the S3 UI. I have only found solutions for one or the other.
The pre-signed url option sounded great but appears to only work with their SDKs and I'm not about to tell my client to install python on their computer to upload a file.
The browser based upload requires me to make my own front end html form and run in on a server just to upload (lol).
Can I not simply create a pre-signed url which navigates the user to the S3 console and allows them to upload before expiration time? Of course, making the bucket public is not an option either. Why is this so complicated!
Management Console
The Amazon S3 management console will only display S3 buckets that are associated with the AWS account of the user. Also, it is not possible to limit the buckets displayed (it will display all buckets in the account, even if the user cannot access them).
Thus, you certainly don't want to give them access to your AWS management console.
Pre-Signed URL
Your user does not require the AWS SDK to use a pre-signed URL. Rather, you must run your own system that generates the pre-signed URL and makes it available to the user (eg through a web page or API call).
Web page
You can host a static upload page on Amazon S3, but it will not be able to authenticate the user. Since you only wish to provide access to specific people, you'll need some code running on the back-end to authenticate them.
Generate...
You ask: "Can I not simply create a pre-signed url which navigates the user to the S3 console and allows them to upload before expiration time?"
Yes and no. Yes, you can generate a pre-signed URL. However, it cannot be used with the S3 console (see above).
Why is this so complicated?
Because security is important.
So, what to do?
A few options:
Make a bucket publicly writable, but not publicly readable. Tell your customer how to upload. The downside is that anyone could upload to the bucket (if they know about it), so it is only security by obscurity. But, it might be a simple solution for you.
Generate a very long-lived pre-signed URL. You can create a URL that works for months or years. Provide this to them, and they can upload (eg via a static HTML page that you give them).
Generate some IAM User credentials for them, then have them use a utility like the AWS Command-Line Interface (CLI) or Cloudberry. Give them just enough credentials for upload access. This assumes you only have a few customers that need access.
Bottom line: Security is important. Yet, you wish to "avoid the hassle of sending them credentials", nor do you wish to run a system to perform the authentication checks. You can't have security without doing some work, and the cost of poor security will be much more than the cost of implementing good security.
you could deploy a lambda function to call "signed URL" then use that URL to upload the file. here is an example
https://aws.amazon.com/blogs/compute/uploading-to-amazon-s3-directly-from-a-web-or-mobile-application/
We are building a custom application (using LoopBack) that will need to store many large files coming from multiple users, so naturally we're looking at S3. We've done something similar before, with clients uploading files to the server which then processes and uploads them to S3 under one AWS account, but for this new app, we're looking to allow the clients (using a custom iOS app) to use the iOS S3 SDK to upload directly to their own bucket or folder. User accounts will be created on the server.
Is there any way to handle S3 authentication/authorization using custom code? For example, could the iOS client request a temporary token allowing them to upload to a specific S3 bucket or folder? Or would we need to create unique IAM users for each user in our system?
Is that a terrible idea? It sounds like a terrible idea. :)
I found a similar question here but there was no conclusive answer.
Update: I found this article on Temporary Security Credentials that looks very promising. It also suggests using Cognito, which I've never used, if building a mobile app.
Cognito is the way to go. You should definetly not create IAM users for this. IAM is for managing access to the aws services programatically or from the console. Moreover you would need to hardcode the IAM access keys in the ios app, which is not a best practice.
https://docs.aws.amazon.com/cognito/latest/developerguide/what-is-amazon-cognito.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc_cognito.html
If users of your application are already authenticated, you could generate a pre-signed S3 url on your backend using your credentials. This URL can then be returned to the application and used to upload a file.
It would circumvent having to create individual IAM users/permissions and/or managing bucket policies.
Check out the docs on it here.
Not sure how relevant to your situation.
You can create a role that allows upload to s3 and use SAML web-based identity to authenticate and allow privileges to assume the role and get temp credentials and token.
This will keep very limited time authenticated to S3 upload. ie until the temp credentials expire.