We provide REST api to upload user files, on backend we use S3 Bucket to store the user uploads.
As our REST api has timeout of 30sec, user may get timeouts based on filesize and his network.
So we thought of providing him a S3 Pre-Sign URLs for uploads, through which user can upload his large files using AJAX calls from frontend or using backend scheduled script.
Everything looks OK but we don't have any clue about provided s3 Pre-Signs:
- whether he attempted to upload or not:
if attempted whether the upload is success or not
if failed, what was the error (url expired or something other..)
We can know about success case by searching the object key in our bucket. But in case of failures we don't have any clue.
Please let me know if there is anyway to track the S3 pre-sign access/uploads.
You will not know when a pre-signed URL is used, but another couple of options are:
You can configure an Amazon S3 Event to trigger when a new file is uploaded to a bucket. This could trigger an AWS Lambda function that can process the file or, at least, make a log that the file was uploaded.
You can use Amazon S3 Server Access Logging to track access to the bucket.
Related
I am currently looking for a solution to store temporary files in AWS. I want to create a functionality on my app that allows my customers to upload a file and send it by email (something like WeTranfer or Send Anywhere).
I want to save the file temporarily on my AWS storage, for 10 hours and then remove it permanently. If the file has not expired, the user can click the link (provided by AWS) on the email and download the file.
I recently came across S3 Bucket Lifecycle rules, but I can only specify days for the expiration and not hours.
I would appreciate any suggestion. Thank you!
Amazon S3 is the appropriate place to store these files.
If you want access controls (to control which users can access the file) and fine-grained control over when the object 'expires', then you would need to code this yourself.
The files should be stored in a private Amazon S3 bucket. You would then need a back-end app that manages user authentication. When an authorized user requests access to a file, the app can generate an Amazon S3 pre-signed URL, which provides time-limited access to private objects in Amazon S3 (eg 10 hours). This is the link you would put into the email.
Deletion could still be handled by S3 Lifecycle rules, but it is less important when the file is actually deleted because the pre-signed URL would block access to the file after 10 hours anyway.
I have a blogging website and it uses AWS S3 signed URL logic to upload any pictures, used in the blogs, directly from the browser to S3 bucket.
To maintain the security, the request for generating the signed URL goes through the backend which verifies the user authentication and other things, and returns a URL with few configs that must be used to upload the file to S3 bucket from the client application. Here the server returns few metadata to be used in the config. To maintain consistency I used the user's email address as metadata and that will ensure that no random user can upload the file to S3 (though without this too, the security would be maintained but I just added it to add a layer of security).
The problem that I recently found out that (maybe I missed some config) when the file uploaded by particular user abc#example.com is fetched, the response-header includes field:
x-amz-meta-data: {"emailaddress":"abc#example.com"}
Did I miss any configuration in S3 bucket? Or the metadata will be fetched in all the responses?
If yes, how is it a signed URL as all the metadata will be shown in the browser? If no, what configuration am I missing?
If this was expected, how can I transfer all the files to a new bucket with the same policy with modified metadata?
Any help would be appreciated.
Continuing my adventure in AWS, Uploading The File, I am just going to come out and ask it. What is the proper way to configure the security settings for an S3 bucket intended to be the endpoint for a presigned post file upload?
Currently I have created an IAM user with full S3 permissions which can generate the presigned post in response to a get request via an AWS Lambda function. However if I have the block all privacy setting enabled when I use the post I get access denied. If I turn off block all it works but I am worried about security.
So I will just ask, what is the proper way to configure the security settings for an S3 bucket intended to be the endpoint for a presigned post file upload?
I am new to AWS.
I am learning upload a file from client directly to s3.Here are the steps
Client-1 sends request for file upload.
Server generates pre-signed url,keys etc and sends it to client-1.
Client-1 uploads the files directly to s3 using the keys it received from server.
Now how can server (application) know that upload was successful?
How can server access s3 content (Just like it accesses database content-like how many files are there,etc)?
If client-2 wants to access a file uploaded by client-1,how server can
programmatically give access to client-2 ? (Like expiring
tokens,signed URLs.. Like how Facebook uses a long url with access keys for images,etc)
Thank you for answering!
Amazon S3 can be configured to trigger an AWS Lambda function when a new object is created. It will provide the name of the bucket and object. However, there won't be a clear correlation between the client and the upload unless the pre-signed URL enforces a particular filename.
The server can then access the object in Amazon S3 via normal API calls, just like any other object stored in S3.
If client-2 wants to access the object, and the object is private, the server can generate a pre-signed URL to give client-2 access to the object.
See: Amazon S3 pre-signed URLs
My idea was (is) to create an S3 bucket for allowing users to upload binary objects. The next step would be to confirm the upload and the API will then initiate processing of the file.
To make it more secure the client would first request an upload location. The API then allocates and pre-creates a one-time use directory on S3 for this upload, and sets access policy on that directory to allow a file to be dumped in there (but ideally not be read or even overwritten).
After confirmation by the client the API initiates processing and clean-up.
The problem I'm facing is authentication and authorisation. Simplest would be to allow public write with difficult-to-guess bucket directories, eg
s3://bucket/year/month/day/UUID/UUID/filename
Where the date is added in to allow clean-up later for orphaned files (and should volume grow to require it one can add hours/minutes.
The first UUID is not meaningful other than providing a unique upload location. The second identifies the user.
The entire path is created by the API. The API then allows the user access to write into that final directory. (The user should not be allowed to create this directory).
The question I'm stuck with is that from googling it seems that public writable S3 buckets is considered bad practice, even horribly so.
What alternative do I have?
a) provide the client with some kind of access token?
b) create an IAM account for every uploader (I do not want to be tied to Amazon this way)
c) Any other options?
P.S And is it possible to control the actual file name that the client can use to create a file from the policy?
From what I understand, your goals are to:
Securely allow users to upload specific files to an S3 bucket
Limit access by preventing users from reading or writing other files
Ideally, upload the files directly to S3 without going through your server
You can do this by generating presigned PUT URLs server-side and returning those URLs to the client. The client can use those URLs to upload directly to S3. The client is limited to only the filename you specify when signing the URL. It will be limited to PUT only. You keep your AWS access keys secure on the server and never send it to the client.
If you are using the PutObject API, you only need to sign one URL per file. If you are using the multi-part upload API, it's a bit more complicated and you'll need to start and finish the upload server-side and send presigned UploadPart URLs to the client.