Is there a way by which I can get notified when a upload is completed in S3 Bucket? The requirement is that I need to provide link to users after uploading of a video is complete in the bucket. By default now I provide link after 30 minutes of start of video, whether video takes 5 minutes to upload or 40 minutes. So is there any way like any API that provides information that the upload has been completed?
Notifications can be triggered in Amazon S3 when any of the following occur:
s3:ObjectCreated:*
s3:ObjectCreated:Put
s3:ObjectCreated:Post
s3:ObjectCreated:Copy
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectRemoved:*
s3:ObjectRemoved:Delete
s3:ObjectRemoved:DeleteMarkerCreated
s3:ReducedRedundancyLostObject
Notifications can be sent via three destinations:
Amazon Simple Notification Service (SNS), which in-turn can send notifications via email, HTTP/S endpoint, SMS, mobile push notification
Amazon Simple Queueing Service (SQS)
Amazon Lambda (not currently available in all regions)
See: Configuring Amazon S3 Event Notifications
The most appropriate choice depends on your programming preference and how your app is written:
Use SNS to push to an HTTP endpoint to trigger some code in your app
Write some code to periodically check an SQS queue
Write a Lambda function in Node.js or Java
Once triggered, your code would then need to identify who uploaded the video, retrieve their user details, then send them an email notification. This would be easiest if you control the key (filename) of the object being uploaded, since this will assist in determining the user to notify.
You can use Amazon Lambda to post a message to Amazon SNS (or notify you any other way) when a file is uploaded to S3.
Setup an S3 trigger to your Lambda function. See this tutorial: http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser.html
Inside your Lambda function, send out your notification. You can use SNS, SES, SQS, etc.
There is no direct method that can tell that whether the upload is complete or not in S3 bucket. You can do a simple thing which I have followed after lot of research and it is working correctly.
Follow this link and read the size of file after every 30 seconds or so as per your requirement when the file size has not changed for two simultaneous readings once again check the size for surety because it might be due to network congestion that size might not have changed for two simultaneous readings.
Related
I have enabled the S3 notification create:* event for SNS sending to my email from AWS CLI in my Windows servers.
However, there are many notification emails received from the SNS after I copied 10000+ local files such as xml, html, jpg to the S3. Because of creating the folder or the files, it is going to send the event with per creating file request to SNS.
I search the topic discussed in Edit event notification prefix and suffix and I think it just partially works on folder and file.
How can just receive one notification email that indicates it's starting the copy job?
How can just receive one notification email that indicates it's starting the copy job?
You can't. This is not how S3 notifications work. You would have to develop custom solution to that. Maybe instead of SNS, have a lambda function to get triggered. The function would somehow verify if this is the first file being uploaded, and send out email only for that file.
I have tried setting up S3 event notifications via SNS topic & was able to successfully get event notifications when objects are created. However in my use case we have large file uploads from the apps that we don't control. These uploads take time. We want to get notified when upload starts (in progress) as well.
I was not able to find any event type that corresponds to upload start!
For large files multipart uploads are used so we get "multipart upload complete" event but still we don't have a clue about when the upload started!
Is there any other way to detect the uploads (start) on AWS S3?
You can create an Amazon CloudWatch Events rule that triggers on CreateMultiPartUpload and sends a message to an Amazon SNS topic:
From CreateMultipartUpload - Amazon Simple Storage Service:
This action initiates a multipart upload and returns an upload ID.
There is a use case I'm working on and I'm not quite sure how it can be solved. The main goal is to upload an image from a react native app to an Amazon S3 bucket using an AWS Lambda function (API Gateway) in order to use Amazon Rekognition service with another Amazon S3 image depending on some values sent to the lambda.
Since the image could be too large I have to use presigned URLs, which means that i make a request to the lambda to get a presigned S3 url to the client so that the client uploads the image to the bucket straight away. But then, how can i use face Rekognition service within the AWS Lambda?
I know i can trigger a lambda after an S3 upload, so i could do the face Rekognition request right after the user makes the http request with the presigned URL, but how can i get face Rekognition service response from that triggered lambda to the original user?
I've thought about SNS, but sending some text message to the user after an image upload instead of a message in the app seems odd.
Thank you in advance and apologies for the long read
You're on the right track with SNS, maybe not with the service, but with the principal.
This is a problem of asynchronous handling of requests and how you can subsequently inform the user of the server's decision. To start, you're async process will need to store the result of the facial recognition somewhere, if you're already using a database (SQL or NoSQL), that would seem to be the place to do this.
Then you have to get the information to the user. Since your user is running a mobile application, there are only two ways of doing this. Either the user will have to poll the back-end service in order to retrieve the result of the async process, or your back-end will need to push the result to the device. Polling the service is straightforward and is usable depending on the load you expect from your application and the duration of the asynchronous process. You can also use long polling to reduce the number of requests, but this doesn't fix the issue (too many users spamming your service waiting for the result) itself.
If you want to notify the users, you will have to create a notification mechanism that is not based on polling a service. You could for example make use of WebSockets, configure your devices to have an MQTT connection (e.g., with AWS IoT) or use another cloud-based notification service that allows you to push messages to the device. You also do not have to include all the information in the message you push to your devices. The pushed message can be a trigger for the device to retrieve the result from the back-end service e.g., using an HTTP API.
I have a node.js backend that sends out images to a secondary api for transformations and then those images appear in s3 bucket. The problem is that the secondary api doesn't inform my api when the file is created in the bucket.
Is there some sort of long polling in s3 available because spamming get requests doesn't feel right (also will get expensive).
I'm considering adding a trigger on new files in s3, that will invoke a lambda that will put a message into some sort of pub/sub message broker and then I could just subscribe to it but this seems a bit too complicated?
From the S3 notification docs you can be notified via:
Amazon Simple Notification Service (Amazon SNS) topic
Amazon Simple Queue Service (Amazon SQS) queue
AWS Lambda
The relative benefits or each one are up to you but don't poll S3 for changes. Use one of these to be notified of the changes. You can decide to get notices for just new objects or deleted object.
I currently have a setup where my mobile front-end performs an AWS s3 upload of an image. The s3 upload triggers a AWS lambda function that starts a AWS step-function (state-machine) which performs various jobs and actions.
I am looking for the best (and most time-efficient) way to get the output at the end of the step-function back to the mobile devise.
One way is to monitor the executionARN of the state machine and, when it is completed, fetch the data. This seems to be the case with awslabs lambda-refarch-imagerecognition implementation here. However, my front-end is on mobile and I would rather not have to send and receive many request to check if the state-machine is finished.
Another possible solution is to refactor the process so that the s3 upload is a stand-alone event and, once it has been successful, make an API request to an AWS API-gateway that triggers the step-function. The API POST request will then return the response. The problem here is that the app must wait for the s3 response to proceed with starting the state-machine.
Is there a a better way to perform this sequence and receive a response. Ideally, the s3 upload would return the full response from the state-machine. This way there one request (image-upload) and one response.
I would use Amazon SNS -> push notifications. You say you want to avoid making "many requests" (and waiting for responses - or polling).
Amazon SNS allows you to publish to a specific topic.
Anything which is "subscribed" to the topic, will (receive a notification / message), whenever one (a stateless update) is published to the topic.
The "mobile front-end" (device - you mention) "would receive the message" / receive push notifications from the SNS endpoint / topic.
This could be triggered when the "state machine" completes, allowing the mobile device to "get a timely update" via a push notification.
This would avoid polling for a response.