I currently have a setup where my mobile front-end performs an AWS s3 upload of an image. The s3 upload triggers a AWS lambda function that starts a AWS step-function (state-machine) which performs various jobs and actions.
I am looking for the best (and most time-efficient) way to get the output at the end of the step-function back to the mobile devise.
One way is to monitor the executionARN of the state machine and, when it is completed, fetch the data. This seems to be the case with awslabs lambda-refarch-imagerecognition implementation here. However, my front-end is on mobile and I would rather not have to send and receive many request to check if the state-machine is finished.
Another possible solution is to refactor the process so that the s3 upload is a stand-alone event and, once it has been successful, make an API request to an AWS API-gateway that triggers the step-function. The API POST request will then return the response. The problem here is that the app must wait for the s3 response to proceed with starting the state-machine.
Is there a a better way to perform this sequence and receive a response. Ideally, the s3 upload would return the full response from the state-machine. This way there one request (image-upload) and one response.
I would use Amazon SNS -> push notifications. You say you want to avoid making "many requests" (and waiting for responses - or polling).
Amazon SNS allows you to publish to a specific topic.
Anything which is "subscribed" to the topic, will (receive a notification / message), whenever one (a stateless update) is published to the topic.
The "mobile front-end" (device - you mention) "would receive the message" / receive push notifications from the SNS endpoint / topic.
This could be triggered when the "state machine" completes, allowing the mobile device to "get a timely update" via a push notification.
This would avoid polling for a response.
Related
There is a use case I'm working on and I'm not quite sure how it can be solved. The main goal is to upload an image from a react native app to an Amazon S3 bucket using an AWS Lambda function (API Gateway) in order to use Amazon Rekognition service with another Amazon S3 image depending on some values sent to the lambda.
Since the image could be too large I have to use presigned URLs, which means that i make a request to the lambda to get a presigned S3 url to the client so that the client uploads the image to the bucket straight away. But then, how can i use face Rekognition service within the AWS Lambda?
I know i can trigger a lambda after an S3 upload, so i could do the face Rekognition request right after the user makes the http request with the presigned URL, but how can i get face Rekognition service response from that triggered lambda to the original user?
I've thought about SNS, but sending some text message to the user after an image upload instead of a message in the app seems odd.
Thank you in advance and apologies for the long read
You're on the right track with SNS, maybe not with the service, but with the principal.
This is a problem of asynchronous handling of requests and how you can subsequently inform the user of the server's decision. To start, you're async process will need to store the result of the facial recognition somewhere, if you're already using a database (SQL or NoSQL), that would seem to be the place to do this.
Then you have to get the information to the user. Since your user is running a mobile application, there are only two ways of doing this. Either the user will have to poll the back-end service in order to retrieve the result of the async process, or your back-end will need to push the result to the device. Polling the service is straightforward and is usable depending on the load you expect from your application and the duration of the asynchronous process. You can also use long polling to reduce the number of requests, but this doesn't fix the issue (too many users spamming your service waiting for the result) itself.
If you want to notify the users, you will have to create a notification mechanism that is not based on polling a service. You could for example make use of WebSockets, configure your devices to have an MQTT connection (e.g., with AWS IoT) or use another cloud-based notification service that allows you to push messages to the device. You also do not have to include all the information in the message you push to your devices. The pushed message can be a trigger for the device to retrieve the result from the back-end service e.g., using an HTTP API.
I have tried to do some R&D but i couldn't find anything useful the only thing that i found is lambda functions is the only way. I want to write a simple application that execute when the Simple Queue Service receive any message , but i couldn't find a way to do that till now, since i don't want to use lambda. for example if i receive some message on Simple Queue Service and while receiving each messages i can trigger a event that is not lambda but instead of that any HTTP-request.
I think your choices are:
use lambda (which you said you didn't want to use, but its probably the best solution)
use your own app running on ec2 or even on premise to consume the message and invoke the http endpoint
use SNS instead of SQS for message delivery - SNS supports http endpoints.
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint.
from here: https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
A short answer to your question is No, (until today)
Let me tell you the sineros i faced.In general Queue triggering lambda is widely used and for that u have to make sure about proper concurrency (minimum 5) in place and also database I/O if u are performing and any DB calls . But I've a scenario where we cannot use "lambda as a triggering service" as our DB is onprem "ORACLE" so the choices are .
Push to "SNS" and make http "what ever applies"(to a container we have custom Kubernetes routed through NLB ).Also make sure you push a batch of messages as it might make more http noise.
2.Poll the queue and perform the operations.
SQS triggering => Lambda and lambda invoking state machine (step functions)
What's the easiest way to save/log every message published on a AWS SNS topic? I thought there might be a magic setting to automatically push them to S3 or a database, or maybe a database service supporting the HTTP destination automatically, but doesn't seem to be the case. Maybe it needs to be done via a Lambda function?
The purpose is just for basic diagnostics and debugging while setting up some SNS publishing. I don't really care about high scale or fast querying, just want to log and perform basic queries on all the activity for a few minutes at a time.
You can setup a trigger to push your SNS messages to SQS queue. Push is automatic and does not require any code.
According to the docs, SNS can publish to:
http – delivery of JSON-encoded message via HTTP POST
https – delivery of JSON-encoded message via HTTPS POST
email – delivery of message via SMTP
email-json – delivery of JSON-encoded message via SMTP
sms – delivery of message via SMS
sqs – delivery of JSON-encoded message to an Amazon SQS queue
application – delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda – delivery of JSON-encoded message to an AWS Lambda function.
So yes, a simple approach would be to trigger a lambda function to write to S3 or CloudWatch.
I see AWS publish API for sending push notifications to devices.
http://docs.aws.amazon.com/sns/latest/api/API_Publish.html
According to:
https://aws.amazon.com/blogs/aws/push-notifications-to-mobile-devices-using-amazon-sns/
We can
"Send messages directly to a specific device by calling the Publish function with the device’s ARN. You can easily scale this to handle millions of users by storing the endpoint ARNs in Amazon DynamoDB and using multi-threaded code on the server."
If I want to send push notifs to 100K users (who haven't registered to a specific topic), is there a multi-publish (or batch-publish) API, where I don't need to call the "Push notifications" API for every single user?
probably not. the devices need to be registered (i.e. for SNS you actually have to create the endpoint for each device).
After you have the endpoint you can subscribe them to either one or multiple SNS endpoint and start publishing notifications through them.
Is there a way by which I can get notified when a upload is completed in S3 Bucket? The requirement is that I need to provide link to users after uploading of a video is complete in the bucket. By default now I provide link after 30 minutes of start of video, whether video takes 5 minutes to upload or 40 minutes. So is there any way like any API that provides information that the upload has been completed?
Notifications can be triggered in Amazon S3 when any of the following occur:
s3:ObjectCreated:*
s3:ObjectCreated:Put
s3:ObjectCreated:Post
s3:ObjectCreated:Copy
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectRemoved:*
s3:ObjectRemoved:Delete
s3:ObjectRemoved:DeleteMarkerCreated
s3:ReducedRedundancyLostObject
Notifications can be sent via three destinations:
Amazon Simple Notification Service (SNS), which in-turn can send notifications via email, HTTP/S endpoint, SMS, mobile push notification
Amazon Simple Queueing Service (SQS)
Amazon Lambda (not currently available in all regions)
See: Configuring Amazon S3 Event Notifications
The most appropriate choice depends on your programming preference and how your app is written:
Use SNS to push to an HTTP endpoint to trigger some code in your app
Write some code to periodically check an SQS queue
Write a Lambda function in Node.js or Java
Once triggered, your code would then need to identify who uploaded the video, retrieve their user details, then send them an email notification. This would be easiest if you control the key (filename) of the object being uploaded, since this will assist in determining the user to notify.
You can use Amazon Lambda to post a message to Amazon SNS (or notify you any other way) when a file is uploaded to S3.
Setup an S3 trigger to your Lambda function. See this tutorial: http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser.html
Inside your Lambda function, send out your notification. You can use SNS, SES, SQS, etc.
There is no direct method that can tell that whether the upload is complete or not in S3 bucket. You can do a simple thing which I have followed after lot of research and it is working correctly.
Follow this link and read the size of file after every 30 seconds or so as per your requirement when the file size has not changed for two simultaneous readings once again check the size for surety because it might be due to network congestion that size might not have changed for two simultaneous readings.