I created an email form on my website that calls an API Gateway endpoint as my HTML form action. It delivers the payload (a few lines of text generally) to the endpoint which triggers my AWS Lambda func. This works as planned, but it's a little slow (2-5 seconds) as sending email via SES takes a few seconds.
I'd like to use an in-memory datastore like Redis or Memcached to just set the data and close the Lambda func., but this seems expensive for my limited use case - I get 10-15 emails per month.
Is a better use case delivering the payload to an API Gateway endpoint - same as before - but have the AWS Lambda func. save the data immediately to an AWS DynamoDb instance which then closes the connection (terminates the AWS Lambda func.) ... and behind the scenes a second AWS Lambda func. would invoke/trigger that would then deliver the email to the appropriate account?
The delay I'm having appears to be the actual sending of the email using AWS SES so I'm trying to make this faster. I can do the above or is there a better way to invoke an SES instance to send email ... maybe async. somehow?
The usual pattern for this sort of thing is to push the data into a queue and have a second lambda consume it (to actually send the email). For this volume, the free tier should be plenty :)
Related
I would normally handle the task of sending an email after a new DynamoDB entry with Lambda and SES but I'm required to not use Lambda for it.
There's a 'Contact us' section in the website and the email needs to be sent every time a new entry is made. We use API Gateway to post the data to DyanmoDB
Is there a way to carry this out without Lambda?
It's not possible without writing code. Furthermore you may probably want to tailor each email to make it more personal to the user, thus Lambda is a must.
You could design something using EventBridge Pipes which can trigger when a new user is added and can have SNS as a destination which could trigger an email. But that email may not be customizable nor may it send to people not subscribed to the topic
DynamoDB triggers Lambda functions by feeding the records of a DynamoDB Stream to the Lambda function. That is by far the easiest way to process updates to a DynamoDB table, but you can also write other code that processes a DynamoDB outside of Lambda: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
If you really want to do it without lambda you can use the following pattern and glue some AWS services together.
You can consume the dynamodb stream with eventbridge pipes and send it to a stepfunction and there you use the sdk integration to call SES. Instead of stepfunction you can directly use SNS for simpler setups.
You will not have to use lambda for this setup. Further you can transform your message either in the pipe or in the stepfunction.
Be aware that eventbirdge pipes has currently no AWS CDK L2 construct which might make it harder to configure if you use CDK.
There is a use case I'm working on and I'm not quite sure how it can be solved. The main goal is to upload an image from a react native app to an Amazon S3 bucket using an AWS Lambda function (API Gateway) in order to use Amazon Rekognition service with another Amazon S3 image depending on some values sent to the lambda.
Since the image could be too large I have to use presigned URLs, which means that i make a request to the lambda to get a presigned S3 url to the client so that the client uploads the image to the bucket straight away. But then, how can i use face Rekognition service within the AWS Lambda?
I know i can trigger a lambda after an S3 upload, so i could do the face Rekognition request right after the user makes the http request with the presigned URL, but how can i get face Rekognition service response from that triggered lambda to the original user?
I've thought about SNS, but sending some text message to the user after an image upload instead of a message in the app seems odd.
Thank you in advance and apologies for the long read
You're on the right track with SNS, maybe not with the service, but with the principal.
This is a problem of asynchronous handling of requests and how you can subsequently inform the user of the server's decision. To start, you're async process will need to store the result of the facial recognition somewhere, if you're already using a database (SQL or NoSQL), that would seem to be the place to do this.
Then you have to get the information to the user. Since your user is running a mobile application, there are only two ways of doing this. Either the user will have to poll the back-end service in order to retrieve the result of the async process, or your back-end will need to push the result to the device. Polling the service is straightforward and is usable depending on the load you expect from your application and the duration of the asynchronous process. You can also use long polling to reduce the number of requests, but this doesn't fix the issue (too many users spamming your service waiting for the result) itself.
If you want to notify the users, you will have to create a notification mechanism that is not based on polling a service. You could for example make use of WebSockets, configure your devices to have an MQTT connection (e.g., with AWS IoT) or use another cloud-based notification service that allows you to push messages to the device. You also do not have to include all the information in the message you push to your devices. The pushed message can be a trigger for the device to retrieve the result from the back-end service e.g., using an HTTP API.
I have tried to do some R&D but i couldn't find anything useful the only thing that i found is lambda functions is the only way. I want to write a simple application that execute when the Simple Queue Service receive any message , but i couldn't find a way to do that till now, since i don't want to use lambda. for example if i receive some message on Simple Queue Service and while receiving each messages i can trigger a event that is not lambda but instead of that any HTTP-request.
I think your choices are:
use lambda (which you said you didn't want to use, but its probably the best solution)
use your own app running on ec2 or even on premise to consume the message and invoke the http endpoint
use SNS instead of SQS for message delivery - SNS supports http endpoints.
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint.
from here: https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
A short answer to your question is No, (until today)
Let me tell you the sineros i faced.In general Queue triggering lambda is widely used and for that u have to make sure about proper concurrency (minimum 5) in place and also database I/O if u are performing and any DB calls . But I've a scenario where we cannot use "lambda as a triggering service" as our DB is onprem "ORACLE" so the choices are .
Push to "SNS" and make http "what ever applies"(to a container we have custom Kubernetes routed through NLB ).Also make sure you push a batch of messages as it might make more http noise.
2.Poll the queue and perform the operations.
SQS triggering => Lambda and lambda invoking state machine (step functions)
I currently have a setup where my mobile front-end performs an AWS s3 upload of an image. The s3 upload triggers a AWS lambda function that starts a AWS step-function (state-machine) which performs various jobs and actions.
I am looking for the best (and most time-efficient) way to get the output at the end of the step-function back to the mobile devise.
One way is to monitor the executionARN of the state machine and, when it is completed, fetch the data. This seems to be the case with awslabs lambda-refarch-imagerecognition implementation here. However, my front-end is on mobile and I would rather not have to send and receive many request to check if the state-machine is finished.
Another possible solution is to refactor the process so that the s3 upload is a stand-alone event and, once it has been successful, make an API request to an AWS API-gateway that triggers the step-function. The API POST request will then return the response. The problem here is that the app must wait for the s3 response to proceed with starting the state-machine.
Is there a a better way to perform this sequence and receive a response. Ideally, the s3 upload would return the full response from the state-machine. This way there one request (image-upload) and one response.
I would use Amazon SNS -> push notifications. You say you want to avoid making "many requests" (and waiting for responses - or polling).
Amazon SNS allows you to publish to a specific topic.
Anything which is "subscribed" to the topic, will (receive a notification / message), whenever one (a stateless update) is published to the topic.
The "mobile front-end" (device - you mention) "would receive the message" / receive push notifications from the SNS endpoint / topic.
This could be triggered when the "state machine" completes, allowing the mobile device to "get a timely update" via a push notification.
This would avoid polling for a response.
I have been experimenting with AWS API Gateway and AWS Lambda to try out a serverless architecture. Have been going through blogs and AWS documentation. Have tried out sample GET/POST. But, I have the following requirement w.r.t tracking user events from my custom application
Events are posted from my application to API end point
I wanted the API to respond back with a custom response (Say {'fine'})
(acknowledging that the request has been received)
After the response is sent, hand over the event payload to a AWS Lambda function
As per the documentation, I understand,
a) I can post events to API end point
b) On GET/POST trigger an AWS Lambda Function
- Respond back from AWS Lambda function to API request
I wanted to change the above and modify it to
a) Post events to API end point
a.0) Respond back acknowledging that request is received [Say {'fine'} ]
b) Trigger AWS Lambda function to process the event payload
Please share across suggestions on how to achieve the same.
Another asynchronous model many customers have used:
Set up an API configured to send requests to Amazon Kinesis. This API could acknowledge the request.
Set up AWS Lambda to consume your Kinesis stream.
This setup has some advantages for high workload APIs as fetches from the Kinesis stream can be batched and don't require a 1-to-1 scaling of both your API Gateway limits and Lambda limits.
Update
To answer your questions about scalability:
Kinesis
Kinesis scales by adding what it calls "shards" to the stream. Each shard handles a portion of your traffic, based on a partition key. Each shard scales up to 1000 rps or 1MBps (see limits). Even with the lower default 25 shards, this would support up to 25,000 rps or 25MBps with an evenly distributed partition key.
API Gateway
API Gateway has a default account level limit of 500 rps, but this can easily be extended by requesting a limit increase. We have customers in production that are using the service at limits above your current suggested scale.
If you want a fast response from the API and not have to wait for the processing of data, you could:
post an event to an API Gateway endpoint
trigger an AWS Lambda Function A
call asynchronously a Lambda Function B using the AWS SDK in the Lambda Function A
Call context.succeed() or context.done() or the callback function in the Lambda Function A so it respond back to API Gateway
the Lambda Function B can process the data while API Gateway already received a response
You should first run some tests to see what type of real world response times you are getting from having your lambda function complete all the logic. If the times are above what you feel are acceptable for your use case, here is another asynchronous solution utilizing an SNS Topic to trigger a secondary Lambda function.
Client Request to API Gateway -> Calls Lambda function A
Lambda A verifies payload and then publishes to SNS Topic X
Lambda A returns {fine} success message -> API Gateway -> client
SNS Topic X triggers Lambda function B
Lambda function B implements given logic