My question is not technical but I am looking for architectural input on which tech stack to use. So my use case is: I have subscribers to whom I send a newsletter daily at some time t. I have configured a lambda function to run and send emails. The issue is, currently per second the lambda function sends 3 email, and given the time limit on lambda is 15 minutes, that means I can send roughly 2700 emails. Clearly if the users scale up to 5k or 10k, lambda function will time out.
So I am looking for alternatives to send the email. I am reluctant to use ec2 instance mainly because of cost reasons and want to use lambda function. The other idea i am currently trying is to create another lambda function, which creates an event stream and sends the event to aws sqs queue, which then triggers the lambda function to send the email, but I am highly doubtful this will scale.
Any technical input is highly appreciated.
If you want to broadcast the same message to all users then you can create an AWS SNS topic.
If you have an email list in S3 or dynamodb then you can create one lambda to read the email ID, then send the email ID to SQS and then create another lambda to send mail.
DynamoDB --> Lambda (to read email-ID) --> SQS ---> Lambda (To send email-ID)
Related
I am developing an email client that uses SMTP server from my customers. I stock the credentials in an AWS RDS database. Customers are able to create mail campaigns, and email have to be sent asynchronously after the campagin creation.
I want to configure a SQS queue to make messages available with a delay of 2 minutes between each message. The purpose is to send the message to a Lambda function able to send the message through SMTP (using the credentials in the database). Thanks to this delay between messages, I can optimize the delivery of the emails. The order of the delivery is not really important.
Unfortunately, I do not find a way to do that. The timers available in SQS are for the entire Queue, or have to be specific for each message. In my case, if i put 100 messages in the SQS queue, I need the first one to be sent immediately, the second 2 minutes after and so on.
Does someone has solutions with this issue ? If SQS is not the right service to manage this need, is there another one available on the AWS platform ?
We can probably achieve this combination of lambda event source mapping config for sqs and lambda reserved concurrency setting.
Event source Mapping Batch size: 1
This will allow only 1 message to be sent to Lambda at once.
Event source Mapping Batch window: 2 mins
This will allow lambda to be invoked only once every 2 minutes.
Lambda Reserved Concurrency as 1
This will allow only one lambda thread to be called at once.
Thanks #luk2302 for your answer, it is the good one accordingly to the AWS support team. Using the EventBridge service to trigger the lambda at define timing works perfectly.
I'm using AWS lambda to build my web app (next.js) and I'm successfully sending email using AWS SES.
I set a CloudWatch event to run every day, that runs a lambda function that sends personal email to some of the users.
The problem is that lambda has running time limit of max 30sec so I can't use it to send emails to a lot of users.
What can I do to solve this issue?
After looking in the CloudWatch logs, I saw it took ~5sec to complete this lambda for ONE user.
So, I'm guessing it will not handle 100 users or more.
At first, it is strange that sending an email takes 5 seconds unless there is a heavy data preparation to format each email.
As for the lambda timeout, to have a scalable design, I recommend that you make lambda A --> SQS ---> lambda B.
Lambda A just writes messages to sqs (recipient's email, template name, whatever necessary info to format the email). Please note that you can write a batch of messages (10 messages) to SQS.. based on one of my previous implementations, it is easy to write 5000+ sqs messages in 30 sec. You can set a longer time up to 15 minutes if lambda is not behind an API.
lambda B will be triggered by SQS to format the email and send it to SES. You can set batch size and sqs config the best suits your design and quota. This way, you can always send emails regardless of the volume as long as quotas of AWS Account allows. Please note that you can also introduce a random delay to each message written to SQS to make sure the limit of SES is not violated (14/sec <== can be raised easily).
you may consider a better design too based on your app architecture :)
I have a scheduled daemon lambda that is called once a day to analyze data and create needed alert messages for the users of the app for the further sending.
The problem I try to solve is to avoid "sparks" in email sending since if I try to send all the created emails at one moment I will get natural reject from SES. So I plan to store created emails somewhere (SQS, DynamoDB, etc) and then schedule a run of another lambda that will take a portion of emails, call SES.sendEmail(), then if there are still emails to send, schedule another call of itself in a few seconds.
I planned to create CloudWatch event inside lambda and use Cron to schedule lambda call but realized that Cron has 1 minute precision, which is not enough for my purposes. I'm not going to wait inside any lambda (since its just pay for nothing approach), so Lambda.Invoke() is also inappropriate. Are there any other alternatives to recursively schedule call of lambda from itself with conditional delay and seconds precision?
You could have a lambda that publishes the emails to sqs, which allows you to set a delivery delay, and have another lambda which is triggered by messages in the queue.
I have a requirement to send email and SMS based on some conditions to users, i want to publish a message to AWS (Any service) with time and message at the time of user creation, is there any way to call a lambda function based on my scheduled time along with message?
Sounds like what you are saying is that you want to store a message and a 'time to send' someplace and then when that time comes, send out that message via SMS and/or SES, correct?
Lots of ways to accomplish it, but one way would be to store your messages into the database of your choice (perhaps dynamodb), and have a lambda function that gets called periodically (every minute or whatever frequency you determine) to find messages that are ready to send.
In this scenario you could use cloudwatch events to call the lambda function at the interval you decide (but no more frequent than once per minute).
Possible enhancement (especially if you have a huge number events) would be to have the lambda function not actually process the sms/ses sends - but just find those messages that are ready to send - and post those messages to an SNS topic and have a different lambda function that takes care of the actual processing (sending) of those messages.
You can use CloudWatch scheduled events for this. It allows you to specify cron expression. The event itself can trigger your lambda that then checks any preconditions you might have and then sends notification via SNS or some other way.
My system run on an Amazon autoscaling group and one feature allows user to user messaging and I have the following use case to resolve.
A new message is sent in my application between users.
A message to notify the the user by e-mail is dropped into a queue with a 60 second delay. This delay allows time for a realtime chat client (faye/angularjs) to see the message and mark it as viewed.
After the delay the message is picked up, the "read" status is checked and if it has not been read by the client an e-mail is dispatched.
Originally I was going to use a cronjob on each application server poll the message queue however it occurs to me it would be more efficient to use SNS to call some kind of e-mail sending endpoint (perhaps in Lambda).
I can't see any way to have SNS poll SQS however, can anybody suggest how this could be done? Essentially I want SNS with a delay so that I don't spam somebody in a "live" chat with e-mail alerts.
Thanks
Unfortunately this is not yet available out of the box. The missing part is the generation of Amazon SNS notifications on message arrival/visibility by an Amazon SQS queue, be it via push (similar to Amazon S3 notifications, or via poll similar to Amazon Kinesis subscriptions (see The Pull/Push Event Models for more on the difference), which would both allow to directly connect an AWS Lambda function to the resp. SQS delay queue events, see e.g.:
Lambda with SQS
That being said, you can work around this limitations in a few ways, for example:
trigger your Lambda function on schedule (e.g. once per minute), and poll your SQS delay queue from there
scheduled Lambda functions are an eagerly awaited missing Lambda feature in turn, but it is more easily worked around, be it either by a cron job of yours, or Eric Hammond's Unreliable Town Clock (UTC) for example
The AWS Lambda team has delivered many/most similar feature requests over recent month' btw., so I would expect them to offer both SQS event handling and scheduled Lambda functions over the course of the year still.
In early 2019, this problem can be solved in a few different ways:
SQS as an Event Source to Lambda (finally announced 2018-06-28),
similar to the OP's original design.
AWS Step Functions (announced 2016-12-01), using a wait step for
the delay.
DynamoDB Streams with Lambda triggers (announced 2017-02-17),
using TTL expiration on items to fire the Lambda trigger.
As SNS has a topic limit of 100,000 per account, I would recommend using Amazon SES to send the emails (62,000 free emails/month could help with implementation cost decisions).