I have a requirement to send email and SMS based on some conditions to users, i want to publish a message to AWS (Any service) with time and message at the time of user creation, is there any way to call a lambda function based on my scheduled time along with message?
Sounds like what you are saying is that you want to store a message and a 'time to send' someplace and then when that time comes, send out that message via SMS and/or SES, correct?
Lots of ways to accomplish it, but one way would be to store your messages into the database of your choice (perhaps dynamodb), and have a lambda function that gets called periodically (every minute or whatever frequency you determine) to find messages that are ready to send.
In this scenario you could use cloudwatch events to call the lambda function at the interval you decide (but no more frequent than once per minute).
Possible enhancement (especially if you have a huge number events) would be to have the lambda function not actually process the sms/ses sends - but just find those messages that are ready to send - and post those messages to an SNS topic and have a different lambda function that takes care of the actual processing (sending) of those messages.
You can use CloudWatch scheduled events for this. It allows you to specify cron expression. The event itself can trigger your lambda that then checks any preconditions you might have and then sends notification via SNS or some other way.
Related
I have a situation in which I have a message published to sns topic, an sqs queue subscribed to receive all messages sent to the topic, then a lambda that polls the sqs for messages, the performs some processing (in this case saving to dynamodb). Then there is another lambda that gets triggered off the dynamodb change stream.
For a particular event (identified by a unique id) it would be nice to able to visualize where the event is in this process (i.e. whether it is in the first queue, whether it is in a dlq, etc). I know i can use x-ray or structured logs combined with queries to find this information, but I was wondering if I could use step functions to do this instead.
I know I can send messages to sns with step functions, and I know I can have the next step after that being receive messages from sqs and invoke lambda. However, this seems to just call "receive message" once on the queue, and doesn't connect to the specific event (which would be the point of the visualization).
Does anyone know if this is possible with step functions, or would I need to build my own ui or rely on x-ray or cloudwatch logs insights queries?
Thanks
I am developing an email client that uses SMTP server from my customers. I stock the credentials in an AWS RDS database. Customers are able to create mail campaigns, and email have to be sent asynchronously after the campagin creation.
I want to configure a SQS queue to make messages available with a delay of 2 minutes between each message. The purpose is to send the message to a Lambda function able to send the message through SMTP (using the credentials in the database). Thanks to this delay between messages, I can optimize the delivery of the emails. The order of the delivery is not really important.
Unfortunately, I do not find a way to do that. The timers available in SQS are for the entire Queue, or have to be specific for each message. In my case, if i put 100 messages in the SQS queue, I need the first one to be sent immediately, the second 2 minutes after and so on.
Does someone has solutions with this issue ? If SQS is not the right service to manage this need, is there another one available on the AWS platform ?
We can probably achieve this combination of lambda event source mapping config for sqs and lambda reserved concurrency setting.
Event source Mapping Batch size: 1
This will allow only 1 message to be sent to Lambda at once.
Event source Mapping Batch window: 2 mins
This will allow lambda to be invoked only once every 2 minutes.
Lambda Reserved Concurrency as 1
This will allow only one lambda thread to be called at once.
Thanks #luk2302 for your answer, it is the good one accordingly to the AWS support team. Using the EventBridge service to trigger the lambda at define timing works perfectly.
I have a question related to AWS on how to trigger a Lambda function based on time.
I have a booking system where users can book a ride. I want to send an automated notification for them e.g. 10 mins from the time rider should arrive in case the rider is late.
How can I trigger the lambda function to run (as an example 02:30 and the current time is 12:00)
Presumably you could have hundreds of users who need to receive notifications at any particular time. Therefore, merely triggering the Lambda function at a particular time probably isn't the way to accomplish this.
Instead, you could:
Use a database to maintain the times that users should be notified
Trigger an AWS Lambda function every minute
The Lambda function would:
Consult the database to find any notification that is due to be sent now (or previously) and has not already been sent
Send the notification
Mark the notification as sent (or delete it from the database)
Note that the function should look for any notification that is due, even if it was for a previous time. This will handle situations where notifications were not successfully sent previously, or where the Lambda function doesn't run for some reason.
The Lambda function can be triggered on a schedule by using Amazon CloudWatch Events.
I have a scheduled daemon lambda that is called once a day to analyze data and create needed alert messages for the users of the app for the further sending.
The problem I try to solve is to avoid "sparks" in email sending since if I try to send all the created emails at one moment I will get natural reject from SES. So I plan to store created emails somewhere (SQS, DynamoDB, etc) and then schedule a run of another lambda that will take a portion of emails, call SES.sendEmail(), then if there are still emails to send, schedule another call of itself in a few seconds.
I planned to create CloudWatch event inside lambda and use Cron to schedule lambda call but realized that Cron has 1 minute precision, which is not enough for my purposes. I'm not going to wait inside any lambda (since its just pay for nothing approach), so Lambda.Invoke() is also inappropriate. Are there any other alternatives to recursively schedule call of lambda from itself with conditional delay and seconds precision?
You could have a lambda that publishes the emails to sqs, which allows you to set a delivery delay, and have another lambda which is triggered by messages in the queue.
I have the following infrastructure:
I have an EC2 instance with a NodeJS+Express process listening on a port for messages (process 1). Every time the process receives a message it sends it to an SQS queue. Then I have another process in the same machine reading the queue using long polling (process 2). When it finds a message in the queue it inserts the data in a MariaDB database sitting on an RDS instance.
(Just to clarify, messages are generated by users, they send a chunk of data which can contain arbitrary information to the endpoint where the process 1 is listening)
Now I want to put the process that reads the SQS (process 2) in a Lambda function so that the process that writes to the queue and the one that reads from the queue are completely independent. The problem is that I don't know if this is possible.
I know that Lambda function are invoked in response to an event, and the events supported at the moment are S3, SNS, SES, DynamoDB, Kinesis, Cognito, CloudWatch and Cloudformation but NOT SQS.
I was thinking in using SNS notifications to invoke the Lambda function so that every time a message is pushed to the queue, an SNS notification is fired and invokes the Lambda function but after playing a bit with it I've realised that is not possible to create an SNS notification from SQS, it's only possible to write SNS notifications to the queue.
Right now I'm a bit stuck because I don't know how to continue. I have the feeling that is not possible to create this infrastructure due to the current limitations in the AWS services. Is there another way to do what I want or am I in a dead-end?
Just to extend my question with some research I've made, this github repo shows how to read an SQS queu from a Lambda function but the lambda function works only if is fired from the command line:
https://github.com/robinjmurphy/sqs-to-lambda
In the readme, the author mentions the following:
Update: Lambda now supports SNS notifications as an event source,
which makes this hack entirely unneccessary for SNS notifcations. You
might still find it useful if you like the idea of using a Lambda
function to process jobs on an SQS queue.
But I think this doesn't solve my problem, an SNS notification can invoke the Lambda function but I don't see how I can create a notification when a message is received in the SQS queue.
Thanks
There are couple of Strategies which can be used to connect the dots, (A)Synchronously or Run-Sleep-Run to keep the data process flow between SNS, SQS, Lambda.
Strategy 1 : Have a Lambda function listen to SNS and process it in real time [Please note that an SQS Queue can subscribe to an SNS Topic - which would may be helpful for logging / auditing / retry handling]
Strategy 2 : Given that you are getting data sourced to SQS Queue. You can try with 2 Lambda Functions [Feeder & Worker].
Feeder would be scheduled lambda function whose job is to take items
from SQS (if any) and push it as an SNS topic (and continue doing it forever)
Worker would be linked to listen the SNS topic which would do the actual data processing
We can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.
Further details:
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
AWS SQS is one of the oldest products of Amazon, which only supported polling (long and short) up until June 2018. As mentioned in this answer, AWS SQS now supports the feature of triggering lambda functions on new message arrival in SQS. A complete tutorial for this is provided in this document.
I used to tackle this problem using different mechanisms, and given below are some approaches you can use.
You can develop a simple polling application in Lambda, and use AWS CloudWatch to invoke it every 5 mins or so. You can make this near real-time by using CloudWatch events to invoke lambda with short downtimes. Use this tutorial or this tutorial for this purpose. (This could cost more on Lambdas)
You can consider that SQS is redundant if you don't need to persist the messages nor guarantee the order of delivery. You can use AWS SNS (Simple Notification Service) to directly invoke a lambda function and do whatever the processing required. Use this tutorial for this purpose. This will happen in real-time. But the main drawback is the number of lambdas that can be initiated per region at a given time. Please read this and understand the limitation before following this approach. Nevertheless AWS SNS Guarantees the order of delivery. Also SNS can directly call an HTTP endpoint and store the message in your DB.
I had a similar situation (and now have a working solution deploed). I have addressed it in a following manner:
i.e. publishing events to SNS; which then get fanned-out to Lambda and SQS.
NOTE: This is not applicable to the events that have to be processed in a certain order.
That there are some gotchas (w/ possible solutions) such as:
racing condition: lambda might get invoked before messages is deposited into the queue
distributed nature of SQS queue may lead to returning no messages even though there is a message note1.
The solution to both cases would be to do long-polling of SQS queue; but this does make your lambda bill more expensive.
note1
Short poll is the default behavior where a weighted random set of machines is sampled on a ReceiveMessage call. This means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
We had some similar requirements so we ended up building a library and open sourcing it to help with SQS to Lambda async. I'm not sure if this fills your particular set of requirements, but thought it might be worth a look: https://read.iopipe.com/sqs-lambda-teaming-up-92c4096be49c