I have a Lambda function that takes a list of tasks to be run at the time specified. This time can vary.
I am using SNS to trigger another Lambda function that in turn runs the tasks.
These tasks need to be run at specified time. Is it possible to publish a message to SNS using Lambda at the specified time?
Or send the message to SNS, but SNS in turn triggers Lambda at the specified time?
Any option would do.
P.S. I know there is an option of using Cloud Watch events, but I do not want to use any more services.
It appears that your requirement is to trigger an AWS Lambda function at specific times.
This can be done by using Amazon CloudWatch Events, which can take a cron-like expression to run a Lambda function at desired time intervals or specific times. This functionality was originally in the Lambda console, but was moved to CloudWatch Events when more services added scheduling capabilities.
However, CloudWatch Events cannot trigger an Amazon SNS message. If you need SNS to trigger Lambda, then you'll need CloudWatch Events to trigger a Lambda function that sends a message to SNS (which then triggers Lambda functions). Obviously, it would be cleaner to avoid SNS altogether unless you specifically need to fan-out the message to multiple subscriptions/Lambda functions.
Related
I am wondering about how to configure AWS Lambda, SNS, and SQS for processing background tasks.
There are three ways I thought.
Option 1. A function called consumer can execute workers by receiving tasks from the queue.
Option 2. Send all tasks to SNS. One worker and one SQS receive and work from SNS.
Option 3. Directly forward the task to one SQS and one lambda from the APP.
The biggest concern is whether to directly invoke Lambda in the app or use task consumer using SQS or SNS.
My idea is from Triggering multiple lambda functions from one SQS trigger
It depends on your current and future requirements:
Options 1: Choosing consumer lambda will allow you to add validations and manipulation in the event.
But your consumer lambda will be running until your worker lambdas are running.
Option 2: SNS gives you flexibility to add new events in future and new subscribers as well and your App will have to deal with only SNS.
Option 3: If you are sure in future there will be no such other lambdas. In this case your app need to have the configuration which type of event will go to which SQS.
You can choose any option based on your requirement but I will suggest you to choose option 2 as your app will be required to push notification to SNS only(Single integration). In SNS you can add filters for different types of event.
From SNS you can directly trigger lambda as well.
If you do not need output of lambda functions in your App you should use SNS/SQS for async processing.
The typical pattern is:
Push jobs/tasks to an Amazon SQS queue
Configure an AWS Lambda function to subscribe to the SQS queue
Lambda will automatically execute the Lambda function for each message in the SQS queue
I guess this matches your Option 1, but with the AWS Lambda service acting as the "consumer" that triggers the individual Lambda functions.
If there are different types of inputs (eg three different tasks) that each require a different Lambda function, then create 3 separate queues each linked to its own Lambda function (your Option 3).
Inserting Amazon SNS in-between (shown in your Option 2) makes it easier to 'fork' information, such as adding another subscriber to each message in case they need to be processed in parallel. Otherwise, it is not necessary.
I have a Scheduled Lambda function (via CloudWatch event rule) which is triggered every minute.
This lambda picks up a request from SQS queue, process the parameters and triggers AWS step functions workflow.
Now, ONLY 1 Lambda function instance is running every minute. How can I trigger multiple (e.g. 10) concurrent Lambda functions like this?
One way I can think of is to create 10 Cloudwatch event rule which runs every 1 minute, but I am not sure if that is the right way of doing it. Also, if I use this way, 10 lambda would be called even if I don't have entries in my SQS queue.
You can use the lambda step function.
Event trigger first function. Then it will call multiple functions parallel.
Some useful links:
https://www.youtube.com/watch?v=c797gM0f_Pc
https://medium.com/soluto-nashville/simplifying-workflows-with-aws-step-functions-57d5fad41e59
since your lambda function fetching data from SQS so you can create event source mapping between lambda and SQS so whenever message published to SQS, your lambda function will invoke concurrently depending on number of messages in queue so you do not need to invoke lamnda from cloudwatch event
I have a service which generates a burst of Cloud Watch Events once every hour. These Cloud Watch events (which could be in thousands) each will trigger an AWS Lambda function and ultimately number of concurrent lambdas running can cross the maximum limit. How can I modify my system such that these cloud watch events will be handled gracefully by Lambda functions or if possible somehow I can distribute all these cloud watch events over the rest period of the first service.
P.S. I do not want to modify the limit on concurrent running lambdas.
have you thought about adding these events to SQS instead of consuming Lambda directly and then configure the SQS to call the Lambda function?
This is how you can trigger a Lambda function by SQS queue
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html
and you can define the delay in queue consumption by using this article
https://cloudaffaire.com/how-to-configure-delay-queue-in-sqs/
As prior stated you could use SQS, but I too would advise against this because you would still have the same concurrency issue (more here).
Depending on how quickly you require your processing, it may be a good idea to have lambda "poll" an SQS every couple of minutes, then send the batch of messages to a lambda to process via SNS (rather than have lambda trigger off of SQS PutMessage API Calls).
I'm using an AWS Lambda function that is triggered from an SNS event trigger to consume from an SQS queue. When the Lambda function executes, it pulls 10 messages from the queue, processes them, pulls another 10, and so on and so forth - up to a certain time limit that's coded into the Lambda function (less than the max of 5 minutes, obviously).
It's my understanding that a Lambda function triggered by an SNS event is one-to-one, is that correct? In other words, one SNS event won't trigger multiple Lambda functions (up to the maximum concurrent execution limit). There's no scaling based on load.
Are there any other potential solutions, leveraging Lambda, that would let me consume from SQS as frequently/fast as possible? I had considered trying to auto-scale my Lambda functions by leveraging CloudWatch alarms (and SNS event triggers) based on SQS queue size, but it seems like those alarms can fire, at most, every 5 minutes. I've also considered developing a master Lambda function that can automatically execute (many) slave Lambdas based on querying the queue size.
I understand that the more optimal design may be to leverage Kinesis instead of SNS. I may consider incorporating Kinesis in the future, but let's just pretend that Kinesis is not an option at this time.
There is no best way to do this. One approach (which you've kind of already mentioned) is to use CloudWatch and schedule a Lambda function to run every minute (that's the minimum schedule time for Lambda). This Lambda function will then look for new SQS messages and invoke other Lambda functions to handle new message(s). Here is a very good article for that use case: https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/
Personally, I do not recommend triggering your Lambda by SNS for this use case, because SNS doesn't give a full guarantee for delivery and recommend sending the SNS notifications to SQS - which does not solve your problem. From the FAQ's:
[...] If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).
Source: https://aws.amazon.com/sns/faqs/
For this kind of processing, instead of SQS if you push messages to Kinesis Stream you should be able to flexibly process(In batches of needed size) the messages.
Note: If you use SQS, after triggering a Lambda function through SNS (or using a Scheduled Lambda), it can invoke inner Lambda functions to check the queue where multiple concurrent inner Lambdas are spawned. However the problem is that its not practical to process SQS items in batches.
I am using AWS Lambda function to process the messages in Queue it's working fine. But i need to execute this Lambda function when messages available or added in SQS queue.
Is it possible to trigger the Lambda function based on SQS queue.Please suggest one method to achieve this goal.
Invoking Lambda functions from SQS queues is not directly supported. You can see the list of available triggers here: http://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html
Possible solutions:
Replace SQS queue with Kinesis or DynamoDB. Both can trigger Lambda functions on updates.
Inject SNS before SQS. SNS can add items to SQS and trigger Lambda function.
If you don't need near real-time processing, these two options are also valid:
Create CloudWatch Event Rule that will trigger the Lambda function every N minutes (e.g. every minute).
Create CloudWatch alarm watching ApproximateNumberOfMessagesVisible parameter for your SQS queue. This alarm should publish to an SNS topic, which in turn will trigger Lambda function.
Lambda now supports SQS as a native event source
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html