I have a Scheduled Lambda function (via CloudWatch event rule) which is triggered every minute.
This lambda picks up a request from SQS queue, process the parameters and triggers AWS step functions workflow.
Now, ONLY 1 Lambda function instance is running every minute. How can I trigger multiple (e.g. 10) concurrent Lambda functions like this?
One way I can think of is to create 10 Cloudwatch event rule which runs every 1 minute, but I am not sure if that is the right way of doing it. Also, if I use this way, 10 lambda would be called even if I don't have entries in my SQS queue.
You can use the lambda step function.
Event trigger first function. Then it will call multiple functions parallel.
Some useful links:
https://www.youtube.com/watch?v=c797gM0f_Pc
https://medium.com/soluto-nashville/simplifying-workflows-with-aws-step-functions-57d5fad41e59
since your lambda function fetching data from SQS so you can create event source mapping between lambda and SQS so whenever message published to SQS, your lambda function will invoke concurrently depending on number of messages in queue so you do not need to invoke lamnda from cloudwatch event
Related
I know about the provisioned instance configuration for lambda functions. Is it possible to run multiple instances of a lambda function on a timer basis? I know generally we use CloudWatch Events for this, just not how to specify multiple instances.
To be clear, I want something like: I want 10 instances of my function to run at "2022-02-02 10:10:10".
Some options:
Create 10 identical CloudWatch events
Create a new Lambda that is triggered by your single CloudWatch event. The new Lambda would invokes your worker Lambda function 10 times asynchronously
Create a Step Functions state machine that triggers 10 Lambda invocations, and trigger the step function on a schedule
Is it possible to create a lambda which is triggered manually to read from an SQS.
I have an SQS queue which is constantly receiving messages, I want to process them when I want to (not continuous sync processing).
Can I have something like a start/stop lambda, where I start the lambda, and it consumes actively from SQS and when I stop it stops consuming
Normally, an AWS Lambda function is configured to "trigger" from an Amazon SQS queue. Whenever a message arrives, a Lambda function would be triggered.
However, instead of configuring a trigger, you could code an AWS Lambda function to call ReceiveMessages() on the queue itself. The Lambda function would then be responsible for calling DeleteMessage() after the message has been processed.
You can invoke a Lambda function at any time by using the Invoke() command.
The only decision you would need to make is whether the Lambda function should process:
One message per invocation, or
One batch of messages (up to 10) per invocation, or
Run a loop that keeps retrieving messages from the queue until the queue is empty or until the Lambda function times-out (which could leave a message partially processed, so it isn't a good idea)
I have a Lambda function that takes a list of tasks to be run at the time specified. This time can vary.
I am using SNS to trigger another Lambda function that in turn runs the tasks.
These tasks need to be run at specified time. Is it possible to publish a message to SNS using Lambda at the specified time?
Or send the message to SNS, but SNS in turn triggers Lambda at the specified time?
Any option would do.
P.S. I know there is an option of using Cloud Watch events, but I do not want to use any more services.
It appears that your requirement is to trigger an AWS Lambda function at specific times.
This can be done by using Amazon CloudWatch Events, which can take a cron-like expression to run a Lambda function at desired time intervals or specific times. This functionality was originally in the Lambda console, but was moved to CloudWatch Events when more services added scheduling capabilities.
However, CloudWatch Events cannot trigger an Amazon SNS message. If you need SNS to trigger Lambda, then you'll need CloudWatch Events to trigger a Lambda function that sends a message to SNS (which then triggers Lambda functions). Obviously, it would be cleaner to avoid SNS altogether unless you specifically need to fan-out the message to multiple subscriptions/Lambda functions.
I'm using an AWS Lambda function that is triggered from an SNS event trigger to consume from an SQS queue. When the Lambda function executes, it pulls 10 messages from the queue, processes them, pulls another 10, and so on and so forth - up to a certain time limit that's coded into the Lambda function (less than the max of 5 minutes, obviously).
It's my understanding that a Lambda function triggered by an SNS event is one-to-one, is that correct? In other words, one SNS event won't trigger multiple Lambda functions (up to the maximum concurrent execution limit). There's no scaling based on load.
Are there any other potential solutions, leveraging Lambda, that would let me consume from SQS as frequently/fast as possible? I had considered trying to auto-scale my Lambda functions by leveraging CloudWatch alarms (and SNS event triggers) based on SQS queue size, but it seems like those alarms can fire, at most, every 5 minutes. I've also considered developing a master Lambda function that can automatically execute (many) slave Lambdas based on querying the queue size.
I understand that the more optimal design may be to leverage Kinesis instead of SNS. I may consider incorporating Kinesis in the future, but let's just pretend that Kinesis is not an option at this time.
There is no best way to do this. One approach (which you've kind of already mentioned) is to use CloudWatch and schedule a Lambda function to run every minute (that's the minimum schedule time for Lambda). This Lambda function will then look for new SQS messages and invoke other Lambda functions to handle new message(s). Here is a very good article for that use case: https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/
Personally, I do not recommend triggering your Lambda by SNS for this use case, because SNS doesn't give a full guarantee for delivery and recommend sending the SNS notifications to SQS - which does not solve your problem. From the FAQ's:
[...] If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).
Source: https://aws.amazon.com/sns/faqs/
For this kind of processing, instead of SQS if you push messages to Kinesis Stream you should be able to flexibly process(In batches of needed size) the messages.
Note: If you use SQS, after triggering a Lambda function through SNS (or using a Scheduled Lambda), it can invoke inner Lambda functions to check the queue where multiple concurrent inner Lambdas are spawned. However the problem is that its not practical to process SQS items in batches.
I am using AWS Lambda function to process the messages in Queue it's working fine. But i need to execute this Lambda function when messages available or added in SQS queue.
Is it possible to trigger the Lambda function based on SQS queue.Please suggest one method to achieve this goal.
Invoking Lambda functions from SQS queues is not directly supported. You can see the list of available triggers here: http://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html
Possible solutions:
Replace SQS queue with Kinesis or DynamoDB. Both can trigger Lambda functions on updates.
Inject SNS before SQS. SNS can add items to SQS and trigger Lambda function.
If you don't need near real-time processing, these two options are also valid:
Create CloudWatch Event Rule that will trigger the Lambda function every N minutes (e.g. every minute).
Create CloudWatch alarm watching ApproximateNumberOfMessagesVisible parameter for your SQS queue. This alarm should publish to an SNS topic, which in turn will trigger Lambda function.
Lambda now supports SQS as a native event source
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html