I'm not sure if I understand AWS Lambda - SQS triggers correctly. Can I possibly configure it in such a way that one SQS queue can trigger different lambda functions depending on the message body or a message attribute?
My use case: I have three different lambda functions (processCricket, processFootball, processTennis) each of which perform a unique function. I have a single queue (processGame) which receives messages. Each message on the queue has a attribute "type" which is either "Cricket", "Football" or "Tennis". Can I invoke a different lambda function depending on the "type" on the message?
Option 1: Configure SQS to trigger a different lambda function depending on the type (Not sure if I can do this)
Option 2: Configure one lambda function which can check type and then call the other lambda functions depending on its type
Option 3: Create separate queues for each lambda. Control which lambda processes the message by adding the message to the appropriate queue.
Option 1: Configure SQS to trigger a different lambda function depending on the type
You can't know about the type until it is consumed by the lambda. So this one is not possible.
Option 2: Configure one lambda function which can check type and then call the other lambda functions depending on its type
Yes it is the "possible" way of first option. but it may cost "more" depending on your usage. When you consume the sqs in batch mode, then you have to invoke multiple lambdas by making multiple checks.
Option 3: Create separate queues for each lambda. Control which lambda processes the message by adding the message to the appropriate queue.
In my opinion, this could be the best option. You can configure different DLQ for each queue, set different batch size depending on your business rules, no need for extra lambda to increase "complexity".
You should not configure multiple Lambda functions as triggers for a single SQS queue. This is because the message in SQS will be delivered to any one consumer and while this message is being processed by that consumer, it would not be visible to others. Thus, you wouldn't be able to decide which "type" of message goes to which function, so Option 1 is invalid.
Both Option 2 and 3 should work fine. I would select Option 2 if you do not expect that many messages to be delivered to your queue, thus not having to worry about Lambda scaling. Also note, multiple messages can be delivered in a single batch to the Lambda trigger, so you would have to implement your logic accordingly.
If you're expecting a large number of messages, then Option 3 would be better suited.
Your best option here would be to not send the messages directly to the queue at all. You can use either SNS or EventBridge as the destination for the message. Then you should have one queue for each type of message. You can then subscribe each queue to the source (SNS or EventBridge) and only receive the messages that make sense for that queue. With EventBridge you can do a fair amount of filtering on the entire payload. For SNS you'd need to add the type to the attributes so it can be used for filtering.
You should probably post to an SNS topic and have multiple lambdas subscribe to the topic and process the event as they wish.
Multiple lambda subscribing to SNS with different subscription filters looks best option here.
Benefits.
Minimum infra to manage.
Subscribers (Lambdas) do not have to worry about the filtering after receiving the message, each lambda will get specific message for processing based on the condition given in subscription filters.
SNS will take care of routing part based on the subscription filter.
Only additional piece you have to take care is make the type field available in Message Headers to apply to the subscription filters.
I was asking the same question and found article below while I was considering EventBridge vs SQS.
In short, Option 1 is possible i.e. trigger a specific function based on event type. This came in Nov 2021.
https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/
For more around price comparison see here: https://dev.to/aws-builders/should-we-consider-migrate-to-amazon-eventbridge-from-amazon-sns-sqs--4dgi
Related
As per my existing solution I have two lambda function which gets triggered by the different SQS message and create a folder structure in S3.
Now, I have the requirement where I need to use the single SQS message to trigger both the lambda function.
Is it possible to trigger multiple lambda function via a single SQS message if yes then can you please explain the process and how efficient it would be?
If is there any other approach I can follow please let me know.
Thanks!
No, you can't do that. The best way is to create fan out setup with SNS + two SQS queues.
Otherwise, you have to develop other solution, e.g. one lambda gets triggered by sqs, and then invokes the second one passing the message as input.
I'm having a use case where I have an Amazon SQS fifo queue with lambda function. I need to make sure that fifo triggers the lambda only when the previous lambda execution is completed (also the events come in order). As from aws docs, fifo supports exactly once processing but it does not mention anywhere that it would not push more event on lambda untill the first message is completely processed.
I need to make sure that the next message is processed only when the previous message is completely processed by the lambda function.
Is there are way to ensure that message 2 is only processed by lambda when message 1 is completely processed by lambda?
fifo supports exactly once processing but it does not mention anywhere
that it would not push more event on lambda untill the first message
is completely processed.
SQS never pushes anything anywhere. You have to poll SQS for messages. When you configure Lambda integration with SQS Lambda is actually running a process behind the scenes to poll SQS for you.
AWS FIFO queues allow you to force messages to be processed in order by specifying a Message Group ID. When you specify the same Message Group ID for multiple messages, then the FIFO queue will only make one of those messages available at a time (in first-in-first-out) order. Only after the first message is removed from the queue is the second message made available, etc...
In addition to this, you should configure AWS Lambda SQS integration with a Batch Size of 1, so that it doesn't try to wait for multiple messages to be available before processing. And you could configure the Reserved Concurrency on the Lambda function to 1, as mentioned in the other answer, so that only one instance of the Lambda function can be running at a time.
It is actually pretty easy to do this. It is not clarified, since it will by default simply use up the available account concurrency and handle as many messages in parallel as is possible.
You can influence this by setting the reserved concurrency for the lambda function to 1. This will ensure no more than 1 lambda function will be executed at the same time.
I am wondering about how to configure AWS Lambda, SNS, and SQS for processing background tasks.
There are three ways I thought.
Option 1. A function called consumer can execute workers by receiving tasks from the queue.
Option 2. Send all tasks to SNS. One worker and one SQS receive and work from SNS.
Option 3. Directly forward the task to one SQS and one lambda from the APP.
The biggest concern is whether to directly invoke Lambda in the app or use task consumer using SQS or SNS.
My idea is from Triggering multiple lambda functions from one SQS trigger
It depends on your current and future requirements:
Options 1: Choosing consumer lambda will allow you to add validations and manipulation in the event.
But your consumer lambda will be running until your worker lambdas are running.
Option 2: SNS gives you flexibility to add new events in future and new subscribers as well and your App will have to deal with only SNS.
Option 3: If you are sure in future there will be no such other lambdas. In this case your app need to have the configuration which type of event will go to which SQS.
You can choose any option based on your requirement but I will suggest you to choose option 2 as your app will be required to push notification to SNS only(Single integration). In SNS you can add filters for different types of event.
From SNS you can directly trigger lambda as well.
If you do not need output of lambda functions in your App you should use SNS/SQS for async processing.
The typical pattern is:
Push jobs/tasks to an Amazon SQS queue
Configure an AWS Lambda function to subscribe to the SQS queue
Lambda will automatically execute the Lambda function for each message in the SQS queue
I guess this matches your Option 1, but with the AWS Lambda service acting as the "consumer" that triggers the individual Lambda functions.
If there are different types of inputs (eg three different tasks) that each require a different Lambda function, then create 3 separate queues each linked to its own Lambda function (your Option 3).
Inserting Amazon SNS in-between (shown in your Option 2) makes it easier to 'fork' information, such as adding another subscriber to each message in case they need to be processed in parallel. Otherwise, it is not necessary.
I need to call AWS Lambda hundreds of thousands times, with different parameters. Is there a way to somehow execute it in bulk, passing a long list of parameters or e.g. a path to an S3 object with one line per payload?
You can pass all of the parameters through an Amazon's SQS. You can specify the batch size (i.e the number of messages that would be sent simultaneously in a group). Only issue is that the maximum number of messages sent at once is 10. Although, this should still be more efficient than processing them one by one. Alternatively you can also encode all the parameters within a single message, just keep in mind that maximum message size is 256KB.
A common technique is:
Create an Amazon SQS queue
Configure the queue to trigger your AWS Lambda function
Send messages to the queue. Each message would contain a set of input parameters that will be used by the Lambda function.
The format of the parameters is your choice because you will need to write code in the Lambda function to retrieve the parameters from the event record that is passed to the handler function.
You can configure a batch size which controls how many messages are sent to the Lambda function. You can set it to 1 to process a single message, or make it bigger and have the Lambda function loop through the messages that have been provided.
I recommend that you test the process with just a few messages before putting all the messages in the queue.
See:
Using AWS Lambda with Amazon SQS - AWS Lambda
AWS Lambda function scaling - AWS Lambda
I need to call AWS Lambda hundreds of thousands times, with different parameters.
for this you can use aws EventBridge where you can schedule a lambda as per your convenience using cron expression ( for fine granularity), or if you want to send events from different aws sources or third party that can also be done using an event pattern.
after selecting target as aws lambda you can configure input as constant jsontext,matched event, part of matched event or Input transformer
note:- eventbroidge also allows to retry policy and deal letter queue
I have this architectural dilemma, and thought, maybe this is solved problem, or solvable multiple ways. I've got an SQS queue (one of many), which is triggering a Lambda function (one of many). This queue requires slightly different processing of messages, based on one key in the payload. What would be the best way for sorting out the queue messages before actual processing by Lambda?
Should it be separate Lambda, which will check for the key, and then place the message into separate queue, which will trigger corresponding Lambda?
Should it be just bunch of if statements in primary Lambda?
Is there maybe automated way to deal with such situation?
Thanks!
It appears that your situation is:
An external process sends messages to an Amazon SNS topic
An Amazon SQS queue is subscribed to the SNS topic
An AWS Lambda function is subscribed to the SQS queue
There are various sub-types of messages, each of which should be processed slightly differently
One option is to use Amazon SNS Message Filtering, which can deliver messages differently based upon a message attribute. It could, for example, send a subset of messages to an SNS topic or Lambda function. However, this would require the messages being sent to the SNS topic to have a message attribute defined.
If this is not the case, then possible options are:
Use one Lambda function to process all message types (You can include quite complex code in multiple files within a Lambda function!), or
Use a Lambda function to determine the message sub-type and then send it to a specific Lambda function (or, send it via an SQS queue to a Lambda function for added resiliency)