I have this architectural dilemma, and thought, maybe this is solved problem, or solvable multiple ways. I've got an SQS queue (one of many), which is triggering a Lambda function (one of many). This queue requires slightly different processing of messages, based on one key in the payload. What would be the best way for sorting out the queue messages before actual processing by Lambda?
Should it be separate Lambda, which will check for the key, and then place the message into separate queue, which will trigger corresponding Lambda?
Should it be just bunch of if statements in primary Lambda?
Is there maybe automated way to deal with such situation?
Thanks!
It appears that your situation is:
An external process sends messages to an Amazon SNS topic
An Amazon SQS queue is subscribed to the SNS topic
An AWS Lambda function is subscribed to the SQS queue
There are various sub-types of messages, each of which should be processed slightly differently
One option is to use Amazon SNS Message Filtering, which can deliver messages differently based upon a message attribute. It could, for example, send a subset of messages to an SNS topic or Lambda function. However, this would require the messages being sent to the SNS topic to have a message attribute defined.
If this is not the case, then possible options are:
Use one Lambda function to process all message types (You can include quite complex code in multiple files within a Lambda function!), or
Use a Lambda function to determine the message sub-type and then send it to a specific Lambda function (or, send it via an SQS queue to a Lambda function for added resiliency)
Related
I want to use a native AWS queuing mechanism to implement a pub/sub process where there one publisher and multiple consumers, but for each message, there is only a single possible receiver. Is this possible using SQS or SNS, or do these always go to all consumers? Would I need to go to Amazon-MQ? Or maybe a combination of SNS and an ELB?
Thanks,
Stan
The default (simple) implementation sens all SNS messages to all subscribers (in your case SQS).
Since I don't know what exactly is your case, I would say this.
If the message content determines which subscriber should get the message, then use SNS filters.
If there is some logic or rules that need to be built, use MQ.
In my application we are using a SQS to queue messages to be processed by another module. SQS doesn't send notification that a message has come and I don't want to make my application to go to check on it every "X times". So I'm trying to use a lambda trigger to make a http request to my module and make it pool messages from SQS when a message got there.
The problem is SQS deletes the sent messages if there is no error on the lambda function (as far I know). Forcing an error just to keep the messages on the pool can't be right. So I need a way to keep messages on the SQS after the lambda was triggered.
Maybe I should move the code that process the message to the lambda function, but I'm looking for ways to keep it there.
Anyone could give some guidance?
Thanks in advance
SQS is built to be a single producer to consumer for its queues so the intended functionality is happening.
However, there is a solution available for this exact scenario but it will require you to update your architecture.
The solution is to use a fanout architecture.
You would instead publish to an SNS topic, which has your SQS queue subscribed to it. Then create additional SQS queues for parallel channels (1 per each unique Lambda).
Add each Lambda function as consumer of its own SQS queue, each with their own processing.
I'm not sure if I understand AWS Lambda - SQS triggers correctly. Can I possibly configure it in such a way that one SQS queue can trigger different lambda functions depending on the message body or a message attribute?
My use case: I have three different lambda functions (processCricket, processFootball, processTennis) each of which perform a unique function. I have a single queue (processGame) which receives messages. Each message on the queue has a attribute "type" which is either "Cricket", "Football" or "Tennis". Can I invoke a different lambda function depending on the "type" on the message?
Option 1: Configure SQS to trigger a different lambda function depending on the type (Not sure if I can do this)
Option 2: Configure one lambda function which can check type and then call the other lambda functions depending on its type
Option 3: Create separate queues for each lambda. Control which lambda processes the message by adding the message to the appropriate queue.
Option 1: Configure SQS to trigger a different lambda function depending on the type
You can't know about the type until it is consumed by the lambda. So this one is not possible.
Option 2: Configure one lambda function which can check type and then call the other lambda functions depending on its type
Yes it is the "possible" way of first option. but it may cost "more" depending on your usage. When you consume the sqs in batch mode, then you have to invoke multiple lambdas by making multiple checks.
Option 3: Create separate queues for each lambda. Control which lambda processes the message by adding the message to the appropriate queue.
In my opinion, this could be the best option. You can configure different DLQ for each queue, set different batch size depending on your business rules, no need for extra lambda to increase "complexity".
You should not configure multiple Lambda functions as triggers for a single SQS queue. This is because the message in SQS will be delivered to any one consumer and while this message is being processed by that consumer, it would not be visible to others. Thus, you wouldn't be able to decide which "type" of message goes to which function, so Option 1 is invalid.
Both Option 2 and 3 should work fine. I would select Option 2 if you do not expect that many messages to be delivered to your queue, thus not having to worry about Lambda scaling. Also note, multiple messages can be delivered in a single batch to the Lambda trigger, so you would have to implement your logic accordingly.
If you're expecting a large number of messages, then Option 3 would be better suited.
Your best option here would be to not send the messages directly to the queue at all. You can use either SNS or EventBridge as the destination for the message. Then you should have one queue for each type of message. You can then subscribe each queue to the source (SNS or EventBridge) and only receive the messages that make sense for that queue. With EventBridge you can do a fair amount of filtering on the entire payload. For SNS you'd need to add the type to the attributes so it can be used for filtering.
You should probably post to an SNS topic and have multiple lambdas subscribe to the topic and process the event as they wish.
Multiple lambda subscribing to SNS with different subscription filters looks best option here.
Benefits.
Minimum infra to manage.
Subscribers (Lambdas) do not have to worry about the filtering after receiving the message, each lambda will get specific message for processing based on the condition given in subscription filters.
SNS will take care of routing part based on the subscription filter.
Only additional piece you have to take care is make the type field available in Message Headers to apply to the subscription filters.
I was asking the same question and found article below while I was considering EventBridge vs SQS.
In short, Option 1 is possible i.e. trigger a specific function based on event type. This came in Nov 2021.
https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/
For more around price comparison see here: https://dev.to/aws-builders/should-we-consider-migrate-to-amazon-eventbridge-from-amazon-sns-sqs--4dgi
guys need small help, I have a use case, where I want to set up a communication service.
using SQS, SQs is going to receive a different type of events to be communicated. Now we have a single lambda function which does a single communication. let's say one email Lambda, Slack lambda, etc.
how I can invoke different lambda based on queue attributes. I was planning to use SQS as an event source and something kind of this architecture link to sample architeture
here in the above, we can handle rate limiting and concurrency at the lambda service level
simplified works if event type is A invoke Lambda A if the event type is B invoke a lambda B
and both events are in same SQS
all suggestions are welcome
Your problem is a SQS message can only be read by one service at a time. When it is being read, it is invisible to anyone else. You can only have one Lambda consumer and there isn't any partitioning or routing in SQS besides setting up another SQS topic. Multiple consumers are implemented Kensis or AWS MSK (Kafka)
What you are trying to accomplish is called a fan out. This is a common cloud architecture. What you probably want to do is publish initially to SNS. Then with SNS you can filter and route to multiple SQS topics for each of the message types and each SQS topic would then be consumed by it's own Lambda.
Check out a tutorial here:
https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html
I have the following infrastructure:
I have an EC2 instance with a NodeJS+Express process listening on a port for messages (process 1). Every time the process receives a message it sends it to an SQS queue. Then I have another process in the same machine reading the queue using long polling (process 2). When it finds a message in the queue it inserts the data in a MariaDB database sitting on an RDS instance.
(Just to clarify, messages are generated by users, they send a chunk of data which can contain arbitrary information to the endpoint where the process 1 is listening)
Now I want to put the process that reads the SQS (process 2) in a Lambda function so that the process that writes to the queue and the one that reads from the queue are completely independent. The problem is that I don't know if this is possible.
I know that Lambda function are invoked in response to an event, and the events supported at the moment are S3, SNS, SES, DynamoDB, Kinesis, Cognito, CloudWatch and Cloudformation but NOT SQS.
I was thinking in using SNS notifications to invoke the Lambda function so that every time a message is pushed to the queue, an SNS notification is fired and invokes the Lambda function but after playing a bit with it I've realised that is not possible to create an SNS notification from SQS, it's only possible to write SNS notifications to the queue.
Right now I'm a bit stuck because I don't know how to continue. I have the feeling that is not possible to create this infrastructure due to the current limitations in the AWS services. Is there another way to do what I want or am I in a dead-end?
Just to extend my question with some research I've made, this github repo shows how to read an SQS queu from a Lambda function but the lambda function works only if is fired from the command line:
https://github.com/robinjmurphy/sqs-to-lambda
In the readme, the author mentions the following:
Update: Lambda now supports SNS notifications as an event source,
which makes this hack entirely unneccessary for SNS notifcations. You
might still find it useful if you like the idea of using a Lambda
function to process jobs on an SQS queue.
But I think this doesn't solve my problem, an SNS notification can invoke the Lambda function but I don't see how I can create a notification when a message is received in the SQS queue.
Thanks
There are couple of Strategies which can be used to connect the dots, (A)Synchronously or Run-Sleep-Run to keep the data process flow between SNS, SQS, Lambda.
Strategy 1 : Have a Lambda function listen to SNS and process it in real time [Please note that an SQS Queue can subscribe to an SNS Topic - which would may be helpful for logging / auditing / retry handling]
Strategy 2 : Given that you are getting data sourced to SQS Queue. You can try with 2 Lambda Functions [Feeder & Worker].
Feeder would be scheduled lambda function whose job is to take items
from SQS (if any) and push it as an SNS topic (and continue doing it forever)
Worker would be linked to listen the SNS topic which would do the actual data processing
We can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.
Further details:
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
AWS SQS is one of the oldest products of Amazon, which only supported polling (long and short) up until June 2018. As mentioned in this answer, AWS SQS now supports the feature of triggering lambda functions on new message arrival in SQS. A complete tutorial for this is provided in this document.
I used to tackle this problem using different mechanisms, and given below are some approaches you can use.
You can develop a simple polling application in Lambda, and use AWS CloudWatch to invoke it every 5 mins or so. You can make this near real-time by using CloudWatch events to invoke lambda with short downtimes. Use this tutorial or this tutorial for this purpose. (This could cost more on Lambdas)
You can consider that SQS is redundant if you don't need to persist the messages nor guarantee the order of delivery. You can use AWS SNS (Simple Notification Service) to directly invoke a lambda function and do whatever the processing required. Use this tutorial for this purpose. This will happen in real-time. But the main drawback is the number of lambdas that can be initiated per region at a given time. Please read this and understand the limitation before following this approach. Nevertheless AWS SNS Guarantees the order of delivery. Also SNS can directly call an HTTP endpoint and store the message in your DB.
I had a similar situation (and now have a working solution deploed). I have addressed it in a following manner:
i.e. publishing events to SNS; which then get fanned-out to Lambda and SQS.
NOTE: This is not applicable to the events that have to be processed in a certain order.
That there are some gotchas (w/ possible solutions) such as:
racing condition: lambda might get invoked before messages is deposited into the queue
distributed nature of SQS queue may lead to returning no messages even though there is a message note1.
The solution to both cases would be to do long-polling of SQS queue; but this does make your lambda bill more expensive.
note1
Short poll is the default behavior where a weighted random set of machines is sampled on a ReceiveMessage call. This means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
We had some similar requirements so we ended up building a library and open sourcing it to help with SQS to Lambda async. I'm not sure if this fills your particular set of requirements, but thought it might be worth a look: https://read.iopipe.com/sqs-lambda-teaming-up-92c4096be49c