Does AWS lambda provide support for listening to SQS queue? I found some examples which says one can do that but I am not sure if AWS lambda explicity provide support for that. When I create the lambda function, then I found one blueprint for SQS. So,
I linked to it in your other thread - these are the supported event sources. Notice that cloudwatch events are one of the possible event types. You could set up a Lambda to, for example, run every minute and poll an SQS queue. You cannot directly trigger a Lambda off of an SQS queue.
Good news, this feature was released yesterday!
28 JUN 2018: AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources
Read the announcement blog post here: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
AWS Serverless Model supports a new event source as following:
Type: SQS
PropertiesProperties:
QueueQueue: arn:aws:sqs:us-west-2:012345678901:my-queue arn:aws:sqs:us-west-2:0123456789 # NOTE: FIFO SQS Queues are not yet supported
BatchSize: 10
And this is how it can be configured from the AWS Console UI:
You can make your lambda function poll the queue using the SQS API. You could use SNS to trigger the Lambda function.
Update: AWS Lambda can now be triggered from Amazon SQS queues.
Old answer:
Rather than having AWS Lambda poll an Amazon SQS queue, the application that sends the message to SQS queue should instead directly invoke the Lambda function. This could be done in several ways:
Direct invocation via an AWS API call
Sending a message to an Amazon SNS topic, with the Lambda function subscribed to the topic
Calling a function via AWS API Gateway, which can then trigger a Lambda function
The extra step of putting a message into an SQS queue is not necessary.
Related
I have set up a lambda function and an SQS service. I want the events from the SQS service to get consumed by the lambda function.
SQS configuration
All the other configuration is set to match the default.
Lambda trigger
Lambda configuration
The code used is from the sqs-poller template (and the configuration too)
Code configuration
I'm using the following code to send the event. I run the code with the following command
AWS_SDK_LOAD_CONFIG=true AWS_SHARED_CREDENTIALS_FILE=./credentials node sqs.js
That works fine because I'm seeing the messages in the monitoring panel of the SQS service.
Any idea why events are not being consumed by my lambda function?
It would appear that you have two competing concepts in your architecture.
Amazon SQS and AWS Lambda
When an Amazon SQS queue is configured as a trigger to an AWS Lambda function, the AWS service polls the SQS queue looking for messages. When message(s) are found, the Lambda function is invoked, with the messages being passed to the function via the event variable.
The Lambda function can then process those messages, reading the detail of the messages from the event variable.
If the Lambda function completes without error, the AWS service will automatically delete the messages from the SQS queue. If there was an error in the function, the messages will automatically reappear on the SQS queue after the invisibility period has expired.
At no time does the AWS Lambda function actually call the Amazon SQS service to receive or delete messages. Rather, it is given the messages when it is invoked.
SQS Poller
You mention that you are using an sqs-poller class. I'm not sure whether you are referring to Receiving Messages Using the QueuePoller Class in Amazon SQS - AWS SDK for Ruby or #jishimi/sqs-poller - npm.
Nonetheless, polling is a traditional way that worker processes retrieve messages from an SQS queue, and then delete the messages after they are processed. The process is:
They ask check whether there are messages available in the SQS queue
If so, they invoke a worker
When the worker is finished, they delete the message
You should notice that these are the same steps that the AWS Lambda service does when SQS is configured as a trigger for an AWS Lambda function. Therefore, using a polling architecture is incompatible with using SQS as a trigger for an AWS Lambda function.
You should pick one or the other, not both.
I am using
Client > SNS > SQS > Lambda Trigger
As per my understanding, SNS > Lamda will be an asynchronous invocation
However, SNS > SQS > Lambda supposed to be synchronous.
Is my understanding correct?
I have configured DLQ on the SQS source but doesn’t store failed message into DLQ. Also saw 2x retried attempt on Lambda cloudwatch.
If SNS > SQS > Lambda is synchronous, I think this should not happen am I right?
#jellycsc already answered your question in the comment section, but i would like to expand it. Shortly it is poll based which uses synchronous invocation for it.
There are three type of invocation models for AWS Lambda
Synchronous
Elastic Load Balancing
Alexa
Cognito
API Gateway (there is an async version of it too)
Asynchronous
S3
SNS
Cloudwatch/Eventbridge events
Poll-based
Kinesis
SQS
DynamoDB streams
According to the this blogpost
AWS will manage the poller on your behalf and perform Synchronous invokes of your function with this type of integration. The retry behavior for this model is based on data expiration in the data source. For example, Kinesis Data streams store records for 24 hours by default (up to 168 hours).
I have a lambda function with 2 aliases.
Dev - Points to the $LATEST version
Test - Points to a specific version.
I have setup my SQS queue to trigger both aliases when it receives a message in the queue. However, it only seems to trigger one of them (The one that got registered the latest) and not both. Has anyone else come across this issue?
arn:aws:lambda:us-east-1:XXXXXXXXXXXXXX:function:Amoel:Dev
arn:aws:lambda:us-east-1:XXXXXXXXXXXXXX:function:Amoel:Test
A message on the queue would be consumed only once, by whichever Lambda function grabs it.
If you wish to send the same message to multiple AWS Lambda functions, combine it with Amazon SNS:
SQS queue -> SNS topic -> 2 x Lambda function subscriptions
Be sure to activate Raw Message Delivery on the Amazon SNS topic to preserve the original format of the message from Amazon SQS.
I want to trigger Lambda function whenever new message added to SQS.
Note that I don't want to add new message (events) to SQS.
What I'm trying to do:
My app will send message to SQS
Whenever new message added to queue CloudWatch event gets generated
CloudWatch Event triggers lambda
Problem:
In AWS console while configuring CloudWatch Events I haven't found any option to add source of event i.e. URL or Name of my SQS queue.
I'm not sure if this use case is valid but please help me out.
EDIT: AWS now supports SQS as an event source to trigger Lambda functions. See this blog post for more details.
ORIGINAL ANSWER:
SQS is not supported as a direct event source for AWS Lambda functions. If there are properties of a queueing system that you need for your use case, then you could have a "cron-job" type Lambda function that runs on a schedule, receives messages from the queue, and calls your worker Lambda function in response to each message received. The problem with this approach is that you must continually poll SQS even during periods when you don't expect messages, which incurs unnecessary cost.
The easiest approach is to use SNS instead. Create a topic, publish events to that topic instead of adding a message to an SQS queue, and have your Lambda function subscribe to that SNS topic. It will then be invoked each time a message is published to that SNS topic. There's a tutorial on this approach here:
http://docs.aws.amazon.com/lambda/latest/dg/with-sns-example.html
I would recommend to change your approach.
Your application should publish a message to an existing SNS topic. Your SQS and Lambda should than subscribe to this SNS topic.
Application -> publish -> SNS_TOPIC
-> SQS is notified
-> Lambda is notified
I am using AWS Lambda function to process the messages in Queue it's working fine. But i need to execute this Lambda function when messages available or added in SQS queue.
Is it possible to trigger the Lambda function based on SQS queue.Please suggest one method to achieve this goal.
Invoking Lambda functions from SQS queues is not directly supported. You can see the list of available triggers here: http://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html
Possible solutions:
Replace SQS queue with Kinesis or DynamoDB. Both can trigger Lambda functions on updates.
Inject SNS before SQS. SNS can add items to SQS and trigger Lambda function.
If you don't need near real-time processing, these two options are also valid:
Create CloudWatch Event Rule that will trigger the Lambda function every N minutes (e.g. every minute).
Create CloudWatch alarm watching ApproximateNumberOfMessagesVisible parameter for your SQS queue. This alarm should publish to an SNS topic, which in turn will trigger Lambda function.
Lambda now supports SQS as a native event source
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html