I have a lambda function is responsible for checking the server status. It needs to be called when SQS receives new messages and It is not allowed to change anything in SQS. I tried using SQS Lambda trigger but it will push the message into lambda function => that changed SQS queue.
I am looking the way to handle this problem. I try to use CloudWatch to handle this but I don't know is this possible or not? How Cloudwatch can trigger Lambda functions when SQS receives new messages?
Thanks in advance.
This will be difficult because, if the message is consumed quickly, it might not have an impact on Amazon CloudWatch metrics. You'll need to experiment to see whether this is the case. For example, set a metric for the maximum number of messages received in a 1-minute time period and try to trigger a CloudWatch Alarm when it is greater than zero.
Alternatively, have the system that sends the SQS message send it to Amazon SNS instead. Then, both the SQS queue and the Lambda function can subscribe to the SNS topic and both can be activated.
In fact, I know somebody who always uses SNS in front of SQS "just in case" this type of thing is necessary.
Related
I have a lambda that makes web requests depending on attributes of a message coming in via API Gateway. When the web request fails, I drop the event in a queue to be processed at a later time.
Since there is a likelihood that the external service is down as the reason for the failed request, I want to retry the request but not immediately.
I know I can have the queue be a trigger for the lambda, but I don't want it to trigger immediately on a new message arriving. I'd rather have it wait for 5 minutes or so and then have the SQS trigger the lambda.
My current solution has another lambda, that is triggered by a CloudWatch event, pull from the queue and then resend the messages to the lambda that makes requests. I feel like this solution is sloppy since I'm building a cloud watch event and another lambda just to handle a retry.
Is there a way for the SQS to trigger the Lambda on a time interval rather than on enqueue? Is there a better way to handle this?
Is there a way for the SQS to trigger the Lambda on a time interval rather than on enqueue? Is there a better way to handle this?
Yes, you can setup SQS delay queues with the delay of up to 15 minutes.
I have a Lambda function that is triggered by an SNS topic. What would happen to the messages being published to the SNS topic if Lambda reaches its limit of maximum concurrent executions and is not able to scale further?
For example, consider a situation where my SNS topic is receiving 1000 messages per second but Lambda is able to scale only up to processing 600 messages per second. From what I understand about SNS, it is a pub/sub mechanism and there can be no backlog in it (unlike SQS, Kinesis etc.). So what will happen to the extra 400 messages per second?
Also, how can I monitor if my Lambda is able to process at the rate at which SNS is receiving messages?
To answer your first question you need to understand the retry behavior of AWS Lambda. Please see the following quote out of the documentation.
Asynchronous invocation – Asynchronous events are queued before being
used to invoke the Lambda function. If AWS Lambda is unable to fully
process the event, it will automatically retry the invocation twice,
with delays between retries. If you have specified a Dead Letter Queue
for your function, then the failed event is sent to the specified
Amazon SQS queue or Amazon SNS topic. If you don't specify a Dead
Letter Queue (DLQ), which is not required and is the default setting,
then the event will be discarded. For more information, see Dead
Letter Queues.
To answer your second question:
You could use AWS CloudWatch.
There are two metrics interesting for you:
AWS/Lambda - Invocations
AWS/SNS - NumberOfMessagesPublished
I want to trigger Lambda function whenever new message added to SQS.
Note that I don't want to add new message (events) to SQS.
What I'm trying to do:
My app will send message to SQS
Whenever new message added to queue CloudWatch event gets generated
CloudWatch Event triggers lambda
Problem:
In AWS console while configuring CloudWatch Events I haven't found any option to add source of event i.e. URL or Name of my SQS queue.
I'm not sure if this use case is valid but please help me out.
EDIT: AWS now supports SQS as an event source to trigger Lambda functions. See this blog post for more details.
ORIGINAL ANSWER:
SQS is not supported as a direct event source for AWS Lambda functions. If there are properties of a queueing system that you need for your use case, then you could have a "cron-job" type Lambda function that runs on a schedule, receives messages from the queue, and calls your worker Lambda function in response to each message received. The problem with this approach is that you must continually poll SQS even during periods when you don't expect messages, which incurs unnecessary cost.
The easiest approach is to use SNS instead. Create a topic, publish events to that topic instead of adding a message to an SQS queue, and have your Lambda function subscribe to that SNS topic. It will then be invoked each time a message is published to that SNS topic. There's a tutorial on this approach here:
http://docs.aws.amazon.com/lambda/latest/dg/with-sns-example.html
I would recommend to change your approach.
Your application should publish a message to an existing SNS topic. Your SQS and Lambda should than subscribe to this SNS topic.
Application -> publish -> SNS_TOPIC
-> SQS is notified
-> Lambda is notified
I am using AWS Lambda function to process the messages in Queue it's working fine. But i need to execute this Lambda function when messages available or added in SQS queue.
Is it possible to trigger the Lambda function based on SQS queue.Please suggest one method to achieve this goal.
Invoking Lambda functions from SQS queues is not directly supported. You can see the list of available triggers here: http://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html
Possible solutions:
Replace SQS queue with Kinesis or DynamoDB. Both can trigger Lambda functions on updates.
Inject SNS before SQS. SNS can add items to SQS and trigger Lambda function.
If you don't need near real-time processing, these two options are also valid:
Create CloudWatch Event Rule that will trigger the Lambda function every N minutes (e.g. every minute).
Create CloudWatch alarm watching ApproximateNumberOfMessagesVisible parameter for your SQS queue. This alarm should publish to an SNS topic, which in turn will trigger Lambda function.
Lambda now supports SQS as a native event source
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
I would like to implement an Amazon SNS topic which first delivers messages to a SQS queue that is a subscriber on the topic, and then executes an AWS Lambda function that is also a subscriber on the same topic. The Lambda function can then read messages from the SQS queue and process several of them in parallel (hundreds).
My question is whether there is any way to guarantee that messages sent to the SNS topic would first be delivered to the SQS queue, and only then to the Lambda function?
The purpose of this is to scale to a large number of messages without having to execute the Lambda function separately for every single message.
For this purpose, triggering the lambda could be better and efficient if used from a cloud watch alert. With the cloud watch alert set at a buffer limit on the SQS, that could fire the lambda to start and process the full queue.
What you're looking for is currently not possible with one SNS Topic. If you subscribe your Lambda to a SNS Topic that particular Lambda gets executed each time that SNS Topic receives a message, in parallel.
Solution might be to have two SNS Topics and publish messages to the first one and have your SQS subscribe to it. After successful submission of messages to this first topic you could send a message to the second SNS Topic to execute your Lambda to process messages the first SNS Topic stored to SQS.
Another possible solution might be the above, you could just send some periodic message to the second topic to run the subscribed Lambda. This would allow you to scale your Lambda SQS Workers.
Subscribing both an SQS queue and a Lambda function to an SNS topic is a good way to have your Lambda function process SNS messages with low latency. I've tested this process just now. With every attempt the lambda function is invoked after the SQS message is inserted. I wouldn't expect this to always be the case, but it fixes the latency problem as best I am willing to measure. It's not guaranteed and you will need a CloudWatch scheduled event to pick up any missed messages.