Is it possible to create a lambda which is triggered manually to read from an SQS.
I have an SQS queue which is constantly receiving messages, I want to process them when I want to (not continuous sync processing).
Can I have something like a start/stop lambda, where I start the lambda, and it consumes actively from SQS and when I stop it stops consuming
Normally, an AWS Lambda function is configured to "trigger" from an Amazon SQS queue. Whenever a message arrives, a Lambda function would be triggered.
However, instead of configuring a trigger, you could code an AWS Lambda function to call ReceiveMessages() on the queue itself. The Lambda function would then be responsible for calling DeleteMessage() after the message has been processed.
You can invoke a Lambda function at any time by using the Invoke() command.
The only decision you would need to make is whether the Lambda function should process:
One message per invocation, or
One batch of messages (up to 10) per invocation, or
Run a loop that keeps retrieving messages from the queue until the queue is empty or until the Lambda function times-out (which could leave a message partially processed, so it isn't a good idea)
Related
I deployed a lambda, SQS standard queue and Dead letter queue on AWS. And I configured maxReceiveCount in the queue to retry before putting events to DLQ. Lambda pulls events from SQS queue in batch and process each event sequently. My question is about how retry works in case of error. There are two retries, one is on lambda maximumRetryAttempts, the other is on SQS and DLQ. Should I disable the lambda one?
In function, when it processes one event it calls deleteMessage on sqs to delete it. If there is any event that throws exception, the function throws it to lambda to make the retry happen so that it won't retry the success events.
But lambda itself has a maximumRetryAttempts and should I set it to 0? otherwise, will it retry before return to SQS? If I don't disable it, will the retry to process the whole batch of events including the success one?
Not sure which maximumRetryAttempts on lambda you are referring to. But When you use SQS with Lambda through event source mapping, as its done by default, there is no any retry parameter on lambda.
The only retry that applies is set at SQS, not lambda.
The retry option for lambda I can think of, and maybe you are thinning off as well, is for asynchronous invocation. This does not apply for SQS, as your lambda is invoked synchronously with SQS:
Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages.
Lambda Function can be invoked in three different ways:
Lambda reads from and invokes function. Ex: From SQS, Kinesis, etc.
Function invoked synchronously. Ex: From ApiGateway, ELB, etc.
Function invoked asynchronously. Ex: From S3 Events, SNS, Cloudwatch events etc.
Below Retry attempts is applicable for Asynchronous invocations(option 3 above)
For SQS Failures, we have two options:
DLQ on SQS itself.
Destination on Lambda. This could be SNS, another lambda, event bridge and another SQS queue. With this option, we can send both failures and success events.
Note: We don't need to call deleteMessage within lambda, lambda poller will delete message from SQS, when lambda returns success.
When a file is added to my S3 bucket an S3PUT Event is triggered which puts a message into SQS. I've configured a Lambda to be triggered as soon as a message is available.
In the lambda function, I'm sending an API request to run a task on an ECS Fargate container with environment variables containing the message received from SQS. In the container I'm using the message to download the file from S3, do processing and on successful processing I wish to delete the message from SQS.
However the message gets deleted from SQS automatically after my lambda executes.
Is there any way that I can configure the lambda not to automatically delete the SQS message (other than raising an exception and failing the lambda purposely), so that I can programmatically delete the message from my container?
Update:
Consider this scenario which I wish to achieve.
Message enters SQS queue
Lambda takes the message & runs ECS API and finishes without deleting the msg from queue.
Msg is in-flight.
ECS container runs the task and deletes msg from queue on successful processing.
If container fails, after the visibility timeout the message will re-enter the queue and the lambda will be triggered again and the cycle will repeat from step 1.
If container fails more than a certain number of times, only then will message go from in-flight to DLQ.
This all currently works only if I purposely raise an exception on the lambda and I'm looking for a similar solution without doing this.
The behaviour is intended and as long as SQS is configured as a Lambda trigger, once the function returns (i.e. completes execution) the message is automatically deleted.
The way I see it, to achieve the behaviour you're describing you have 4 options:
Remove SQS as Lambda trigger and instead execute the Lambda Function on a schedule and poll the queue yourself. The lambda will read messages that are available but unless you delete them explicitly they will become available again once their visibility timeout is expired. You can achieve this with a CloudWatch schedule.
Remove SQS as Lambda trigger and instead execute the Lambda Function explicitly. Similar to the above but instead of executing on a schedule all the time, the Lambda function could be triggered by the producer of the message itself.
Keep the SQS Lambda trigger and store the message in an alternative SQS Queue (as suggested by #jarmod in a comment above).
Configure the producer of the message to publish a message to an SNS Topic and subscribe 2 SQS Queue to this topic. One of the two queues will trigger a Lambda Function, the other one will be used by your ECS tasks.
Update
Based on the new info provided, you have another option:
Leave the event flow as it is and let the message in the SQS be deleted by Lambda. Then, in your ECS Task, handle the failure state and put a new message in the SQS with the same payload/body. This will allow you to retry indefinitely.
There's no reason why the SQS message has to be the exact same, what you're interested is the body/payload.
You might want to consider adding a mechanism to set a limit to these retries and post a message to a DLQ.
One solution I can think of is: remove lambda triggered by the sqs queue, create an alarm that on sqs queue. When the alarm triggers, scale out the ecs task. When there's no item in the queue, scale down the ecs task. Let the ecs task just poll the queue and handle all the messages.
I have lambda using SQS events as inputs. The SQS queue also has a DLQ.
The lambda function invokes a downstream Restful API (call this operation DoPostToAPI())
I need to guarantee that the lambda function attempts to call DoPostToAPI() at least 2 times (before message goes to DLQ)
What configuration of Lambda Retries and SQS Redrive policy would I need to set in order to accomplish the above requirement?
I need to be 100% certain that messages that arrive on the DLQ only arrive because they have attempted to been sent to downstream API DoPostToAPI() 2 times, and that messages dont arrive in DLQ for any other reason, if possible.
To me, it makes sense that messages should only arrive on the DLQ if the operation was attempted, and not for other reasons (i.e. I dont want messages to arrive on DLQ purely because of throttling, since the DoPostToAPI() should be attempted first before sending to DLQ) Why would I want messages on DLQ if the lambda function operation wasnt even attempted? In order words, I need the lambda operation to be guaranteed to be invoked before item moves to DLQ.
Can I get some help on this? Is it possible to guarantee that messages on the DLQ have arrived because of failed DoPostToAPI() api calls? Or is it (more unfortunate) possible that messages arrive on DLQ for reasons other than failed calls to downstream API?
From what I have read online so far, its possible that lambda , after doing receive on SQS message and moving the message to invisibile on the queue, could run into throttling issues and re-attempt the lambda invocation. But if it runs into lambda throttling again, it could end up back on main queue, which if it reaches its max receive count, could place the message on the DLQ without the lambda having been attempted at all. Is this correct?
For simplicity lets imagine the following inputs
SQSQueue1
SQSQueue1DLQ
LambdaFunction1 --> ServiceClient1.DoPostToAPI()
What is the interplay between the lambda "maximum_retry_attempts" and the SQS redrive_policy "maxReceiveCount"
In order to ensure your lambda attempts retries when using SQS, You only need set the SQS property
maxReceiveCount
This value controls how many lambda invocations will be attempted for a given batch before a message goes to the Dead Letter queue.
Unfortunately, the lambda property
maximum_retry_attempts
Does not apply for lambda functions using SQS as function event trigger.
I have a Scheduled Lambda function (via CloudWatch event rule) which is triggered every minute.
This lambda picks up a request from SQS queue, process the parameters and triggers AWS step functions workflow.
Now, ONLY 1 Lambda function instance is running every minute. How can I trigger multiple (e.g. 10) concurrent Lambda functions like this?
One way I can think of is to create 10 Cloudwatch event rule which runs every 1 minute, but I am not sure if that is the right way of doing it. Also, if I use this way, 10 lambda would be called even if I don't have entries in my SQS queue.
You can use the lambda step function.
Event trigger first function. Then it will call multiple functions parallel.
Some useful links:
https://www.youtube.com/watch?v=c797gM0f_Pc
https://medium.com/soluto-nashville/simplifying-workflows-with-aws-step-functions-57d5fad41e59
since your lambda function fetching data from SQS so you can create event source mapping between lambda and SQS so whenever message published to SQS, your lambda function will invoke concurrently depending on number of messages in queue so you do not need to invoke lamnda from cloudwatch event
I am trying to trigger my lambda function from SQS delay queue and it is triggering lambda but no message in queue. Even when I check on aws console there is a message delay and when that delay time is over. It trigger my lambda function but when my lambda try to get the list of messages it shows empty list. The other thing is when I remove the lambda trigger after that whenever I send a message to queue it shows message available after the delay time. So it's working as expected without adding a trigger to lambda but when I add trigger my lambda is not receiving any message to process.
I have tried various things but nothing worked out. my Default Visibility Timeout: 30seconds and Receive Message Wait Time: 0 seconds and Delivery Delay: 10 seconds.
Using below code to fetch the messages from sqs and it always return empty list :
final ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(SQS_URL);
final List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
I am new to aws and don't know what to do, due to this issue stuck in a deadlock. Please help.
If you have configured the Amazon SQS queue to trigger an AWS Lambda function, then the function should not call ReceiveMessage().
Instead, the message is automatically taken from the SQS queue and is passed to the Lambda function via the event parameter.
For sample code, see: Sample Amazon SQS Function Code - AWS Lambda
The Lambda function should loop through the messages passed to the function. When the function ends, the messages will automatically be deleted.