We are using recently released feature from AWS Lambda and SQS integration.
Whenever there is a message in SQS, Lambda is triggered and process the message.
However, In case of Lambda failure, it retires to process the message again.
Is there a way to configure interval between retries?
The answer to your question is no, but to overcome this problem you can put your Lambda inside a state machine (using AWS Step Functions), and using the machine's configuration you can control the retires behavior and even disable retries if you want.
This blog post I've written gives a more general explanation.
Related
I have a machine that's reacting to commands from my cloud, in an AWS SQS queue. The machine reports status back using AWS SNS topic, which may activate Lambda functions that can do some stuff (i.e. database lookups etc.) and send a new command to the machine.
Sometimes things go wrong, or the machine sends invalid data due to user input etc. This will result in an exception being thrown in the Lambda functions.
Is it possible somehow, to make the SNS Trigger catch that exception from the Lambda, so I can handle it and make a uniformed way to send error messages back to the machine?
It's not possible with a vanilla SNS trigger. The trigger has no knowledge of what happens after it fires. The easiest solution is to modify your Lambda function to send a message to your machine during exception handling. Alternately, you can wrap your Lambda execution in a step function and handle errors that way.
If my lambda function fails, is there any way in AWS to invoke the same function after 3-4 hours.
If yes what would be the flow to do so?
It depends on the failure. Clients such as the AWS CLI and the AWS SDK retry on client timeouts, throttling errors (429), and other errors that aren't caused by a bad request. Read more about auto retries in here.
If you want a custom retry logic, Dead Letter Queue can be an option. See more details in here https://aws.amazon.com/pt/blogs/compute/robust-serverless-application-design-with-aws-lambda-dlq/
Or you can use CloudWatch event to trigger on lambda failure.
Here is a good article explains that approach.
https://aws.amazon.com/blogs/mt/get-notified-specific-lambda-function-error-patterns-using-cloudwatch/
Above is my serverless config for my lambda. We want only limited parallel lambda(10) running, since it has db operations, using this configuration we were expecting Lambda to only pick 10 messages(reserved concurrency) at a time and only 1 message in each request(batchSize)
However as soon as I publish bulk messages to lambda, there are many messages InFlight. I was expecting only 10 messages to be InFlight.
Based on below monitoring it seems like lambda is getting invoked many times but gets throttled and the concurrent executions are always 10.
Questions: What is the concept behind this behavior? Also, are the throttled lambda instances waiting for others to finish? Does this impact other lambda's running under the same account? AWS Documentation doesn't give much information regarding the functioning.
I have created a lambda function which is triggered through cloudwatch event cron.
While testing I found that lambda retry is not working in case of timeout.
I want to understand what is the expected behaviour.Should retry happen in case of timeout?
P.S I have gone through the document on the aws site but still can't figure out
https://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html
Found the aws documentation on this,
"Error handling for a given event source depends on how Lambda is invoked. Amazon CloudWatch Events is configured to invoke a Lambda function asynchronously."
"Asynchronous invocation – Asynchronous events are queued before being used to invoke the Lambda function. If AWS Lambda is unable to fully process the event, it will automatically retry the invocation twice, with delays between retries."
So the retry should happen in this case. Not sure what was wrong with my lambda function , I just deleted and created again and retry worked this time.
Judging from the docs you linked to it seems that the lambda function is called again if it has timed out and the timeout is because it is waiting for another resource (i.e. is blocked by network):
The function times out while trying to reach an endpoint.
As a cron event is not stream based (if it is synchronous or asynchronous seems not be be clear from the docs) it will be retried.
CloudWatch Event invokes a Lambda function asynchronously.
For asynchronous invocation, Lambda manages the function's asynchronous event queue and attempts to retry two more times on errors including timeout.
https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html
So with the default configuration, your function should retry with timeout errors. If it doesn't, there might be some other reasons as follows:
The function doesn't have enough concurrency to run and events are throttled. Check function's reserved concurrency setting. It should be at least 1.
When above happens, events might also be deleted from the queue without being sent to the function. Check function's asynchronous invocation setting, make sure it has enough age to keep the events in the queue and retry attempts is not zero.
I have a topic at AWS SNS which sends messages to an AWS Lambda function. This function may sometimes fail and depending on which attempt is currently running I need to store some information about the failure. After looking for a while I didn't find any way to do that. Is it possible to retrieve which attempt is running?
I tried to use the delivery status feature and retrieve the attempt number using CloudWatch Events, but apparently SNS always deliver successfully its notifications to Lambda functions.
When a Lambda function fails it retries like explained here and not using delivery polices defined at the SNS topic (because it was delivered successfully).
In this particular case I guess it must be implemented mannually.