If my lambda function fails, is there any way in AWS to invoke the same function after 3-4 hours.
If yes what would be the flow to do so?
It depends on the failure. Clients such as the AWS CLI and the AWS SDK retry on client timeouts, throttling errors (429), and other errors that aren't caused by a bad request. Read more about auto retries in here.
If you want a custom retry logic, Dead Letter Queue can be an option. See more details in here https://aws.amazon.com/pt/blogs/compute/robust-serverless-application-design-with-aws-lambda-dlq/
Or you can use CloudWatch event to trigger on lambda failure.
Here is a good article explains that approach.
https://aws.amazon.com/blogs/mt/get-notified-specific-lambda-function-error-patterns-using-cloudwatch/
Related
I have a lambda function for which the timeout is set to 10 seconds. This lambda is triggered from an API Gateway. Now in my case, I could see in the cloudwatch logs that I am getting a Time out error Task timed out after 10.00 seconds which is fine. But the Response code I am getting in my API gateway logs is Http-200.
I read few AWS docs and answers on Stack Overflow regarding this issue that if this is something which is expected or there is some issue with my code, but none of them seems to give clear answer as many of the questions are too old to follow.
Also I did not find anything substantial in AWS docs as well.
As per AWS,
For Lambda custom integrations, you must map errors returned by Lambda
in the integration response to standard HTTP error responses for your
clients. Otherwise, Lambda errors are returned as 200 OK responses by
default and the result is not intuitive for your API users.
Error Handling here
You have to explicitly handle such errors.
I recently stumbled across this as well.
In my scenario I was generally wondering what API Gateway returns when the lambda execution times out, runs out of memory etc. and found an answer in the aws forum:
AWS Forum
The Lambda error regex is only applied when an execution result failed like the exception was thrown or marked as failed inside your Lambda function. If the exception is failed by Lambda service like access denied, throttled, etc, the regex will not be applied.
One possible solution could be to use the API Gateway as a lambda proxy. Then it actually maps all the service errors to an HTTP-502 response.
The thread in the forum I linked also mentions to use the invocation type event. Maybe that helps.
We are using recently released feature from AWS Lambda and SQS integration.
Whenever there is a message in SQS, Lambda is triggered and process the message.
However, In case of Lambda failure, it retires to process the message again.
Is there a way to configure interval between retries?
The answer to your question is no, but to overcome this problem you can put your Lambda inside a state machine (using AWS Step Functions), and using the machine's configuration you can control the retires behavior and even disable retries if you want.
This blog post I've written gives a more general explanation.
I have an AWS Lambda that was triggered by SNS message. Many time, it has reached the max duration allowed by AWS, and AWS killed it immediately.
I have to either dig into the Lambda logs or the lambda duration chart to find out about the error.
Are there a better way to report this kind of errors?
Yes, there are some 3rd party tools that help you monitor your environment and provide exactly that - filter on specific errors and drill down to what happened there (the input event, the outgoing HTTP requests etc.).
Moreover, you can also configure alerts on specific errors that you will get via slack/mail.
Disclosure: I work for Lumigo, a company that does exactly that.
I have a AWS Lambda function running some process in my infrastructure. The Lambda is triggered every 8 hours using a CloudWatch rule. I am trying to raise a notification if any error happens into the Lambda process. I tried to use SES but that service is not available in that Region.
I will like to know any suggestions for this problem:
How to setup notifications when an error occurs in my Lambda functions ?
I am looking for suggestions. This questions never asked for doing my task. I will appreciate any official documentation but either way, any help is welcome.
Some suggestions:
Dead Letter Queues:
If your error causes failed invocations, you can use a Lambda Dead Letter Queue to send the event to an SNS topic or an SQS queue. If you send it to an SNS topic, you can directly subscribe to the topic via SNS or Email to get notified any time a message is published to that topic.
Multi-region SES:
If you're really set on using SES directly, SES clients can be instantiated with an explicit region provided -- as long as your lambda's execution role has the appropriate permissions, you can send email to SES from a different region. Here's documentation for instantiating the JS SES Client.
CloudWatch Logs:
If your error does not cause the invocation to fail, another option is using a CloudWatch Logs metric filter to aggregate failures and potentially alarm on them. If you're using NodeJS, you can simply log out via console.log(), console.error(), etc. and it will be written out to CWLogs. More details here.
You can subscribe an SNS topic to CloudWatch Alarms, and notify yourself in the same way as the DLQ.
As you gain experience with the error and learn how to process common errors, you could also subscribe another lambda to the SNS topic from the DLQ/CWLogs example to process it as it happens.
My current AWS Lambda function invokes another AWS Lambda function but I want to make sure that the invoke succeeded. After looking at concurrent execution limits for AWS Lambda I am trying to figure out what would happen if the concurrent limit is hit and I tried to invoke the Lambda from another Lambda.
For now, I am solving this problem by putting messages in an SNS but I rather prefer invoking Lambda directly avoiding the indirection.
The best way to handle the concurrent limit is to use a Kinesis stream rather than SNS.
The number of shards will limit the number of lambda invoked. And if it pertinent for you, you can take several messages at once, which you can't do with SNS, and that can lead to hit the concurrent limit.
Can you elaborate a little? Not sure I Understand what you are trying to achieve.
Lambda limits can be viewed under AWS console / EC2 page, top left corner has menu item called Limits, there you should see the limit.
When you hit the limit, lambda will stop being Invoked, and if my memory serves me right you will see an error in the logs saying something about limit being hit.
From the AWS Lambda FAQs:
Q: What happens if my account exceeds the default throttle limit on concurrent executions?
On exceeding the throttle limit, AWS Lambda functions being invoked
synchronously will return a throttling error (429 error code). Lambda
functions being invoked asynchronously can absorb reasonable bursts of
traffic for approximately 15-30 minutes, after which incoming events
will be rejected as throttled. In case the Lambda function is being
invoked in response to Amazon S3 events, events rejected by AWS Lambda
may be retained and retried by S3 for 24 hours. Events from Amazon
Kinesis streams and Amazon DynamoDB streams are retried until the
Lambda function succeeds or the data expires. Amazon Kinesis and
Amazon DynamoDB Streams retain data for 24 hours.
Inside the AWS Console you can always create a Service Limit Increase for AWS Lambda concurrent executions at no additional cost. This answer explains this in more detail.
I believe you're handling it correctly currently. I was just reading an article that was explaining how you shouldn't invoke lambda from another lambda because:
"If you do, the first will run until the second is finished executing, and you’re double billing yourself. Instead, use SNS or SQS to send a message to the other Lambda."
http://web.archive.org/web/20160713113906/http://www.appliedsoftwaredesign.com/archives/aws-lambda-pro-tips/