Let's say my service is live in production using AWS Lambda in place? Now, I find that there is a bug in AWS Lambda, say I fix it and update lambda function, will some messages during the update would fail? How does it handle it internally?
Messages would continue going to the old version until the new version is fully deployed.
Since Lambda is distributed and stateless, no messages will be lost. Depending on the size of your Lambda function, it may take a few seconds for AWS to deploy, but it won't be accepting requests until it is ready to do so (has uploaded and validated successfully).
Worst case scenario, the moment you push the update, there may be several seconds where your Lambda is accepting requests on your old version.
Related
I have two lambda functions in my AWS. One acts as a custom authorizer and the other acts as a notification service which calls the firebase FCM notification service.
When a request is made first time in a day to the notification lambda there is no response. The lambda does not work and hence does not call the firebase service.
It seemed like a cold start problem to me so I added the provisioned concurrency for both auth and notification lambda to 1 in the hope that it will work. But the problem persists.
Cloudwatch logs are of no help at all since nothing gets printed to it which I can use to figure out the issue. Either the authorizer lambda goes cold and does not response or the primary notification lambda goes cold and does not response or even both of them have issues.
After the first call to lambda fails any subsequent calls then work smoothly like a charm.
I do not want to install any plugin which will keep the lambda warm (not an option from the client) so is there some other way I can diagnose this problem and fix it?
We are using recently released feature from AWS Lambda and SQS integration.
Whenever there is a message in SQS, Lambda is triggered and process the message.
However, In case of Lambda failure, it retires to process the message again.
Is there a way to configure interval between retries?
The answer to your question is no, but to overcome this problem you can put your Lambda inside a state machine (using AWS Step Functions), and using the machine's configuration you can control the retires behavior and even disable retries if you want.
This blog post I've written gives a more general explanation.
I am using AWS Lambda to check the health status and then send out an email. If the health is down I want it to send an email only once.
This Lambda function runs every 20minutes or so and I would like to prevent it from sending out multiple emails in interval if things have broken. Is there a way store environment variables or something in the AWS eco system so that it knows the state between each lambda function runs. (that way it doesnt send out an email and knows it has sent an email already).
I have looked into creating an alarm and sending out notifications but the email sent out through alarm wont do and I would like to have a custom email sent out, so I am using AWS SES through lambda. There is a cloud watch alarm that turns on when there is an error but I cant seem to fetch the state of alarm through the aws-sdk (its apparently not there).
I have written the function in NodeJS
Any suggestions ?
I've implemented something like this a little differently. I too do not care for getting an email for each error, since the errors I receive from my AWS Lambdas do not require immediate attention. I prefer to get them once an hour.
So I write all the errors I receive to an SQS queue. I configure the AWS Lambdas, which are throwing the errors, to send certain errors (configurable via environment variables) to certain SQS queues. Cloudwatch rules (running whenever), configured to pull from specific SQS queues in the Cloudwatch rule definition, then execute an AWS Lambda passing in the rule definition containing the SQS queue to pull from. The Lambda called by the CloudWatch rule handles reading from the SQS queue then emailing the results.
For your case you could modify that process to read all the errors from SQS, then filter that data down to the results you want to send. I use SQS because the "errors" I get don't need to be persisted.
I could see two quick ways to store something like a "last_email_sent" value. The first would be in DynamoDB. This is part of the AWS "serverless" environment that doesn't require you to do much more than interact with it. You didn't indicate your development environment but there are multiple development environments that are supported.
The second would be with the SSM Parameter Store. You can store any number of parameters there too.
There are likely other ways to do this too. Both of these are a bit of overkill but they would work to store what you need.
Alright, I found a better way that is simpler without dealing with other constraints. The NodeJS sdk is limited as it is. When the service is down create an alarm through the sdk and the next time the lambda gets triggered check if the alarm exists and send an email. That way if you want to do some notification through alarm it is possible too.
I think in my question I said this was not possible (last part), which I will retract.
Here is the link for the sdk reference: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatch.html
If I:
Trigger an AWS Lambda deployment/update
Trigger a request to AWS Lambda prior to step #1 finishing
Will the request just hit the old lambda? Will it error out?
So far in my testing it seems like there is no "downtime", that it swaps out the old for the new almost instantly--although the first request on the new lambda does have to do a cold start.
You are correct. IIRC, each function invocation uses a specific function ARN, which changes when you update the function. When you invoke the function, you're using the new ARN, which invokes only the new version.
I believe it's possible to continue using the old function, using the old ARN explicitly (though you might not be able to do this from the Lambda console.)
For more info, see http://docs.aws.amazon.com/lambda/latest/dg/versioning-aliases.html
It is now possible to have even more fine-grained control over how requests are handled between lambda function versions using the new traffic shifting feature announced recently at AWS's re:Invent conference:
https://aws.amazon.com/about-aws/whats-new/2017/11/aws-lambda-supports-traffic-shifting-and-phased-deployments-with-aws-codedeploy/
I deployed a service written in Python2.7 using AWS Lambda, and it's about extracting data from some pages and sending results to a web app. The service is triggered by the AWS CloudWatch event (fixed rate of 5 mins).
However, I found out sometimes the service was triggered twice at a time. I got this because there were two log stream printed the same data and result but with different RequestID's. And the database had duplicate data, which showed that both worked successfully. It looked like the service was triggered twice almost at the same time for no reasons.
Does anyone experience the same thing, and how do you fix it? Or, is there a way to limit only one function can be executed at a time.
Yes. Some AWS services have SLA of at least once delivery. I have experienced this with CloudWatch and CloudTrail. I do not know if you can limit it only once. You have to check if the data has been processed already. I overcame this by making boto3 calls in my python code before processing the data. Without knowing your situation, it is difficult to suggest a solution.