I have a API Gateway integrated with lambda. When I hit rest API, I notice the logs sometimes goes to existing log stream of lambda and sometimes it creates separate log streams.
SO how aws decides when to create new log stream?
My assumption is for each API request, a separate lambda container will be launched to execute lambda function,hence the logs will also go to new log stream. But this is not the case.
TL;DR, when the lambda function reuses the execution environment, logs go to an existing log stream. Cold start always creates a new execution environment and logs will go to a new log stream.
Each instance of a Lambda function has a dedicated log stream. If a function scales up, each concurrent instance has its own log stream. Each time an execution environment is reaped and a new environment is created in response to an invocation, this generates a new log stream.
Ref: https://docs.aws.amazon.com/lambda/latest/operatorguide/log-structure.html
Occasionally lambda may reuse the same container if requests are coming in short span and in such cases, it is possible to have logs in same log stream.
Related
We are making use of AWS Lambda and have configured cloudwatch for logging. There is a cron job running every 5 minutes which is triggering the lambda function. The logs that are generated for the hit are getting created in different log streams. For reference, please check the image attached here:
So, let's say there is an API hit at 11:45, then for checking the logs I have to go through the log streams having last event time 2022-05-05 11:43:10 (UTC+05:30) , 2022-05-05 11:43:00 (UTC+05:30), 2022-05-05 11:38:11 (UTC+05:30) and 2022-05-05 11:38:02 (UTC+05:30) and so on. The reason is, for a single hit logs are getting created in different log streams. Some of the logs are in first log stream, some are in second, a few are in third one. Previously, all the logs were created in single log stream corresponding to a single hit. Is there anything that can be done to avoid this? as this makes debugging a time taking process.
This is how Lambda works: each Lambda execution environment gets its own log stream. If you need to look at logs across log streams, then the best "built-in" solution is CloudWatch Logs Insights, which works at the log-group level.
Update: this document describes the Lambda execution environment, and the conditions that cause creation/destruction of an environment.
Here's what I know, or think I know.
In AWS Lambda, the first time you call a function is commonly called a "cold start" -- this is akin to starting up your program for the first time.
If you make a second function invocation relatively quickly after your first, this cold start won't happen again. This is colloquially known as a "warm start"
If a function is idle for long enough, the execution environment goes away, and the next request will need to cold start again.
It's also possible to have a single AWS Lambda function with multiple triggers. Here's an example of a single function that's handling both API Gateway requests and SQS messages.
My question: Will AWS Lambda reuse (warm start) an execution environment when different event triggers come in? Or will each event trigger have it's own cold start? Or is this behavior that's not guaranteed by Lambda?
Yes, different triggers will use the same containers since the execution environment is the same for different triggers, the only difference is the event that is passed to your Lambda.
You can verify this by executing your Lambda with two types of triggers (i.e. API Gateway and simply the Test function on the Lambda Console) and looking at the CloudWatch logs. Each Lambda container creates its own Log Stream inside of your Lambda's Log Group. You should see both event logs going to the same Log Stream which means the 2nd event is successfully using the warm container created by the first event.
I have a scheduled error handling lambda, I would like to use Serverless technology here as opposed to a spring boot service or something.
The lambda will read from an s3 bucket and process accordingly. The problem is at times the s3 bucket may have high volume of data to be processed. long running operations aren't suited to lambdas.
One solution I can think of is have the lambda read and process one item from the bucket and on success trigger another instance of the same lambda unless the bucket is empty/fully-processed. The thing i don't like is that this is synchronous and quite slow. I also need to be conscious of running too many lambdas at the same time as we are hitting a REST endpoint as part of the error flow and don't want to overload it with too many requests.
I am thinking it would be nice to have maybe 3 instances of the lambdas running at the same time until the bucket is empty but not really sure, I am wondering if anyone has any nice patterns that could be used here or suggestions on best practices?
Thanks
Create a S3 bucket for processing your files.
Enable a trigger S3 -> Lambda, on every new file in the bucket lambda will be invoked to process the file, every file is processed separately. https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-event-notifications.html
Once the file is processed you could either delete or move file to other place.
About concurrency please have a look at provisioned concurrency https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html
Update:
As you still plan to use a scheduler lambda and S3
Lambda reads/lists only the filenames and puts messages into SQS to process the file.
A new Lambda to consume SQS messages and process the file.
Note: I would recommend using SQS initially if the files/messages are not so big, it has built it recovery mechanics, DLQ , delays, visibility etc which you could benefit more than the simple S3 storage, second way is just create message with file reference and still use SQS.
I'd separate the lambda that is called by the scheduler from the lambda that is doing the actual processing. When the scheduler calls the first lambda, it can look at the contents of the bucket, then spawn the worker lambdas to process the objects. This way you have control over how many objects you want per worker.
Given your requirements, I would recommend:
Configure an Amazon S3 Event so that a message is pushed to an Amazon SQS queue when the objects are created in the S3 bucket
Schedule an AWS Lambda function at regular intervals that will:
Check that the external service is working
Invoke a Lambda function to process one message from the queue, and keep looping
The hard part would be throttling the second Lambda function so that it doesn't try to send all request at once (which might impact that external service).
You could probably do this by using a Step Function to trigger Lambda and then, if it was successful, trigger another Lambda function. This could even be done in parallel, such as allowing up to three parallel Lambda executions. The benefit of using Step Functions is that there is no cost for "waiting" for each Lambda to finish executing.
So, the Step Function flow would be something like:
Invoke a "check external service" Lambda function
If it fails, then quit the flow
Invoke the "processing" Lambda function
Get one message
Process the message
If successful, remove the message from the queue
Return success/fail
If it was successful, keep looping until the queue is empty
We have created a Lambda Function, which has to trigger after every minute. It is working as expected and showing the correct result. But logs stream, which are getting through Cloudwatch events, contains multiple Lambda trigger logs in a single Cloudwatch Log stream.
Event Rule: -
Is it possible to create 1 cloudwatch log for 1 Lambda Trigger ??
As per the AWS Lambda documentation here, a log stream represents an instance of your function. In other words, a log stream represents the logs from a single execution environment for your Lambda Function... The execution environment is also called the context (one of the arguments you can pass in to your handler) and the reason you're not getting a new log stream with each invocation is because of the context in which Lambda functions execute.
When you invoke your Lambda function, AWS loads the container that holds your function code and provisions the requested resources that are required to enable your function to execute: CPU, memory, networking etc. These all form the functions execution environment which is also called the context. These resources take time to provision and this results in increased latency for the execution of the function. This is commonly known as a "cold start".
In order to mitigate this undesired latency, or cold start, with each invocation, after your function has completed its initial execution, instead of terminating the execution environment, AWS keeps the container and execution environment running and the resources such as cpu, memory and networking, provisioned and ready for and in anticipation of, the next invocation. This is known as keeping the function "warm". When the container is warm, subsequent invocations of your function are executed in the same execution environment, or context, as the previous invocation, and because the invocation was executed by the same instance of the function the logs are written to the same log stream as the previous invocation(s), which is the log stream that represents that instance / execution environment / context of the function.
Notwithstanding this, it's worth pointing out that AWS does not keep the container running indefinitely. If there is no subsequent invocation within a given period of time (there's no exact period of time but it's generally considered to be between 30 and 45 minutes, source) AWS will terminate the container, and release the resources for use by another function. The next time the Lambda function is invoked, AWS will repeat the provisioning process for the function and a new execution environment will be created, and this will cause your functions logs to be written to a new log stream which represents the new execution environment / context / instance of your function.
You can read more about Lambda's execution context here.
Rachit,
Your Lambda function comes with a CloudWatch Logs log group, with a
log stream for each instance of your function. The runtime sends
details about each invocation to the log stream, and relays logs and
other output from your function's code.
Moreover, From the AWS Cloudwatch documentation you can see that a log stream is created each time the logs come from a different event source. In case of Lambda, it's one stream per Lambda container where each container might process multiple events.
A log stream is a sequence of log events that share the same source.
Each separate source of logs into CloudWatch Logs makes up a separate
log stream.
Ref:
https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-logging.html
For example I have lambda functions that consume messages from a KinesisStream. How do stop and resume the function so that I don't incur charges and I don't loose data in the stream.
I know that if the events keep failing, Kinesis will keep retrying and the cost can be very high.
I cannot delete the function because there is lots of automation around it through CloudFormation. Is there a way to stop and restart the function?
SOLUTION: http://alestic.com/2015/11/aws-lambda-kinesis-pause-resume
NOTE: Event sources for rules, log streaming, cannot be disable using the event source. You will not event get it in the list when calling the API using the SDK. For those you have to disable the Event Rule, or the Log Subscription.
The updated Lambda console on AWS supports this in the UI now. Click on the Kinesis stream feeding your lambda function, toggle the "Enabled/Disabled" toggle at the bottom, and Save. This will essentially pause/resume your function.Screenshot - Toggling Kinesis input into Lambda
Let's talk about Kinesis for a moment. When you pull records off the stream, Kinesis will not 'delete' those records until you 'checkpoint' the stream. You can read the same records over and over until you confirm with Kinesis that you don't need them anymore.
AWS Lambda does not checkpoint the stream until the function completes its execution without an error. (context.success())
If you deploy a Lambda function and it is broken in some way (exits with an exception/error), the Lambda function will not checkpoint the stream, and your records will stay in the stream for as long until retention period expires (24 hours, by default). The 'un-checkpointed' records can then be read in a subsequent Lambda execution.
During deployment, the same thing applies. Any currently executing Lambdas that are interrupted will not checkpoint the stream, and any currently executing Lambdas that complete successfully will checkpoint as you expect.