I would like to create a Scheduled Events to run a Lambda to execute an api call every 1 minute (cron-line behaviour).
The caveat to this setup is that; the external api is un-reliable / slow and the api call sometimes could last longer than 1 minute.
So, my question here is; given the setup & scenario - would AWS run another Scheduled Event and execute the lambda before the previous executing finished? I.e. overlap?
If it does; is there a way to configure the scheduled event to not "overlap"?
I did some initial research into this and came across this article:
https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
It looks like you can set concurrency limits at function level? Is this the way to achieve non-overlapping scheduled lambda executions? i.e. set the function's concurrency limit to 1?
Yes by default it will execute your Lambda function every 1 minute, regardless if the previous invocation has completed or not.
To enforce no more than one running instance of your Lambda function at a time, set the Concurrency setting of your Lambda function to 1.
Related
I am writing an AWS-lambda function that reads past 1-minute data from a DB, converts it to JSON, and then pushes it to a Kafka topic.
The lambda will run every 1 minute.
So if a scenario comes like this:
at t1, a lambda process is invoked let's say P1, it is reading data from t1 to t1-1.
at t2, if P1 is not finished then will a new lambda process will be invoked or we will wait for P1 to finish so that we can invoke another process?
I understand that lambda support up to 1000 parallel processes in a region but in this situation, the lambda function will already be running in a process.
Lambda does support multi-threading and multi-process in the same execution (see an example).
However, from the context that you gave, that doesn't seem to be the underlying question.
Concurrency allows you to configure how many instances of a single Lambda function can be in execution at a given time. Quoting the relevant part of the documentation:
Concurrency is the number of requests that your function is serving at any given time. When your function is invoked, Lambda allocates an instance of it to process the event. When the function code finishes running, it can handle another request. If the function is invoked again while a request is still being processed, another instance is allocated, which increases the function's concurrency. The total concurrency for all of the functions in your account is subject to a per-region quota.
Triggers define when a Lambda is executed. If you have multiple events coming (from SQS for example) to a lambda that has concurrency>1, then it's likely that there will be multiples instances of that given Lambda running at the same time.
With concurrency=1, if you trigger the Lambda every 1 minute and it takes more than 1 minute to execute and finish, then your processing will lag behind. In other words, future Lambdas will be processing t-2, t-3, and so on.
With concurrency=1, if you want something to be processed every 1 minute, you have to make sure it doesn't take more than 1 minute to process it. With extra concurrency it can take longer.
I have an AWS Lambda that polls from an external server for new events every 6 hours. On every call, if there are any new events, it publishes the updated total number of events polled to a SNS. So I essentially need to call the lambda on fixed intervals but also pass a counter state across calls.
I'm currently considering the following options:
Store the counter somewhere on a EFS/S3, but it seems an
overkill for a simple number
EventBridge, which would be ok to schedule the execution, but doesn't store state across calls
A step function with a loop + wait on the the lambda would do it, but it doesn't seem to be the most efficient/cost effective way to do it
use a SQS with a delay so that the lambda essentially
triggers itself, passing the updated state. Again I don't think
this is the most effective, and to actually get to the 6 hours delay
I would have to implement some checks/delays within the lambda, as the max delay for SQS is 15 minutes
What would be the best way to do it?
For scheduling Lambda at intervals, you can use CloudWatch Events. Scheduling Lambda using Serverless framework is a breeze. A cronjob type statement can schedule your lambda call. Here's a guide on scheduling: https://www.serverless.com/framework/docs/providers/aws/events/schedule
As for saving data, you can use AWS Systems Manager Parameter Store. It's a simple Key value pair storate for such small amount of data.
https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html
OR you can also save it in DynamoDB. Since the data is small and frequency is less, you wont be charged much and there's no hassle of reading files or parsing.
I have the follow schema (using AWS-IoTCore and AWS-Lambdas) that starts upon some MQTT event. At the end of the main process a flag success is saved in the database. This process takes around 5 min. I would like to call a Lambda function 24hs after the beginning of the process to check if the flag exists in the database. How can I set this timeout or delay in the Lambda function without stepfunctions?
There are two possible paths to "wait 24 hours" until execution of an AWS Lambda function.
Using a 'wait' step in AWS Step Functions
AWS Step Functions is ideal for orchestrating multi-step Lambda functions. There is a Wait state that can be triggered, and the flow could then trigger another Lambda function. Step Functions will track each individual request and you can see each execution and their current state. Sounds good!
Schedule your own check
Use Amazon CloudWatch Events to trigger an AWS Lambda function at regular intervals. The Lambda function could query the database to locate records for processes that did not complete successfully. However, since your code needs to find "non-successful" processes, each process would first need to be added to the database (eg at the start of the 'Process' function).
Alternate approach: Only track failures
The architecture in the above diagram stores 'success' in the database, which is then checked 24 hours later. Based on this diagram, it appears as though the only purpose of the database is to track successful processes.
Instead, I would recommend:
Changing the top line to only store 'failed' processes, and the time of their failure (assuming failure can be detected)
Trigger an AWS Lambda function at regular intervals that will:
Check the database
Retrieve failed processes that are older that 24 hours
Submit them for re-processing
Delete the records from the database, or mark them as 'retried' to avoid future retrievals
Basically, the database is used to track failures, rather than successes. The "wait 24 hours" step is replaced with a regular database check for failed processes.
I'm not sure why the systems wants to wait 24 hours before sending a 'success' email, but I presume that is an intentional part of your design.
You can use CloudWatch Events (or EventBridge) to schedule a Lambda function to run once, at a set time.
To create a CloudWatch Event rule, you can use the PutRule API operation to create the rule and then add the Lambda function as the target using PutTargets. The Lambda function should also allow CloudWatch Events to invoke the function in its function policy.
I also tested this out with the following cron expression: cron(0 0 1 1 ? 2021)
This would trigger the target Lambda function once at Fri, 01 Jan 2021 00:00:00 GMT. Also note, function may be invoked more than once due to CW invoking the function asynchronously. After the Lambda function has been invoked, you can delete the rule from the same function for clean-up.
I've created a simple lambda that reads data from dynamodb.
First time I call the lambda it takes about 1500ms to complete, but then after I run the lambda again it takes about 150ms. How is it possible?
What type of caching response does AWS preform to achieve this?
AWS Lambda is provision infrastructure on your first call and it's required time also AWS needs to start a JVM with the code to be able to call the function. Starting the JVM takes time and thus will incur some overhead.
Another issue is cold ,if there is no idle container available waiting to run the code. This is all invisible to the user and AWS has full control over when to kill containers.
So above steps are involved during first call and you can see 1500 ms
Next call you have everything on place so lambda give you response in 150 ms or less .
This is as per design of serverless to save infrastructure cost ,only provision infrastructure when needed and get first call.
I would suggest please read documents
- https://aws.amazon.com/lambda/
This happens due to cold start. This happens mainly when we invoke the lambda for the first time after deployment or when a lambda function is idle for sometime.
These articles explains about how language, memory or size of the lambda affects the cold start
https://read.acloud.guru/does-coding-language-memory-or-package-size-affect-cold-starts-of-aws-lambda-a15e26d12c76
https://mikhail.io/serverless/coldstarts/aws/
I have defined a lambda function that is invoked from API Gateway with proxy integration. Thus, I have defined an eager resource path for it:
And referenced my lambda function:
My lambda is able to process request like GET /myresource, POST /myresource.
I have tried this strategy to keep it warm, described in acloudguru. It consists of setting up a CloudWatch event rule that invokes the lambda every 5 minutes to keep it warm. Unfortunately it isn't working.
This is the behaviour I have seen:
After some period, let's say 20 minutes, I call GET /myresource from API Gateway and it takes around 15 seconds. Subsequent requests last ~30ms. The CloudWatch event is making no difference...
Let's suppose another long period without calling the gateway. If I go to the Lambda console and invoke it directly (test button) it answers right away (less than 1ms) with a 404 (that's normal because my lambda expects GET /myresource or POST /myresource).
Immediately after this lambda console execution I call GET /myresource from API Gateway and it still takes ~20 seconds. That is to say, the function was still cold despite having being invoked from the Lambda console. This might explain why the CloudWatch event doesn't work since it calls the lambda without setting the method/resource-url.
So, how can I make this particular case with API Gateway with proxy integration + Lambda stay warm to prevent those slow first request?
As of now (2019-02-27) [1], A periodic CloudWatch event rule does not deterministically solve the cold start issue. But a periodic CloudWatch event rule will reduce the probability of cold starts.
The reason is it's upto the Lambda server to decide whether to use a new Lambda container instead of an existing container to process an incoming request. Some of the related details regarding how Lambda containers are reused is explained in [1]
In order to reduce the cold start time (not to reduce the number cold starts), can you try followings? 1. increasing the memory allocated to the function, 2. reduce the deployment package size (eg- remove unnecessary dependencies), and 3. use a language like NodeJS, Python instead of Java, .Net
[1]According to reinvent session, (39:50 at https://www.youtube.com/watch?v=QdzV04T_kec), the Lambda team expects to improve the VPC cold start latency in Lambda.
[2] https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/
Denis is quite right about the non deterministic lambda behaviour regarding the number of containers hit by CloudWatch events. I'll follow his advice to improve the startup time.
On the other hand I have managed to make my CloudWatch events hit the lambda function properly, reducing (in many cases) the number of cold starts.
I just had to add an additional controller mapped to "/" with a hardcoded response:
#Controller("/")
class WarmUpController {
private val logger = LoggerFactory.getLogger(javaClass)
#Get
fun warmUp(): String {
logger.info("Warming up")
return """{"message" : "warming up"}"""
}
}
With this in place the default (/) invocation from CloudWatch does keep the container warm most of the time.