I'm working on a project were I'm using a lambda function to connect to a relational database and to DynamoDB at the same time. To access that function I'm using API Gateway, but I found a problem: My lambda function, written in Java takes more than 10 seconds to start due to the creation of both database connections.
I know API Gateway timeout is 10 seconds, and that's not a problem executing my function witch takes less than 1 second, but the problem is when it has to start.
I would like to know how to catch this timeout exception and notify to the user that he needs to start the request again.
Is there a way to do so without moving to Node.js or accessing lambda function directly?
Since the cost of establishing a connection to a relational database is so high, I would encourage you to open the connection in the initialization code of your Lambda function (outside of the handler).
The database connection will then be re-used across multiple invocations for the lifetime of the Lambda container. Within your Lambda function handler you may want to ensure the connection is alive and hasn't timed out, and re-open as required.
The first call through API Gateway may timeout, but subsequent calls will reuse the connection for the lifetime of the container.
Another trick is to create a scheduled function to periodically call your function to keep the container "warm".
Cheers,
Ryan
Related
Im a little confused since AWS has a lot of features and I do not know what to use.
So, I was creating a Lambda function that does a lot of stuff with a remote web, process could last at least a minute.
So my idea was creating an API that calls this lambda, have lambda create an unique ID and return a response right away to the client with a token., save this token to a DB.
Then have lambda process all this stuff with a remote web and, when it finishes, save the results to the DB and a bucket (a file), so this result is ready to deliver when the client makes another call to another API that makes a query to the DB to know the status of this process.
Thing is, it seems that if a response is sent from the handler, lambda terminates the execution, Im afraid the processing with the remote web will never finish.
I have read that step functions is the way to go, but I cant figure out which service will take the processing, ¿maybe another lambda?
Is there another service that is more suitable for this type of work?, this process involves scrapping a page and downloading files, is written in python.
I have read that step functions is the way to go, but I cant figure
out which service will take the processing, ¿maybe another lambda?
Yes, another Lambda function would do the background processing. You could also just have the first Lambda invoke a second Lambda directly, without using Step Functions. Or the first Lambda could place the task info in an SQS queue that another Lambda polls.
Is there another service that is more suitable for this type of work?
Lambda functions are fine for this. You just need to split your task into multiple functions somehow, either by one Lambda calling the other directly, or by using Step Functions, or an SQS queue or something.
Our current implementation is to open one database connection outside of the Lambda handler. When the backing Lambda container terminates, the connection is then left open/idle.
Can I make the new container close the previous old container's database connection?
Are there any hooks available like an onContainerClose()?
How can we close the previous open connection which cannot be used anymore, when the Lambda cold starts?
In the background, AWS Lambda functions execute in a container that isolates them from other functions & provides the resources, such as memory, specified in the function’s configuration.
Any variable outside the handler function will be 'frozen' in between Lambda invocations and possibly reused. Possibly because depending on the volume of executions, the container is almost always reused though this is not guaranteed.
You can personally test this by invoking a Lambda with the below source code multiple times & taking a look at the response:
let counter = 0
exports.handler = async (event) => {
counter++
const response = {
statusCode: 200,
body: JSON.stringify(counter),
};
return response;
};
This also includes database connections that you may want to create outside of the handler, to maximise the chance of reuse between invocations & to avoid creating a new connection every time.
Regardless of if the Lambda function is reused or not, a connection made outside of the handler will eventually be closed when the Lambda container is terminated by AWS. Granted, the issue of "zombie" connections are much less when the connection is reused but it is still there.
When you start to reach a high number of concurrent Lambda executions, the main question is how to end the unused connections leftover by terminated Lambda function containers. AWS Lambda is quite good at reliably terminating connections when the container expires but you may still run into issues getting close to your max_connections limit.
How can we close the previous open connection which cannot be used anymore, when the Lambda cold starts?
There is no native workaround via your application code or Lambda settings to completely getting rid of these zombie connections unless you handle opening and closing them yourself, and take the added duration hit of creating a new connection (still a very small number).
To clear zombie connections (if you must), a workaround would be to trigger a Lambda which would then list, inspect & kill idle leftover connections. You could either trigger this via an EventBridge rule operating on a schedule or trigger it when you are close to maxing out the database connections.
These are also great guidelines to follow:
Ensure your Lambda concurrent executions limit does not exceed your database maximum connections limit: this is to prevent the database from maxing out connections
Reduce database timeouts (if supported): limit the amount of time that connections can be idle & left open, for example in MySQL tweaking the wait_timeout variable from the default 28800s (8 hour) to 900 seconds (15 minutes) can be a great start
Reduce the number of database connections: try your best to reduce the connections you need to make to the database via good application design & caching
If all else fails, look into increasing the max connections limit on the databe
I have defined a lambda function that is invoked from API Gateway with proxy integration. Thus, I have defined an eager resource path for it:
And referenced my lambda function:
My lambda is able to process request like GET /myresource, POST /myresource.
I have tried this strategy to keep it warm, described in acloudguru. It consists of setting up a CloudWatch event rule that invokes the lambda every 5 minutes to keep it warm. Unfortunately it isn't working.
This is the behaviour I have seen:
After some period, let's say 20 minutes, I call GET /myresource from API Gateway and it takes around 15 seconds. Subsequent requests last ~30ms. The CloudWatch event is making no difference...
Let's suppose another long period without calling the gateway. If I go to the Lambda console and invoke it directly (test button) it answers right away (less than 1ms) with a 404 (that's normal because my lambda expects GET /myresource or POST /myresource).
Immediately after this lambda console execution I call GET /myresource from API Gateway and it still takes ~20 seconds. That is to say, the function was still cold despite having being invoked from the Lambda console. This might explain why the CloudWatch event doesn't work since it calls the lambda without setting the method/resource-url.
So, how can I make this particular case with API Gateway with proxy integration + Lambda stay warm to prevent those slow first request?
As of now (2019-02-27) [1], A periodic CloudWatch event rule does not deterministically solve the cold start issue. But a periodic CloudWatch event rule will reduce the probability of cold starts.
The reason is it's upto the Lambda server to decide whether to use a new Lambda container instead of an existing container to process an incoming request. Some of the related details regarding how Lambda containers are reused is explained in [1]
In order to reduce the cold start time (not to reduce the number cold starts), can you try followings? 1. increasing the memory allocated to the function, 2. reduce the deployment package size (eg- remove unnecessary dependencies), and 3. use a language like NodeJS, Python instead of Java, .Net
[1]According to reinvent session, (39:50 at https://www.youtube.com/watch?v=QdzV04T_kec), the Lambda team expects to improve the VPC cold start latency in Lambda.
[2] https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/
Denis is quite right about the non deterministic lambda behaviour regarding the number of containers hit by CloudWatch events. I'll follow his advice to improve the startup time.
On the other hand I have managed to make my CloudWatch events hit the lambda function properly, reducing (in many cases) the number of cold starts.
I just had to add an additional controller mapped to "/" with a hardcoded response:
#Controller("/")
class WarmUpController {
private val logger = LoggerFactory.getLogger(javaClass)
#Get
fun warmUp(): String {
logger.info("Warming up")
return """{"message" : "warming up"}"""
}
}
With this in place the default (/) invocation from CloudWatch does keep the container warm most of the time.
I'm trying to create app with serverless framework.
Every functions works fine on my local machine.
But remote machine(Lambda) gives to below error message. (This take a 30sec)
{
"message": "Endpoint request timed out"
}
Code size is 37.5 MB, and instance memory size is 3008 MB.
Any advice or suggestion would be appreciated. Thank you in advance
I solved my problems with set to callbackWaitsForEmptyEventLoop = false.
By default calling the callback() function in a NodeJS Lambda function does not end the function execution. It will continue running until the event loop is empty. A common issue with NodeJS Lambda functions continuing to run after callback is called occurs when you are holding on to open database connections. You haven't posted any code, so I can't give specific recommendations, but you would need to determine if you are leaving database connections open in your code or something similar.
Here's what the documentation says about the behavior of callbackWaitsForEmptyEventLoop:
callbackWaitsForEmptyEventLoop
The default value is true. This property is useful only to modify the
default behavior of the callback. By default, the callback will wait
until the Node.js runtime event loop is empty before freezing the
process and returning the results to the caller. You can set this
property to false to request AWS Lambda to freeze the process soon
after the callback is called, even if there are events in the event
loop. AWS Lambda will freeze the process, any state data and the
events in the Node.js event loop (any remaining events in the event
loop processed when the Lambda function is called next and if AWS
Lambda chooses to use the frozen process). For more information about
callback, see Using the Callback Parameter.
Can we use JDBC connection pooling with AWS Lambda ? AS AWS lambda function get called on a specific event, so its life time persist even after it finishing one of its call ?
No. Technically, you could create a connection pool outside of the handler function but since you can only make use of any one single connection per invocation so all you would be doing is tying up database connections and allocating a pool of which you could only ever use 1.
After uploading your Lambda function to AWS, the first time it is invoked AWS will create a container and run the setup code (the code outside of your handler function that creates the pool- let's say N connections) before invoking the handler code.
When the next request arrives, AWS may re-use the container again (or may not. It usually does, but that's down to AWS and not under your control).
Assuming it reuses the container, your handler function will be invoked (the setup code will not be run again) and your function would use one of N the connections to your database from the pool (held at the container level). This is most likely the first connection from the pool, number 1 as it is guaranteed to not be in use, since it's impossible for two functions to run at the same time within the same container. Read on for an explanation.
If AWS does not reuse the container, it will create a new container and your code will allocate another pool of N connections. Depending on the turnover of containers, you may exhaust the database pool entirely.
If two requests arrive concurrently, AWS cannot invoke the same handler at the same time. If this were possible, you'd have a shared state problem with the variables defined at the container scope level. Instead, AWS will use two separate containers and these will both allocate a pool of N connections each, i.e. 2N connections to your database.
It's never necessary for a single invocation function to require more than one connection (unless of course you need to communicate to two independent databases within the same context).
The only time a connection pool would be useful is if it were at one level above the container scope, that is, handed down by the AWS environment itself to the container. This is not possible.
The best case you can hope for is to have a single connection per container. Even then you would have to manage this single connection to ensure the database server hasn't disconnect or rebooted. If it does, your container's connection will die and your handler will never be able to connect again (until the container dies), unless you write some code in your function to check for dropped connections. On a busy server, the container might take a long time to die.
Also keep in mind that if your handler function fails, for example half way through a transaction or having locked a table, the next request invocation will get the dirty connection state from the container. The first invocation may have opened a transaction and died. The second invocation may commit and include all the previous queries up to the failure.
I recommend not managing state outside of the handler function at all, unless you have a specific need to optimise. If you do, then use a single connection, not a pool.
Yes, the lambda is mostly persistent, so JDBC connection pooling should work. The first time a lambda function is invoked, the environment will be created and it may or may not get reused. But in practice, subsequent invocations will often reuse the same lambda process along with all program state if your triggering events occur often.
This short lambda function demonstrates this:
package test;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
public class TestLambda implements RequestHandler<String, String> {
private int invocations = 0;
public String handleRequest(String request, Context context) {
invocations++;
System.out.println("invocations = " + invocations);
return request;
}
}
Invoke this from the AWS console with any string as the test event. In the CloudWatch logs, you'll see the invocations number increment each time.
Kudos to the AWS RDS proxy, now you can used pooled MySql and postgrese connections without any extra configs in your Java or other any code specific to AWS Lambda. All you need is to create and Add a Database proxy your AWS Lambda function you want to reuse/pool connections. See how-to here.
Note: AWS RDS proxy is not included in the Free-Tier (more here).
It has caveat
There is no destroy method which ensures closing pool. One may say DB connection idle time would handle.
What if same DB being used for other use cases like pool maintain in regular machine Luke EC2.
As many say, if there is sudden spike in requests, create chaos to DB as there will be always some maximum connection setting at database side per user.