Consider the following AWS Lambda function:
var i = 0;
exports.handler = function (event, context) {
context.succeed(++i);
};
Executing this function multiple times, I end up with an output similar like the following:
> 0
> 1
> 2
> 0
> 1
> 0
> 3
> 2
> 4
> 1
> 2
As you can see, it seems like there are 3 singletons of the script, and I am randomly ending up in one of them when I execute the function.
Is this an expected behaviour? I couldn't find any related information on the documentation.
I'm asking this because I intend to connect to MySQL and keep a connection pool:
var MySQL = require('mysql');
var connectionPool = MySQL.createPool({
connectionLimit: 10,
host: '*****',
user: '*****',
pass: '*****',
database: '*****'
});
function logError (err, callback) {
console.error(err);
callback('Unable to perform operation');
}
exports.handler = function (event, context) {
connectionPool.getConnection(function (err, connection) {
err && logError(err, context.fail);
connection.query('CALL someSP(?)', [event.user_id], function (err, data) {
err && logError(err, context.fail);
context.succeed(data[0]);
connection.release();
});
});
};
The connection pool needs to be disposed using connectionPool.end() but where shall I execute this?
If I add it at the end of the script (after the handler), then the connection pool will be closed immediately when the lambda function first executes.
If I dispose the connection pool inside the handler, then the connection pool will be closed for future requests.
Furthermore, should I dispose it? If I don't dispose it, the connections will be kepts in the pool and in memory, but as you have seen in the first code sample, AWS keeps ~ 3 singletons of my module, that would mean that I'd end up with 3 different connection pools, with 10 connections each.
Unless I am badly misunderstanding your question, this is well documented and expected behavior for lambda. See here: https://aws.amazon.com/lambda/faqs/
Lambda spins up instances of your container to match the usage patterns of your lambda function. If it is not being used at the moment, then it will spin it down, if it is being used heavily, then more containers will be created. You should never depend on persistent state in a lambda function. It is ok to use state if it is for the lifecycle of your function, or you are optimizing something.
As far as I know, you can not control the number of function instances in memory at any given time, so if you are worried about using up your mysql connections, you should design accordingly.
From the documentation:
"AWS Lambda can start as many copies of your function as needed without lengthy deployment and configuration delays. There are no fundamental limits to scaling a function. AWS Lambda will dynamically allocate capacity to match the rate of incoming events."
As applies directly to your mysql question, I would always return your connection to the pool when you are finished using it. Then I would do some calculations on how many concurrent requests you expect to have and plan accordingly with your mysql server configuration.
Related
I have an AWS Lambda Function which is called using API gateway. This has a default timeout of 30 seconds.
The processing I need to do takes longer than 30 seconds so I have a second Lambda which is called by the first like this.
using (var client = new AmazonLambdaClient(RegionEndpoint.EUWest2))
{
var lambdaRequest = new InvokeRequest
{
FunctionName = "****LambdaFunction",
InvocationType = "Event",
Payload = JsonConvert.SerializeObject(callProcessingRequest)
};
client.InvokeAsync(lambdaRequest);
}
I've got a couple of issues with it.
First is the documentation says that InvocationType="Event" should call the Lambda async, but that doesn't seem to be the case, it takes about 10 seconds to run that invoke call.
The second more urgent issue is i'm getting intermittant errors logged like this.
1. Lambda encountered an UnobservedTaskException via 'TaskScheduler.UnobservedTaskException' event:
2. A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Signature expired: 20230119T093505Z is now earlier than 20230119T093648Z (20230119T094148Z - 5 min.))
Problem
I'm using mssql v6.2.0 in a Lambda that is invoked frequently (consistently ~25 concurrent invocations under standard load).
I seem to be having trouble with connection pooling or something because I keep having tons of open DB connections which overwhelm my database (SQL Server on RDS) causing the Lambdas to just time out waiting for query results.
I have read the docs, various similar questions, Github issues, etc. but nothing has worked for this particular issue.
Things I've Learned Already
I did learn that pooling is possible across invocations due to the fact that variables outside the handler function are shared across invocations in the same container. This makes me think I should see just a few connections for each container running my Lambda, but I don't know how many that is so it's hard to verify. Bottom line is that pooling should keep me from having tons and tons of open connections, so something isn't working right.
There are several different ways to use mssql and I have tried several of them. Notably I've tried specifying max pool size with both large and small values but got the same results.
AWS recommends that you check to see if there's already a pool before trying to create a new one. I tried that to no avail. It was something like pool = pool || await createPool()
I know that RDS Proxy exists to help with situations like this, but it appears it isn't offered (at this time) for SQL Server instances.
I do have the ability to slow down my data a bit, but this has a slight impact on the performance of the product as a whole, so I don't want to do that just to avoid solving a DB connections issue.
Left unchecked, I saw as many as 700 connections to the DB at once, leading me to think there's a leak of some kind and it's maybe not just a reasonable result of high usage.
I didn't find a way to shorten the TTL for connections on the SQL Server side as recommended by this re:Invent slide. Perhaps that is part of the answer?
Code
'use strict';
/* Dependencies */
const sql = require('mssql');
const fs = require('fs').promises;
const path = require('path');
const AWS = require('aws-sdk');
const GeoJSON = require('geojson');
AWS.config.update({ region: 'us-east-1' });
var iotdata = new AWS.IotData({ endpoint: process.env['IotEndpoint'] });
/* Export */
exports.handler = async function (event) {
let myVal= event.Records[0].Sns.Message;
// Gather prerequisites in parallel
let [
query1,
query2,
pool
] = await Promise.all([
fs.readFile(path.join(__dirname, 'query1.sql'), 'utf8'),
fs.readFile(path.join(__dirname, 'query2.sql'), 'utf8'),
sql.connect(process.env['connectionString'])
]);
// Query DB for updated data
let results = await pool.request()
.input('MyCol', sql.TYPES.VarChar, myVal)
.query(query1);
// Prepare IoT Core message
let params = {
topic: `${process.env['MyTopic']}/${results.recordset[0].TopicName}`,
payload: convertToGeoJsonString(results.recordset),
qos: 0
};
// Publish results to MQTT topic
try {
await iotdata.publish(params).promise();
console.log(`Successfully published update for ${myVal}`);
//Query 2
await pool.request()
.input('MyCol1', sql.TYPES.Float, results.recordset[0]['Foo'])
.input('MyCol2', sql.TYPES.Float, results.recordset[0]['Bar'])
.input('MyCol3', sql.TYPES.VarChar, results.recordset[0]['Baz'])
.query(query2);
} catch (err) {
console.log(err);
}
};
/**
* Convert query results to GeoJSON for API response
* #param {Array|Object} data - The query results
*/
function convertToGeoJsonString(data) {
let result = GeoJSON.parse(data, { Point: ['Latitude', 'Longitude']});
return JSON.stringify(result);
}
Question
Please help me understand why I'm getting runaway connections and how to fix it. For bonus points: what's the ideal strategy for handling high DB concurrency on Lambda?
Ultimately this service needs to handle several times the current load -- I realize this becomes a quite intense load. I'm open to options like read replicas or other read-performance-boosting measures as long as they're compatible with SQL Server, and they're not just a cop out for writing proper DB access code.
Please let me know if I can improve the question. I know there are similar ones out there but I have read/tried a lot of them and didn't find them to help. Thanks in advance!
Related Material
https://forums.aws.amazon.com/thread.jspa?messageID=678029 (old, but similar)
https://www.slideshare.net/AmazonWebServices/best-practices-for-using-aws-lambda-with-rdsrdbms-solutions-srv320 re:Invent slide deck
https://www.jeremydaly.com/reuse-database-connections-aws-lambda/ Relevant info but for MySQL instead of SQL Server
Answer
I finally found the answer after 4 days of effort. All I needed to do was scale up the DB. The code is actually fine as-is.
I went from db.t2.micro to db.t3.small (or 1 vCPU, 1GB RAM to 2 vCPU and 2GB RAM) at a net cost of roughly $15/mo.
Theory
In my case, the DB probably couldn't handle the processing (which involves several geographic calculations) for all my invocations at once. I did see CPU go up, but I assumed that was a result of the high open connections. When the queries slowed down, the concurrent invocations pile up as Lambdas start to wait for results, finally causing them to time out and not close their connections properly.
Comparisions:
db.t2.micro:
200+ DB connections (goes up continuously if you leave it running)
50+ concurrent invocations
5000+ ms Lambda duration when things slow down, ~300ms under no load
db.t3.small:
25-35 DB connections (constantly)
~5 concurrent invocations
~33 ms Lambda duration <-- ten times faster!
CloudWatch Dashboard
Summary
I think this issue was confusing to me because it didn't smell like a capacity issue. Almost every time I've dealt with high DB connections in the past, it has been a code error. Having tried options there, I thought it was "some magical gotcha of serverless" that I needed to understand. In the end it was as simple as changing DB tiers. My takeaway is that DB capacity issues can manifest themselves in ways other than high CPU and memory usage, and that high connections may be a result of something besides a code bug.
Update (4 months in)
This continues to work very well. I'm impressed that doubling the DB resources seems to have given > 2x performance. Now, when due to load (or a temporary bug during development), the db connections get really high (even over 1k) the DB handles it. I'm not seeing any issues at all with db connections timing out or the database getting bogged down due to load. Since the original time of writing I've added several CPU-intensive queries to support reporting workloads, and it continues to handle all these loads simultaneously.
We've also deployed this setup to production for one customer since the time of writing and it handles that workload without issue.
So a connection pool is no good on Lambda at all what you can do is reuse connections.
Trouble is every Lambda execution opens a pool it'll just flood the DB like you're getting, you want 1 connection per lambda container, you can use a db class like so (this is rough but lemmy know if you've got questions)
export default class MySQL {
constructor() {
this.connection = null
}
async getConnection() {
if (this.connection === null || this.connection.state === 'disconnected') {
return this.createConnection()
}
return this.connection
}
async createConnection() {
this.connection = await mysql.createConnection({
host: process.env.dbHost,
user: process.env.dbUser,
password: process.env.dbPassword,
database: process.env.database,
})
return this.connection
}
async query(sql, params) {
await this.getConnection()
let err
let rows
[err, rows] = await to(this.connection.query(sql, params))
if (err) {
console.log(err)
return false
}
return rows
}
}
function to(promise) {
return promise.then((data) => {
return [null, data]
}).catch(err => [err])
}
What you need to understand is A lambda execution is a little virtual machine that does a task and then stops, it does sit there for a while and if anyone else needs it then it gets reused along with the container and connection for a single task there's never multiple connections to a single lambda.
Hope this helps let me know if ya need any more detail! Oh and welcome to stackoverflow, that's a well-constructed question.
I am trying to create a google cloud task from one of my Google Cloud Functions. This function gets triggered when a new object is added to one of my Cloud Storage buckets.
I followed the instructions given here to create my App Engine (App Engine Quickstart Guide)
Then in my Cloud Function, I added the following code to create a cloud task (as described here - Creating App Engine Tasks)
However, there is something wrong with my task or App Engine call (not sure what).
I am getting the following errors every now and then. Sometimes it works and sometimes it does not.
{ Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded at Object.exports.createStatusError (/srv/node_modules/grpc/src/common.js:91:15) at Object.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:1204:28) at InterceptingListener._callNext (/srv/node_modules/grpc/src/client_interceptors.js:568:42) at InterceptingListener.onReceiveStatus (/srv/node_modules/grpc/src/client_interceptors.js:618:8) at callback (/srv/node_modules/grpc/src/client_interceptors.js:845:24) code: 4, metadata: Metadata { _internal_repr: {} }, details: 'Deadline Exceeded' }
Do let me know if you need more information and I will add them to this question here.
I had the same problem with firestore, trying to write one doc at time; I solve it by returning the total promise. That is because cloud function needs to know when is convenient to terminate the function but if you do not return anything maybe cause this error.
My example:
data.forEach( d => {
reports.doc(_date).collection('data').doc(`${d.Id}`).set(d);
})
This was the problem with me, I was writing document 1 by 1 but I wasn't returning the promise. So I solve it doing this:
const _datarwt = [];
data.forEach( d => {
_datarwt.push( reports.doc(_date).collection('data').doc(`${d.Id}`).set(d) );
})
const _dataloaded = await Promise.all( _datarwt );
I save the returned promise in an array and await for all the promises. That solved it for me. Hope been helpful.
I have this scenario where I have a WebApi and an endpoint that when triggered does a lot of work (around 2-5min). It is a POST endpoint with side effects and I would like to limit the execution so that if 2 requests are sent to this endpoint (should not happen, but better safe than sorry), one of them will have to wait in order to avoid race conditions.
I first tried to use a simple static lock inside the controller like this:
lock (_lockObj)
{
var results = await _service.LongRunningWithSideEffects();
return Ok(results);
}
this is of course not possible because of the await inside the lock statement.
Another solution I considered was to use a SemaphoreSlim implementation like this:
await semaphore.WaitAsync();
try
{
var results = await _service.LongRunningWithSideEffects();
return Ok(results);
}
finally
{
semaphore.Release();
}
However, according to MSDN:
The SemaphoreSlim class represents a lightweight, fast semaphore that can be used for waiting within a single process when wait times are expected to be very short.
Since in this scenario the wait times may even reach 5 minutes, what should I use for concurrency control?
EDIT (in response to plog17):
I do understand that passing this task onto a service might be the optimal way, however, I do not necessarily want to queue something in the background that still runs after the request is done.
The request involves other requests and integrations that take some time, but I would still like the user to wait for this request to finish and get a response regardless.
This request is expected to be only fired once a day at a specific time by a cron job. However, there is also an option to fire it manually by a developer (mostly in case something goes wrong with the job) and I would like to ensure the API doesn't run into concurrency issues if the developer e.g. double-sends the request accidentally etc.
If only one request of that sort can be processed at a given time, why not implement a queue ?
With such design, no more need to lock nor wait while processing the long running request.
Flow could be:
Client POST /RessourcesToProcess, should receive 202-Accepted quickly
HttpController simply queue the task to proceed (and return the 202-accepted)
Other service (windows service?) dequeue next task to proceed
Proceed task
Update resource status
During this process, client should be easily able to get status of requests previously made:
If task not found: 404-NotFound. Ressource not found for id 123
If task processing: 200-OK. 123 is processing.
If task done: 200-OK. Process response.
Your controller could look like:
public class TaskController
{
//constructor and private members
[HttpPost, Route("")]
public void QueueTask(RequestBody body)
{
messageQueue.Add(body);
}
[HttpGet, Route("taskId")]
public void QueueTask(string taskId)
{
YourThing thing = tasksRepository.Get(taskId);
if (thing == null)
{
return NotFound("thing does not exist");
}
if (thing.IsProcessing)
{
return Ok("thing is processing");
}
if (!thing.IsProcessing)
{
return Ok("thing is not processing yet");
}
//here we assume thing had been processed
return Ok(thing.ResponseContent);
}
}
This design suggests that you do not handle long running process inside your WebApi. Indeed, it may not be the best design choice. If you still want to do so, you may want to read:
Long running task in WebAPI
https://blogs.msdn.microsoft.com/webdev/2014/06/04/queuebackgroundworkitem-to-reliably-schedule-and-run-background-processes-in-asp-net/
I am trying to call a DynamoDB client method and get one item from the DynamoDB table. I am using AWS Lambda. However, I keep getting the message:
"Process exited before completing request."
I have increased the timeout just to make sure, but the processing time is less than the timeout. Any advice?
console.log('Loading event');
var AWS = require('aws-sdk');
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
exports.handler = function(event, context) {
dynamodb.listTables(function(err, data) {
});
var params = {
"TableName": "User",
"Key":
{"User Id" : {"S":event.objectId}
},
"AttributesToGet" : ["First Name","Last Name", "Latitude", "Longitude"],
"ConsistentRead" : true
}
dynamodb.getItem(params, function(response,result) {
response.on('data', function(chunk){
console.log(""+chunk);
console.log("test1")
context.done(result);
});
result.on('ready', function(data){
console.log("test2")
console.log("Error:" + data.error);
console.log("ConsumedCapacityUnits:" + data.ConsumedCapacityUnits);
context.done('Error',data);
// ...
});
});
};
Take a look at your memory consumption (included in last log line). I got the same message when I assigned too little memory to my lambda function.
The message "Process exited before completing request" means that the Javascript function exited before calling context.done (or context.succeed, etc.). Usually, this means that there is some error in your code.
I'm not a Javascript expert (at all) so there may be more elegant ways to find the error but my approach has been to put a bunch of console.log messages in my code, run it, and then look at the logs. I can usually zero in on the offending line and, if I look at it long enough, I can usually figure out my mistake.
I see you have some logging already. What are you seeing in the output?
I have used callback, instead of context.
More recent examples on aws website use callback instead of context.
To complete request, either of the below must be called:
callback(error); // This is used when there is an error
// or
callback(null, data); // This is used when there is a success
// 'data' will contain success result, like some JSON object
When lambda execution completes the request,
failing to call one of the above callbacks,
you will see below error:
"Process exited before completing request."
Error in your code. Remove the last }); and don't use context it is there for backward compatibility, use callbacks on node.js 4.3 and 6.1 runtime.
Maybe you are not following aws lamda standard of using the function
check this Golang code.
package main
import "github.com/aws/aws-lambda-go/lambda"
func main() {
lambda.Start(yourFunction)
}
func yourFunction(){
// do your stuff
}
Check your lamda memory usage, for me this error was occurred because of lambda was using 201 MB memory, which was greater than allowed 200 MB of memory for its execution.
First verify your code and if it is ok, increase memory allotment to this lambda from configuration > General Configuration > Edit > Increase memory