I have deployed my lambda function via Serverless framework. When i invoke the function locally it works fine. But on AWS Lambda environment it is unable to make connection to MYSQL which is hosted on remotemysql.com. It gives timeout error every time.
Tried to increase timeout but nothing works
sequelize = new Sequelize(
process.env.DB_NAME,
process.env.DB_USER,
process.env.DB_PASSWORD,
{
host: process.env.DB_HOST,
dialect: "mysql",
logging: false,
connectTimeout: 60000
}
);
sequelize
.authenticate()
.then(() => {
logger.info("Database connection established");
// do my work
// some api calls to xys hosts outside servers
})
.catch(error => {
logger.error("Database connection failed", {
code: error.original.code,
errno: error.original.errno
});
process.exit(1);
});
My function is not inside any VPC and it has internet access allowed as I verified it is returning API responses which I am making to some other services outside AWS.
I am not sure if it is because of TCP connection or something else.
Please advise
Ok, so I have figured out what was causing the issue. It was not because of the connectivity but something wrong with the async handler.
my handler was like
exports.handler = async () => { ... }
I removed async and now it is working fine.
Related
I am running my nodejs application on google cloudrun. My application connects to google memorystore redis. Every few mins am getting the following error
Error: read Connection Reset
Followed by
AbortError: Redis connection lost and command aborted. It might have been processed.
Please help what am I missing?
My nodejs code
const redis = require('redis')
const redisClient = redis.createClient({host:'xxx', port: 6379})
redisClient.on('error, function (err) {
console.log(err)
}
const data = await redisClient.getExAsync('key')
Use "setInterval" function in order to invoke Redis operation every minute.
async function RedisKA() {
client.get("key2", (err, reply) => {
console.log(`${kaCount} redis keep `);
});
}
let updateIntervalId = setInterval(RedisKA, 60000);
If you want to avoid the request timeout on the Cloud Run side, which is 5 minutes by default then set your value based on your requirement.
The issue may be caused due a socket time out. This is expected to happen when there is no activity for a period of time.
This could be avoided by periodically executing any command on the connection, for example one command per minute, so it will keep the socket alive and will not abort the connection.
I am working on multiple Firebase cloud functions (all hosted in the same region) that connect with a GCP hosted Redis instance in the same region, using a VPC connector. I am using version 3.0.2 of the nodejs library for Redis. In the cloud functions' debug logs, I am seeing frequent connection reset logs, triggered for each cloud function with no fixed pattern around the timeline for the connection reset. And each time, the error captured in the error event handler is ECONNRESET. While creating the Redis instance, I have provided a retry_strategy to reconnect after 5 ms with maximum of 10 such attempts, along with the retry_unfulfilled_commands set to true, expecting that any unfulfilled command at the time of connection reset will be automatically retried (refer the code below).
const redisLib = require('redis');
const client = redisLib.createClient(REDIS_PORT, REDIS_HOST, {
enable_offline_queue: true,
retry_unfulfilled_commands: true,
retry_strategy: function(options) {
if (options.error && options.error.code === "ECONNREFUSED") {
// End reconnecting on a specific error and flush all commands with
// a individual error
return new Error("The server refused the connection");
}
if (options.attempt > REDIS_CONNECTION_RETRY_ATTEMPTS) {
// End reconnecting with built in error
console.log('Connection retry count exceeded 10');
return undefined;
}
// reconnect after 5 ms
console.log('Retrying connection after 5 ms');
return 5;
},
});
client.on('connect', () => {
console.log('Redis instance connected');
});
client.on('error', (err) => {
console.error(`Error connecting to Redis instance - ${err}`);
});
exports.getUserDataForId = (userId) => {
console.log('getUserDataForId invoked');
return new Promise((resolve, reject) => {
if(!client.connected) {
console.log('Redis instance not yet connected');
}
client.get(userId, (err, reply) => {
if(err) {
console.error(JSON.stringify(err));
reject(err);
} else {
resolve(reply);
}
});
});
}
// more such exports for different operations
Following are the questions / issues I am facing.
Why is the connection getting reset intermittently?
I have seen logs that even if the cloud function is being executed, the connection to Redis server lost resulting in failure of the command.
With retry_unfulfilled_commands set to true, I hoped it will handle the scenario as mentioned in point number 2 above, but as per debug logs, the cloud function times out in such scenario. This is what I observed in the logs in that case.
getUserDataForId invoked
Retrying connection after 5 ms
Redis instance connected
Function execution took 60002 ms, finished with status: 'timeout' --> coming from wrapper cloud function
Should I, instead of having a Redis connection instance at global level, try to have a connection created during each such Redis operation? It might have some performance issues as well as issues around number of concurrent Redis connections (since I have multiple cloud functions and all those will be creating Redis connections for each simultaneous invocation), right?
So, how to best handle it since I am facing all these issues during development itself, so not really sure if it's code related issue or some infrastructure configuration related issue.
This behavior could be caused by background activities.
"Background activity is anything that happens after your function has
terminated"
When the background activity interferes with subsequent invocations in Cloud Functions, unexpected behavior and errors that are hard to diagnose may occur. Accessing the network after a function terminates usually leads to "ECONNRESET" errors.
To troubleshoot this, make sure that there is no background activity by searching the logs for entries after the line saying that the invocation finished. Background activity can sometimes be buried deeper in the code, especially when asynchronous operations such as callbacks or timers are present. Review your code to make sure all asynchronous operations finish before you terminate the function.
Source
I have a serverless project: AWS + Angular on the frontend. Currently, I get the data when page is initialized and refresh the data when press "update" button. However, I want to monitor changes in the table constantly. In Firebase there is onSnapShot() method, which sends the new data when a collection is updated.
I want to make something similar with AWS. However, in official documentation, I do not see how to correctly do it.
So here are 2 questions:
How can I connect to the WebSocket with aws-sdk? (Currently, I can connect only from the terminal with wscat -c myurl call. Or shall I simply send http.Post with websocket url?
is it possible to pass invoke in the callback URL? - I want to get data from DynamoDB when page initialize and then invoke it again and again (with a callback URL)
My Lambda function looks like this:
exports.handler = async (event, context) => {
let params = {
TableName: "documents"
}
let respond = await db.scan(params).promise();
return respond;
};
On the front-end I have:
ngOnInit(): void {
AWS.config.credentials = new AWS.Credentials({
accessKeyId: '//mykey', secretAccessKey: '//mysecretkey'
})
AWS.config.update({
region:'//myregion'
})
this.updateTable() // triggers post request to APi Gateway => lambda and receives a response with data.
}
From my understanding, you will need to set up a DynamoDB stream and a lambda function that respond to the database CRUD events, send the updated data to the WebSocket connection if the event data matches the criteria (document id for example), through AWS.ApiGatewayManagementApi. (FYI: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/ApiGatewayManagementApi.html)
I am using AWS - ECS service and I have 5 running tasks on the cluster that has initiated as awsvpc network mode.
The problem is that the task is supposed to send request to Twilio for the SMS code but the request to Twilio is being timed out.
const twilioClient = require('twilio')(accountSid, authToken)
try {
await twilioClient.messages.create({
body: `${code}`,
from: phoneNumber,
to: userInput.phone
})
} catch (err) {
console.log('Twilio Error: ', err)
return false
}
The error below shows the error I have logged on CloudWatch.
Twilio Error: {
Error: ETIMEDOUT
at Timeout._onTimeout (/srv/node_modules/request/request.js:849:19)
ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10) code: 'ETIMEDOUT', connect: true
}
The problem is that the same code works in case of default network mode for Task on AWS ECS.
I am using EC2 mode, not Fargate mode.
Looking forward to the right help on this.
Cheers.
I have a multi-endpoint webservice written in Flask and running on API Gateway and Lambda thanks to Zappa.
I have a second, very tiny, lambda, written in Node, that periodically hits one of the webservice endpoints. I do this by configuring the little lambda to have Internet access then use Node's https.request with these options:
const options = {
hostname: 'XXXXXXXXXX.execute-api.us-east-1.amazonaws.com',
port: 443,
path: '/path/to/my/endpoint',
method: 'POST',
headers: {
'Authorization': `Bearer ${s3cretN0tSt0r3d1nTheC0de}`,
}
};
and this works beautifully. But now I am wondering whether I should instead make the little lambda invoke the API endpoint directly using the AWS SDK. I have seen other S.O. questions on invoking lambdas from lambdas but I did not see any examples where the target lambda was a multi-endpoint webservice. All the examples I found used new AWS.Lambda({...}) and then called invokeFunction with params.
Is there a way to pass, say, an event to the target lambda which contained the path of the specific endpoint I want to call? (and the auth headers, etc.) * * * * OR * * * * is this just a really dumb idea, given that I have working code already? My thinking is that a direct SDK lambda invocation might (is this true?) bypass API Gateway and be cheaper, BUT, hitting the endpoint directly via API Gateway is better for logging. And since the periodic lambda runs once a day, it's probably free anyway.
If what I have now is best, that's a fine answer. A lambda invocation answer would be cool too, since I've not been able to find a good example in which the target lambda had multiple https endpoints.
You can invoke the Lambda function directly using the invoke method in AWS SDK.
var params = {
ClientContext: "MyApp",
FunctionName: "MyFunction",
InvocationType: "Event",
LogType: "Tail",
Payload: <Binary String>,
Qualifier: "1"
};
lambda.invoke(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
/*
data = {
FunctionError: "",
LogResult: "",
Payload: <Binary String>,
StatusCode: 123
}
*/
});
Refer the AWS JavaScript SDK lambda.invoke method for more details.