We have a NodeJS Lambda deployed in AWS. Works fine but whenever errors happen, the entire and details of the error is not shown in AWS Cloudwatch. CW shows all console.info output but does not show the stack trace of the exception. If we run the Lambda in our local machines, the console logs would look like the one below:
****START METHOD EXECUTION****
****END METHOD EXECUTION******
/Users/john.doe/Documents/workspace/myproject/user-service/dbutility.js:45
await connection.query('INSERT INTO users SET ?', record, async function (error, results, fields) {
^
at Handshake.onConnect (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Pool.js:58:9)
at Handshake.<anonymous> (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Connection.js:526:10)
at Handshake._callback (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Connection.js:488:16)
at Sequence.end (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
at Protocol.handleNetworkError (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/protocol/Protocol.js:369:14)
at Connection._handleNetworkError (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Connection.js:418:18)
at Socket.emit (node:events:513:28)
at Socket.emit (node:domain:489:12)
at emitErrorNT (node:internal/streams/destroy:151:8)
But when the same Lambda is deployed in AWS, the Cloudwatch logs we see is only the one below
****START METHOD EXECUTION****
****END METHOD EXECUTION******
In our codes, we catch error using the usual try catch:
try {
} catch (err) {
console.error(err);
}
How can we display error stack trace or details in the CW logs?
Related
Intermittently getting the following error when connecting to an AWS keyspace using a lambda layer
All host(s) tried for query failed. First host tried, 3.248.244.53:9142: Host considered as DOWN. See innerErrors.
I am trying to query a table in a keyspace using a nodejs lambda function as follows:
import cassandra from 'cassandra-driver';
import fs from 'fs';
export default class AmazonKeyspace {
tpmsClient = null;
constructor () {
let auth = new cassandra.auth.PlainTextAuthProvider('cass-user-at-xxxxxxxxxx', 'zzzzzzzzz');
let sslOptions1 = {
ca: [ fs.readFileSync('/opt/utils/AmazonRootCA1.pem', 'utf-8')],
host: 'cassandra.eu-west-1.amazonaws.com',
rejectUnauthorized: true
};
this.tpmsClient = new cassandra.Client({
contactPoints: ['cassandra.eu-west-1.amazonaws.com'],
localDataCenter: 'eu-west-1',
authProvider: auth,
sslOptions: sslOptions1,
keyspace: 'tpms',
protocolOptions: { port: 9142 }
});
}
getOrganisation = async (orgKey) => {
const SQL = 'select * FROM organisation where organisation_id=?;';
return new Promise((resolve, reject) => {
this.tpmsClient.execute(SQL, [orgKey], {prepare: true}, (err, result) => {
if (!err?.message) resolve(result.rows);
else reject(err.message);
});
});
};
}
I am basically following this recommended AWS documentation.
https://docs.aws.amazon.com/keyspaces/latest/devguide/using_nodejs_driver.html
It seems that around 10-20% of the time the lambda function (cassandra driver) cannot connect to the endpoint.
I am pretty familiar with Cassandra (I already use a 6 node cluster that I manage) and don't have any issues with that.
Could this be a timeout or do I need more contact points?
Followed the recommended guides. Checked from the AWS console for any errors but none shown.
UPDATE:
Update to the above question....
I am occasionally (1 in 50 if I parallel call the function (5 concurrent calls)) getting the below error:
"All host(s) tried for query failed. First host tried,
3.248.244.5:9142: DriverError: Socket was closed at Connection.clearAndInvokePending
(/opt/node_modules/cassandra-driver/lib/connection.js:265:15) at
Connection.close
(/opt/node_modules/cassandra-driver/lib/connection.js:618:8) at
TLSSocket.
(/opt/node_modules/cassandra-driver/lib/connection.js:93:10) at
TLSSocket.emit (node:events:525:35)\n at node:net:313:12\n at
TCP.done (node:_tls_wrap:587:7) { info: 'Cassandra Driver Error',
isSocketError: true, coordinator: '3.248.244.5:9142'}
This exception may be caused by throttling in the keyspaces side, resulting the Driver Error that you are seeing sporadically.
I would suggest taking a look over this repo which should help you to put measures in place to either prevent the occurrence of this issue or at least reveal the true cause of the exception.
Some of the errors you see in the logs you will need to investigate Amazon CloudWatch metrics to see if you have throttling or system errors. I've built this AWS CloudFormation template to deploy a CloudWatch dashboard with all the appropriate metrics. This will provide better observability for your application.
A System Error indicates an event that must be resolved by AWS and often part of normal operations. Activities such as timeouts, server faults, or scaling activity could result in server errors. A User error indicates an event that can often be resolved by the user such as invalid query or exceeding a capacity quota. Amazon Keyspaces passes the System Error back as a Cassandra ServerError. In most cases this a transient error, in which case you can retry your request until it succeeds. Using the Cassandra driver’s default retry policy customers can also experience NoHostAvailableException or AllNodesFailedException or messages like yours "All host(s) tried for query failed". This is a client side exception that is thrown once all host in the load balancing policy’s query plan have attempted the request.
Take a look at this retry policy for NodeJs which should help resolve your "All hosts failed" exception or pass back the original exception.
The retry policies in the Cassandra drivers are pretty crude and will not be able to do more sophisticated things like circuit breaker patters. You may want to eventually use a "failfast" retry policy for the driver and handle the exceptions in your application code.
I have a try {} catch (err) {} clause in my jenkins ci/cd pipeline that deploys a yaml to CloudFormation in aws. However when I've tested this out and purposely made something incorrect on the yaml file, I can see the stack fails to upload in cloudformation but jenkins pipeline still carries on to next stage and eventually marks the build as successful which is not the desired outcome.
I wish to fail the pipeline completely if my cloudformation stack (or any external process) fails. How can I do this? Is it possible to do so?
In your catch block, you can check for a specific error message and fail the build. SOmething like the below.
script {
try {
sh 'letsGenerateAError'
}
catch (error) {
if(error.getMessage().contains("SOme Error you want to check")) {
throw error
}
echo "Error occurred while Running, But ignoring it. Message : ${error.getMessage()}"
}
}
Is it possible to capture a thrown error message from a lambda function in Amazon Connect?
Example lambda throwing an error:
exports.handler = async (event) => {
throw new Error('some error message')
};
I would like to use the error message in a voice prompt, without catching the error.
When a synchronous invocation returns an error, Amazon Connect retries up to 3 times, for a maximum of 8 seconds. At that point, the flow will progress down the Error branch.
An Error branch, is this what you want? see document
Yes, just return it in the response and then access via contact attribute.
Problem
When I add the Idempotency configuration of aws-lambda-powertools my function code is not executed propertly.
The AWS lambda serves as message handler for a MS Teams chatbot when the function performs a cold start the async code within the handler is not executed and no message is returned to the user. I also don't see any logs so it seems that the code in the async handler is not executed at all.
Could this be due to the way I handle my async handler?
Code
#idempotent(persistence_store=persistence_layer, config=cfg)
def lambda_handler(event:dict, context: dict):
asyncio.get_event_loop().run_until_complete(lambda_messages(event))
payload = json.loads(event["body"])
return {"status": 400, "payload": payload}
The issue was due to the timeout of my aws sam function not being configured properly. Because of aws-labmda-powertools it was hard to debug as the error was not easily vissible.
I'm developing a chatbot on AWS Lex and I want to use Lambda function to branch my intent.
In order to do so, I created a Lambda as follows:
exports.handler = async (event) => {
console.log(event); //capture Lex params
/*
let { name, slots } = event.currentIntent
if(slots.MeetingType.toLowerCase() === 'on-line') {
return {
dialogAction: {
type: "ElicitSlot",
intentName: name,
slotToElicit: "InvitationLink",
slots
}
}
}
return {
dialogAction: {
type: "Delegate",
slots
}
}
*/
};
But as you can see, even when the function does nothing but log Lex output, I'm getting this error message in Lex:
An error has occurred: The server encountered an error processing the
Lambda response
Any help would be appreciated.
Because you are trying to build a Lex chatbot using JavaScript, please refer to this use case in the AWS SDK for JavaScript DEV Guide. It will walk you through this use case:
Building an Amazon Lex chatbot
Once you get this working, you can port the logic to a Lambda function.
Amazon Lex is giving you this error message because the Lambda function has failed during execution.
Enable CloudWatch logging for your Lambda function and check the logs after Lex has called it. The logs should provide you with more specific details about what's caused the code to break/fail. From there you should have a better idea of how to resolve the issue.
Feel free to post the output from the logs if you need more assistance with debugging the issue.