Lambda.FunctionError In my elasticsearch service log - amazon-web-services
I have an aws lambda function that connects to Kinesis Firehose delivery streams. In my logs the lambda function is executing perfectly and returning the data I want.
On my Kinesis Firehose delivery streams dashboard in the monitoring section it looks like I am getting Incoming bytes and Incoming records since there is data in those graphs. there is also data in the ExecuteProcessing Duration graph but then the ExecuteProcessing Success graph shows a line at 0, so I am guessing it is failing.
In the Elasticsearch logs I am getting a
Lambda.FunctionError with a message that says: The Lambda function
was successfully invoked but it returned an error result.
I am new to working with AWS and I am having trouble debugging this error code. Any help is appreciated.
The first thing that you must check is the return output of function. Remember that you must return a array with the same number of records in the following structure:
output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': base64.b64encode(json.dumps(payload).encode('utf-8')).decode('utf-8')
}
output.append(output_record)
**return {'records': output}**
if you forget return this array , you will get this error message.
If the Lambda function is returning an error, then you should be able to find more information in the CloudWatch logs for the function in question. These docs describe the different ways to access the logs. If the logs don't provide enough information, you might considering altering the function to write more information to stdout or stderr.
Related
Cloud Function Can not Get Correct Pub/Sub MEssage
I am setting up a pub/sub trigger-based cloud function in GCP, the topic the cloud function listen to is us-pubsub1. When I deployed the cloud function and used the testing panel to send messages like:
{"index":123,
"video_name":'test.mp4'}
in cloud function to processing the message with key index and video_name, it has no issue. But when I sent a real message to us-pubsub1 to trigger the cloud function, it always failed for not able to find the 'index' in the message body. and when reading the pub/sub message in cloud function, it return me messages like:
{'#type': 'type.googleapis.com/google.pubsub.v1.PubsubMessage',
'attributes': None, 'data':
'eyJzZXNzaW9uX2lkIjogImUzYjM0MTJiLWQxNWUtNDM5My05YjEyLWI3ZGY1ZGE4MTQ0NCIsICJzZXNzaW9uX25hbWUiOiAiU0VTU0lPTl9DMjIjMwMTI1VDIwMTI1NCIsICJzaXRlX25hbWUiOiAiVkEgTUVESUNBTCBDRU5URVIgLSBQQUxPIEFMVE8gLSc3RlY3RvbXkiLCAiaHViX3NlcmlhbF9udW1iZXIiOiAiQzIyNC0wMDIwNSIsICJjYXN0X2FwcF92ZXJzaW9uIjogIjEyLjAuMzIuNyIsICJkdl9zeXN0ZW0iOiAiVUFUU0syMDA3IiwgImludGVybmFsX2tleSI6ICJtNjQzY2U1NS0zNjNiLQ=='}
I checked that the message arrive in us-pubsub1 correctly and just failed to process in cloud function.
Is there any thing I have missed for fetching the message body for real cloud function?
It's normal. You have a pubsub message enveloppe and your content is base64 encoded in the data field. Here the documentation details
FWIW, here the real content of your sample
{"session_id": "e3b3412b-d15e-4393-9b12-b7df5da81444", "session_name": "SESSION_C22#3#UC##SB"'6FUR#%dTD44TDU"Dstectomy", "hub_serial_number": "C224-00205", "cast_app_version": "12.0.32.7", "dv_system": "UATSK2007", "internal_key": "m643ce55-363b-
it is truncated, you might not share the whole content ;)
Kinesis put records not returned in response from get records request
I have a Scala app using the aws-java-sdk-kinesis to issue a series of putRecord requests to a local kinesis stream.
The response returned after each putRecord request indicates its successfully putting the records into the stream.
The scala code making the putRecordRquest:
def putRecord(kinesisClient: AmazonKinesis, value: Array[Byte], streamName: String): Try[PutRecordResult] = Try {
val putRecordRequest = new PutRecordRequest()
putRecordRequest.setStreamName(streamName)
putRecordRequest.setData(ByteBuffer.wrap(value))
putRecordRequest.setPartitionKey("integrationKey")
kinesisClient.putRecord(putRecordRequest)
}
To confirm this I have a small python app that basically consumes from the stream (initialStreamPosition: LATEST). And prints the records it finds by iterating through the shard-iterators. But unexpectedly however it returns an empty set of records for each obtained shardIterator.
Trying this using the aws cli tool, I do however get records returned for the same shardIterator. I am confused? How can that be?
Running the python consumer (with LATEST), returns:
Shard-iterators: ['AAAAAAAAAAH9AUYVAkOcqkYNhtibrC9l68FcAQKbWfBMyNGko1ypHvXlPEuQe97Ixb67xu4CKzTFFGoLVoo8KMy+Zpd+gpr9Mn4wS+PoX0VxTItLZXxalmEfufOqnFbz2PV5h+Wg5V41tST0c4X0LYRpoPmEnnKwwtqwnD0/VW3h0/zxs7Jq+YJmDvh7XYLf91H/FscDzFGiFk6aNAVjyp+FNB3WHY0d']
Records: []
If doing the "same" with the aws cli tool however I get:
> aws kinesis get-records --shard-iterator AAAAAAAAAAH9AUYVAkOcqkYNhtibrC9l68FcAQKbWfBMyNGko1ypHvXlPEuQe97Ixb67xu4CKzTFFGoLVoo8KMy+Zpd+gpr9Mn4wS+PoX0VxTItLZXxalmEfufOqnFbz2PV5h+Wg5V41tST0c4X0LYRpoPmEnnKwwtqwnD0/VW3h0/zxs7Jq+YJmDvh7XYLf91H/FscDzFGiFk6aNAVjyp+FNB3WHY0d --endpoint-url http://localhost:4567
Returns:
{"Records":[{"SequenceNumber":"49625122979782922897342908653629584879579547704307482626","ApproximateArrivalTimestamp":1640263797.328,"Data":{"type":"Buffer","data":[123,34,116,105,109,101,115,116,97,109,112,34,58,49,54,52,48,50,54,51,55,57,55,44,34,100,116,109,34,58,49,54,52,48,50,54,51,55,57,55,44,34,101,34,58,34,101,34,44,34,116,114,97,99,107,101,114,95,118,101,114,115,105,111,110,34,58,34,118,101,114,115,105,111,110,34,44,34,117,114,108,34,58,34,104,116,116,112,115,58,47,47,116,101,115,116,46,99,111,109,34,44,34,104,99,99,34,58,102,97,108,115,101,44,34,115,99,34,58,49,44,34,99,111,110,116,101,120,116,34,58,123,34,101,116,34,58,34,101,116,34,44,34,100,101,118,34,58,34,100,101,118,34,44,34,100,119,101,108,108,34,58,49,44,34,111,105,100,34,58,49,44,34,119,105,100,34,58,49,44,34,115,116,97,116,101,34,58,123,34,108,99,34,58,123,34,99,111,100,101,34,58,34,115,111,109,101,45,99,111,100,101,34,44,34,105,100,34,58,34,115,111,109,101,45,105,100,34,125,125,125,44,34,121,117,105,100,34,58,34,102,53,101,52,57,53,98,102,45,100,98,102,100,45,52,102,53,102,45,56,99,56,98,45,53,97,56,98,50,56,57,98,52,48,49,97,34,125]},"PartitionKey":"integrationKey"},{"SequenceNumber":"49625122979782922897342908653630793805399163707871723522","ApproximateArrivalTimestamp":1640263817.338,"Data":{"type":"Buffer","data":[123,34,116,105,109,101,115,116,97,109,112,34,58,49,54,52,48,50,54,51,56,49,55,44,34,100,116,109,34,58,49,54,52,48,50,54,51,56,49,55,44,34,101,34,58,34,101,34,44,34,116,114,97,99,107,101,114,95,118,101,114,115,105,111,110,34,58,34,118,101,114,115,105,111,110,34,44,34,117,114,108,34,58,34,104,116,116,112,115,58,47,47,116,101,115,116,46,99,111,109,34,44,34,104,99,99,34,58,102,97,108,115,101,44,34,115,99,34,58,49,44,34,99,111,110,116,101,120,116,34,58,123,34,101,116,34,58,34,101,116,34,44,34,100,101,118,34,58,34,100,101,118,34,44,34,100,119,101,108,108,34,58,49,44,34,111,105,100,34,58,49,44,34,119,105,100,34,58,49,44,34,115,116,97,116,101,34,58,123,34,108,99,34,58,123,34,99,111,100,101,34,58,34,115,111,109,101,45,99,111,100,101,34,44,34,105,100,34,58,34,115,111,109,101,45,105,100,34,125,125,125,44,34,121,117,105,100,34,58,34,102,53,101,52,57,53,98,102,45,100,98,102,100,45,52,102,53,102,45,56,99,56,98,45,53,97,56,98,50,56,57,98,52,48,49,97,34,125]},"PartitionKey":"integrationKey"},{"SequenceNumber":"49625122979782922897342908653632002731218779711435964418","ApproximateArrivalTimestamp":1640263837.347,"Data":{"type":"Buffer","data":[123,34,116,105,109,101,115,116,97,109,112,34,58,49,54,52,48,50,54,51,56,51,55,44,34,100,116,109,34,58,49,54,52,48,50,54,51,56,51,55,44,34,101,34,58,34,101,34,44,34,116,114,97,99,107,101,114,95,118,101,114,115,105,111,110,34,58,34,118,101,114,115,105,111,110,34,44,34,117,114,108,34,58,34,104,116,116,112,115,58,47,47,116,101,115,116,46,99,111,109,34,44,34,104,99,99,34,58,102,97,108,115,101,44,34,115,99,34,58,49,44,34,99,111,110,116,101,120,116,34,58,123,34,101,116,34,58,34,101,116,34,44,34,100,101,118,34,58,34,100,101,118,34,44,34,100,119,101,108,108,34,58,49,44,34,111,105,100,34,58,49,44,34,119,105,100,34,58,49,44,34,115,116,97,116,101,34,58,123,34,108,99,34,58,123,34,99,111,100,101,34,58,34,115,111,109,101,45,99,111,100,101,34,44,34,105,100,34,58,34,115,111,109,101,45,1pre05,100,34,125,125,125,44,34,121,117,105,100,34,58,34,102,53,101,52,57,53,98,102,45,100,98,102,100,45,52,102,53,102,45,56,99,56,98,45,53,97,56,98,50,56,57,98,52,48,49,97,34,125]},"PartitionKey":"integrationKey"}],"NextShardIterator":"AAAAAAAAAAE+9W/bI4CsDfzvJGN3elplafFFBw81/cVB0RjojS39hpSglW0ptfsxrO6dCWKEJWu1f9BxY7OZJS9uUYyLn+dvozRNzKGofpHxmGD+/1WT0MVYMv8tkp8sdLdDNuVaq9iF6aBKma+e+iD079WfXzW92j9OF4DqIOCWFIBWG2sl8wn98figG4x74p4JuZ6Q5AgkE41GT2Ii2J6SkqBI1wzM","MillisBehindLatest":0}
The actual python consumer I have used in many other settings to introspec other kinesis streams we have and its working as expected. But for some reason here its not working.
Does anyone have a clue what might be going on here?
So I was finally able to identify the issue, and perhaps it will be useful for someone else with similar problem. In my setup, I am using a local kinesis stream (kinesalite) which doesn't support CBOR. You have to disable this explicitly otherwise I was seeing the following error when trying to deserialize the received record. Unable to unmarshall response (We expected a VALUE token but got: START_OBJECT). Response Code: 200, Response Text: OK In my case, setting the environment variable: AWS_CBOR_DISABLE=1 did the trick
"LAMBDA_RUNTIME" Error on high-volume Lambda Function
I'm currently using a Lambda Function written in Javascript that is setup with an SQS event source to automatically pull messages from an SQS Queue and do some basic processing on the message contents. I cannot show the code but the summary of the lambda function's execution is basically: For each message in the batch it receives as part of the event: It parses the body, which is a JSON string, into a Javascript object. It reads an object from S3 that is listed in the object using getObject. It puts a record into a DynamoDB table using put. If there were no errors, it deletes the individual SQS message that was processed from the Queue using deleteMessage. This SQS queue is high-volume and receives messages in-bulk, regularly building up a backlog of millions of messages. The Lambda is normally able to scale to process hundreds of thousands of messages concurrently. This solution has worked well for me with other applications in the past but I'm now encountering the following intermittent error that reliably begins to appear as the Lambda scales up: [ERROR] [#############] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 400. I've been unable to find any information anywhere about what this error means and what causes it. There appears to be not discernible pattern as to which executions encounter it. The function is usually able to run for a brief period without encountering the error and scale to expected levels. But then, as you can see, the error starts to appear quite suddenly and completely destroys the Lambda throughput by forcing it to auto-scale down: Does anyone know what this "LAMBDA_RUNTIME" error means and what might cause it? My Lambda Function runtime is Node v12.
Your function is being invoked asynchronously, so when it finishes it signals the caller if it was sucessful. You should have an error some milliseconds earlier, probably an unhandled exception not being logged. If that's the case, your functions ends without knowing about the exception and tries to post a success response.
I have this error only that I get:
[ERROR] [1638918279694] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413.
I went to the lambda function on aws console and ran the test with a custom event I build and the error I got there was:
{
"errorMessage": "Response payload size exceeded maximum allowed payload size (6291556 bytes).",
"errorType": "Function.ResponseSizeTooLarge"
}
So this is the actual error that cloudwatch doesn't return but the testing section of the lambda function console do.
I think I'll have to return info to an S3 file or something, but that's another matter.
DynamoDB Trigger Lambda Function PROBLEM: Function call failed
I enabled streams on my dynamoDB table. As items are modified, a lambda function is triggered. I think I set up everything correctly both on the lambda trigger side, permissions, and dynamodb side. I also ran my lambda function with test data and it succeeded. However, when items are modified in the table, the trigger did not start my lambda function. Instead, I got the following error: Batch size: 100 Last processing result: PROBLEM: Function call failed Any idea what's the best way to debug this? I went on CloudWatch logs but there were no logs associated with the trigger/stream. Thanks. Edit: Logs for the lambda function (not its dynamodb trigger). The trigger didn't generate any log statements. START RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67 Version: $LATEST 18:16:28 END RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67 18:16:28 REPORT RequestId: 3a08eedc-f0de-11e8-9008-033b48d2cb67 Duration: 81.85 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 30 MB
I ran into this issue today. I debugged it by manually triggering the lambda with the Test button on the top of the main lambda page. It showed the error output trying to run my lambda. The reason I had an error was the handler parameter as I had a non-standard javascript function name and I forgot to configure that in my lambda.
In my case, my lambda role did not have permissions to write to the SNS and the lambda code was writing to a SNS. So i added a policy to the lambda role giving it permissions to write to any SNS topic.
In my case, the problem came from the stream batch size 100. In the lambda code I was checking the event and I exit if the event doesn't meet the requirement.
In my case, the handler method was not configured correctly for the Lambda function written in Java. The format that I used for setting the handler is as follows
packageName.className::handlerMethod
For example, for my handler class
package com.example;
public class App implements
RequestHandler<DynamodbEvent, String> {
public String handleRequest(DynamodbEvent ddbEvent, Context context) {
the handler should be defined as
com.example.App::handleRequest
This sounds like a possible use-case for Rookout if you need to follow variable values in your live Lambda in a situation where you're not able to generate logs and running it locally isn't going to give you real-world event trigger data.
created cloudwatch event from lambda but nothing in Cloudwatch Events
I have used the aws api's to create a cloudwatch event from within my lambda. I have logged the successful message back from calling 'putEvents' and this returns: 2017-12-08T15:08:22.582Z a1d35179-dc29-11e7-ae3c-9354b3005c70 Success So it was obviously successful but when I try and view the event in Cloudwatch there is nothing there? Where has it gone?
You need to have a Rule to read the Custom Event you are pushing to the CW Event. Then you will be able to see the metrics TriggeredRules showing number of successful PutEvent api you are making.
Another reason could be that you pass on an incorrect timestamp. It's a quick fix, I ran into a bug where we used int(time.time()) in Python for the timestamp parameter, instead of: int(time.time() * 1000)