I am working in AWS GovCloud I have the following configuration in AWS Lambda:
A Lambda function which decodes a payload
A Kinesis Stream set as a trigger for the aforementioned function
A Lambda Destination (we have tried Lambda functions as well as SQS, SNS)
No matter the configuration, I cannot seem to get Lambda to trigger the destination function (or queue in the event of SQS).
Here is the current Lambda Function. I have tried many permutations of the result/return payload without avail.
import base64
import json
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
for record in event['Records']:
payload = base64.b64decode(record['kinesis']['data']).decode('utf-8', 'ignore')
print("Success")
result = {
"statusCode": 202,
"headers": {
#'Content-Type': 'application/json',
},
"body": '{payload}'
}
return json.dumps(result)
I then send a message to Kinesis with the AWS CLI (I have noted that "Test" in the console does not observe desintations as per Jared Short ).
Every 0.1s: aws kinesis put-records --stream-name test-stream --records Data=SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IGZyb20gdGhlIEFXUyBDTEkh,PartitionKey=partitionkey1 Thu Jul 8 19:03:54 2021
{
"FailedRecordCount": 0,
"Records": [
{
"SequenceNumber": "49619938447946944252072058244333476686328287240252293122",
"ShardId": "shardId-000000000000"
}
]
}
Using Cloudwatch metrics and logs I am able to observe the function being triggered by the messages sent to Kinesis every .1 second.
The metrics charts indicate a success (as I expect).
Here is an example log from Cloudwatch:
START RequestId: 0cf3fb87-06e6-4e35-9de8-b30147e7be9d Version: $LATEST
Loading function
Success
END RequestId: 0cf3fb87-06e6-4e35-9de8-b30147e7be9d
REPORT RequestId: 0cf3fb87-06e6-4e35-9de8-b30147e7be9d Duration: 1.27 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB Init Duration: 113.64 ms
START RequestId: e663fa4a-2d0b-42d6-9e38-599712b71101 Version: $LATEST
Success
END RequestId: e663fa4a-2d0b-42d6-9e38-599712b71101
REPORT RequestId: e663fa4a-2d0b-42d6-9e38-599712b71101 Duration: 1.04 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB
START RequestId: b1373bbe-d2c6-49fb-a71f-dcedaf9210eb Version: $LATEST
Success
END RequestId: b1373bbe-d2c6-49fb-a71f-dcedaf9210eb
REPORT RequestId: b1373bbe-d2c6-49fb-a71f-dcedaf9210eb Duration: 0.98 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 51 MB
START RequestId: e0382653-9c33-44d6-82a7-a82f0f416297 Version: $LATEST
Success
END RequestId: e0382653-9c33-44d6-82a7-a82f0f416297
REPORT RequestId: e0382653-9c33-44d6-82a7-a82f0f416297 Duration: 1.05 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB
START RequestId: f9600ef5-419f-4271-9680-7368ccc5512d Version: $LATEST
Success
However, viewing the cloudwatch logs/metrics for the destination lambda function or SQS queue clearly show that the destination is not being triggered.
Over the course of troubleshooting, I have over-provisioned IAM policies to the Lambda function execution role so I am fairly confident that it is not an IAM related issue. Additionally, both functions are sharing the same execution role.
One thing I am not clear on after reviewing AWS documentation and 3rd party information is the criteria by which AWS determines success or failure for a given function. I am currently researching the invokation docs in search of what might be wrong here - but my interpretation is that AWS knows our function is successful based on the above Cloudwatch metrics showing a 100% success rate.
Does anyone know what I am doing wrong or how to try to troubleshoot the destination trigger for lambda?
Edit: As pointed out, the code is not correct for multiple record events. This is a function of senseless troubleshooting/changes to the code to get the Destination to trigger. Even something as simple as this does not invoke the destination.
import base64
import json
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# for record in event['Records']:
# payload = base64.b64decode(record['kinesis']['data']).decode('utf-8', 'ignore')
# print("Success")
# result = {
# "statusCode": 202,
# "headers": {
# 'Content-Type': 'application/json',
# },
# "body": '{"Success":True, payload}'
# }
return { "result": "OK" }
So, the question: Can someone demonstrate it is possible to have a Kinesis Stream Event Source Trigger a Lambda Function which successfully triggers a Lambda destination in AWS Govcloud?
Related
I'm doing a test where I want my function to fail on timeout.
So i set my timeout to 1 sec
My expectation is that it will fail when I invoke the function, however it still succeeds with a duration over the set timeout
Duration: 22744.46 ms Billed Duration: 22745 ms
Memory Size: 128 MB Max Memory Used: 74 MB Init Duration: 639.17 ms
When timeout, you will see line like this
2022-04-01T23:59:10.675Z xxxx Task timed out after 3.00 seconds
next the duration, memory stuffs you post this above
I used following python code, hopes help
import json
import time
def lambda_handler(event, context):
time.sleep(4)
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
I am using AWS Lambda to host a nodeJs service that fetch my open invoices on Stripe and execute a payment and update my database.
The problem is that most of the time, but not all the time (sometimes everything goes how it should), it hang on the call of invoice list and do nothing.
Here's the part of the code where log stops :
const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY, {
maxNetworkRetries: 1,
timeout: 2000
});
[other imports]
const microservice = async (event, context, callback) => {
[some code including database connection]
console.log('retrieving all open invoices...')
let invoices;
try {
invoices = await stripe.invoices.list({
status: 'open',
limit: 100,
});
console.log(invoices.data.length + ' data retrieved.');
} catch (error) {
console.log('Unable fetch stripe invoices : ', error);
console.log('Exiting due to stripe connection error.');
reports.push(new Report('Unable fetch stripe invoices', 'ERROR'));
return {
statusCode: 500,
};
}
[code that process invoices]
return {};
};
module.exports.microservice = microservice;
And here the log output :
START RequestId: d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 Version: $LATEST
2021-10-26T00:04:05.741Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO Connecting to database...
2021-10-26T00:04:05.929Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO Executing (default): SELECT 1+1 AS result
2021-10-26T00:04:05.931Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO Connection has been established successfully.
2021-10-26T00:04:05.931Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO retrieving all open invoices...
END RequestId: d628aa1e-dee6-4cc6-9ce0-f7c11cf73249
REPORT RequestId: d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 Duration: 15015.49 ms Billed Duration: 15000 ms Memory Size: 400 MB Max Memory Used: 40 MB
2021-10-26T00:04:20.754Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 Task timed out after 15.02 seconds
And when it gooes all right it's like that :
START RequestId: e5fb6b08-adf9-433f-b1da-fd9ec29dde31 Version: $LATEST
2021-10-25T14:35:03.369Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Connecting to database...
2021-10-25T14:35:03.590Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Executing (default): SELECT 1+1 AS result
2021-10-25T14:35:03.600Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Connection has been established successfully.
2021-10-25T14:35:03.600Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO retrieving all open invoices...
2021-10-25T14:35:04.011Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO 0 data retrieved.
2021-10-25T14:35:04.011Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Everything went smoothly !
END RequestId: e5fb6b08-adf9-433f-b1da-fd9ec29dde31
REPORT RequestId: e5fb6b08-adf9-433f-b1da-fd9ec29dde31 Duration: 646.58 ms Billed Duration: 647 ms Memory Size: 400 MB
I don't get why it hangs with no error or log...
Network issues can happen due to various reasons. In your case, what you can try doing is to reduce the limit (e.g. limit : 30) and set your client library to retry the connection again by setting maxNetworkRetries : 3 or number that fits your application. When this is set, Stripe will retry the connection when the timeout error occurs.
This is a perfect match for Step functions use cases. It will allow you to orchestrate the steps of getting the invoices and processing them and easily design a retry mechanism in case of errors.
It is not really a solution, but what is causing the issue is that Stripe take a very long time to return less than 100 results.
We found workaround in order to not fetch this list.
A help because are not writing the file to the S3 bucket
What did I do:
import time
import boto3
query = 'SELECT * FROM db_lambda.tb_inicial limit 10'
DATABASE = 'db_lambda'
output = 's3: // bucket-lambda-test1 / result /'
def lambda_handler (event, context):
client = boto3.client ('athena')
# Execution
response = client.start_query_execution (
QueryString = query,
QueryExecutionContext = {
Database: DATABASE
},
ResultConfiguration = {
'OutputLocation': output,
}
)
return response
return
IAM role created with:
AmazonS3FullAccess
AmazonAthenaFullAccess
CloudWatchLogsFullAccess
AmazonVPCFullAccess
AWSLambda_FullAccess
When running Lambda message:
Response:
{
"statusCode": 200,
"body": "\" Hello from Lambda! \ ""
}
Request ID:
"f2dd5cd2-070c-41ea-939f-d4909ce39fd0"
Function logs:
START RequestId: f2dd5cd2-070c-41ea-939f-d4909ce39fd0 Version: $ LATEST
END RequestId: f2dd5cd2-070c-41ea-939f-d4909ce39fd0
REPORT RequestId: f2dd5cd2-070c-41ea-939f-d4909ce39fd0 Duration: 0.84 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 52 MB
How I did the test:
Configure test event
A function can have a maximum of 10 test events. The events are maintained, so that you can change your computer or web browser and test the function with the same events.
Create new test event
Edit saved test events
Test event saved
{
}
The "Hello from Lambda" message is the default code in a Lambda function. It would appear that you did not click 'Deploy' before testing the function. Clicking Deploy will save the Lambda code.
Also, once you get it running, please note that start_query_execution() will simply start the Athena query. You will need to use get_query_results() to obtain the results.
I am attempting to write a simple Lambda function to query a table in Athena. But after a few seconds I see "Status: FAILED" in the Cloudwatch logs.
There is no descriptive error message on the cause of failure.
My test code is below:
import json
import time
import boto3
# athena constant
DATABASE = 'default'
TABLE = 'test'
# S3 constant
S3_OUTPUT = 's3://test-output/'
# number of retries
RETRY_COUNT = 1000
def lambda_handler(event, context):
# created query
query = "SELECT * FROM default.test limit 2"
# % (DATABASE, TABLE)
# athena client
client = boto3.client('athena')
# Execution
response = client.start_query_execution(
QueryString=query,
QueryExecutionContext={
'Database': DATABASE
},
ResultConfiguration={
'OutputLocation': S3_OUTPUT,
}
)
# get query execution id
query_execution_id = response['QueryExecutionId']
print(query_execution_id)
# get execution status
for i in range(1, 1 + RETRY_COUNT):
# get query execution
query_status = client.get_query_execution(QueryExecutionId=query_execution_id)
query_execution_status = query_status['QueryExecution']['Status']['State']
if query_execution_status == 'SUCCEEDED':
print("STATUS:" + query_execution_status)
break
if query_execution_status == 'FAILED':
#raise Exception("STATUS:" + query_execution_status)
print("STATUS:" + query_execution_status)
else:
print("STATUS:" + query_execution_status)
time.sleep(i)
else:
# Did not encounter a break event. Need to kill the query
client.stop_query_execution(QueryExecutionId=query_execution_id)
raise Exception('TIME OVER')
# get query results
result = client.get_query_results(QueryExecutionId=query_execution_id)
print(result)
return
The logs show the following:
2020-08-31T10:52:12.443-04:00
START RequestId: e5434651-d36e-48f0-8f27-0290 Version: $LATEST
2020-08-31T10:52:13.481-04:00
88162f38-bfcb-40ae-b4a3-0b5a21846e28
2020-08-31T10:52:13.500-04:00
STATUS:QUEUED
2020-08-31T10:52:14.519-04:00
STATUS:RUNNING
2020-08-31T10:52:16.540-04:00
STATUS:RUNNING
2020-08-31T10:52:19.556-04:00
STATUS:RUNNING
2020-08-31T10:52:23.574-04:00
STATUS:RUNNING
2020-08-31T10:52:28.594-04:00
STATUS:FAILED
2020-08-31T10:52:28.640-04:00
....more status: FAILED
....
END RequestId: e5434651-d36e-48f0-8f27-0290
REPORT RequestId: e5434651-d36e-48f0-8f27-0290 Duration: 30030.22 ms Billed Duration: 30000 ms Memory Size: 128 MB Max Memory Used: 72 MB Init Duration: 307.49 ms
2020-08-31T14:52:42.473Z e5434651-d36e-48f0-8f27-0290 Task timed out after 30.03 seconds
I think I have the right permissions for S3 bucket access given to the role (if not, I would have seen the error message). There are no files created in the bucket either. I am not sure what is going wrong here. What am I missing?
Thanks
The last line in your log shows
2020-08-31T14:52:42.473Z e5434651-d36e-48f0-8f27-0290 Task timed out after 30.03 seconds
To me this looks like the timeout of the Lambda Function is set to 30 seconds. Try increasing it to more than the time the Athena query needs (the maximum is 15 minutes).
Below code is copied from AWS documentation, but my code is almost the same except for the queue URL definition part.
I want to print out the message body in JSON format, but it seems it has some things extra. How can I get rid of them without using substring?
# Create SQS client
# blah blah
# Receive message from SQS queue
response = sqs.receive_message(
QueueUrl=queue_url,
AttributeNames=[
'SentTimestamp'
],
MaxNumberOfMessages=1,
MessageAttributeNames=[
'All'
],
VisibilityTimeout=0,
WaitTimeSeconds=0
)
message = response['Messages'][0]
receipt_handle = message['ReceiptHandle']
print('Received and deleted message: %s' % message)
This printed message has following format:
START RequestId: fe107bc8-3829-4600-9bfc-df89f59b0c70 Version: $LATEST
{JSON body}
END RequestId: fe107bc8-3829-4600-9bfc-df89f59b0c70
REPORT RequestId: fe107bc8-3829-4600-9bfc-df89f59b0c70 Duration: 914.38 ms Billed Duration: 1000 ms Memory Size: 128 MB Max Memory Used: 71 MB Init Duration: 247.03 ms
What I really want is just the {JSON body}. How can I get rid of the rest?
Unfortunately you can't remove
START RequestId: fe107bc8-3829-4600-9bfc-df89f59b0c70 Version: $LATEST
END RequestId: fe107bc8-3829-4600-9bfc-df89f59b0c70
REPORT RequestId: fe107bc8-3829-4600-9bfc-df89f59b0c70 Duration: 914.38 ms Billed Duration: 1000 ms Memory Size: 128 MB Max Memory Used: 71 MB Init Duration: 247.03 ms
from the CloudWatch Logs. This is standard print out behavior for a lambda function.
However, you can use log event filters in console which can help with locating specific {JSON body} of interest. It is the most basic and fastest to solution to use though.
More complex filtering of your logs is also possible, but I think this is not what you are after.