AWS Lambda reading Athena database file and writing S3 to no avail - amazon-web-services

A help because are not writing the file to the S3 bucket
What did I do:
import time
import boto3
query = 'SELECT * FROM db_lambda.tb_inicial limit 10'
DATABASE = 'db_lambda'
output = 's3: // bucket-lambda-test1 / result /'
def lambda_handler (event, context):
client = boto3.client ('athena')
# Execution
response = client.start_query_execution (
QueryString = query,
QueryExecutionContext = {
Database: DATABASE
},
ResultConfiguration = {
'OutputLocation': output,
}
)
return response
return
IAM role created with:
AmazonS3FullAccess
AmazonAthenaFullAccess
CloudWatchLogsFullAccess
AmazonVPCFullAccess
AWSLambda_FullAccess
When running Lambda message:
Response:
{
"statusCode": 200,
"body": "\" Hello from Lambda! \ ""
}
Request ID:
"f2dd5cd2-070c-41ea-939f-d4909ce39fd0"
Function logs:
START RequestId: f2dd5cd2-070c-41ea-939f-d4909ce39fd0 Version: $ LATEST
END RequestId: f2dd5cd2-070c-41ea-939f-d4909ce39fd0
REPORT RequestId: f2dd5cd2-070c-41ea-939f-d4909ce39fd0 Duration: 0.84 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 52 MB
How I did the test:
Configure test event
A function can have a maximum of 10 test events. The events are maintained, so that you can change your computer or web browser and test the function with the same events.
Create new test event
Edit saved test events
Test event saved
{
}

The "Hello from Lambda" message is the default code in a Lambda function. It would appear that you did not click 'Deploy' before testing the function. Clicking Deploy will save the Lambda code.
Also, once you get it running, please note that start_query_execution() will simply start the Athena query. You will need to use get_query_results() to obtain the results.

Related

AWS Lambda cutting my payload in lambda invocation

I fried my brain a long day with a weird problem. I'm trying to send a .docx file document encoded in base64, initially, I thought that the AWS Api Gateway was cutting my body request, but invoking directly via awscli I check the lambda is cutting part of the body payload. Let me show you:
Here a snippet of my lambda function:
def lambda_handler(event, context):
if event["httpMethod"] == "GET":
return {
'statusCode': 200,
'body': json.dumps('Hello from PDFTron! Please send base64 doc to use this Lambda function!')
}
elif event["httpMethod"] == "POST":
try:
output_path = "/tmp/"
body = json.loads(str(event["body"])) # <--- the error is thrown here
base64str = body["file"]["data"]
filename = body["file"]["filename"]
base64_bytes = base64decode(base64str)
input_filename = filename.split('.')[0] + "docx"
Can you check the full file in lambda_function.py
Ok, now I try call my function via awscli with this command line:
aws lambda invoke \
--profile lab \
--region us-east-1 \
--function-name pdf_sign \
--cli-binary-format raw-in-base64-out \
--log-type Tail \
--query 'LogResult' \
--output text \
--payload file://payload.json response.json | base64 -d
payload.json
And receive the error:
C/ZcP1zcsiKiso3qnIUFO0Bk9/LjB7EOzkNAA7GgFDYu2A7RzzmPege9ijNyW/K0LvQKiYYtd21rNKycfuvBIr8syxsO7wi2gebCTwnZmHG+x/9N2jid9MWX+uApnxQ19L5TCPJbetnNGoe94JNV1A5VV5seZPWJ7BMTa7WFKCvBRyBeXWiivCsFH5FY7lRQGmmC8rq6Ezzj4rP3ndEKabbyq9HBRddi8TQILtJ7wfMQQU1sQL8FgwdJFXIqvhhL9a8EHwEJC2oblN8d1U1MbLTqYEnty1Z1EQT/bRCPoNJq18okfXuc70GjC0U0P2m5l6z4riKkoS3YXgWjLLIxbCQD7nzEIGuDHeWe+ADzsBybqyRyBOeBAxk0ED5XN1SITy31hv8QW+ViBw2j1ExOruxU44+sS9d7ZQ9yvXqog7O0v6MhDfxHfPa1W6ULOY7y3Jgt/9XgbuOVptXclLf5GWQesSErNLTXaTWTQTxSI6FL+emt3UJzivnbkQ7rZfxnZXU9K+kbLulko3uYfib5CwAA//8DAFBLAwQUAAYACAAAACEA+36uyJIBAAAiAwAAEQAIAWRvY1Byb3BzL2NvcmUueG1sIKIEASigAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAfJJNa9wwEEDvhf4Ho7tX0uZjg/E6tAmhhAYCdUnoTZUmGzW2JKRJnP33HdtrbzaEgA4zmqeHNKPy/LVtsheIyXq3ZnIhWAZOe2PdZs1+11f5GcsSKmdU4x2s2RYSO6++fil1KLSPcBt9gIgWUkYmlwod1uwRMRScJ/0IrUoLIhwVH3xsFVIaNzwo/aQ2wJdCnPIWUBmFivfCPMxGtlMaPSvDc2wGgdEcGmjBYeJyIfmeRYht+vDAUHlDtha3AT5Ep+JMvyY7g13XLbqjAaX7S35/8/PX8NTcur5XGlhVGl2gxQaqku9DitLz33+gcdyeE4p1BIU+VtcqeZf9UDFaCgZsKvVNf4Jt56NJJDjICDOQdLQBaZSj/mCD6EYlvKHZPlgw37fVtya7pqGmwfSu1uMRXmz/L6rjgZjTSXUbrUMw1VJIkYtlLk9rIQshaP2ZnRNU7iYzPgZMRh0txv5Plbuji8v6iu19ZzXJ5Gr0vTu/F7a7W39uPMnFKpermnQnx4fGSTC29PBXV/8BAAD//wMAUEsDBBQABgAIAAAAIQDKYulMAAEAAOQBAAAUAAAAd29yZC93ZWJTZXR0aW5ncy54bWyU0VFLwzAQB/B3we9Q8r6mHSpS1g5EBj6rHyDNrl1YLhfuMuv89IY6J+LLfLtwuR93/Ffrd/TFG7A4Cq2qy0oVECxtXRhb9fqyWdyrQpIJW+MpQKuOIGrdXV+tpmaC/hlSyj+lyEqQBm2rdinFRmuxO0AjJUUIuTkQo0n5yaNGw/tDXFjCaJLrnXfpqJdVdadODF+i0DA4C49kDwghzfOawWeRguxclG9tukSbiLeRyYJIvgf9l4fGhTNT3/yB0FkmoSGV+ZjTRjOVx+tqrtD/ALf/A5ZnAG3zNAZi0/scQd6kyJjqcgYUk0P3ARviB6ZJgHW30r+y6T4BAAD//wMAUEsDBBQABgAIAAAAIQA1ZRhYpAAAAP4AAAATACgAY3VzdG9tWG1sL2l0ZW0xLnhtbCCiJAAooCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACsz00KgzAQhuF9oXcIOYCRLlyICkLd1kKgq26SOJpAfiQZQW/fIKUn6HL4Hl6YRtY8bFFBIhwsKISJ42Ghpe/+2XOzox4mgyb4cZ6NgtFb46HYk6XkhA/hMs6WkhfElGFLK0p2Z32qZUs14lozlpQGJ1IRVvB5m0N0AvMZFxbO8D2ozYFHdivLikkjrQlLFKs+vrG/pLqG/R7urpcPAAAA//8DAFBLAQItABQABgAIAAAAIQCWW+dDiQEAADwGAAATAAAAAAAAAAAAAAAAAAAAAABbQ29udGVudF9UeXBlc10ueG1sUEsBAi0AFAAGAAgAAAAhAB6RGrfvAAAATgIAAAsAAAAAAAAAAAAAAAAAwgMAAF9yZWxzLy5yZWxzUEsBAi0AFAAGAAgAAAAhAFRBdBwzAQAASwUAABwAAAAAAAAAAAAAAAAA4gYAAHdvcmQvX3JlbHMvZG9jdW1lbnQueG1sLnJlbHNQSwECLQAUAAYACAAAACEA5fxODOgMAADnMAAAEQAAAAAAAAAAAAAAAABXCQAAd29yZC9kb2N1bWVudC54bWxQSwECLQAKAAAAAAAAACEArp7hlQsEAQALBAEAFQAAAAAAAAAAAAAAAABuFgAAd29yZC9tZWRpYS9pbWFnZTMuZ2lmUEsBAi0AFAAGAAgAAAAhAJa1reLxBQAAUBsAABUAAAAAAAAAAAAAAAAArBoBAHdvcmQvdGhlbWUvdGhlbWUxLnhtbFBLAQItAAoAAAAAAAAAIQAZDo2bmmgCAJpoAgAVAAAAAAAAAAAAAAAAANAgAQB3b3JkL21lZGlhL2ltYWdlMS5wbmdQSwECLQAUAAYACAAAACEABNjgzVrSAgCcAgcAFQAAAAAAAAAAAAAAAACdiQMAd29yZC9tZWRpYS9pbWFnZTIuZW1mUEsBAi0AFAAGAAgAAAAhABb3sCgZCAAArB0AABEAAAAAAAAAAAAAAAAAKlwGAHdvcmQvc2V0dGluZ3MueG1sUEsBAi0AFAAGAAgAAAAhALqKxOGRAgAAZgkAABIAAAAAAAAAAAAAAAAAcmQGAHdvcmQvZm9udFRhYmxlLnhtbFBLAQItABQABgAIAAAAIQB0Pzl6wgAAACgBAAAeAAAAAAAAAAAAAAAAADNnBgBjdXN0b21YbWwvX3JlbHMvaXRlbTEueG1sLnJlbHNQSwECLQAUAAYACAAAACEASUab3eAAAABVAQAAGAAAAAAAAAAAAAAAAAA5aQYAY3VzdG9tWG1sL2l0ZW1Qcm9wczEueG1sUEsBAi0AFAAGAAgAAAAhAIIHldWPDAAA+ncAAA8AAAAAAAAAAAAAAAAAd2oGAHdvcmQvc3R5bGVzLnhtbFBLAQItABQABgAIAAAAIQCiDemz4QEAAOMDAAAQAAAAAAAAAAAAAAAAADN3BgBkb2NQcm9wcy9hcHAueG1sUEsBAi0AFAAGAAgAAAAhAPt+rsiSAQAAIgMAABEAAAAAAAAAAAAAAAAASnoGAGRvY1Byb3BzL2NvcmUueG1sUEsBAi0AFAAGAAgAAAAhAMpi6UwAAQAA5AEAABQAAAAAAAAAAAAAAAAAE30GAHdvcmQvd2ViU2V0dGluZ3MueG1sUEsBAi0AFAAGAAgAAAAhADVlGFikAAAA/gAAABMAAAAAAAAAAAAAAAAARX4GAGN1c3RvbVhtbC9pdGVtMS54bWxQSwUGAAAAABEAEQBdBAAAQn8GAAAA'}}}
[ERROR] Runtime.MarshalError: Unable to marshal response: Object of type KeyError is not JSON serializable
Traceback (most recent call last):END RequestId: a26b93bd-bcf2-4072-99da-29ffcc3bc350
REPORT RequestId: a26b93bd-bcf2-4072-99da-29ffcc3bc350 Duration: 11.92 ms Billed Duration: 12 ms Memory Size: 1024 MB Max Memory Used: 93 MB
When I add a debug (print command) on my code I can check that the payload is cutted and in the log error conntains the the final part of my payload (eg. RX4GAGN1c3RvbVhtbC9pdGVtMS54bWxQSwUGAAAAABEAEQBdBAAAQn8GAAAA'}}}.
Reading the aws doc about lambda quotas and limits, we can see that the invocation payload limit is 6MB, very different than my payload size (232kb)
Resource
Quota
Invocation payload (request and response)
6 MB each for request and response (synchronous); 256 KB (asynchronous)
Source: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
I really think I'm doing all things, into the rules, someone here can tell me and help me, where am i going wrong or misunderstand?
I expect the payload of my request arrives correct in my python function.
How you logged event in your Lambda function?
It seems like the problem is this line:
body = json.loads(str(event["body"]))
You are using the CLI and based on the docs, the Event will be sent as a String. So you need to JSON parse the event object first.
output_path = "/tmp/"
e = json.loads(event)
body = str(e["body"])
base64str = body["file"]["data"]
filename = body["file"]["filename"]
base64_bytes = base64decode(base64str)
input_filename = filename.split('.')[0] + "docx"

Why my 'AWS Lambda Invoke Function' task in Azure DevOps Build Pipeline doesn't fail if the Lambda returns 400?

I have this python code inside Lambda:
#This script will run as a Lambda function on AWS.
import time, json
cmdStatus = "Failed"
message = ""
statusCode = 200
def lambda_handler(event, context):
time.sleep(2)
if(cmdStatus=="Failed"):
message = "Command execution failed"
statusCode = 400
elif(cmdStatus=="Success"):
message = "The script execution is successful"
statusCode = 200
else:
message = "The cmd status is: " + cmdStatus
statusCode = 500
return {
'statusCode': statusCode,
'body': json.dumps(message)
}
and I am invoking this Lambda from Azure DevOps Build Pipeline - AWS Lambda Invoke Function.
As you can see in the above code - have intentionally put that cmdStatus to Failed to make that Lambda fail but when executed from Azure DevOps Build Pipeline - the task succeeds. Strange.
How can I make the pipeline to fail in this case? Please help.
Thanks
I have been working with a similar issue myself and it looks like a bug in the task itself. It was reported in 2019 and nothing happened since so I wouldn't hold out much hope.
https://github.com/aws/aws-toolkit-azure-devops/issues/175
My workaround to this issue was to instead use the AWS CLI task with
Command: lambda
Subcommand: invoke
Options and Parameters: --function-name {nameOfYourFunction} response.json
Followed immediately by a bash task with an inline bash script
cat response.json
if grep -q "errorType" "response.json"; then
echo "An error was found"
exit 1
fi
echo "No error was found"

AWS Lambda hang until it timout on stipe invoices list

I am using AWS Lambda to host a nodeJs service that fetch my open invoices on Stripe and execute a payment and update my database.
The problem is that most of the time, but not all the time (sometimes everything goes how it should), it hang on the call of invoice list and do nothing.
Here's the part of the code where log stops :
const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY, {
maxNetworkRetries: 1,
timeout: 2000
});
[other imports]
const microservice = async (event, context, callback) => {
[some code including database connection]
console.log('retrieving all open invoices...')
let invoices;
try {
invoices = await stripe.invoices.list({
status: 'open',
limit: 100,
});
console.log(invoices.data.length + ' data retrieved.');
} catch (error) {
console.log('Unable fetch stripe invoices : ', error);
console.log('Exiting due to stripe connection error.');
reports.push(new Report('Unable fetch stripe invoices', 'ERROR'));
return {
statusCode: 500,
};
}
[code that process invoices]
return {};
};
module.exports.microservice = microservice;
And here the log output :
START RequestId: d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 Version: $LATEST
2021-10-26T00:04:05.741Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO Connecting to database...
2021-10-26T00:04:05.929Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO Executing (default): SELECT 1+1 AS result
2021-10-26T00:04:05.931Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO Connection has been established successfully.
2021-10-26T00:04:05.931Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 INFO retrieving all open invoices...
END RequestId: d628aa1e-dee6-4cc6-9ce0-f7c11cf73249
REPORT RequestId: d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 Duration: 15015.49 ms Billed Duration: 15000 ms Memory Size: 400 MB Max Memory Used: 40 MB
2021-10-26T00:04:20.754Z d628aa1e-dee6-4cc6-9ce0-f7c11cf73249 Task timed out after 15.02 seconds
And when it gooes all right it's like that :
START RequestId: e5fb6b08-adf9-433f-b1da-fd9ec29dde31 Version: $LATEST
2021-10-25T14:35:03.369Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Connecting to database...
2021-10-25T14:35:03.590Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Executing (default): SELECT 1+1 AS result
2021-10-25T14:35:03.600Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Connection has been established successfully.
2021-10-25T14:35:03.600Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO retrieving all open invoices...
2021-10-25T14:35:04.011Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO 0 data retrieved.
2021-10-25T14:35:04.011Z e5fb6b08-adf9-433f-b1da-fd9ec29dde31 INFO Everything went smoothly !
END RequestId: e5fb6b08-adf9-433f-b1da-fd9ec29dde31
REPORT RequestId: e5fb6b08-adf9-433f-b1da-fd9ec29dde31 Duration: 646.58 ms Billed Duration: 647 ms Memory Size: 400 MB
I don't get why it hangs with no error or log...
Network issues can happen due to various reasons. In your case, what you can try doing is to reduce the limit (e.g. limit : 30) and set your client library to retry the connection again by setting maxNetworkRetries : 3 or number that fits your application. When this is set, Stripe will retry the connection when the timeout error occurs.
This is a perfect match for Step functions use cases. It will allow you to orchestrate the steps of getting the invoices and processing them and easily design a retry mechanism in case of errors.
It is not really a solution, but what is causing the issue is that Stripe take a very long time to return less than 100 results.
We found workaround in order to not fetch this list.

AWS (GovCloud) Lambda Destination Not Triggering

I am working in AWS GovCloud I have the following configuration in AWS Lambda:
A Lambda function which decodes a payload
A Kinesis Stream set as a trigger for the aforementioned function
A Lambda Destination (we have tried Lambda functions as well as SQS, SNS)
No matter the configuration, I cannot seem to get Lambda to trigger the destination function (or queue in the event of SQS).
Here is the current Lambda Function. I have tried many permutations of the result/return payload without avail.
import base64
import json
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
for record in event['Records']:
payload = base64.b64decode(record['kinesis']['data']).decode('utf-8', 'ignore')
print("Success")
result = {
"statusCode": 202,
"headers": {
#'Content-Type': 'application/json',
},
"body": '{payload}'
}
return json.dumps(result)
I then send a message to Kinesis with the AWS CLI (I have noted that "Test" in the console does not observe desintations as per Jared Short ).
Every 0.1s: aws kinesis put-records --stream-name test-stream --records Data=SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IGZyb20gdGhlIEFXUyBDTEkh,PartitionKey=partitionkey1 Thu Jul 8 19:03:54 2021
{
"FailedRecordCount": 0,
"Records": [
{
"SequenceNumber": "49619938447946944252072058244333476686328287240252293122",
"ShardId": "shardId-000000000000"
}
]
}
Using Cloudwatch metrics and logs I am able to observe the function being triggered by the messages sent to Kinesis every .1 second.
The metrics charts indicate a success (as I expect).
Here is an example log from Cloudwatch:
START RequestId: 0cf3fb87-06e6-4e35-9de8-b30147e7be9d Version: $LATEST
Loading function
Success
END RequestId: 0cf3fb87-06e6-4e35-9de8-b30147e7be9d
REPORT RequestId: 0cf3fb87-06e6-4e35-9de8-b30147e7be9d Duration: 1.27 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB Init Duration: 113.64 ms
START RequestId: e663fa4a-2d0b-42d6-9e38-599712b71101 Version: $LATEST
Success
END RequestId: e663fa4a-2d0b-42d6-9e38-599712b71101
REPORT RequestId: e663fa4a-2d0b-42d6-9e38-599712b71101 Duration: 1.04 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB
START RequestId: b1373bbe-d2c6-49fb-a71f-dcedaf9210eb Version: $LATEST
Success
END RequestId: b1373bbe-d2c6-49fb-a71f-dcedaf9210eb
REPORT RequestId: b1373bbe-d2c6-49fb-a71f-dcedaf9210eb Duration: 0.98 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 51 MB
START RequestId: e0382653-9c33-44d6-82a7-a82f0f416297 Version: $LATEST
Success
END RequestId: e0382653-9c33-44d6-82a7-a82f0f416297
REPORT RequestId: e0382653-9c33-44d6-82a7-a82f0f416297 Duration: 1.05 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB
START RequestId: f9600ef5-419f-4271-9680-7368ccc5512d Version: $LATEST
Success
However, viewing the cloudwatch logs/metrics for the destination lambda function or SQS queue clearly show that the destination is not being triggered.
Over the course of troubleshooting, I have over-provisioned IAM policies to the Lambda function execution role so I am fairly confident that it is not an IAM related issue. Additionally, both functions are sharing the same execution role.
One thing I am not clear on after reviewing AWS documentation and 3rd party information is the criteria by which AWS determines success or failure for a given function. I am currently researching the invokation docs in search of what might be wrong here - but my interpretation is that AWS knows our function is successful based on the above Cloudwatch metrics showing a 100% success rate.
Does anyone know what I am doing wrong or how to try to troubleshoot the destination trigger for lambda?
Edit: As pointed out, the code is not correct for multiple record events. This is a function of senseless troubleshooting/changes to the code to get the Destination to trigger. Even something as simple as this does not invoke the destination.
import base64
import json
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# for record in event['Records']:
# payload = base64.b64decode(record['kinesis']['data']).decode('utf-8', 'ignore')
# print("Success")
# result = {
# "statusCode": 202,
# "headers": {
# 'Content-Type': 'application/json',
# },
# "body": '{"Success":True, payload}'
# }
return { "result": "OK" }
So, the question: Can someone demonstrate it is possible to have a Kinesis Stream Event Source Trigger a Lambda Function which successfully triggers a Lambda destination in AWS Govcloud?

AWS Lambda function fails while query Athena

I am attempting to write a simple Lambda function to query a table in Athena. But after a few seconds I see "Status: FAILED" in the Cloudwatch logs.
There is no descriptive error message on the cause of failure.
My test code is below:
import json
import time
import boto3
# athena constant
DATABASE = 'default'
TABLE = 'test'
# S3 constant
S3_OUTPUT = 's3://test-output/'
# number of retries
RETRY_COUNT = 1000
def lambda_handler(event, context):
# created query
query = "SELECT * FROM default.test limit 2"
# % (DATABASE, TABLE)
# athena client
client = boto3.client('athena')
# Execution
response = client.start_query_execution(
QueryString=query,
QueryExecutionContext={
'Database': DATABASE
},
ResultConfiguration={
'OutputLocation': S3_OUTPUT,
}
)
# get query execution id
query_execution_id = response['QueryExecutionId']
print(query_execution_id)
# get execution status
for i in range(1, 1 + RETRY_COUNT):
# get query execution
query_status = client.get_query_execution(QueryExecutionId=query_execution_id)
query_execution_status = query_status['QueryExecution']['Status']['State']
if query_execution_status == 'SUCCEEDED':
print("STATUS:" + query_execution_status)
break
if query_execution_status == 'FAILED':
#raise Exception("STATUS:" + query_execution_status)
print("STATUS:" + query_execution_status)
else:
print("STATUS:" + query_execution_status)
time.sleep(i)
else:
# Did not encounter a break event. Need to kill the query
client.stop_query_execution(QueryExecutionId=query_execution_id)
raise Exception('TIME OVER')
# get query results
result = client.get_query_results(QueryExecutionId=query_execution_id)
print(result)
return
The logs show the following:
2020-08-31T10:52:12.443-04:00
START RequestId: e5434651-d36e-48f0-8f27-0290 Version: $LATEST
2020-08-31T10:52:13.481-04:00
88162f38-bfcb-40ae-b4a3-0b5a21846e28
2020-08-31T10:52:13.500-04:00
STATUS:QUEUED
2020-08-31T10:52:14.519-04:00
STATUS:RUNNING
2020-08-31T10:52:16.540-04:00
STATUS:RUNNING
2020-08-31T10:52:19.556-04:00
STATUS:RUNNING
2020-08-31T10:52:23.574-04:00
STATUS:RUNNING
2020-08-31T10:52:28.594-04:00
STATUS:FAILED
2020-08-31T10:52:28.640-04:00
....more status: FAILED
....
END RequestId: e5434651-d36e-48f0-8f27-0290
REPORT RequestId: e5434651-d36e-48f0-8f27-0290 Duration: 30030.22 ms Billed Duration: 30000 ms Memory Size: 128 MB Max Memory Used: 72 MB Init Duration: 307.49 ms
2020-08-31T14:52:42.473Z e5434651-d36e-48f0-8f27-0290 Task timed out after 30.03 seconds
I think I have the right permissions for S3 bucket access given to the role (if not, I would have seen the error message). There are no files created in the bucket either. I am not sure what is going wrong here. What am I missing?
Thanks
The last line in your log shows
2020-08-31T14:52:42.473Z e5434651-d36e-48f0-8f27-0290 Task timed out after 30.03 seconds
To me this looks like the timeout of the Lambda Function is set to 30 seconds. Try increasing it to more than the time the Athena query needs (the maximum is 15 minutes).