I'm trying to publish a bucket notification to an SQS queue when an object is created in S3. I have read that I might be trying to use boto3 in mixed ways (i.e. as client & resource but neither will work).
s3 = boto3.client('s3')
s3.create_bucket(Bucket='my-bucket',
CreateBucketConfiguration={
'LocationConstraint': 'eu-west-2',
},
)
bucket_notification = s3.BucketNotification('my-bucket')
s3_notification_config = {
'QueueConfigurations': [
{
'QueueArn': 'arn:aws:sqs:location:number:number',
'Events': [
's3:ObjectCreated:*',
],
},
],
}
response = bucket_notification.put(NotificationConfiguration=s3_notification_config)
This gives me the following error:
AttributeError: 'S3' object has no attribute 'BucketNotification'
When I change the first line of code to be:
s3 = boto3.resource('s3')
I get the following error:
An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations
I understand the first error but not sure how to work around it by using resource instead of client when presented with the second error.
Related
I've observed for failed multipart upload (like crash or stop in the middle), the partially uploaded object still exist in storage.
I want to configure lifecycle rules for these incomplete objects via either minio or S3 C++ SDk.
I want to configure something like
{
"ID": "bucket-lifecycle-incomplete-chunk-upload",
"Status": "Enabled",
"NoncurrentVersionExpiration": {
"NoncurrentDays": 1
},
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 1
}
},
My C++ code looks like the following:
Aws::S3::Model::AbortIncompleteMultipartUpload incomplete_upload_config;
incomplete_upload_config.SetDaysAfterInitiation(days);
Aws::S3::Model::NoncurrentVersionExpiration version_expire;
version_expire.SetNoncurrentDays(1);
auto status = Aws::S3::Model::ExpirationStatus::Enabled;
Aws::S3::Model::LifecycleRule rule;
rule.SetID("bucket-lifecycle-incomplete-chunk-upload");
rule.SetStatus(std::move(status));
rule.SetNoncurrentVersionExpiration(std::move(version_expire));
rule.SetAbortIncompleteMultipartUpload(std::move(incomplete_upload_config));
Aws::S3::Model::BucketLifecycleConfiguration bkt_config;
bkt_config.AddRules(std::move(rule));
Aws::S3::Model::PutBucketLifecycleConfigurationRequest config_req{};
config_req.SetBucket(bucket);
config_req.SetLifecycleConfiguration(std::move(bkt_config));
auto outcome = client->PutBucketLifecycleConfiguration(config_req);
And I get the following result:
Received HTTP return code: 400; Failed to update config for bucket <bucket-name> because MalformedXML: Unable to parse ExceptionName: MalformedXML Message:
The pain point for this error is: I cannot find which additional or missing fields lead to this error.
I am trying to upload a binary Image to S3 and update a DynamoDB table in the same AWS Lambda Function. The problem is, whenever I try to make an API call, I get the following error in postman:
{
"errorMessage": "'itemId'",
"errorType": "KeyError",
"requestId": "bccaead6-cb60-4a5e-9fc7-14ff25380451",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 14, in lambda_handler\n s3_upload = s3.put_object(Bucket=bucket, Key=event[\"itemId\"] + \".png\", Body=decode_content)\n"
]
}
My events section takes in 3 Strings and whenever I try and access those strings, I get this error. However, if I try and access them without trying to upload to an S3 Bucket, everything works fine. My Lambda Function looks like this:
import json
import boto3
import base64
dynamoclient = boto3.resource("dynamodb")
s3 = boto3.client("s3")
table = dynamoclient.Table("Items")
bucket = "images"
def lambda_handler(event, context):
get_file_content = event["content"]
decode_content = base64.b64decode(get_file_content)
s3_upload = s3.put_object(Bucket=bucket, Key=event["itemId"] + ".png", Body=decode_content)
table.put_item(
Item={
'itemID': event["itemId"],
'itemName': event['itemName'],
'itemDescription': event['itemDescription']
}
)
return {
"code":200,
"message": "Item was added successfully"
}
Again, if I remove everything about the S3 file upload, everything works fine and I am able to update the DynamoDB table successfully. As for the API Gateway side, I have added the image/png to the Binary Media Types section. Additionally, for the Mapping Templates section for AWS API Gateway, I have added the content type image/png. In the template for the content type, I have the following lines:
{
"content": "$input.body"
}
For my Postman POST request, in the headers section, I have put this:
Finally, for the body section, I have added the raw event data with this:
{
"itemId": "0fx170",
"itemName": "Mouse",
"itemDescription": "Smooth"
}
Lastly, for the binary section, I have uploaded my PNG file.
What could be going wrong?
I have a .Net core client application using amazon Textract with S3,SNS and SQS as per the AWS Document , Detecting and Analyzing Text in Multipage Documents(https://docs.aws.amazon.com/textract/latest/dg/async.html)
Created an AWS Role with AmazonTextractServiceRole Policy and added the Following Trust relation ship as per the documentation (https://docs.aws.amazon.com/textract/latest/dg/api-async-roles.html)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "textract.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Subscribed SQS to the topic and Given Permission to the Amazon SNS Topic to Send Messages to the Amazon SQS Queue as per the aws documentation .
All Resources including S3 Bucket, SNS ,SQS are in the same us-west2 region
The following method shows a generic error "InvalidParameterException"
Request has invalid parameters
But If the NotificationChannel section is commented the code is working fine and returning the correct job id.
Error message is not giving a clear picture about the parameter. Highly appreciated any help .
public async Task<string> ScanDocument()
{
string roleArn = "aws:iam::xxxxxxxxxxxx:instance-profile/MyTextractRole";
string topicArn = "aws:sns:us-west-2:xxxxxxxxxxxx:AmazonTextract-My-Topic";
string bucketName = "mybucket";
string filename = "mytestdoc.pdf";
var request = new StartDocumentAnalysisRequest();
var notificationChannel = new NotificationChannel();
notificationChannel.RoleArn = roleArn;
notificationChannel.SNSTopicArn = topicArn;
var s3Object = new S3Object
{
Bucket = bucketName,
Name = filename
};
request.DocumentLocation = new DocumentLocation
{
S3Object = s3Object
};
request.FeatureTypes = new List<string>() { "TABLES", "FORMS" };
request.NotificationChannel = channel; /* Commenting this line work the code*/
var response = await this._textractService.StartDocumentAnalysisAsync(request);
return response.JobId;
}
Debugging Invalid AWS Requests
The AWS SDK validates your request object locally, before dispatching it to the AWS servers. This validation will fail with unhelpfully opaque errors, like the OP.
As the SDK is open source, you can inspect the source to help narrow down the invalid parameter.
Before we look at the code: The SDK (and documentation) are actually generated from special JSON files that describe the API, its requirements and how to validate them. The actual code is generated based on these JSON files.
I'm going to use the Node.js SDK as an example, but I'm sure similar approaches may work for the other SDKs, including .NET
In our case (AWS Textract), the latest Api version is 2018-06-27. Sure enough, the JSON source file is on GitHub, here.
In my case, experimentation narrowed the issue down to the ClientRequestToken. The error was an opaque InvalidParameterException. I searched for it in the SDK source JSON file, and sure enough, on line 392:
"ClientRequestToken": {
"type": "string",
"max": 64,
"min": 1,
"pattern": "^[a-zA-Z0-9-_]+$"
},
A whole bunch of undocumented requirements!
In my case the token I was using violated the regex (pattern in the above source code). Changing my token code to satisfy the regex solved the problem.
I recommend this approach for these sorts of opaque type errors.
After a long days analyzing the issue. I was able to resolve it .. as per the documentation topic only required SendMessage Action to the SQS . But after changing it to All SQS Action its Started Working . But Still AWS Error message is really misleading and confusing
you would need to change the permissions to All SQS Action and then use the code as below
def startJob(s3BucketName, objectName):
response = None
response = textract.start_document_text_detection(
DocumentLocation={
'S3Object': {
'Bucket': s3BucketName,
'Name': objectName
}
})
return response["JobId"]
def isJobComplete(jobId):
# For production use cases, use SNS based notification
# Details at: https://docs.aws.amazon.com/textract/latest/dg/api-async.html
time.sleep(5)
response = textract.get_document_text_detection(JobId=jobId)
status = response["JobStatus"]
print("Job status: {}".format(status))
while(status == "IN_PROGRESS"):
time.sleep(5)
response = textract.get_document_text_detection(JobId=jobId)
status = response["JobStatus"]
print("Job status: {}".format(status))
return status
def getJobResults(jobId):
pages = []
response = textract.get_document_text_detection(JobId=jobId)
pages.append(response)
print("Resultset page recieved: {}".format(len(pages)))
nextToken = None
if('NextToken' in response):
nextToken = response['NextToken']
while(nextToken):
response = textract.get_document_text_detection(JobId=jobId, NextToken=nextToken)
pages.append(response)
print("Resultset page recieved: {}".format(len(pages)))
nextToken = None
if('NextToken' in response):
nextToken = response['NextToken']
return pages
Invoking textract with Python, I received the same error until I truncated the ClientRequestToken down to 64 characters
response = client.start_document_text_detection(
DocumentLocation={
'S3Object':{
'Bucket': bucket,
'Name' : fileName
}
},
ClientRequestToken= fileName[:64],
NotificationChannel= {
"SNSTopicArn": "arn:aws:sns:us-east-1:AccountID:AmazonTextractXYZ",
"RoleArn": "arn:aws:iam::AccountId:role/TextractRole"
}
)
print('Processing started : %s' % json.dumps(response))
I am trying to use the boto3 SDK to enable server access logging through python. However, I keep getting the error of:
You must give the log-delivery group WRITE and READ_ACP permissions to the target bucket
I know I need to add permissions to that group, but I don't know how to do that through the Python SDK.
I've tried following Enabling Logging Programmatically - Amazon Simple Storage Service but I was unable to convert it to Python.
I've additionally tried putting the Grantee and Permissions inside of the put_bucket_logging call, but to no avail.
Listed below is my function to attempt to do this resulting in the aforementioned error:
def enableAccessLogging(clientS3, bucketName, storageBucket,
targetPrefix):
#Give the group log-delievery WRITE and READ_ACP permisions to the
#target bucket
acl = get_bucket_acl(clientS3, storageBucket)
new_grant = {
'Grantee': {
'ID' : 'LogDelivery',
'Type' : 'Group'
},
'Permission': 'FULL_CONTROL',
}
modified_acl = copy.deepcopy(acl)
modified_acl['Grants'].append(new_grant)
setBucketAcl(clientS3, bucketName, modified_acl)
response = clientS3.put_bucket_logging(
Bucket=bucketName,
BucketLoggingStatus={
'LoggingEnabled': {
'TargetBucket': storageBucket,
'TargetPrefix': targetPrefix
}
}
)
I figured it out, I made the new acl correctly, but when I applied it, I applied it to the source bucket not the targetBucket so for anyone else doing this, the correct code is below:
def enableAccessLogging(clientS3, bucketName, storageBucket,
targetPrefix):
#Give the group log-delievery WRITE and READ_ACP permisions to the
#target bucket
acl = get_bucket_acl(clientS3, storageBucket)
new_grant = {
'Grantee': {
'URI': "http://acs.amazonaws.com/groups/s3/LogDelivery",
'Type' : 'Group'
},
'Permission': 'FULL_CONTROL',
}
modified_acl = copy.deepcopy(acl)
modified_acl['Grants'].append(new_grant)
setBucketAcl(clientS3, storageBucket, modified_acl)
response = clientS3.put_bucket_logging(
Bucket=bucketName,
BucketLoggingStatus={
'LoggingEnabled': {
'TargetBucket': storageBucket,
'TargetPrefix': targetPrefix
}
}
)
How do I use boto3 to simulate the Add Event Source action on the AWS GUI Console in the Event Sources tab.
I want to programatically create a trigger such that if an object is created in MyBucket, it will call MyLambda function(qualified with an alias).
The relevant api call that I see in the Boto3 documentation is create_event_source_mapping but it states explicitly that it is only for AWS Pull Model while I think that S3 belongs to the Push Model. Anyways, I tried using it but it didn't work.
Scenarios:
Passing a prefix filter would be nice too.
I was looking at the wrong side. This is configured on S3
s3 = boto3.resource('s3')
bucket_name = 'mybucket'
bucket_notification = s3.BucketNotification(bucket_name)
response = bucket_notification.put(
NotificationConfiguration={'LambdaFunctionConfigurations': [
{
'LambdaFunctionArn': 'arn:aws:lambda:us-east-1:033333333:function:mylambda:staging',
'Events': [
's3:ObjectCreated:*'
],
},
]})