I'm writing a custom S3 bucket policy using AWS that requires canonical ID of the account as a key parameter. I can get the account ID programmatically using cdk core. You may refer the python sample below.
cid = core.Aws.ACCOUNT_ID
Is there any way that we can get the same for canonical-ID.
Update:
I've found a workaround using S3API call. I've added the following code in my CDK stack. May be helpful to someone.
def find_canonical_id(self):
s3_client = boto3.client('s3')
return s3_client.list_buckets()['Owner']['ID']
I found 2 ways to get the canonical ID (boto3):
Method-1 Through List bucket API (also mentioned by author in the update)
This method is recommended by AWS as well.
import boto3
client = boto3.client("s3")
response = client.list_buckets()
canonical_id = response["Owner"]["ID"]
Method-2 Through Get bucket ACL API
import boto3
client = boto3.client("s3")
response = client.get_bucket_acl(
Bucket='sample-bucket' # should be in your acct
)
canonical_id = response["Owner"]["ID"]
Related
I am following the steps in the SageMaker Monitoring Tutorial here:
https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_model_monitor/introduction/SageMaker-ModelMonitoring.html
And for the line:
bucket.Object(code_prefix + "/preprocessor.py").upload_file("preprocessor.py")
I get the error:
TypeError: expected string or bytes-like object
Which I dont understand, because the input to the upload_file() function is "preprocessor.py" which is a string.
To unblock you quickly, please take the sagemaker session route to upload the artifact to S3 bucket -
import sagemaker
from sagemaker import get_execution_role
# S3 bucket for saving code and model artifacts.
# Feel free to specify a different bucket and prefix
bucket = sagemaker.Session().default_bucket()
artifact_name = "preprocessor.py"
prefix = "code" // change as required
sample_url = sagemaker.Session().upload_data(
artifact_name, bucket=bucket, key_prefix=prefix"
)
I am trying to receive Stack resources ARN information using boto3.
I tried to use:
import boto3
client = boto3.resource('cloudformation',
aws_access_key_id='xxxxxxxx',
aws_secret_access_key='xxxxxxxxxxxx')
response = client.list_stack_resources(
StackName='ORG-ROLES')
I get "AttributeError: 'cloudformation.ServiceResource' object has no attribute 'list_stack_resources'"
This Stack runs 9 resources, I want to get one resource ARN information.
Hope you can help me.
You're mixing up the client-level and resource-level APIs. You need to use one or the other. Here's an example of each.
import boto3
session = boto3.Session(profile_name='xxxx', region_name='us-east-1')
STACK_NAME = 'ORG-ROLES'
# Use client-level API
client = session.client('cloudformation')
response = client.list_stack_resources(StackName=STACK_NAME)
print('Client API:', response['StackResourceSummaries'])
# Use resource-level API
resource = session.resource('cloudformation')
stack = resource.Stack(STACK_NAME)
print('Resource API:', list(stack.resource_summaries.all()))
i have n number of rules for a s3 bucket. I need to delete one of the rule which i configured using boto3.
but I am not finding the command for that. if I use
response = s3.bucket_lifecycle.delete() or
response = s3.delete_bucket_lifecycle(Bucket='examplebucket',)
how it will find a particular rule which i need to delete ?
You can use the boto3.resource to do this. See documentation.
import boto3
s3 = boto3.resource('s3')
bucket_lifecycles = s3.BucketLifecycle('bucket_name').rules
# pick a lifecycle and delete
for bucket_lifecycle in bucket_lifecycles:
if bucket_lifecycle['ID'] == 'something':
bucket_lifecycle.delete()
I have a s3 bucket with multiple folders. How can I generate s3 presigned URL for a latest object using python boto3 in aws for each folder asked by a user?
You can do something like
import boto3
from botocore.client import Config
import requests
bucket = 'bucket-name'
folder = '/' #you can add folder path here don't forget '/' at last
s3 = boto3.client('s3',config=Config(signature_version='s3v4'))
objs = s3.list_objects(Bucket=bucket, Prefix=folder)['Contents']
latest = max(objs, key=lambda x: x['LastModified'])
print(latest)
print (" Generating pre-signed url...")
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': bucket,
'Key': latest['Key']
}
)
print(url)
response = requests.get(url)
print(response.url)
here it will give the latest last modified file from the whole bucket however you can update login and update prefix value as per need.
if you are using Kubernetes POD, VM, or anything you can pass environment variables or use the python dict to store the latest key if required.
If it's a small bucket then recursively list the bucket, with prefix as needed. Sort the results by timestamp, and create the pre-signed URL for the latest.
If it's a very large bucket, this will be very inefficient and you should consider other ways to store the key of the latest file. For example: trigger a Lambda function whenever an object is uploaded and write that object's key into a LATEST item in DynamoDB (or other persistent store).
Is there a way to upload file to AWS S3 with Tags(not add Tags to an existing File/Object in S3). I need to have the file appear in S3 with my Tags , ie in a single API call.
I need this because I use a Lambda Function (that uses these S3 object Tags) is triggered by S3 ObjectCreation
You can inform the Tagging attribute on the put operation.
Here's an example using Boto3:
import boto3
client = boto3.client('s3')
client.put_object(
Bucket='bucket',
Key='key',
Body='bytes',
Tagging='Key1=Value1'
)
As per the docs, the Tagging attribute must be encoded as URL Query parameters. (For example, "Key1=Value1")
Tagging — (String) The tag-set for the object. The tag-set must be
encoded as URL Query parameters. (For example, "Key1=Value1")
EDIT: I only noticed the boto3 tag after a while, so I edited my answer to match boto3's way of doing it accordingly.
Tagging directive is now supported by boto3. You can do the following to add tags if you are using upload_file()
import boto3
from urllib import parse
s3 = boto3.client("s3")
tags = {"key1": "value1", "key2": "value2"}
s3.upload_file(
"file_path",
"bucket",
"key",
ExtraArgs={"Tagging": parse.urlencode(tags)},
)
If you're uploading a file using client.upload_file() or other methods that have the ExtraArgs parameter, you specify the tags differently you need to add tags in a separate request. You can add metadata as follows, but this is not the same thing. For an explanation of the difference, see this SO question:
import boto3
client = boto3.client('s3')
client.upload_file(
Filename=path_to_your_file,
Bucket='bucket',
Key='key',
ExtraArgs={"Metadata": {"mykey": "myvalue"}}
)
There's an example of this on the S3 docs, but you have to know that "metadata" corresponds to tags be aware that metadata is not exactly the same thing as tags though it can function similarly.
s3.upload_file(
"tmp.txt", "bucket-name", "key-name",
ExtraArgs={"Metadata": {"mykey": "myvalue"}}
)