i have n number of rules for a s3 bucket. I need to delete one of the rule which i configured using boto3.
but I am not finding the command for that. if I use
response = s3.bucket_lifecycle.delete() or
response = s3.delete_bucket_lifecycle(Bucket='examplebucket',)
how it will find a particular rule which i need to delete ?
You can use the boto3.resource to do this. See documentation.
import boto3
s3 = boto3.resource('s3')
bucket_lifecycles = s3.BucketLifecycle('bucket_name').rules
# pick a lifecycle and delete
for bucket_lifecycle in bucket_lifecycles:
if bucket_lifecycle['ID'] == 'something':
bucket_lifecycle.delete()
Related
I have a S3 bucket by name "archive_A". I have created a lambda function to retrieve meta data info for any object "creation" or "permanently delete" from S3 bucket as triggers to my lambda function (python) and insert the meta data collected into DynamoDB.
For S3 bucket archive_A, I have manually added the triggers, one for "creation" and another one for "permanently delete" in my lambda function via GUI.
import boto3
from uuid import uuid4
def lambda_handler(event, context):
s3 = boto3.client("s3")
dynamodb = boto3.resource('dynamodb')
for record in event['Records']:
bucket_name = record['s3']['bucket']['name']
object_key = record['s3']['object']['key']
size = record['s3']['object'].get('size', -1)
event_name = record ['eventName']
event_time = record['eventTime']
dynamoTable = dynamodb.Table('S3metadata')
dynamoTable.put_item(
Item={'Resource_id': str(uuid4()), 'Bucket': bucket_name, 'Object': object_key,'Size': size, 'Event': event_name, 'EventTime': event_time})
In the future there could be more S3 buckets like archive_B, archive_C etc. In that case I have to keep adding triggers manually for each S3 bucket which is bit cumbersome.
Is there any dynamic way or adding triggers to lambda for S3 buckets with name "archive_*" and hence any future S3 bucket with name like "archive_G" will have a dynamically added triggers to lambda.
Please suggest. I am quite new to AWS too. Any example would be easier to follow.
There is no in-built way to automatically add triggers for new buckets.
You could probably create an Amazon EventBridge rule that triggers on CreateBucket and calls an AWS Lambda function with details of the new bucket.
That Lambda function could then programmatically add a trigger on your existing Lambda function.
I'm writing a custom S3 bucket policy using AWS that requires canonical ID of the account as a key parameter. I can get the account ID programmatically using cdk core. You may refer the python sample below.
cid = core.Aws.ACCOUNT_ID
Is there any way that we can get the same for canonical-ID.
Update:
I've found a workaround using S3API call. I've added the following code in my CDK stack. May be helpful to someone.
def find_canonical_id(self):
s3_client = boto3.client('s3')
return s3_client.list_buckets()['Owner']['ID']
I found 2 ways to get the canonical ID (boto3):
Method-1 Through List bucket API (also mentioned by author in the update)
This method is recommended by AWS as well.
import boto3
client = boto3.client("s3")
response = client.list_buckets()
canonical_id = response["Owner"]["ID"]
Method-2 Through Get bucket ACL API
import boto3
client = boto3.client("s3")
response = client.get_bucket_acl(
Bucket='sample-bucket' # should be in your acct
)
canonical_id = response["Owner"]["ID"]
I have a s3 bucket with multiple folders. How can I generate s3 presigned URL for a latest object using python boto3 in aws for each folder asked by a user?
You can do something like
import boto3
from botocore.client import Config
import requests
bucket = 'bucket-name'
folder = '/' #you can add folder path here don't forget '/' at last
s3 = boto3.client('s3',config=Config(signature_version='s3v4'))
objs = s3.list_objects(Bucket=bucket, Prefix=folder)['Contents']
latest = max(objs, key=lambda x: x['LastModified'])
print(latest)
print (" Generating pre-signed url...")
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': bucket,
'Key': latest['Key']
}
)
print(url)
response = requests.get(url)
print(response.url)
here it will give the latest last modified file from the whole bucket however you can update login and update prefix value as per need.
if you are using Kubernetes POD, VM, or anything you can pass environment variables or use the python dict to store the latest key if required.
If it's a small bucket then recursively list the bucket, with prefix as needed. Sort the results by timestamp, and create the pre-signed URL for the latest.
If it's a very large bucket, this will be very inefficient and you should consider other ways to store the key of the latest file. For example: trigger a Lambda function whenever an object is uploaded and write that object's key into a LATEST item in DynamoDB (or other persistent store).
Is there a way to upload file to AWS S3 with Tags(not add Tags to an existing File/Object in S3). I need to have the file appear in S3 with my Tags , ie in a single API call.
I need this because I use a Lambda Function (that uses these S3 object Tags) is triggered by S3 ObjectCreation
You can inform the Tagging attribute on the put operation.
Here's an example using Boto3:
import boto3
client = boto3.client('s3')
client.put_object(
Bucket='bucket',
Key='key',
Body='bytes',
Tagging='Key1=Value1'
)
As per the docs, the Tagging attribute must be encoded as URL Query parameters. (For example, "Key1=Value1")
Tagging — (String) The tag-set for the object. The tag-set must be
encoded as URL Query parameters. (For example, "Key1=Value1")
EDIT: I only noticed the boto3 tag after a while, so I edited my answer to match boto3's way of doing it accordingly.
Tagging directive is now supported by boto3. You can do the following to add tags if you are using upload_file()
import boto3
from urllib import parse
s3 = boto3.client("s3")
tags = {"key1": "value1", "key2": "value2"}
s3.upload_file(
"file_path",
"bucket",
"key",
ExtraArgs={"Tagging": parse.urlencode(tags)},
)
If you're uploading a file using client.upload_file() or other methods that have the ExtraArgs parameter, you specify the tags differently you need to add tags in a separate request. You can add metadata as follows, but this is not the same thing. For an explanation of the difference, see this SO question:
import boto3
client = boto3.client('s3')
client.upload_file(
Filename=path_to_your_file,
Bucket='bucket',
Key='key',
ExtraArgs={"Metadata": {"mykey": "myvalue"}}
)
There's an example of this on the S3 docs, but you have to know that "metadata" corresponds to tags be aware that metadata is not exactly the same thing as tags though it can function similarly.
s3.upload_file(
"tmp.txt", "bucket-name", "key-name",
ExtraArgs={"Metadata": {"mykey": "myvalue"}}
)
The following code sadly lists all buckets of all regions and not only from "eu-west-1" as specified. How can I change that?
import boto3
s3 = boto3.client("s3", region_name="eu-west-1")
for bucket in s3.list_buckets()["Buckets"]:
bucket_name = bucket["Name"]
print(bucket["Name"])
s3 = boto3.client("s3", region_name="eu-west-1")
connects to S3 API endpoint in eu-west-1. It doesn't limit the listing to eu-west-1 buckets. One solution is to query the bucket location and filter.
s3 = boto3.client("s3")
for bucket in s3.list_buckets()["Buckets"]:
if s3.get_bucket_location(Bucket=bucket['Name'])['LocationConstraint'] == 'eu-west-1':
print(bucket["Name"])
If you need a one liner using Python's list comprehension:
region_buckets = [bucket["Name"] for bucket in s3.list_buckets()["Buckets"] if s3.get_bucket_location(Bucket=bucket['Name'])['LocationConstraint'] == 'eu-west-1']
print(region_buckets)
The solution above does not always work for buckets in some US regions because the 'LocationConstraint' can be null. Here is another solution:
s3 = boto3.client("s3")
for bucket in s3.list_buckets()["Buckets"]:
if s3.head_bucket(Bucket=bucket['Name'])['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region'] == 'us-east-1':
print(bucket["Name"])
The SDK method:
s3.head_bucket(Bucket=[INSERT_BUCKET_NAME_HERE])['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region']
... should always give you the bucket region. Thanks to sd65 for the tip: https://github.com/boto/boto3/issues/292