I have a s3 bucket and this is path where I will upload a file dev/uploads/excel . Now I want to add a trigger to invoke a my already made lambda function. is there any specific code I have to run once to enable trigger for this function using boto3 ?or need to paste somewhere ? I am confuse how It will work ?
You need to add a S3 trigger on the Lambda function and handle the S3 event in your code.
To create the S3 trigger select the Add trigger option on the left-hand side on the Lambda console.
Since you want to trigger off of new uploads, you can create this event to trigger off of the PUT event.
For the prefix you can add the path you want: dev/uploads/excel
A Lambda example in Python:
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# Get the object from the event and show its content type
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
try:
response = s3.get_object(Bucket=bucket, Key=key)
print("CONTENT TYPE: " + response['ContentType'])
return response['ContentType']
except Exception as e:
print(e)
print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
raise e
Also, there a lot of docs explaining this, like the following one: Tutorial: Using an Amazon S3 trigger to invoke a Lambda function.
Related
I need to add a s3 trigger in the lambda function in the source code itself instead of creating a trigger in the aws console. I need that trigger to read a file when it is uploaded on a particular folder of S3 bucket. I have done this using the creating the s3 trigger in the console itself with help of prefix. Can someone help with of creating this s3 trigger in the lambda function source code itself. Below is the source code of the lambda function for reading the file.
import json
import urllib.parse
import boto3
print('Loading function')
s3 = boto3.client('s3')
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
# Get the object from the event and show its content type
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
try:
response = s3.get_object(Bucket=bucket, Key=key)
print("CONTENT TYPE: " + response['ContentType'])
return response['ContentType']
except Exception as e:
print(e)
print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
raise e
The AWS Lambda function would presumably only be run when the trigger detects a new object.
However, you are asking how to create the trigger from the AWS Lambda function, but that function will only be run when a trigger exists.
So, it is a Cart before the horse - Wikipedia situation.
Instead, you can create the trigger in Amazon S3 from the AWS Lambda console or via an API call, but this is done as part of the definition of the Lambda function -- it can't be done from within the source code of the function, since it isn't run until the trigger already exists!
I have a S3 bucket by name "archive_A". I have created a lambda function to retrieve meta data info for any object "creation" or "permanently delete" from S3 bucket as triggers to my lambda function (python) and insert the meta data collected into DynamoDB.
For S3 bucket archive_A, I have manually added the triggers, one for "creation" and another one for "permanently delete" in my lambda function via GUI.
import boto3
from uuid import uuid4
def lambda_handler(event, context):
s3 = boto3.client("s3")
dynamodb = boto3.resource('dynamodb')
for record in event['Records']:
bucket_name = record['s3']['bucket']['name']
object_key = record['s3']['object']['key']
size = record['s3']['object'].get('size', -1)
event_name = record ['eventName']
event_time = record['eventTime']
dynamoTable = dynamodb.Table('S3metadata')
dynamoTable.put_item(
Item={'Resource_id': str(uuid4()), 'Bucket': bucket_name, 'Object': object_key,'Size': size, 'Event': event_name, 'EventTime': event_time})
In the future there could be more S3 buckets like archive_B, archive_C etc. In that case I have to keep adding triggers manually for each S3 bucket which is bit cumbersome.
Is there any dynamic way or adding triggers to lambda for S3 buckets with name "archive_*" and hence any future S3 bucket with name like "archive_G" will have a dynamically added triggers to lambda.
Please suggest. I am quite new to AWS too. Any example would be easier to follow.
There is no in-built way to automatically add triggers for new buckets.
You could probably create an Amazon EventBridge rule that triggers on CreateBucket and calls an AWS Lambda function with details of the new bucket.
That Lambda function could then programmatically add a trigger on your existing Lambda function.
I want to replicate few prefixes into another bucket.
I do not want to use the sync and replication service. I want to do it with lambda only and I have ongoing migration running so data is coming in every 20 minutes, no transformation is required.
Can someone help me how I can use AWS Lambda for sync and copy?
You can configure a Trigger for an AWS Lambda function so that it is invoked when an object is added to the Amazon S3 bucket. The function will receive information about the object that triggered the function, including its Bucket and Key.
You should code the Lambda function to call CopyObject() to copy that object to the desired destination bucket. Here's an example in Python:
import boto3
import urllib
def lambda_handler(event, context):
TARGET_BUCKET = 'my-target-bucket'
# Get the bucket and object key from the Event
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])
# Copy object
s3_client = boto3.client('s3')
s3_client.copy_object(
Bucket = TARGET_BUCKET,
Key = key,
CopySource= {'Bucket': bucket, 'Key': key}
)
I have three s3 buckets that are invoking a lambda function whenever there is a change in the content of specific objects inside the buckets.
Does anyone know if it is possible, using boto3, to retrieve those objects that have triggered the function?
Thanks!
UPDATE
I would like to get the objects that have triggered the lambda function from the response contents. I have tried to get it from the response of the get_function method of the lambda client but to no avail:
import boto3
lam = boto3.client('lambda')
response = lam.get_function(FunctionName='mylambdafunction')
Here's some sample code to retrieve the object that triggered the AWS Lambda function invocation:
import urllib
import boto3
# Connect to S3
s3_client = boto3.client('s3')
# This handler is executed every time the Lambda function is triggered
def lambda_handler(event, context):
# Get the bucket and object key from the Event
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])
localFilename = '/tmp/foo.txt'
# Download the file from S3 to the local filesystem
s3_client.download_file(bucket, key, localFilename)
# Do other stuff here
Basically, it extracts the Bucket and Key (filename) from the event data that is passed to the function, then calls download_file().
I am trying to get AWS CloudFormation to create a template that will allow me to attach an event to an existing S3 Bucket that will trigger a Lambda Function whenever a new file is put into a specific directory within the bucket. I am using the following YAML as a base for the CloudFormation template but cannot get it working.
---
AWSTemplateFormatVersion: '2010-09-09'
Resources:
SETRULE:
Type: AWS::S3::Bucket
Properties:
BucketName: bucket-name
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:Put
Filter:
S3Key:
Rules:
- Name: prefix
Value: directory/in/bucket
Function: arn:aws:lambda:us-east-1:XXXXXXXXXX:function:lambda-function-trigger
Input: '{ CONFIGS_INPUT }'
I have tried rewriting this template a number of different ways to no success.
Since you have mentioned that those buckets already exists, this is not going to work. You can use CloudFormation in this way but only to create a new bucket, not to modify existing bucket if that bucket was not created via that template in the first place.
If you don't want to recreate your infrastructure, it might be easier to just use some script that will subscribe lambda function to each of the buckets. As long as you have a list of buckets and the lambda function, you are ready to go.
Here is a script in Python3. Assuming that we have:
2 buckets called test-bucket-jkg2 and test-bucket-x1gf
lambda function with arn: arn:aws:lambda:us-east-1:605189564693:function:my_func
There are 2 steps to make this work. First, you need to add function policy that will allow s3 service to execute that function. Second, you will loop through the buckets one by one, subscribing lambda function to each one of them.
import boto3
s3_client = boto3.client("s3")
lambda_client = boto3.client('lambda')
buckets = ["test-bucket-jkg2", "test-bucket-x1gf"]
lambda_function_arn = "arn:aws:lambda:us-east-1:605189564693:function:my_func"
# create a function policy that will permit s3 service to
# execute this lambda function
# note that you should specify SourceAccount and SourceArn to limit who (which account/bucket) can
# execute this function - you will need to loop through the buckets to achieve
# this, at least you should specify SourceAccount
try:
response = lambda_client.add_permission(
FunctionName=lambda_function_arn,
StatementId="allow s3 to execute this function",
Action='lambda:InvokeFunction',
Principal='s3.amazonaws.com'
# SourceAccount="your account",
# SourceArn="bucket's arn"
)
print(response)
except Exception as e:
print(e)
# loop through all buckets and subscribe lambda function
# to each one of them
for bucket in buckets:
print("putting config to bucket: ", bucket)
try:
response = s3_client.put_bucket_notification_configuration(
Bucket=bucket,
NotificationConfiguration={
'LambdaFunctionConfigurations': [
{
'LambdaFunctionArn': lambda_function_arn,
'Events': [
's3:ObjectCreated:*'
]
}
]
}
)
print(response)
except Exception as e:
print(e)
You could write a custom resource to do this, in fact that's what I've ended up doing at work for the same problem. At the simplest level, define a lambda that takes a put bucket notification configuration and then just calls the put bucket notification api with the data that was passed it.
If you want to be able to control different notifications across different cloudformation templates, then it's a bit more complex. Your custom resource lambda will need to read the existing notifications from S3 and then update these based on what data was passed to it from CF.