I have a lambda function that I need to run eveytime there is a change in my s3 Bucket. I have added the trigger and it is working just fine, but I was wondering if there is any way to limit the scope the lambda function is to be run... for example Instead of running over the entire bucket, it runs only in the folder (inside the bucket) that change has been made?! or something like that..!
You can specify rules:
- s3:
bucket: photos
event: s3:ObjectCreated:*
rules:
- prefix: uploads/
- suffix: .jpg
See the functions/events/s3 section in the yml definition.
Per this AWS announcement, you can add prefix or suffix restrictions for S3 event triggers.
Related
I want to create a Lambda function that is triggered from a S3 bucket created within the same CloudFormation stack but cannot get the syntax quite right.
The event should only be fired when an object is uploaded to /uploads. I also need to specify some bucket properties (CORS).
S3 bucket definition in resources
resources:
Resources:
myBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket
# CORS properties...
Event in function definition:
events:
- s3:
bucket: myBucket
event: s3:ObjectCreated:Put
rules:
- prefix: uploads/
I do not want to use existing: true because it creates some helper objects for this simple task. I cannot find any documentation or examples that fit my case.
The existing:true flag only relates to S3 buckets created outside of your serverless project, for buckets that already exist, which is not the case here.
The situation you face is that you can't use the typical serverless framework convenience of defining the bucket in the Lambda event trigger, like this:
functions:
users:
handler: users.handler
events:
- s3:
bucket: photos
event: s3:ObjectRemoved:*
The reason that you can't use that method is that it creates the photos bucket and does not allow you to supply additional bucket configuration, e.g. CORS or bucket policy.
The solution to this is to create the S3 bucket in the S3 provider configuration, with CORS policy, and then refer to the bucket from your Lambda function event configuration. For example:
provider:
s3:
photosBucket:
name: photos
versioningConfiguration:
Status: Enabled
corsConfiguration:
CorsRules
- rule1 here
I have a problem that so far I'm unable to identify the root cause.
I have an AWS step machine that should be invoked once a file is uploaded to an S3 bucket.
So far when I upload the file to the S3 bucket, the lambda function that is defined in the StartAt key (StartAt: ImgUploadedEvent) starts as I can see in the lambda logs.
Here is the code:
stepFunctions:
stateMachines:
ValidateImageStateMachine:
loggingConfig:
level: ALL
includeExecutionData: true
destinations:
- Fn::GetAtt: [ StepFuncLogGroup, Arn ]
definition:
Comment: "This state function validates the images after users upload them to S3"
StartAt: ImgUploadedEvent
States:
ImgUploadedEvent:
Type: Task
Resource:
Fn::GetAtt: [ImgUploaded, Arn]
End: true
Below is the lambda function that is declared as the start of the StepMachine
This lambda function as I can see from the logs indeed get called once I modified an Object in S3
functions:
ImgUploaded:
handler: src/stepfunctions/imageWasUploadedEvent.handler
events:
- s3:
bucket: !Ref AttachmentsBucket
existing: true
iamRoleStatements:
- Effect: "Allow"
Action:
- "states:StartExecution"
Resource:
- "*"
To check that the Step Function was working I created a log group and added it to the Step Function.
resources:
Resources:
StepFuncLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /aws/stepfunctions/${self:service}-${self:provider.stage}
I see this cloud watch log group correctly associated with the Step function in the AWS Console.
However when I upload an object to S3, I do see in the logs of the lambda functions that it was invoked, but I can not see any logs on the Step Function log group
My question is:
Is the Step Function indeed working and it is just an issue with the logs in the Step Function?
Or it is that the Step Function itself is not working and the lambda function is working just a lambda function totally independent of the Step Function?
What I have to do so the lambda function gets trigger as part of the Step Function?
BR
After studying how StepFunctions work I finally arrived to the conclusion that this is a wrong pattern.
In no place on the documentation of the plugins or Amazon it says that what I did is a pattern
StepFunctions can be started by events on CloudWatch and those events could be originated on a change on S3. That is not what I did here
There is no a direct link between an action on S3 and a StepFunction
The link in AWS documentation could confuse someone to think otherwise. Here is the title Starting a State Machine Execution in Response to Amazon S3 Events
But the state machine does not start because an S3 events but because the CloudWatch log that this event generates
A lambda proxy function is a good way of invoking an State Machine. This is a easy to use pattern and very common as we can use it with SQS etc
So the correct response to this question is
The state machine does not start because it is never invoked. We just called a lambda function that was used as the StartAt for an StateMachine. This does mean I invoked the State Machine.
That is the reason why there is no logs for the state machine meanwhile there are correct logs for the lambda function
Hope this response helps
I will add more details and reference to this response
BR
Or it is that the Step Function itself is not working and the lambda function is working just a lambda function totally independent of the Step Function?
To verify whether lambda was invoked as part of step function or not can't you just check execution history from the step function console. Also unless you have explicitly configured s3 to publish events to lambda, your lambda will not be automatically invoked upon uploading files to s3.
What I have to do so the lambda function gets trigger as part of the Step Function?
To be able to call trigger step function on file-upload to s3 you can follow this tutorial: https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html
Here's a snippet of my YAML file:
functions:
users:
handler: incoming.handler
events:
- s3:
bucket: mybucket
event: s3:ObjectCreated:*
rules:
- prefix: incoming/
- suffix: .zip
This will match for objects created named incoming/*.zip.
I want to add another key with date time for easy partitioning using Athena. So the full "path" in S3 will be something like this:
incoming/date=<timestamp>/<filename>.zip
Using the rules defined above do not seem to match with the result that the function does not fire.
How do I match using this technique?
If you cannot get this to work using pure Serverless framework functionality, you could try an alternative approach:
Setup the bucket so that SNS notification is triggered on object creation. You can then hook up your Serverless lambda to trigger from SNS.
See the docs here: https://docs.aws.amazon.com/en_pv/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html
Then use SNS filtering to look for .zip (or whatever else you want to use) to trigger only on the files you are interested in: https://serverless.com/framework/docs/providers/aws/events/sns/
good luck!
I have a s3 bucket.
I am using serverless to create a lambda function to read from s3.
https://serverless.com/
Everything works very well.
I want to add a lifecycle rule to this s3 bucket to delete files from s3 after few days using the serverless framework.
How can this be done in serverless.yml file ?
You can follow the aws documentation to configure the life cycle rule for s3 :
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-lifecycleconfig-rule.html
ex in YAML format :
# configure object lifecyle
LifecycleConfiguration:
Rules:
-
ExpirationInDays: 30
Prefix: '/'
Status: Enabled
I want to know if we can create a cloudwatch event that triggered on S3 bucket every time that a change occurs on S3. For example, if a file is uploaded on s3 we recieve an email.
I am using serverless framework, i found on the serverless documentation only stuff related to ec2, but not much things on s3. So please if anyone knows how to use cloudwatch with s3 i am all ears
https://serverless.com/framework/docs/providers/aws/events/s3/
e.g.
functions:
emailOnUpload:
handler: email.handler
events:
- s3:
bucket: photos
event: s3: ObjectCreated:*