S3 can be configured to invoke lambda when an object arrives in it.
Is it possible to invoke a REST API (endpoint of a microservice running in EKS) when an object arrives in S3?
From November 2021 is possible to integrate S3 with Amazon EventBridge.
So you can create an EventBridge rule which is triggered on bucket object creation and has API destination as a target.
For this, the option Bucket Properties -> Amazon EventBridge -> "Send notifications to Amazon EventBridge for all events in this bucket" should be enabled.
Then, on EventBridge create the rule with an event pattern like this:
{
"source": ["aws.s3"],
"detail-type": ["Object Created"],
"detail": {
"bucket": {
"name": ["<my-bucket-name>"]
}
}
}
And configure the target as API destination endpoint (configure http method, endpoint, authorization).
You can set up an SNS topic as a target for the event from S3. In the SNS topic, you can add an HTTP/s subscriber, which can be your API endpoint.
Have the Lambda hit the REST API for you.
Related
I am trying to create an AWS Cloud watch event which will trigger an email whenever a S3 bucket is created or modified to allow public access.
I have created the cloud trail, log stream and am tracking all the S3 events logs. When i am trying to create a custom event by giving the pattern to detect S3 buckets with public access i am not able to fetch any response or the event doesn't get triggered even if i create bucket with public access. Can you help me out with the custom pattern for the same ?
I have tried giving GetPublicAccessBlock, PutPublicAccessBlock etc in event type but no luck. Please suggest accordingly.
you need to do the following in order to receive a notification
Enable CloudTrail for management events
Create an EventBridge Rule with an event pattern
AWS events or EventBridge partner events
Use Pattern from AWS Service, Simple Storage Service(S3) and Event Type as "AWS API Call via CloudTrail"
Note: This only works if you are turning off for an existing bucket (not for a new bucket)
The reason being when we create a bucket with public access, there are only two events generated, which are CreateBucket and PutBucketEncryption and they don't seem to have information regarding public access being turned on. However if we create a bucket with no public access then it generates an additional PutBucketPublicAccessBlock event with CreateBucket and PutBucketEncryption.
{
"source": ["aws.s3"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["s3.amazonaws.com"],
"eventName": ["PutBucketPublicAccessBlock", "DeleteBucketPublicAccessBlock"],
"requestParameters": {
"PublicAccessBlockConfiguration": {
"$or": [{
"RestrictPublicBuckets": [false]
}, {
"BlockPublicPolicy": [false]
}, {
"BlockPublicAcls": [false]
}, {
"IgnorePublicAcls": [false]
}]
}
}
}
}
I have a lambda function deployed in us-east-1 which runs every time an EC2 instance is started.
The lambda function is triggered with the following EventBridge configuration:
{
"detail-type": [
"AWS API Call via CloudTrail"
],
"source": [
"aws.ec2"
],
"detail": {
"eventName": [
"RunInstances"
]
}
}
The lambda function is working great. Now, I'm looking to extend this so that my lambda function is triggered even when an EC2 instance is launched in a different region (e.g. us-east-2).
How can I achieve this?
One option is to put SNS as an event target and subscribe the lambda to the SNS topic. SNS supports cross region subscriptions.
Another option is to use cross region event busses. You create a rule that forwards the event to another region and create another event rule in that region that invokes a lambda. More info here: https://aws.amazon.com/blogs/compute/introducing-cross-region-event-routing-with-amazon-eventbridge/
There was a recently announced new functionality that can help with cross region use cases with aws lambda: https://aws.amazon.com/blogs/compute/introducing-cross-region-event-routing-with-amazon-eventbridge/
Amazon eventBridge is a great way for cross region (and cross-account) event processing
After setting the EventBridge, S3 put object event still cannot trigger the StepFuction.
However, I tried to change the event rule to EC2 status. It's working !!!
I also try to change the rule to S3 all event, but it still not working.
Amazon EventBridge:
Event pattern:
{
"source": ["aws.s3"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["s3.amazonaws.com"],
"eventName": ["PutObject"],
"requestParameters": {
"bucketName": ["MY_BUCKETNAME"]
}
}
Target(s):
Type:Step Functions state machine
ARN:arn:aws:states:us-east-1:xxxxxxx:stateMachine:MY_FUNCTION_NAME
Reference:https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html
Your step function isn't being triggered because the PutObject events aren't being published to cloudtrail. S3 operations are classified as data events so you must enable Data events when creating your cloudTrail. The tutorial says next next and create which seems to suggest no additional options need to be selected. By default, Data Events on the next step (step 2 - Choose log events - as of this writing) is not checked. You have to check it and fill up the bottom part to specify if all buckets/events are to be logged.
I'm attempting to set up a Cloudwatch Event Rule to notify on any AWS IAM actions like DeleteUser or CreateUser. But when I tried to create an event pattern I couldn't find IAM in the service Name list even though when I searched in the AWS documentation i cant's find a mention of IAM not being supported by Cloudwatch event rules. So I tried to create a custom event but i didn't receive any email from SNS (my target), and yes I made sure cloudwatch has permissions to invoke SNS as we already have other working events, any idea on why this is not working ?
{
"source":[
"aws.iam"
],
"detail-type":[
"AWS API Call via CloudTrail"
],
"detail":{
"eventSource":[
"iam.amazonaws.com"
],
"eventName":[
"CreateUser",
"DeleteUser"
]
}
}
I figure it out, IAM emits cloudtrail events only in us-eas-1 and I'm using a different region, it worked when I created the Cloudwatch event in N. Virgenia
The source parameter needs to be "aws.cloudtrail" not "aws.iam".
IAM policy is a global service. It can only report in US-East-1(N.Virginia).
I have same exact config and the region is same as well but creating a new user still don't trigger the event as there is event in clouldtrail as well as in the monitoring of the event rule created. I see that they say in document that cloudtrail has to be enabled but when I create a rule for security group modification which is ec2 events then it is working fine but not with iam one. Is there any permission that I am missing for aws events to send logs to clould trail , if so how did you guys resolved it.
I'm trying to create an amazon cloudWatch rule which triggers whenever an object is uploaded into a bucket. I know that to do this I need to trigger on the PutObject Event, however best I can tell that requires enabling object level logging on the bucket. I will be using a multitude of buckets and want to be able to automate that process, and because of how most of the system is set up using boto3 seems to make the most sense. So how can I turn object-level logging on using boto3?
The only AWS official resource I've been able to find so far is: How Do I Enable Object-Level Logging for an S3 Bucket with AWS CloudTrail Data Events?
Which explains how to enable object level logging through the GUI.
I've also looked through the boto3 library documentation
Both have ultimately not been helpful based on my understanding.
My chief goal is to enable object-level logging through boto3, if that's something that can be done.
You can configure an Amazon S3 Event so that, when a new object is created, it can:
Trigger an AWS Lambda function
Put a message in an Amazon SQS queue
Send a message to an Amazon SNS topic
See: Configuring Amazon S3 Event Notifications
You can use the put_event_selectors() function in CloudTrail service.
client = boto3.client('s3')
client.put_event_selectors(
TrailName='TrailName',
EventSelectors=[
{
'ReadWriteType': 'All',
'IncludeManagementEvents': True,
'DataResources': [
{
'Type': 'AWS::S3::Object',
'Values': [
'arn:aws:s3:::your_bucket_name/',
]
},
]
},
])