Testing Event pattern in AWS lambda - python-2.7

I am pretty new to AWS and trying to update VPC security group when an Event is triggered by ELB
my Event pattern is something like below
{
"userIdentity": {
"invokedBy": [
"elasticloadbalancing.amazonaws.com"
]
},
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"CreateNetworkInterface"
]
}
and sample event is
{
"eventVersion": "1.05",
"userIdentity": {
"type": "Root",
"principalId": "233000205727",
"arn": "arn:aws:iam::233000205727:root",
"accountId": "233000205727",
"userName": "jmfe",
"invokedBy": "elasticloadbalancing.amazonaws.com"
},
"eventTime": "2017-04-15T09:10:30Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "CreateNetworkInterface",
"awsRegion": "us-east-1",
"sourceIPAddress": "elasticloadbalancing.amazonaws.com",
"userAgent": "elasticloadbalancing.amazonaws.com",
"requestParameters": {
"subnetId": "subnet-2dbdf874",
"description": "ELB app/prod-artifactory-b-01-alb/73090a08b6f70469",
"groupSet": {
"items": [
{
"groupId": "sg-3a3b9a47"
}
]
},
"privateIpAddressesSet": {}
},
"responseElements": {
"networkInterface": {
"networkInterfaceId": "eni-4ad99293",
"internalInterfaceId": "interface-4ad99293",
"subnetId": "subnet-2dbdf874",
"vpcId": "vpc-876a77e2",
"availabilityZone": "us-east-1d",
"description": "ELB app/prod-artifactory-b-01-alb/73090a08b6f70469",
"ownerId": "233000205727",
"requesterId": "210368014644",
"requesterManaged": true,
"status": "pending",
"macAddress": "0e:61:f3:36:52:da",
"privateIpAddress": "10.19.84.120",
"privateDnsName": "ip-10-19-84-120.ec2.internal",
"sourceDestCheck": true,
"groupSet": {
"items": [
{
"groupId": "sg-3a3b9a47",
"groupName": "prod-artifactory-b-artifactory-01_client_sg"
}
]
},
"attachableToInstanceBySet": {},
"associableWithElasticIpBySet": {},
"privateIpAddressesSet": {
"item": [
{
"privateIpAddress": "10.19.84.120",
"privateDnsName": "ip-10-19-84-120.ec2.internal",
"primary": true
}
]
},
"ipv6AddressesSet": {},
"tagSet": {}
}
},
"requestID": "064477de-13d2-4e9c-b579-b69a2e6c4882",
"eventID": "0928e0db-89c3-4b33-abd5-62020854599a",
"eventType": "AwsApiCall",
"recipientAccountId": "233000205727"
}
{
"userIdentity": {
"invokedBy": ["elasticloadbalancing.amazonaws.com"]
},
"eventSource": ["ec2.amazonaws.com"],
"eventName": ["CreateNetworkInterface"]
}
I have created a rule with the above event pattern and want to call an AWS lambda function , the code is in python 3.6 and its just a simple hello world
def lambda_handler(event, context):
# TODO implement
return 'Hello from Lambda test'
When ever I test the lambda function with example event (from AWS console) it always Shows 'Hello from Lambda test' , I think I am not able to test the event and event pattern matching , I have changes the event and event pattern with garbage values and still the result is always pass, question is how to test the event and event pattern and see if the event pattern is working correctly within the rule

Related

How to format Filter Criteria on Lambda Functions that are triggered via Kinesis Streams for DynamoDB?

I have a dynamodb table that has a Kinesis stream attached to it. See the relevant cloudformation configuration here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html#cfn-dynamodb-table-kinesisstreamspecification
Recently, AWS announced Filtering Event Sources for AWS Lambda
My goal would be to filter all events that begins with a specific string.
For example - say the original table has a document like:
"dynamodb": {
"ApproximateCreationDateTime": 1640276115300,
"Keys": {
"pk": "foo:random",
"sk": "bar:something"
},
.....
I want to filter all events that start with bar:. The data comes in this format in the lambda function logs:
{
"Records": [
{
"kinesis": {
"kinesisSchemaVersion": "1.0",
"partitionKey": "E7DF48140C98F2557BDAF0126B8443AC",
"sequenceNumber": "49624912313474127477164618281231039365742153203189809218",
"data": "eyJhd3NSZWdpb24iOiJ1cy1lYXN0LTEiLCJldmVudElEIjoiYTJlYTlmNGEtNWU5Zi00MzAwLWE0ZjItOWFlY2Y3ZTM2ZTA0IiwiZXZlbnROYW1lIjoiTU9ESUZZIiwidXNlcklkZW50aXR5IjpudWxsLCJyZWNvcmRGb3JtYXQiOiJhcHBsaWNhdGlvbi9qc29uIiwidGFibGVOYW1lIjoicnBwLXJlY29uLXdvcmstb3JkZXIiLCJkeW5hbW9kYiI6eyJBcHByb3hpbWF0ZUNyZWF0aW9uRGF0ZVRpbWUiOjE2NDAyNzYxMTUzMDAsIktleXMiOnsicGsiOnsiUyI6IndvcmtvcmRlcjo0MzA4NDY5I1FJTTEifSwic2siOnsiUyI6Im9mZmVyaW5nIn19LCJOZXdJbWFnZSI6eyJlbnRpdHlfdHlwZSI6eyJTIjoib2ZmZXJpbmcifSwid29ya19vcmRlcl9rZXkiOnsiUyI6IjQzMDg0NjkjUUlNMSJ9LCJidXllcl9yZXBfaWQiOnsiUyI6IjAifSwic2VsbGVyX2dyb3VwX2NvZGUiOnsiUyI6IkRMUiJ9LCJzZWxsZXJfZGVhbGVyaWQiOnsiUyI6IjU0Mzc3NzcifSwidXBkYXRlZCI6eyJOIjoiMTYzNzY1OTE5NS4zMzEwMjM2OTMwODQ3MTY3OTY4NzYifSwiYnV5ZXJfbmV0Ijp7IlMiOiIwLjAwIn0sInZpbiI6eyJTIjoiMkMzQ0NBQ0cwQ0gzNDE0ODUifSwiYnV5ZXJfbnVtYmVyIjp7IlMiOiIwIn0sInNrIjp7IlMiOiJvZmZlcmluZyJ9LCJzYmx1Ijp7IlMiOiI0MzA4NDY5In0sInNlbGxlcl9uYW1lIjp7IlMiOiJFUEVBTCBBVVRPIFNBTEVTIERCQTIifSwicGsiOnsiUyI6IndvcmtvcmRlcjo0MzA4NDY5I1FJTTEifSwiYnV5ZXJfZmVlIjp7IlMiOiIwLjAwIn0sImJ1eWVyX2FkaiI6eyJTIjoiMC4wMCJ9LCJzaXRlX2lkIjp7IlMiOiJRSU0xIn0sImJ1eWVyX3VuaXZlcnNhbCI6eyJTIjoiMCJ9fSwiT2xkSW1hZ2UiOnsiZW50aXR5X3R5cGUiOnsiUyI6Im9mZmVyaW5nIn0sIndvcmtfb3JkZXJfa2V5Ijp7IlMiOiI0MzA4NDY5I1FJTTEifSwiYnV5ZXJfcmVwX2lkIjp7IlMiOiIwIn0sInNlbGxlcl9ncm91cF9jb2RlIjp7IlMiOiJETFIifSwic2VsbGVyX2RlYWxlcmlkIjp7IlMiOiI1NDM3Nzc3In0sInVwZGF0ZWQiOnsiTiI6IjE2Mzc2NTkxOTUuMzMxMDIzNjkzMDg0NzE2Nzk2ODc1In0sImJ1eWVyX25ldCI6eyJTIjoiMC4wMCJ9LCJ2aW4iOnsiUyI6IjJDM0NDQUNHMENIMzQxNDg1In0sImJ1eWVyX251bWJlciI6eyJTIjoiMCJ9LCJzayI6eyJTIjoib2ZmZXJpbmcifSwic2JsdSI6eyJTIjoiNDMwODQ2OSJ9LCJzZWxsZXJfbmFtZSI6eyJTIjoiRVBFQUwgQVVUTyBTQUxFUyBEQkEyIn0sInBrIjp7IlMiOiJ3b3Jrb3JkZXI6NDMwODQ2OSNRSU0xIn0sImJ1eWVyX2ZlZSI6eyJTIjoiMC4wMCJ9LCJidXllcl9hZGoiOnsiUyI6IjAuMDAifSwic2l0ZV9pZCI6eyJTIjoiUUlNMSJ9LCJidXllcl91bml2ZXJzYWwiOnsiUyI6IjAifX0sIlNpemVCeXRlcyI6NjM0fSwiZXZlbnRTb3VyY2UiOiJhd3M6ZHluYW1vZGIifQ==",
"approximateArrivalTimestamp": 1640276115.796
},
"eventSource": "aws:kinesis",
"eventVersion": "1.0",
"eventID": "shardId-000000000004:49624912313474127477164618281231039365742153203189809218",
"eventName": "aws:kinesis:record",
"invokeIdentityArn": "arn:aws:iam::111111111111:role/acct-managed/foo-bar-role",
"awsRegion": "us-east-1",
"eventSourceARN": "arn:aws:kinesis:us-east-1:111111111111:stream/foo-bar-stream-role/consumer/foo-bar-consumer:1638560626"
}
]
}
Once data is decoded it looks like:
{
"awsRegion": "us-east-1",
"eventID": "a2ea9f4a-5e9f-4300-a4f2-9aecf7e36e04",
"eventName": "MODIFY",
"userIdentity": null,
"recordFormat": "application/json",
"tableName": "foo-bar",
"dynamodb": {
"ApproximateCreationDateTime": 1640276115300,
"Keys": {
"pk": "foo:random",
"sk": "bar:something"
},
"NewImage": {...},
"OldImage": {...},
"SizeBytes": 634
},
"eventSource": "aws:dynamodb"
}
What I have tried so far:
FilterCriteria:
Filters:
- Pattern: "{\"data\": { \"sk\": [ { \"prefix\": \"bar:\"} ] }}"
Filters:
- Pattern: "{\"data\": { \"dynamodb\": { \"sk\": [ { \"prefix\": \"bar:\"} ] }} }"
I was able to get it working using the following:
FilterCriteria:
Filters:
- Pattern: "{\"data\": { \"dynamodb\": { \"NewImage\": { \"sk\": { \"S\": [{ \"prefix\": \"rims:\" }] }}}}}"

Taxonomy of event- driven Lambda messages?

I'm building event driven AWS stacks with Lambda+APIGateway+SQS+SNS+S3+DynamoDB.
One of my constant frustrations is that, if you bind any of the above to Lambda (either through event notifications or event source mappings), the formats of the event messages received by the Lambda are completely different - so a message sent by S3 is completely different to one sent by SQS which is completely different to one sent by DynamoDB etc.
Normally I have to set up a Cloudformation stack with an event source + event source mapping + Lambda, then push a message onto the event source to see what message actually results. What a giant pain.
Is there not a single combined resource out there which lists the different schema formats of different event messages ? Hoping someone can point me in the right direction.
Lambda console provides some example events in Configure test event. Here are the examples from the console for the services you mentioned.
APIGateway (aws proxy)
{
"body": "eyJ0ZXN0IjoiYm9keSJ9",
"resource": "/{proxy+}",
"path": "/path/to/resource",
"httpMethod": "POST",
"isBase64Encoded": true,
"queryStringParameters": {
"foo": "bar"
},
"multiValueQueryStringParameters": {
"foo": [
"bar"
]
},
"pathParameters": {
"proxy": "/path/to/resource"
},
"stageVariables": {
"baz": "qux"
},
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, sdch",
"Accept-Language": "en-US,en;q=0.8",
"Cache-Control": "max-age=0",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Host": "1234567890.execute-api.us-east-1.amazonaws.com",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Custom User Agent String",
"Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
"X-Forwarded-For": "127.0.0.1, 127.0.0.2",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https"
},
"multiValueHeaders": {
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
],
"Accept-Encoding": [
"gzip, deflate, sdch"
],
"Accept-Language": [
"en-US,en;q=0.8"
],
"Cache-Control": [
"max-age=0"
],
"CloudFront-Forwarded-Proto": [
"https"
],
"CloudFront-Is-Desktop-Viewer": [
"true"
],
"CloudFront-Is-Mobile-Viewer": [
"false"
],
"CloudFront-Is-SmartTV-Viewer": [
"false"
],
"CloudFront-Is-Tablet-Viewer": [
"false"
],
"CloudFront-Viewer-Country": [
"US"
],
"Host": [
"0123456789.execute-api.us-east-1.amazonaws.com"
],
"Upgrade-Insecure-Requests": [
"1"
],
"User-Agent": [
"Custom User Agent String"
],
"Via": [
"1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)"
],
"X-Amz-Cf-Id": [
"cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA=="
],
"X-Forwarded-For": [
"127.0.0.1, 127.0.0.2"
],
"X-Forwarded-Port": [
"443"
],
"X-Forwarded-Proto": [
"https"
]
},
"requestContext": {
"accountId": "123456789012",
"resourceId": "123456",
"stage": "prod",
"requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
"requestTime": "09/Apr/2015:12:34:56 +0000",
"requestTimeEpoch": 1428582896000,
"identity": {
"cognitoIdentityPoolId": null,
"accountId": null,
"cognitoIdentityId": null,
"caller": null,
"accessKey": null,
"sourceIp": "127.0.0.1",
"cognitoAuthenticationType": null,
"cognitoAuthenticationProvider": null,
"userArn": null,
"userAgent": "Custom User Agent String",
"user": null
},
"path": "/prod/path/to/resource",
"resourcePath": "/{proxy+}",
"httpMethod": "POST",
"apiId": "1234567890",
"protocol": "HTTP/1.1"
}
}
SQS
{
"Records": [
{
"messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
"receiptHandle": "MessageReceiptHandle",
"body": "Hello from SQS!",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1523232000000",
"SenderId": "123456789012",
"ApproximateFirstReceiveTimestamp": "1523232000001"
},
"messageAttributes": {},
"md5OfBody": "7b270e59b47ff90a553787216d55d91d",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
"awsRegion": "us-east-1"
}
]
}
SNS
{
"Records": [
{
"EventSource": "aws:sns",
"EventVersion": "1.0",
"EventSubscriptionArn": "arn:aws:sns:us-east-1:{{{accountId}}}:ExampleTopic",
"Sns": {
"Type": "Notification",
"MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
"TopicArn": "arn:aws:sns:us-east-1:123456789012:ExampleTopic",
"Subject": "example subject",
"Message": "example message",
"Timestamp": "1970-01-01T00:00:00.000Z",
"SignatureVersion": "1",
"Signature": "EXAMPLE",
"SigningCertUrl": "EXAMPLE",
"UnsubscribeUrl": "EXAMPLE",
"MessageAttributes": {
"Test": {
"Type": "String",
"Value": "TestString"
},
"TestBinary": {
"Type": "Binary",
"Value": "TestBinary"
}
}
}
}
]
}
S3 (put)
{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "1970-01-01T00:00:00.000Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "EXAMPLE"
},
"requestParameters": {
"sourceIPAddress": "127.0.0.1"
},
"responseElements": {
"x-amz-request-id": "EXAMPLE123456789",
"x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "testConfigRule",
"bucket": {
"name": "example-bucket",
"ownerIdentity": {
"principalId": "EXAMPLE"
},
"arn": "arn:aws:s3:::example-bucket"
},
"object": {
"key": "test/key",
"size": 1024,
"eTag": "0123456789abcdef0123456789abcdef",
"sequencer": "0A1B2C3D4E5F678901"
}
}
}
]
}
DynamoDB
{
"Records": [
{
"eventID": "c4ca4238a0b923820dcc509a6f75849b",
"eventName": "INSERT",
"eventVersion": "1.1",
"eventSource": "aws:dynamodb",
"awsRegion": "us-east-1",
"dynamodb": {
"Keys": {
"Id": {
"N": "101"
}
},
"NewImage": {
"Message": {
"S": "New item!"
},
"Id": {
"N": "101"
}
},
"ApproximateCreationDateTime": 1428537600,
"SequenceNumber": "4421584500000000017450439091",
"SizeBytes": 26,
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"eventSourceARN": "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTableWithStream/stream/2015-06-27T00:48:05.899"
},
{
"eventID": "c81e728d9d4c2f636f067f89cc14862c",
"eventName": "MODIFY",
"eventVersion": "1.1",
"eventSource": "aws:dynamodb",
"awsRegion": "us-east-1",
"dynamodb": {
"Keys": {
"Id": {
"N": "101"
}
},
"NewImage": {
"Message": {
"S": "This item has changed"
},
"Id": {
"N": "101"
}
},
"OldImage": {
"Message": {
"S": "New item!"
},
"Id": {
"N": "101"
}
},
"ApproximateCreationDateTime": 1428537600,
"SequenceNumber": "4421584500000000017450439092",
"SizeBytes": 59,
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"eventSourceARN": "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTableWithStream/stream/2015-06-27T00:48:05.899"
},
{
"eventID": "eccbc87e4b5ce2fe28308fd9f2a7baf3",
"eventName": "REMOVE",
"eventVersion": "1.1",
"eventSource": "aws:dynamodb",
"awsRegion": "us-east-1",
"dynamodb": {
"Keys": {
"Id": {
"N": "101"
}
},
"OldImage": {
"Message": {
"S": "This item has changed"
},
"Id": {
"N": "101"
}
},
"ApproximateCreationDateTime": 1428537600,
"SequenceNumber": "4421584500000000017450439093",
"SizeBytes": 38,
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"eventSourceARN": "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTableWithStream/stream/2015-06-27T00:48:05.899"
}
]
}

Splunk props config for AWS Cloudtrail json logs

I need to ingest AWS cloudtrail logs pulled from S3. The files contain a single json payload that contains individual cloudtrail events. However splunk is not recognising the individual events and not splitting correctly. its just one big lump as a single event
Each file contains this format:
"Records": [
{
"apiVersion": "2012-06-01",
"awsRegion": "us-west-1",
"eventID": "c-c245-2c4-32v6-vfff",
"eventName": "DescribeLoadBalancers",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventTime": "2019-11-30T18:15:33Z",
"eventType": "AwsApiCall",
"eventVersion": "1.05",
"recipientAccountId": "redacted",
"requestID": "2xc454xc-2345-234cv5-2345",
"requestParameters": null,
"responseElements": null,
"sourceIPAddress": "1.1.1.1",
"userAgent": "aws-sdk-ruby3/3.75.0 jruby/2.3.3 java aws-sdk-elasticloadbalancing/1.19.0 cloudhealth",
"userIdentity": {
"accessKeyId": "redacted",
"accountId": "redacted",
"arn": "arn:aws:sts::redacted:assumed-role/team/AssumeRoleSession",
"principalId": "redacted:AssumeRoleSession",
"sessionContext": {
"attributes": {
"creationDate": "2019-11-30T17:45:06Z",
"mfaAuthenticated": "false"
},
"sessionIssuer": {
"accountId": "redacted",
"arn": "arn:aws:iam::redacted:team/company",
"principalId": "redacted",
"type": "Role",
"userName": "redacted"
},
"webIdFederationData": {}
},
"type": "AssumedRole"
}
},{
"apiVersion": "2012-06-01",
"awsRegion": "us-west-1",
"eventID": "c-c245-2c4-32v6-vfff",
"eventName": "DescribeLoadBalancers",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventTime": "2019-11-30T18:16:33Z",
"eventType": "AwsApiCall",
"eventVersion": "1.05",
"recipientAccountId": "redacted",
"requestID": "2xc454xc-2345-234cv5-2345",
"requestParameters": null,
"responseElements": null,
"sourceIPAddress": "1.1.1.1",
"userAgent": "aws-sdk-ruby3/3.75.0 jruby/2.3.3 java aws-sdk-elasticloadbalancing/1.19.0 cloudhealth",
"userIdentity": {
"accessKeyId": "redacted",
"accountId": "redacted",
"arn": "arn:aws:sts::redacted:assumed-role/team/AssumeRoleSession",
"principalId": "redacted:AssumeRoleSession",
"sessionContext": {
"attributes": {
"creationDate": "2019-11-30T17:45:06Z",
"mfaAuthenticated": "false"
},
"sessionIssuer": {
"accountId": "redacted",
"arn": "arn:aws:iam::redacted:role/team",
"principalId": "redacted",
"type": "Role",
"userName": "redacted"
},
"webIdFederationData": {}
},
"type": "AssumedRole"
}
}
]
}
My Props looks like this
[cloudtrail]
KV_MODE = json
some further googling and trial and error resulted in this props config which seems to break the events correctly
[cloudtrail]
KV_MODE = json
SHOULD_LINEMERGE=false
LINE_BREAKER=((?<=}),(?={)|[\r\n]+)
SEDCMD-remove_prefix=s/{"Records":\[//g
SEDCMD-remove_suffix=s/\]}//g

Cloud Formation: S3 linked to Lambda gives The ARN is not well formed

I'm trying to use CloudFormation to deploy an S3 bucket that on ObjectCreate invokes a Lambda function.
Here are my resources:
"ExampleFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "index.lambda_handler",
"Code": {
"S3Bucket": "bucketname",
"S3Key": "something.zip"
},
"Runtime": "python3.6",
"Role": {
"Fn::GetAtt": [
"LambdaExecutionRole",
"Arn"
]
}
}
},
"InputDataBucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": "input-data",
"NotificationConfiguration": {
"LambdaConfigurations": [
{
"Function": {
"Ref": "ExampleFunction"
},
"Event": "s3:ObjectCreated:*",
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "suffix",
"Value": "zip"
}
]
}
}
}
]
}
}
},
"LambdaInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"ExampleFunction",
"Arn"
]
},
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
},
"SourceArn": {
"Fn::Join": [
":",
[
"arn",
"aws",
"s3",
"",
"",
{
"Ref": "InputDataBucket"
}
]
]
}
}
}
I've tried to follow the documentation of the Notification Configuration, that says that there can be a circular dependency. However, if I follow the instructions I get the same error. Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-notificationconfig.html
When I try to create the stack, the S3 always breaks it with error "The ARN is not well formed"
I've tried many things, but I always receive this same error.
I can get this to work as long as I know the S3 bucket name in advance (mybucketname below). If you don't know the bucket name in advance, then you can enhance this to request the bucket name as a stack parameter and it should still work. If you need the bucket name to be auto-generated (so you can't predict the name in advance) then this will not work and you'll have to go the create/update route.
Key thing here is to manually create the S3 bucket ARN from the known bucket name, rather than relying on "Ref": "InputDataBucket" to get the bucket name for you.
Also worth reading this support article.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "stackoverflow-48037497",
"Resources" : {
"ExampleFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "index.lambda_handler",
"Code": {
"S3Bucket": "bucketname",
"S3Key": "something.zip"
},
"Runtime": "python3.6",
"Role": {
"Fn::GetAtt": [
"LambdaExecutionRole",
"Arn"
]
}
}
},
"LambdaInvokePermission": {
"Type": "AWS::Lambda::Permission",
"DependsOn": [ "ExampleFunction" ],
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"ExampleFunction",
"Arn"
]
},
"Principal": "s3.amazonaws.com",
"SourceAccount": {
"Ref": "AWS::AccountId"
},
"SourceArn": "arn:aws:s3:::mybucketname"
}
},
"InputDataBucket": {
"Type": "AWS::S3::Bucket",
"DependsOn": [ "ExampleFunction", "LambdaInvokePermission" ],
"Properties": {
"BucketName": "mybucketname",
"NotificationConfiguration": {
"LambdaConfigurations": [
{
"Function": { "Fn::GetAtt" : [ "ExampleFunction", "Arn" ] },
"Event": "s3:ObjectCreated:*"
}
]
}
}
}
}
}

Setup Lambda to trigger from CloudWatch using CloudFormation

I want to use CloudFormation to trigger Lambda when my CloudWatch function is called. I have the below, but it does not work.
CloudWatch rule created fine
"CloudWatchNewEc2": {
"Type": "AWS::Events::Rule",
"DependsOn": ["LambdaNewEc2"],
"Properties": {
"Description": "Triggered on new EC2 instances",
"EventPattern": {
"source": [
"aws.ec2"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"RunInstances"
]
}
},
"Targets": [
{
"Arn": {
"Fn::GetAtt": ["LambdaNewEc2", "Arn"]
},
"Id": "NewEc2AutoTag"
}
]
}
},
Lambda created but is not triggered
"LambdaNewEc2": {
"Type": "AWS::Lambda::Function",
"DependsOn": ["S3Lambda", "IAMRoleLambda"],
"Properties": {
"Code": {
"S3Bucket": {"Ref": "LambdaBucketName"},
"S3Key": "skynet-lambda.zip"
},
"Description": "When new EC2 instances are created, auto tag them",
"FunctionName": "newEc2AutoTag",
"Handler": "index.newEc2_autoTag",
"Role": {"Fn::GetAtt": ["IAMRoleLambda", "Arn"]},
"Runtime": "nodejs6.10",
"Timeout": "30"
}
}
},
It seems like CloudWatch Target is not sufficient?
UPDATE (Full CloudFormation template)
{
"Parameters": {
"Environment": {
"Type": "String",
"Default": "Staging",
"AllowedValues": [
"Testing",
"Staging",
"Production"
],
"Description": "Environment name"
},
"BucketName": {
"Type": "String",
"Default": "skynet-staging",
"Description": "Bucket Name"
},
"LambdaBucketName": {
"Type": "String",
"Default": "skynet-lambda",
"Description": "Lambda Bucket Name"
},
"Owner": {
"Type": "String",
"Description": "Owner"
}
},
"Resources": {
"S3Web": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Ref": "BucketName"
},
"WebsiteConfiguration": {
"IndexDocument": "index.html",
"RoutingRules": [
{
"RedirectRule": {
"ReplaceKeyPrefixWith": "#"
},
"RoutingRuleCondition": {
"HttpErrorCodeReturnedEquals": "404"
}
}
]
},
"AccessControl": "PublicRead",
"Tags": [
{
"Key": "Cost Center",
"Value": "Skynet"
},
{
"Key": "Environment",
"Value": {
"Ref": "Environment"
}
},
{
"Key": "Owner",
"Value": {
"Ref": "Owner"
}
}
]
}
},
"S3Lambda": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": {
"Ref": "LambdaBucketName"
},
"VersioningConfiguration": {
"Status": "Enabled"
},
"Tags": [
{
"Key": "Cost Center",
"Value": "Skynet"
},
{
"Key": "Owner",
"Value": {
"Ref": "Owner"
}
}
]
}
},
"CloudWatchNewEc2": {
"Type": "AWS::Events::Rule",
"DependsOn": ["LambdaNewEc2"],
"Properties": {
"Description": "Triggered on new EC2 instances",
"EventPattern": {
"source": [
"aws.ec2"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"ec2.amazonaws.com"
],
"eventName": [
"RunInstances"
]
}
},
"Targets": [
{
"Arn": {
"Fn::GetAtt": ["LambdaNewEc2", "Arn"]
},
"Id": "NewEc2AutoTag"
}
]
}
},
"IAMRoleLambda": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": "skynet-lambda-role",
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [ "lambda.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/AmazonEC2FullAccess",
"arn:aws:iam::aws:policy/AWSLambdaFullAccess",
"arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess",
"arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
]
}
},
"LambdaNewEc2": {
"Type": "AWS::Lambda::Function",
"DependsOn": ["S3Lambda", "IAMRoleLambda"],
"Properties": {
"Code": {
"S3Bucket": {"Ref": "LambdaBucketName"},
"S3Key": "skynet-lambda.zip"
},
"Description": "When new EC2 instances are created, auto tag them",
"FunctionName": "newEc2AutoTag",
"Handler": "index.newEc2_autoTag",
"Role": {"Fn::GetAtt": ["IAMRoleLambda", "Arn"]},
"Runtime": "nodejs6.10",
"Timeout": "30"
}
}
},
"Outputs": {
"WebUrl": {
"Value": {
"Fn::GetAtt": [
"S3Web",
"WebsiteURL"
]
},
"Description": "S3 bucket for web files"
}
}
}
I managed to deploy your template into a CloudFormation stack (by removing the LambdaBucket and pointing to my own zip file). It seems to create all resource correctly.
It took about 10 minutes for the RunInstances event to appear in CloudTrail. It then successfully triggered the Rule, but the CloudWatch metrics for my rule showed a failed invocation because I faked a Lambda function for your template.
Once I edited the rule to point to a better function and re-tested, it worked fine.
Bottom line: Seems to work!