I have event rule configured in multiple accounts in same region which will get the events from cloudtrail and trigger the events. Since there are so many accounts I cannot configure SNS topic in each and every account and confirm the subscription manually.
How do I have a centralized SNS topic in one account so that the event rule will be sent from all the accounts to that one topic which can trigger the alerts? I haven't worked on this topic earlier and hoping to get some light.
Note: I want to achieve this in CF template.
Here is the cf template that I use.
AWSTemplateFormatVersion: 2010-09-09
Description: ''
Parameters:
DestinationEmailAddress:
Description: ''
Type: String
AllowedPattern: ^[A-Za-z._%+-]+#[A-Za-z.-]+\.[A-za-z]{2,}$
SNSTopicName:
Description: The name of the SNS Topic.
Type: String
AllowedPattern: ^[\w+=,.#/-]+$
Resources:
SnsTopic:
Type: 'AWS::SNS::Topic'
Properties:
Subscription:
- Endpoint: !Ref DestinationEmailAddress
Protocol: email
TopicName: !Ref SNSTopicName
TopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: DefaultStatement
Effect: Allow
Principal:
Service:
- events.amazonaws.com
- cloudwatch.amazonaws.com
- config.amazonaws.com
- cloudformation.amazonaws.com
Action:
- "SNS:GetTopicAttributes"
- "SNS:SetTopicAttributes"
- "SNS:AddPermission"
- "SNS:RemovePermission"
- "SNS:DeleteTopic"
- "SNS:Subscribe"
- "SNS:ListSubscriptionsByTopic"
- "SNS:Publish"
- "SNS:Receive"
Resource: !Ref SnsTopic
Condition:
StringEquals:
"AWS:SourceOwner": 'AWS::'
- Sid: AllowEventRuletoPosttoSNSTopic
Effect: Allow
Principal:
Service: "events.amazonaws.com"
Action: "sns:Publish"
Resource: !Ref SnsTopic
Topics:
- !Ref SnsTopic
EventRule:
Type: 'AWS::Events::Rule'
Properties:
Name: ''
Description: ''
State: ENABLED
Targets:
- Arn: !Ref SnsTopic
Id: SNSTopic
EventPattern:
source:
- aws.kms
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- kms.amazonaws.com
eventName:
- DisableKey
Related
I am pretty new to CloudFormation and playing around with it. I am trying to create a ConfigRule that checks if an EC2 instance does not have monitoring enabled. If it is compliant or non-compliant it will then notify the user via email (SNS). It creates the resources with no errors and even sends a subscription email which I then subscribe to. However, if I change the monitoring of the ec2 to enabled or disabled, no email notification comes to my email. I tried this same scenario manually through the console and had no issues getting email notifications.
CF Designer
Here is my code:
AWSTemplateFormatVersion: "2010-09-09"
Description: ""
Parameters:
TopicName:
Type: String
Description: Topic Name
Default: my-topic
Resources:
snsTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: SNS EC2 Topic
Subscription:
- Endpoint: "test#gmail.com"
Protocol: email
TopicName: my-topic2
snsSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref snsTopic
Protocol: email
Endpoint: test#gmail.com
ConfigRule:
Type: "AWS::Config::ConfigRule"
DependsOn: ConfigurationRecorder
Properties:
ConfigRuleName: "ec2-monitoring-check"
Description: "A Config rule that checks if an instance has monitoring enabled"
Scope:
ComplianceResourceTypes:
- "AWS::EC2::Instance"
Source:
Owner: "AWS"
SourceIdentifier: "EC2_INSTANCE_DETAILED_MONITORING_ENABLED"
ConfigurationRecorder:
Type: "AWS::Config::ConfigurationRecorder"
Properties:
RoleARN:
Fn::GetAtt:
- "IamRoleForAwsConfig"
- "Arn"
RecordingGroup:
AllSupported: true
IncludeGlobalResourceTypes: true
DeliveryChannel:
Type: "AWS::Config::DeliveryChannel"
Properties:
S3BucketName:
Ref: "S3BucketForAwsConfig"
SnsTopicARN: !Ref snsTopic
S3BucketForAwsConfig:
Type: "AWS::S3::Bucket"
Properties: {}
IamRoleForAwsConfig:
Type: "AWS::IAM::Role"
Properties:
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSConfigRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: ""
Effect: "Allow"
Principal:
Service: "config.amazonaws.com"
Action: "sts:AssumeRole"
Policies:
- PolicyName: "allow-access-to-config-s3-bucket"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "s3:PutObject"
Resource:
- Fn::Join:
- ""
-
- Fn::GetAtt:
- "S3BucketForAwsConfig"
- "Arn"
- "/*"
Condition:
StringLike:
s3:x-amz-acl: "bucket-owner-full-control"
- Effect: "Allow"
Action:
- "s3:GetBucketAcl"
Resource:
Fn::GetAtt:
- "S3BucketForAwsConfig"
- "Arn"
myEC2Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-0cff7528ff583bf9a
InstanceType: t2.micro
Monitoring: false
UserData: !Base64 |
#!/bin/bash -ex
# put your script here
Tags:
- Key: Name
Value: TestEC2Instance
Metadata: {}
Conditions: {}
Outputs: {}
I tried a similar thing with CloudWatch events and the same thing occurs. I can subscribe to it but when the rule detects something, I never get the subsequent email notifications.
Anyone run into this issue or know how they can help? Thank you.
I've recently started learning and implementing services using AWS services. So, I guess I'm missing some small steps which I can't figure it.
I'm trying to implement the following diagram using the Cloudformation template. Everything is working fine unless. The Lambda and SQS queue are subscribed to the SNS topic successfully. Whenever a file is stored at the bucket, or even when I publish a message to the SNS topic manually, the lambda function is triggered successfully, but the message is not published to the SQS queue. I've also added the AWS::SQS::QueuePolicy to allow SNS to send messages to SQS, but it still does not work.
template.yml:
...
Resources:
S3ObjectPutTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ${AppName}-vrp-creation-${Environment}-topic
BucketToSNSPermission:
Type: AWS::SNS::TopicPolicy
...
Bucket:
Type: AWS::S3::Bucket
...
Lambda:
Type: AWS::Serverless::Function
...
Queue:
Type: AWS::SQS::Queue
Properties:
DelaySeconds: 0
MaximumMessageSize: 262144
MessageRetentionPeriod: 864000
QueueName: !Sub ${AppName}-${Environment}-queue
ReceiveMessageWaitTimeSeconds: 0
VisibilityTimeout: 90
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
TopicToQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
TopicArn: !Ref S3ObjectPutTopic
Endpoint: !GetAtt Queue.Arn
RawMessageDelivery: true
The full Cloudformation template.yaml file: template.yaml
You have mentioned Service: s3.amazonaws.com instead of Service: sns.amazonaws.com in your SQS policy. Update the template and try.
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
Cloudtrail default logs can be streamed to elasticsearch domain as shown in this image. How do I achieve this using cloudformation template?
Update:
If you are using aws-cli, take a look at my answer here.
Well, after a few hours of exploring and reading a lot of documentation I finally succeeded to create this template.
Designer Overview :
In order to enable the stream logs to elasticsearch we need to create the following resources:
The lambda function will forward the logs from cloudwatch log group to Elasticsearch.
Relevant IAM Role to get logs from cloudwatch and insert to Elasticsearch.
Lambda permission - The AWS::Lambda::Permission resource grants an AWS service or another account permission to use a function to allow the cloudwatch log group to trigger the lambda.
Subscription Filter - The AWS::Logs::SubscriptionFilter resource specifies a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events and have them delivered to a specific destination.
Template usage:
Download LogsToElasticsearch.zip from my Github page.
Update var endpoint = '${Elasticsearch_Endpoint}'; in index.js with your Elasticseatch url e.g - 'search-xxx-yyyy.eu-west-1.es.amazonaws.com';.
Copy the zip file to s3 bucket which will be used in the template (LambdaArtifactBucketName).
Fill relevant Parameters - you can find descriptions to each resource.
Template YAML:
AWSTemplateFormatVersion: 2010-09-09
Description: Enable logs to elasticsearch
Parameters:
ElasticsearchDomainName:
Description: Name of the Elasticsearch domain that you want to insert logs to
Type: String
Default: amitb-elastic-domain
CloudwatchLogGroup:
Description: Name of the log group you want to subscribe
Type: String
Default: /aws/eks/amitb-project/cluster
LambdaName:
Description: Name of the lambda function
Type: String
Default: amitb-cloudwatch-logs
LambdaRole:
Description: Name of the role used by the lambda function
Type: String
Default: amit-cloudwatch-logs-role
LambdaArtifactBucketName:
Description: The bucket where the lambda function located
Type: String
Default: amit-bucket
LambdaArtifactName:
Description: The name of the lambda zipped file
Type: String
Default: LogsToElasticsearch.zip
VPC:
Description: Choose which VPC the Lambda-functions should be deployed to
Type: 'AWS::EC2::VPC::Id'
Default: vpc-1111111
Subnets:
Description: Choose which subnets the Lambda-functions should be deployed to
Type: 'List<AWS::EC2::Subnet::Id>'
Default: 'subnet-123456789,subnet-123456456,subnet-123456741'
SecurityGroup:
Description: Select the Security Group to use for the Lambda-functions
Type: 'List<AWS::EC2::SecurityGroup::Id>'
Default: 'sg-2222222,sg-12345678'
Resources:
ExampleInvokePermission:
Type: 'AWS::Lambda::Permission'
DependsOn: ExampleLambdaFunction
Properties:
FunctionName:
'Fn::GetAtt':
- ExampleLambdaFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: !Sub 'logs.${AWS::Region}.amazonaws.com'
SourceArn: !Sub >-
arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CloudwatchLogGroup}:*
SourceAccount: !Ref 'AWS::AccountId'
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Ref LambdaRole
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: lambda-to-es-via-vpc-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'es:*'
Resource:
- !Sub >-
arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ElasticsearchDomainName}
- PolicyName: logs-and-ec2-permissions
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'ec2:CreateNetworkInterface'
- 'ec2:DescribeNetworkInterfaces'
- 'ec2:DeleteNetworkInterface'
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
ExampleLambdaFunction:
Type: 'AWS::Lambda::Function'
DependsOn: LambdaExecutionRole
Properties:
Code:
S3Bucket: !Ref LambdaArtifactBucketName
S3Key: !Ref LambdaArtifactName
FunctionName: !Ref LambdaName
Handler: !Sub '${LambdaName}.handler'
Role:
'Fn::GetAtt':
- LambdaExecutionRole
- Arn
Runtime: nodejs8.10
Timeout: '300'
VpcConfig:
SecurityGroupIds: !Ref SecurityGroup
SubnetIds: !Ref Subnets
MemorySize: 512
SubscriptionFilter:
Type: 'AWS::Logs::SubscriptionFilter'
DependsOn: ExampleInvokePermission
Properties:
LogGroupName: !Ref CloudwatchLogGroup
FilterPattern: '[host, ident, authuser, date, request, status, bytes]'
DestinationArn:
'Fn::GetAtt':
- ExampleLambdaFunction
- Arn
Results:
Cloudwatch log:
Hope you find it helpfull.
Update 02/09/2020:
node.js 8.10 is now deprecated, you should use node.js 10 or 12.
https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html
Here is example for lambda invocation by event:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: "EventRule"
EventPattern:
source:
- "aws.ec2"
detail-type:
- "EC2 Instance State-change Notification"
detail:
state:
- "stopping"
State: "ENABLED"
Targets:
-
Arn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "EventRule"
- "Arn"
How to modify this tp be able to invoke state machine not lambda? I have replaced "LambdaFunction with "MyStateMachine"
but how modify PermissionForEventsToInvokeLambda? There is no AWS::StepFunctions::Permission in cloudformation :-(
Here's an snippet from the template we used to have to trigger state machine processing from an S3 event:
InboundBucketPutObjectRule:
Type: 'AWS::Events::Rule'
Properties:
Name: !Sub 'inbound-bucket-put-object-rule'
EventPattern:
source:
- aws.s3
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- s3.amazonaws.com
eventName:
- PutObject
- CopyObject
- CompleteMultipartUpload
requestParameters:
bucketName:
- !Ref InboundBucket
Targets:
- Id: ProcessNewObject
Arn: !Ref StateMachine
RoleArn: !GetAtt
- StateMachineStartExecutionRole
- Arn
StateMachineStartExecutionPolicy:
Type: 'AWS::IAM::ManagedPolicy'
Properties:
ManagedPolicyName: !Sub 'state-machine-start-execution'
Roles:
- !Ref StateMachineStartExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'states:StartExecution'
Resource:
- !Ref StateMachine
StateMachineStartExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Sub 'state-machine-start-execution'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: 'sts:AssumeRole'
StateMachine:
Type: 'AWS::StepFunctions::StateMachine'
Properties:
StateMachineName: !Sub 'state-machine'
Taken from real template so I had to anonymize this which might have introduced errors.
Template structure:
InboundBucketPutObjectRule is the CloudWatch event which gets triggered on file uploads.
StateMachineStartExecutionPolicy + StateMachineStartExecutionRole essentially allow CloudWatch event to start execution of a state machine.
StateMachine - the state machine which should be started (definition omitted).
I wrote above that we used to have this - but not anymore. Please see the following answer for our current solution:
https://stackoverflow.com/a/57563395/303810
I'm trying to implement this simple architecture with Serverless Framework :
File Upload to S3 Bucket --> SNS Topic --> 2 Lambda Functions
Actually i'm just testing with just 1 lambda function and here is my code:
service: MyImageLibrary
provider:
name: aws
runtime: python3.6
stage: dev
region: us-west-2
package:
individually: true
functions:
handler: index.lambda_handler
resources:
Resources:
ImageUploadedTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: SNSTopicImageUploadedTopic
ImageUploadedTopicSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint:
Fn::GetAtt: [ TestLambdaFunction , "Arn" ]
Protocol: lambda
TopicArn:
arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
ImageUploadedTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowBucketToPushNotificationEffect
Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sns:Publish
Resource: "*"
Topics:
- arn:aws:sns:us-west-2:xxxxxxxxxxx:SNSTopicImageUploadedTopic
MyImagesBucket:
Type: AWS::S3::Bucket
DependsOn: ImageUploadedTopicPolicy
Properties:
BucketName: ${self:custom.bucketName}
NotificationConfiguration:
TopicConfigurations:
- Event: s3:ObjectCreated:*
Topic:
arn:aws:sns:us-west-2:xxxxxxxxxxxx:SNSTopicImageUploadedTopic
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- PUT
- POST
- HEAD
AllowedOrigins:
- "*"
AllowedHeaders:
- "*"
In the console i can see that the Lambda function is subscribed to the the Topic :
But in the Lambda Function the SNS topic is not set in the Trigger section :
Am i missing something ?
Ok, i GOT IT !
I had to add another permission to allow the Lambda function to be triggered by the SNS Topic :
TestLambdaFunctionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
Principal: sns.amazonaws.com
SourceArn:
Ref: ImageUploadedTopic
FunctionName:
Fn::GetAtt: [ TestLambdaFunction, "Arn" ]
And now in the console i can the SNS as trigger for my Lambda function :)