Related
Say I have a Log Group in cloudWatch, which I want to subscribe to a lambda with a filter (Subscriptions->Stream to AWS Lambda).
I want to achieve it with cloudFormation template, but from cloudFormation doc, it seems that the only two available cloudWatch resources are Alarm/Dashboard.
Questions is:
Is there any way to write cloudWatch Log Group subscription in
cloudFormation?
If not, any alternative way (say from lambda
resource configuration in cloudFormation template)?
Oh, that's a tricky one. I only figured it out by creating one in the console and reverse enginerring - ick. But you're lucky - I have it on hand :P This is the json I was using for subscribing a lambda to a vpc flow log.
Note that the 'VPCFlowLogsGroup' is the logical Id of the log group, the 'FlowLogsCollector' that of the lambda.
"FlowLogsCollectorEventPermission": {
"Type" : "AWS::Lambda::Permission",
"Properties" : {
"Principal" : { "Fn::Sub": "logs.${AWS::Region}.amazonaws.com" },
"Action" : "lambda:InvokeFunction",
"FunctionName" : { "Fn::GetAtt": [ "FlowLogsCollector", "Arn" ] },
"SourceAccount": { "Ref": "AWS::AccountId" },
"SourceArn" : { "Fn::GetAtt": [ "VPCFlowLogsGroup", "Arn" ] }
}
},
"FlowLogsCollectorSubscription": {
"Type" : "AWS::Logs::SubscriptionFilter",
"DependsOn": "FlowLogsCollectorEventPermission",
"Properties" : {
"LogGroupName" : { "Ref" : "VPCFlowLogsGroup" },
"FilterPattern" : "",
"DestinationArn" : { "Fn::GetAtt" : [ "FlowLogsCollector", "Arn" ] }
}
},
I am trying to create RDS cluster and aurora instance using the cloudoformation template below:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "example setup",
"Parameters" : {
"DBInstanceIdentifier" : {
"Type": "String",
"Description": "Name for the DB instance."
},
"DBUser" : {
"Type": "String",
"Description": "Master user"
},
"DBPassword" : {
"Type": "String",
"Description": "Pass"
},
"DBModel" : {
"Type": "String",
"Description": "Instance model to be used for the DB."
}
},
"Resources": {
"RDSCluster": {
"Type": "AWS::RDS::DBCluster",
"Properties": {
"MasterUsername": { "Ref" : "DBUser" },
"MasterUserPassword": { "Ref" : "DBPassword" },
"Engine": "aurora",
"DBClusterParameterGroupName": "default.aurora5.6",
"VpcSecurityGroupIds": [{"Fn::GetAtt" : [ "DBFromSiteSecurityGroup" , "GroupId" ]}]
}
},
"AuroraInstance": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBInstanceIdentifier": { "Ref" : "DBInstanceIdentifier" },
"DBParameterGroupName": "default.aurora5.6",
"Engine": "aurora",
"DBClusterIdentifier": {
"Ref": "RDSCluster"
},
"PubliclyAccessible": "true",
"DBInstanceClass": { "Ref" : "DBModel" }
}
},
"DBFromSiteSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable MySQL",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "3306", "ToPort" : "3306", "CidrIp" : "195.171.102.98/32"}
]
}
},
"DBFromSiteSecurityGroupIngress1" : {
"Type" : "AWS::EC2::SecurityGroupIngress",
"Properties" : {
"GroupName" : { "Ref" : "DBFromSiteSecurityGroup" },
"IpProtocol" : "tcp",
"ToPort" : "3306",
"FromPort" : "3306",
"SourceSecurityGroupName" : { "Ref" : "DBFromSiteSecurityGroup" }
}
}
}
}
The db_model parameter I am passing is "db.t2.medium". The cluster gets created successfully in the cloudformation console however the AWS::RDS::DBInstance creation fails with the following error
"DeletionPolicy:Snapshot cannot be specified for a cluster instance, use deletion policy on the cluster instead."
What's more weird that when I try to run the same CF template in say eu london region, it works fine!!! Is there something wrong with the EU ireland region and aurora?
From AWS Support
This is a known issue and has been reported by other customers as well. The service team is currently working on the fix for this but there is no ETA as to when that would be pushed.
The work-around in the meanwhile is to specify a DeletionPolicy inside the DB instance resource definition that is failing to create, with the value of 'Delete'. [1]
An example below:
"Resources": {
"Database1": {
"DeletionPolicy": "Delete",
"Properties": {...},
"Type": "AWS::RDS::DBInstance"
}
}
References:
[1] DeletionPolicy - http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html#w2ab2c19c23c11c17
An update from AWS Support:
When creating an Amazon Aurora DBInstance in a DB Cluster using AWS
CloudFormation, CloudFormation applies a default Deletion policy of
“Delete”, if a deletion policy is not specified. If a Deletion Policy
of “Snapshot” is specified for an Amazon Aurora DBInstance,
CloudFormation returns an error, because instances in a DB Cluster
cannot be snapshotted individually; Snapshotting must be done at the
DB Cluster level.
As part of a recent deployment, we inadvertently changed the default
deletion policy for an Amazon Aurora DBInstance to “Snapshot”. This
caused our template validation to fail. To remedy this, CloudFormation
is reverting the value of default DeletionPolicy for Amazon Aurora
DBInstances to “Delete”. This fix will be completed by 21st July,
2017. Until this fix is completely rolled out, customers can explicitly override our incorrect defaults by specifying a deletion
policy of “Delete” for Amazon Aurora DBInstances.
We have corrected the gap in our testing that led to this situation,
and will continue to improve our testing to prevent recurrences. We
recognize how critical it is for us to preserve existing behavior for
our customers, and apologize for this inconvenience.
How do I get this DNSRecord as an Email notification from CloudFormation?
"DNSRecord" : {
"Type" : "AWS::Route53::RecordSet",
"Properties" : {
"HostedZoneName" : { "Fn::Join" : [ "", [{"Ref" : "HostedZoneName"}, "." ]]},
"Name" : { "Fn::Join" : [ "", [{"Ref" : "RecordSetName"}, ".", {"Ref" : "HostedZoneName"}, "."]]},
"Type" : "CNAME",
"ResourceRecords" :[ { "Fn::ImportValue" : "DNSName" } ],
"TTL" : "900"
}
}
AWS CloudFormation will not send an email, but your calling script could do so...
An AWS CloudFormation stack can produce Outputs, such as the name of a DNS record created as part of the stack.
If you have automated the creation of a CloudFormation stack (eg you have a script that calls CloudFormation to trigger the stack creation), it can receive back the content of these outputs. Your script could then extract the desired information and send an email.
Alternatively, you could create an AWS Lambda-backed Custom Resources that dependsOn the DNSRecord. You could then write a Lambda function that sends an email message.
Bottom line: No fully-automated method, but you could write your own.
I have a Lambda function deployed to several regions. I would like to publish a message to SNS that will invoke these functions.
Using aws-cli I've created the topics, given Lambda permission to talk to SNS, and create the subscriptions. The subscription appears to be created successfully, and I can see it in the AWS console. But, it doesn't work. The lambda function does not get invoked.
This is CloudFormation based example. You have to add invoke permission for SNS to the Lambda functions:
{
"Type" : "AWS::Lambda::Permission",
"Properties" : {
"FunctionName" : { "Fn::GetAtt" : [ "YourLambda", "Arn" ] },
"Action" : "lambda:InvokeFunction",
"Principal" : "sns.amazonaws.com",
"SourceArn" : { "Ref" : "YourSNSTopicArn" }
}
}
Then you need to subscribe your Lambdas to your SNS topic. This can be done via API call or through CloudFormation.
{
"Type" : "AWS::SNS::Topic",
"Properties" : {
"TopicName" : "YourTopicName",
"Subscription" : [ {
"Endpoint" : { "Fn::GetAtt" : [ "YourLambda", "Arn" ] },
"Protocol": "lambda"
} ]
}
}
If you're missing any of this, your Lambdas won't invoke. Source for the above information is the official blog article Invoking Lambda functions via SNS.
I've got a CloudFormation script that generates a SecurityGroup and an ELB; I'm trying to reference the SecurityGroup in the ELB creation; here's the resources bit:
"ELBSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Security group for the Arena dev stack",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : 80, "ToPort" : 80, "CidrIp" : { "Ref" : "OfficeIp" }}
]
}
},
"ProjectLoadBalancerTest" : {
"Type" : "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties" : {
"AvailabilityZones" : { "Fn::GetAZs" : "" },
"Instances" : [ ],
"Listeners" : [ {
"LoadBalancerPort" : "80",
"InstancePort" : "12345",
"Protocol" : "HTTP"
} ],
"HealthCheck" : {
"Target" : {
"Fn::Join" : [ "", [ "HTTP:", "12345", "/status.json" ] ]
},
"HealthyThreshold" : "2",
"UnhealthyThreshold" : "5",
"Interval" : "60",
"Timeout" : "30"
},
"SecurityGroups" : [
{ "Ref" : "ELBSecurityGroup" }
]
}
}
Unfortunately, this fails with:
Invalid id: "sebelbtest2-ELBSecurityGroup-1F5Z5DIIVQKD1" (expecting "sg-...")
So how can I reference ELBSecurityGroup for use as a property in the ELB creation?
Thanks!
As mytwocents mentioned, the solution is to use Fn::GetAtt. SecurityGroups are now supported by this function: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html
This works on the ELB:
...
"SecurityGroups" : [
{ "Fn::GetAtt" : [ "ELBSecurityGroup", "GroupId" ] }
]
...
Note. If you're putting this into a non-default VPC you'll also need to specify the VPC for the security group, and a subnet ID for the ELB.
As my CloudFormation script is all done within a VPC, I figured out what the problem was - I was creating the Security group, but not specifying the VpcId for it.
Security groups appear to be either normal security groups, or VPC security groups; if you do { "Ref": "MySecurityGroup" } on a normal one, you get the security group name, but not the ID. If you do { "Ref": "MySecurityGroup" } on a VPC one, you get back the sg-abcdef id, which is what is required for the ELB security group parameter.
So the full answer is:
"ELBSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Security group for the ELB",
"VpcId" : { "Ref" : "VpcId" },
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : 80, "ToPort" : 80, "CidrIp" : { "Ref" : "OfficeIp" }}
]
}
},
"MyELB": {
"Type" : "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties" : {
"AvailabilityZones" : { "Fn::GetAZs" : "" },
"Listeners" : [ {
"LoadBalancerPort" : "80",
"InstancePort" : 8000,
"Protocol" : "HTTP"
} ],
"SecurityGroups" : [ { "Ref" : "ELBSecurityGroup" } ]
}
}
This all works perfectly (provided everything you're doing is within your VPC) and in my configuration, will successfully limit access to whatever OfficeIP is set to.
Hmmm…
It appears as though the template is returning the Security Group's name instead of its ID.
Based on the documentation, it would appear that SecurityGroups can only be attached to a load balancer in a VPC.
If it were me, I would do one of two things (or both):
Post my question to the official AWS CloudFormation forum to try and get it answered from someone on that team.
Use the AWS Console, SDKs or CLI tools to set up your environment the way you want, then use the CloudFormer tool to generate a CloudFormation template that matches your environment.
I had the same issue - but decided that since this is just for initial set-up, the elastic load balancer can be managed seperate from cloud formation until they work this bug out. Also, cloud former is not totally reliable especially when you have security groups referencing other security groups.
I was hopeful that a GetAttr function call for the ID would work, but Security Groups are not in the supported list of GetAttr:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html
Try this for your SecurityGroup:
"ELBSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Security group for the Arena dev stack",
"SecurityGroupIngress" : [ {
"IpProtocol" : "tcp",
"FromPort" : 80,
"ToPort" : 80,
"SourceSecurityGroupOwnerId" : { "Fn::GetAtt" : [ "ProjectLoadBalancerTest", "SourceSecurityGroup.OwnerAlias" ] },
"SourceSecurityGroupName" : { "Fn::GetAtt" : [ "ProjectLoadBalancerTest", "SourceSecurityGroup.GroupName" ] }
} ]
}
},
and remove "SecurityGroups" from your ELB definition. This will allow traffic from your ELB to your app servers.
It looks like you're also looking to limit ELB traffic to your office IP. That would be better done with something like apache directives.