AWS - cfn-init not creating a file - amazon-web-services

I am new to cloudformation.
I am using cfn-init to create a file. But doesnt create a file nor my stack fails. Stack successfully gets created with required resources like EC2 instance. Also it installs AWS CLI as mentioned in User data.
But it just does not create file i wish to create.
I tried using Advanced options of not allowing rollback of stack. But the /var/log/cfn-init.log does not get created.
See the template below? Am I doing anything wrong in this?
{
"Parameters" : {
"KeyName" : {
"Description" : "The EC2 Key Pair to allow SSH access to the instance",
"Type" : "AWS::EC2::KeyPair::KeyName"
}
},
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Metadata" : {
"Comment" : "Install a simple application",
"AWS::CloudFormation::Init" : {
"config" : {
"files" : {
"/tmp/setup.mysql" : {
"content" : { "Fn::Join" : ["", ["[default]\n","region=",{"Ref": "AWS::Region"}]]},
"mode" : "000775",
"owner" : "ec2-user",
"group" : "ec2-user"
}
}
}
} },
"Properties" : {
"SecurityGroups" : [ {
"Ref" : "InstanceSecurityGroup" }
],
"IamInstanceProfile" : {"Ref" : "RootInstanceProfile"} ,
"KeyName" : { "Ref" : "KeyName"},
"InstanceType" : "t2.micro",
"ImageId" : "ami-58277d3d",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o awscli-bundle.zip\n",
"unzip awscli-bundle.zip\n",
"sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource Ec2Instance ",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"cfn-signal -e 0",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --region ",
{
"Ref": "AWS::Region"
},
" --resource ",
"Ec2Instance",
"\n"
]
]
}
}
}
},
"RootRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Principal": {
"Service": [ "ec2.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
} ]
},
"Path": "/",
"Policies": [ {
"PolicyName": "root",
"PolicyDocument": {
"Version" : "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Action": ["cloudwatch:PutMetricData"],
"Resource": "*"
} ]
}
} ]
}
},
"RootInstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ {
"Ref": "RootRole"
} ]
}
},
"InstanceSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable SSH access via port 22",
"Tags" : [{ "Key" : "Name", "Value" : "SecurityGr_EC2WithParam" }],
"SecurityGroupIngress" : [ {
"IpProtocol" : "tcp",
"FromPort" : "22",
"ToPort" : "22",
"CidrIp" : "0.0.0.0/0"
} ]
}
}
}
}

As discovered in your comment, the UserData property on your AWS::EC2::Instance Resource requires the first line to be #!/bin/bash\n.
This is necessary in order for the user-data processed by cloud-init to be interpreted as a User-Data Script, as noted in the AWS EC2 documentation section, Running Commands on Your Linux Instance at Launch:
User data shell scripts must start with the #! characters and the path to the interpreter you want to read the script (commonly /bin/bash).
Note also that sudo is not necessary in your user-data script, as also noted in the documentation:
Scripts entered as user data are executed as the root user, so do not use the sudo command in the script.
Finally, note that the AWS CLI comes pre-installed on the Amazon Linux AMI instances by default, which is why you noticed the AWS CLI was still installed on your instance despite your user-data script not running correctly.

Related

Cloudformation - Using parameters in EC2 server

Below is my cloudformation template. Basically I want to use a parameter called "User" in a command in EC2 ubuntu server.
I have tried many solutions, but the mkdir command is not fails, rest of the template is working.
In the below example I have tried to set the "User" value as a tag as well as a property. Both failed. Suggest if you have seen any such examples.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"UserName": {
"Type": "String",
"Description": "Enter the username",
"AllowedPattern": "[a-zA-Z0-9_\\-]+",
"ConstraintDescription": "It should be a combination of characters from [A-Z],[a-z],[0-9],- and _."
},
"KeyName": {
"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
"Type": "AWS::EC2::KeyPair::KeyName",
"ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
}
},
"Resources": {
"EC2Ansiblemaster2743": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"User" : { "Ref": "UserName" },
"AWS::CloudFormation::Init" : {
"configSets" : {
"CreateFile" : [
"create_file"
]
},
"create_file" : {
"files" : {
"/home/ubuntu/healthy.txt": {
"content": "Hello"
}
},
"commands" : {
"mkdir" : {
"command" : "sudo mkdir /home/$User"
}
}
}}
},
"Properties": {
"AvailabilityZone": "us-east-2a",
"ImageId": "ami-05c1fa8df71875112",
"InstanceType": "t2.micro",
"SourceDestCheck": false,
"KeyName" : { "Ref" : "KeyName" },
"Tags" : [
{
"Key" : "User",
"Value" : { "Ref": "UserName" }
}],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"sudo apt-get update\n",
"sudo apt-get -y install python-pip\n",
"sudo pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
"# Install the files and packages from the metadata\n",
"sudo /usr/local/bin/cfn-init ",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource EC2Ansiblemaster2743 ",
" --configsets CreateFile ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n",
"# Signal the status from cfn-init\n",
"sudo /usr/local/bin/cfn-signal -e $? ",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource EC2Ansiblemaster2743 ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
}
},
"SecurityGroupIds": [
{
"Ref": "EC2SG2AnsM0012743"
}
]
}
},
"EC2SG2AnsM0012743": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupName": "SGAnsibleMas2743",
"GroupDescription": "First",
"SecurityGroupIngress": [
{
"CidrIp": "0.0.0.0/0",
"FromPort": "0",
"ToPort": "65535",
"IpProtocol": "tcp",
"Description": "From Anywhere"
},
{
"CidrIp": "0.0.0.0/0",
"FromPort": "22",
"ToPort": "22",
"IpProtocol": "tcp",
"Description": "SSH from anywhere , from and to port will be same - 22"
}
]
},
}
}
}
I used the Fn:Sub to get the parameter to Userdata.
{
"Fn::Sub":
[
"UserName=${variable}\n",
{
"variable": {
"Ref": "UserName"
}
}]
Later I curl 169.254.169.254/latest/user-data to extract the Username string.
Please let me know if any easier solution is there for this issue.
Use the join intrinsic function to properly form the command string by referencing the username in the template.
"command" : "Fn::Join": [ "", "mkdir /home/", { "Ref": "UserName" }]
The issue is that your mkdir command does not resolve as you have it. There is no linkage between the username parameter and the command in the init section. By using the Join function I am forming the command with the correct parameter value.

I am not able to copy S3 file to EC2 instance using Userdata in CloudFormation

I am unable to download an S3 file to my EC2 instance using CloudFormation Userdata property. I have assigned an IAM role but still not able to get it resolved.
I assigned the role inside the template.
I tried passing Access Key and Secret Access Key - same result.
"Parameters": {
"VpcId": {
"Type": "AWS::EC2::VPC::Id",
"Description": "Id of an existing VPC to use for "
},
"SubnetId": {
"Type": "AWS::EC2::Subnet::Id",
"Description": "Id of an existing subnet id to use for "
},
"SecurityGroupIds": {
"Description": "Security groups ",
"Type": "List<AWS::EC2::SecurityGroup::Id>",
"ConstraintDescription": "using existing security be list of EC2 security group ids"
},
"instanceType": {
"Type": "String",
"Default": "t2.micro",
"AllowedValues": [
"t2.micro"
],
"Description": "Enter Instance Type "
},
"AWSREGION": {
"Type": "String",
"Default": "us-east-1",
"AllowedValues": [
"us-east-1"
],
"Description": "Enter AWS_REGION."
}
},
"Resources": {
"InstanceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": [ "ec2.amazonaws.com" ] },
"Action": [ "sts:AssumeRole" ]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName": "S3_Access",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": ["arn:aws:s3:::mybucketlocation/*"]
}
]
}
}
]
}
},
"InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ { "Ref": "InstanceRole" }
]
}
},
"EdgeNode": {
"Type": "AWS::EC2::Instance",
"Properties": {
"IamInstanceProfile": { "Ref": "InstanceProfile" },
"InstanceType": { "Ref" : "instanceType" },
"ImageId": "ami-0cc96feef8c6bbff3",
"SubnetId": { "Ref" : "SubnetId" },
"KeyName": "my-key",
"SecurityGroupIds": {
"Ref": "SecurityGroupIds"
},
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"\n",
[
"#!/bin/bash",
"echo \"\" > /home/xyz/index.txt",
{
"Fn::Join": [
"",
[
"echo \"AWS_REGION: ",
{
"Ref": "AWSREGION"
},
"\" >> /home/xyz/index.txt"
]
]
},
{
"Fn::Join": ["", [
"<script>\n",
"cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" }, " -r Instance --region ", { "Ref" : "AWS::Region" }, "\n",
"</script>"
] ]
}
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Init": {
"config": {
"commands" : {
"Pullcode" : {
"command" : "aws s3 sync s3://mybucketlocation /home/xyz/ --debug"
}
}
}
},
"AWS::CloudFormation::Designer": {
"id": "e37a9183-9f81c2fbd39"
}
}
}
}
In cloud-init-output.log I got this:
/var/lib/cloud/instance/scripts/part-001: line 7: syntax error near unexpected token newline'
/var/lib/cloud/instance/scripts/part-001: line 7:'
Jun 21 11:45:05 cloud-init[4071]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [2]
Jun 21 11:45:05 cloud-init[4071]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Jun 21 11:45:05 cloud-init[4071]: util.py[WARNING]: Running module scripts-
These lines seem strange:
"Fn::Join": ["", [
"<script>\n",
"cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" }, " -r Instance --region ", { "Ref" : "AWS::Region" }, "\n",
"</script>"
You are launching an Amazon EC2 instance. However, these lines look like they were taken from User Data for a Windows instance.
Also, you are prompting the user for a Region, but the script is already running in a specific region, so you can use { "Ref" : "AWS::Region" } to access the value.
You probably want your User Data script to look like this:
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"\n",
[
"#!/bin/bash",
{
"Fn::Sub": "echo AWS_REGION: ${AWS::REGION} >>/home/xyz/index.txt"
},
{
"Fn::Sub": "cfn-init -v -s ${AWS::StackId} -r EdgeNode --region ${AWS::Region}"
},
]
]
}
}
I didn't test it, so you might need to tweak some things.
Userdata is always a dreadful property to get right. You can try cloudkast which is an online cloudformation template generator. It makes it very easy for your to use intrinsic functions in cloudformation which I belive has a bit of a learning curve.

How do you associate a IAM Role with an Aurora Cluster using CloudFormation?

Following the instructions found here, I have created the following IAM Role
"DatabaseS3Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": ["rds.amazonaws.com"]
},
"Action": "sts:AssumeRole"
}
]
},
"Policies": [
{
"PolicyName": "AllowAuroraToReadS3",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket"],
"Resource": {"Fn::Join": ["", [
"arn:aws:s3:::",
{"Fn::Join": ["-",[
{"Ref": "ClientName"},
{"Ref": "SourceBucketName"},
{"Ref": "EnvironmentType"},
{ "Fn::FindInMap" : [ "Regions", { "Ref" : "AWS::Region" }, "Name" ] }
]]} ,
"*"
]]}
}
]
}
}
]
}
}
I am able to add it to a cluster parameter group and associate it using the following.
"RDSDBClusterParameterGroup" : {
"DependsOn": "DatabaseS3Role",
"Type": "AWS::RDS::DBClusterParameterGroup",
"Properties" : {
"Description" : "CloudFormation Aurora Cluster Parameter Group",
"Family" : "aurora5.6",
"Parameters" : {
"time_zone" : "US/Eastern",
"aws_default_s3_role": {"Fn::GetAtt": ["DatabaseS3Role", "Arn"]}
}
}
},
"RDSAuroraCluster" : {
"Type" : "AWS::RDS::DBCluster",
"Properties" : {
"MasterUsername" : { "Ref" : "Username" },
"MasterUserPassword" : { "Ref" : "Password" },
"Engine" : "aurora",
"DBSubnetGroupName" : { "Ref" : "RDSSubnetGroup" },
"DBClusterParameterGroupName" : { "Ref" : "RDSDBClusterParameterGroup" },
"VpcSecurityGroupIds" : [ { "Ref" : "SecurityGroupId" } ],
"Tags" : [
{ "Key" : "Name", "Value" : { "Fn::Join" : [ "-", [
{ "Ref" : "ClientName" },
"aurclstr001",
{"Ref" : "EnvironmentType" },
{ "Fn::FindInMap" : [ "Regions", { "Ref" : "AWS::Region" }, "Name" ] }
] ] } }
]
}
}
However Aurora still isn't able to connect to S3 unless I manually associate a role with the cluster through the console or with the cli command add-role-to-db-cluster.
Digging through the cloud formation documentation does not provide any means of doing this through the template. This documentation does not provide any parameter that allows a role to be associated.
How do I do this without having to add in a manual step to the deployment process?
As of August 29, 2019 this is finally supported!
There is a new attribute named AssociatedRoles that takes an array of DBClusterRoles. These are basically an object with a RoleArn and an optional FeatureName which can currently only be s3Import per this reference showing SupportedFeatureNames.member.N.
Original answer from 2017-06-30:
It's not a great solution but I decided to generate the command that is needed to run in the output. I will open a support request with Amazon to confirm there is no way add a role to the cluster via the DSL.
When I run aws rds describe-db-clusters I see an entry for "AssociatedRoles" that holds an array of objects with a Status and RoleArn.
PostRunCommand:
Description: You must run this awscli command after the stack is created and may also need to reboot the cluster/instance.
Value: !Join [" ", [
"aws rds add-role-to-db-cluster --db-cluster-identifier",
!Ref AuroraSandboxCluster,
"--role-arn",
!GetAtt AuroraS3Role.Arn,
"--profile",
!FindInMap [ AccountNameMap, !Ref AccountNamespace, profile ]
]]
You most likely won't need the last part WRT profile...
A followup after Amazon responded to me. They said:
I understand that you were looking for a way to associate an IAM role with an Aurora cluster in Cloudformation to access other AWS services on your behalf.
As you correctly state there's no role property for a RDS cluster resource as CloudFormation does not support it yet. There's an already open feature request for this as it's a very common issue and I have added your voice to it to add the feature with even more weight.
As usual, I can't provide you with an ETA, however as soon as it's released it should be published in the Cloudformation release history page:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html
However you can create a CFN Template which creates a RDS Aurora setup and in the end of the template a Custom Resource 1, as a Lambda Function, which makes an API call to attach the IAM Role to the RDS Cluster, this way the whole RDS Aurora Cluster setup stay centralized inside the CFN Template without manual actions and the cluster will be able to invoke the Lambda Function.
Please find attached an "example" template of this workaround described above.
I will as well send a feedback on your behalf about the lack of the principal property in the examples which is needed to create a Role to Delegate Permissions to an AWS Service.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "AWS Cloud resources for DevTools services.",
"Metadata": {
"Version": "0.2.0"
},
"Parameters": {
"RDSPassword": {
"Description": "Password for root user on the RDS instance.",
"Type": "String",
"NoEcho":"true"
},
"RDSDatabaseName": {
"Description": "DB Identifier for RDS instance.",
"Type": "String",
"Default": "mydbname"
},
"RDSClass": {
"Description": "RDS Instance Class",
"Type": "String",
"Default": "db.r3.xlarge"
},
"DBIdentifier": {
"Description": "Database Instance Identifier",
"Type": "String"
},
"DBSubnetGroupName": {
"Description": " The Subnet group Group for the RDS instance",
"Type": "String"
},
"RDSSecurityGroupId": {
"Description": "Existing internal SG for RDS instance access",
"Type": "AWS::EC2::SecurityGroup::Id"
},
"RDSRetention": {
"Description": "How long to retain RDS snapshots",
"Type": "String"
},
"RDSVpcId": {
"Description": "VpcId for RDS instance",
"Type": "AWS::EC2::VPC::Id"
},
"PubliclyAccessible": {
"Description": "Set the RDS to be publically available",
"Type": "String",
"AllowedValues" : ["true", "false"],
"Default": "true"
},
"DBClusterIdentifier": {
"Description": "The name of the DBCluster",
"Type": "String"
},
"RDSRoleTag": {
"Description": "sets if the tag for dev/prod use",
"Type": "String",
"Default": "dev"
}
},
"Resources": {
"LambdaRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"AssumeRolePolicyDocument" : {
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Principal" : {
"Service" : [
"lambda.amazonaws.com"
]
},
"Action" : [
"sts:AssumeRole"
]
}
]
}
}
},
"LambdaPolicy": {
"Type" : "AWS::IAM::Policy",
"Properties" : {
"PolicyName" : "LambdaPolicy",
"PolicyDocument" : {
"Version" : "2012-10-17",
"Statement": [ {
"Effect" : "Allow",
"Action" : [
"iam:*",
"ec2:*",
"rds:*",
"logs:*"
],
"Resource" : "*"
} ]
},
"Roles": [ { "Ref": "LambdaRole" } ]
}
},
"LambdaFunction": {
"Type" : "AWS::Lambda::Function",
"DeletionPolicy" : "Delete",
"DependsOn" : [
"LambdaRole"
],
"Properties" : {
"Code" : {
"ZipFile" : {
"Fn::Join" : [
"\n",
[
" var AWS = require('aws-sdk');",
" var rds = new AWS.RDS();",
" var response = require('cfn-response');",
" exports.handler = (event, context, callback) => {",
" var rolearn = event.ResourceProperties.RDSRole;",
" var dbclusteridentifier = event.ResourceProperties.DBClusterIdentifier;",
" var responseData = {};",
" console.log('Role ARN: ' + rolearn);",
" console.log('DBClusterIdentifier: ' + dbclusteridentifier);",
" var addroleparams = {",
" RoleArn: rolearn,",
" DBClusterIdentifier: dbclusteridentifier",
" };",
" if (event.RequestType == 'Delete') {",
" response.send(event, context, response.SUCCESS);",
" return;",
" }",
" rds.addRoleToDBCluster(addroleparams, function(err, data) {",
" if (err) {",
" console.log(err, err.stack); // an error occurred",
" responseData = {Error: 'Create call failed'};",
" response.send(event, context, response.FAILED, responseData);",
" }",
" else {",
" response.send(event, context, response.SUCCESS, responseData);",
" console.log(data); // successful response",
" }",
" });",
" };",
]
]
}
},
"Handler" : "index.handler",
"MemorySize" : 128,
"Role" : {
"Fn::GetAtt" : [
"LambdaRole",
"Arn"
]
},
"Runtime" : "nodejs4.3",
"Timeout" : 10
}
},
"RDSRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": ["rds.amazonaws.com"]
},
"Action": ["sts:AssumeRole"]
}
]
},
"Path": "/"
}
},
"RDSPolicy": {
"Type" : "AWS::IAM::Policy",
"Properties" : {
"PolicyName" : "RDSPolicy",
"PolicyDocument" : {
"Version" : "2012-10-17",
"Statement": [ {
"Effect" : "Allow",
"Action" : [
"lambda:InvokeFunction"
],
"Resource" : "*"
} ]
},
"Roles": [ { "Ref": "RDSRole" } ]
}
},
"RDSDBClusterParameterGroup" : {
"Type" : "AWS::RDS::DBClusterParameterGroup",
"Properties" : {
"Parameters" : {
"aws_default_lambda_role" : { "Fn::GetAtt" : [ "LambdaFunction", "Arn" ] }
},
"Family" : "aurora5.6",
"Description" : "A sample parameter group"
}
},
"RDSDBCluster": {
"Type" : "AWS::RDS::DBCluster",
"DeletionPolicy": "Retain",
"Properties" : {
"BackupRetentionPeriod" : { "Ref": "RDSRetention" },
"DatabaseName": { "Ref": "RDSDatabaseName" },
"DBSubnetGroupName": { "Ref": "DBSubnetGroupName" },
"DBClusterParameterGroupName": { "Ref" : "RDSDBClusterParameterGroup" },
"Engine" : "aurora",
"StorageEncrypted" : true,
"MasterUsername" : "sa",
"MasterUserPassword" : { "Ref": "RDSPassword" },
"Port" : 3306,
"Tags": [
{ "Key": "Role", "Value": { "Ref": "RDSRoleTag" } }
],
"VpcSecurityGroupIds": [{ "Ref": "RDSSecurityGroupId" } ]
}
},
"RDSInstance": {
"Type": "AWS::RDS::DBInstance",
"DeletionPolicy": "Retain",
"Properties": {
"AllowMajorVersionUpgrade": false,
"AutoMinorVersionUpgrade": true,
"DBClusterIdentifier" : { "Ref": "RDSDBCluster" },
"DBInstanceIdentifier": { "Ref": "DBIdentifier" },
"DBInstanceClass": { "Ref": "RDSClass" },
"Engine": "aurora",
"PubliclyAccessible": { "Ref": "PubliclyAccessible" },
"Tags": [
{ "Key": "Role", "Value": { "Ref": "RDSRoleTag" } }
]
}
},
"RDSInstanceSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"DeletionPolicy": "Retain",
"Properties": {
"GroupDescription": "Security group for the RDSInstance resource",
"SecurityGroupEgress": [
{
"IpProtocol": "tcp",
"CidrIp": "127.0.0.1/32",
"FromPort": "1",
"ToPort": "1"
}
],
"SecurityGroupIngress": [
{
"IpProtocol": "tcp",
"SourceSecurityGroupId": { "Ref": "RDSSecurityGroupId" },
"FromPort": "3306",
"ToPort": "3306"
}
],
"VpcId": { "Ref": "RDSVpcId" },
"Tags": [
{ "Key": "Role", "Value": { "Ref": "RDSRoleTag" } }
]
}
},
"AddRoleToDBCluster": {
"DependsOn" : [
"RDSDBCluster",
"RDSInstance"
],
"Type": "Custom::AddRoleToDBCluster",
"Properties" : {
"ServiceToken" : {
"Fn::GetAtt" : [
"LambdaFunction",
"Arn"
]
},
"RDSRole" : { "Fn::GetAtt" : [ "RDSRole", "Arn" ] },
"DBClusterIdentifier" : {"Ref":"RDSDBCluster"}
}
}
}
}

cfn-signal in aws cloudformation with launchConfiguration and AutoScalingGroup

I have the following configuration in my AWS Cloudformation template.
The template creates one EC2 instance based on instance1. I am using the reference to same instance in my LaunchConfiguration to create the instances of same type.
The problem I am facing is by including the CreationPolicy element in my AutoScalingGroup template. I get the following error when ASG launches an instance and waits for the cfn-signal.
+ /opt/aws/bin/cfn-signal -e 0 --stack ss07 --resource Instance1 --region us-west-2
ValidationError: Resource Instance1 is in CREATE_COMPLETE state and cannot be signaled
It seems like somehow the reference is made to an already existing instance1 and not to the instance being created by LaunchConfig. I saw examples which had the LaunchConfig embedded within, but I want to keep the instance details at one place instead of two places.
"instance1": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"install": ["yum_packages","install_cfn"]
},
"yum_packages": {
"packages" : {
"yum" : {
"awslogs" : [],
"ruby" : [],
"wget" : [],
"httpd" : []
}
}
},
"install_cfn": {
"files": {
"/etc/cfn/cfn-hup.conf": {
"content": {
"Fn::Join": [
"",
[
"[main]\n",
"stack=",
{
"Ref": "AWS::StackId"
},
"\n",
"region=",
{
"Ref": "AWS::Region"
},
"\n"
]
]
},
"mode": "000400",
"owner": "root",
"group": "root"
},
"/etc/cfn/hooks.d/cfn-auto-reloader.conf": {
"content": {
"Fn::Join": [
"",
[
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.WebServer.Metadata.AWS::CloudFormation::Init\n",
"action=/opt/aws/bin/cfn-init -v ",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource splitsweetInstance ",
" --configsets install ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
},
"mode": "000400",
"owner": "root",
"group": "root"
}
},
"services": {
"sysvinit": {
"cfn-hup": { "enabled": "true", "ensureRunning": "true", "files": [
"/etc/cfn/cfn-hup.conf",
"/etc/cfn/hooks.d/cfn-auto-reloader.conf"
]
}
}
}
}
}
},
"CreationPolicy": {
"ResourceSignal": {
"Timeout": "PT10M"
}
},
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI", {
"Ref": "AWS::Region"
}, {
"Fn::FindInMap": [
"AWSInstanceType2Arch", {
"Ref": "instanceType1"
},
"Arch"
]
}
]
},
"InstanceType": {"Ref": "instanceType1"},
"KeyName": {"Ref": "KeyName"},
"Monitoring": "false",
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash -xe\n",
"yum install -y aws-cfn-bootstrap\n",
"# Install the files and packages from the metadata\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource splitsweetInstance ",
" --configsets install ",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"wget ", { "Fn::FindInMap": [ "Region2CodeDeployAgent", { "Ref": "AWS::Region"}, "url"] }, "\n",
"chmod +x ./install\n",
"./install auto\n",
"# Signal the status from cfn-init\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource splitsweetInstance ",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]}},
"Tags": [
{
"Key": "Name",
"Value": "inst1"
}
],
"SecurityGroupIds": [
{ "Fn::GetAtt" : [ "instance1Sg", "GroupId" ] }
]
}
}
My Launch Config is as follows -
"LaunchConfig1": {
"Type" : "AWS::AutoScaling::LaunchConfiguration",
"Properties" : {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI", {
"Ref": "AWS::Region"
}, {
"Fn::FindInMap": [
"AWSInstanceType2Arch", {
"Ref": "instanceType1"
},
"Arch"
]
}
]
},
"InstanceId" : { "Ref":"instance1"},
"InstanceMonitoring" : "false",
"InstanceType" : { "Ref": "instanceType1"},
"KeyName" : { "Ref" : "KeyName" },
"SecurityGroups" : [ { "Fn::GetAtt" : [ "instance1Sg", "GroupId" ] } ]
}
}
This is my AutoScalingGroup template -
"AutoScalingGroup1": {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"AvailabilityZones" : { "Fn::GetAZs": { "Ref": "AWS::Region" } },
"Cooldown" : "60",
....
"LaunchConfigurationName" : {"Ref":"LaunchConfig1"},
"MaxSize" : "3",
"MinSize" : "1",
"TargetGroupARNs" : [ {"Ref":"TargetGroup1"} ],
"VPCZoneIdentifier" : [ { "Ref": "subnetCache1" }, { "Ref": "subnetCache2" }, { "Ref": "subnetCache3" } ]
},
"CreationPolicy" : {
"ResourceSignal" : {
"Timeout" : "PT6M",
"Count" : "1"
}
}
}
The problem is that by specifying the InstanceId property in your LaunchConfiguration resource, it is reusing the same UserData that was used to launch the initial EC2 instance, including the hard-coded reference to the Logical Resource signaled by the cfn-signal command. According to the documentation,
When you use an instance to create a launch configuration, all properties are derived from the instance with the exception of BlockDeviceMapping and AssociatePublicIpAddress. You can override any properties from the instance by specifying them in the launch configuration.
To have cfn-signal signal the correct Resource, you will need to override the UserData in your LaunchConfiguration resource to contain a User-Data script that references the Launch Configuration rather than the original EC2 instance. Unfortunately, this will require either duplicating the User-Data script, or rewriting the script to dynamically figure out the Logical Resource associated with the instance the script is currently running on, so the same exact User-Data can be used in both the original EC2 instance and the auto scaling group.
I agree with wjordan that part of the problem you are having is with the InstanceId property in LaunchConfiguration. Also looking at your code sample I could not figure out where splitsweetInstance was referenced.
However my CloudFormation AutoScalingGroup was not set up this way and I still got the same error CREATE_COMPLETE state and cannot be signaled.
The solution for me was setting the DesiredCapacity on the AutoScalingGroup (hopefully this helps others as well) was found here:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-desiredcapacity
As per the docs:
CloudFormation will not mark the Auto Scaling group as successful (by setting its status to CREATE_COMPLETE) until the desired capacity is reached.
This is what that portion of my CloudFormation template looks like:
"WebServerGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"VPCZoneIdentifier" : { "Ref" : "Subnets" },
"LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
"MinSize" : "2",
"MaxSize" : "4",
"DesiredCapacity" : "2",
"TargetGroupARNs" : [ { "Ref" : "ALBTargetGroup" } ]
}
Below is the "signal" portion of UserData section:
"# Signal the status from cfn-init\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource WebServerGroup ",
" --region ", { "Ref" : "AWS::Region" }, "\n",
To solve this problem I ensured that I added both CreationPolicy & UpdatePolicy in Autoscaling group before using cfn-signal in UserData section of the LaunchTemplate
CreationPolicy:
AutoScalingCreationPolicy:
MinSuccessfulInstancesPercent: somepercent
ResourceSignal:
Count: somenumber
Timeout: someminutes
UpdatePolicy:
AutoScalingScheduledAction:
IgnoreUnmodifiedGroupSizeProperties: 'true'
AutoScalingRollingUpdate:
MinInstancesInService: 'somenumber'
MaxBatchSize: 'somenumber'
PauseTime: someminutes
WaitOnResourceSignals: 'true'

How to create a LaunchConfiguration using CloudFormation template which creates a config file?

I want to write a LaunchConfiguration for my AWS stack using CloudFormation template.
I have written it like below.
"LaunchConfiguration": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"files": {
"/etc/test.conf": {
"content": { "Fn::Join": [ "", [
"user: root\n",
"password: password\n"
]]},
"mode": "000400",
"user": "root",
"group": "root"
}
}
}
},
"Properties": {
"ImageId": "ami-*****",
"InstanceType": "*****",
"KeyName": "*****",
"IamInstanceProfile": "*****",
"InstanceMonitoring": "****",
"SecurityGroups": [
{
"Ref": "SecurityGroup"
}
]
}
},
The file is not being created in the EC2 instances created. Can anyone help me on this?
You're missing a couple things. First, you need invoke the cfn-init script from the LaunchConfiguration UserData.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html
"UserData" : { "Fn::Base64" : { "Fn::Join" : [ "", [
"#!/bin/bash -ve\n",
"# Run cfn-init\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ", { "Ref": "AWS::StackName" },
" --resource LaunchConfiguration ",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"# Signal success\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ", { "Ref" : "AWS::StackName" },
" --resource AutoScalingGroup ",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]}}
This example also uses cfn-signal to signal success which notifies the Auto Scaling group that the instance bootstrapping was successful. To use this feature, you will also need to add the CreationPolicy to your AutoScalingGroup resource.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-creationpolicy.html
"CreationPolicy" : {
"ResourceSignal" : {
"Timeout" : "PT10M",
"Count" : "1"
}
}
Lastly, you are missing the default config wrapper around your Metadata.
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"files": {
"/etc/test.conf" : {
"content" : { "Fn::Join": [ "", [
"user: root\n",
"password: password\n"
]]},
"mode" : "000400",
"user" : "root",
"group" : "root"
}
}
}
}
}
You can use something other than config, but you then need to define the configSets attribute.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html#aws-resource-init-configsets
"AWS::CloudFormation::Init" : {
"configSets" : {
"default" : [
"db-config",
"app-config"
]
},
"db-config": {
"files": {
...
}
},
"app-config": {
...
}
}
For more information, this is a detailed overview of bootstrapping instances using CloudFormation.
https://s3.amazonaws.com/cloudformation-examples/BoostrappingApplicationsWithAWSCloudFormation.pdf
Put "files" to "upload" section as
"Metadata" : {
"AWS::CloudFormation::Init" : {
"upload": {
"files": {
"/etc/test.conf": {
"content": { "Fn::Join": [ "", [
"user: root\n",
"password: password\n"
]]},
"mode": "000400",
"user": "root",
"group": "root"
}
}
}
}
},