AWS::CloudFormation::Init not executing commands in metadata - amazon-web-services

This is my template with a test command in the metadata. I can't figure out why the command doesn't get executed. It's supposed to save a string to a file in /tmp.
The machine is Ubuntu 16.04 with cloud-init installed. In UserData I install the helper scripts and execute cfn-init.
Thanks for your help.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Metadata": {
"AWS::CloudFormation::Designer": {
"2af5b799-f6bf-7f20-a6eb-943274f18373": {
"size": {
"width": 60,
"height": 60
},
"position": {
"x": 326,
"y": 118
},
"z": 0,
"embeds": []
}
}
},
"Resources": {
"EC2I3WADD": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "ami-c60b90d1",
"KeyName": "CF-KEY",
"InstanceType": "t2.micro",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"apt-get -y install python-setuptools\n",
"mkdir aws-cfn-bootstrap-latest\n",
"curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1\n",
"easy_install aws-cfn-bootstrap-latest\n",
"/usr/local/bin/cfn-init --stack ",
{
"Ref": "AWS::StackName"
},
" --resource WebServer",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "2af5b799-f6bf-7f20-a6eb-943274f18373"
},
"AWS::CloudFormation::Init": {
"config": {
"commands": {
"test": {
"command": "echo \"$MAGIC\" > /tmp/test.txt",
"env": {
"MAGIC": "I come from the environment!"
}
}
}
}
}
}
}
}
}

The issue is likely being cause by an incorrect --resource argument being passed to the cfn-init command within your UserData. This argument should match the logical resource name of the resource containing the MetaData, in your case EC2I3WADD.
"/usr/local/bin/cfn-init --stack ", { "Ref": "AWS::StackName" }, " --resource EC2I3WADD", " --region ", { "Ref": "AWS::Region" }
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html

Related

Configsets config commands are not executing sequencially cloudformation

This is metadata for cloudformationinit
{
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"setup": [
"setup_docker",
"setup_redis"
]
},
"setup_docker": {
"commands": {
"install_docker": {
"command": {
"Fn::Join": ["", ["amazon-linux-extras install docker"]]
}
},
"start_docker": {
"command": { "Fn::Join": ["", ["service docker start"]] }
},
"start_on_restart": {
"command": { "Fn::Join": ["", ["systemctl enable docker"]] }
}
}
},
"setup_redis": {
"commands": {
"install_redis": {
"command": {
"Fn::Join": ["", ["amazon-linux-extras install redis6"]]
}
},
"change_redis_6379_protected_mode": {
"command": {
"Fn::Join": [
"",
[
"sed -i 's/protected-mode yes/protected-mode no/' /etc/redis/redis.conf"
]
]
}
},
"change_redis_6379_daemonize": {
"command": {
"Fn::Join": [
"",
["sed -i 's/daemonize no/daemonize yes/' /etc/redis/redis.conf"]
]
}
},
"start_redis": {
"command": { "Fn::Join": ["", ["systemctl start redis"]] }
}
}
}
}
}
}
When I execute this template, setup_docker commands got executed in order, first it will execute install_docker, then start_docker, then start_on_restart command, but setup_redis commands are not executing, first two commands got skipped, third command got executed and it throws an error /etc/redis/redis.conf file not found
Why this is happening? commands are wrapped inside setup_redis object only and what is the solution for this?
Thanks in advance
Update
If I separate out the installation part of redis into separate config, it is working properly, does this mean command will not execute sequentially? Why commands are not executing sequentially?
{
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"setup": ["setup_redis", "modify_files"]
},
"setup_redis": {
"commands": {
"install_redis": {
"command": {
"Fn::Join": ["", ["amazon-linux-extras install redis6"]]
}
}
}
},
"modify_files": {
"commands": {
"change_redis_6379_protected_mode": {
"command": {
"Fn::Join": [
"",
[
"sed -i 's/protected-mode yes/protected-mode no/' /etc/redis/redis.conf"
]
]
}
},
"change_redis_6379_daemonize": {
"command": {
"Fn::Join": [
"",
["sed -i 's/daemonize no/daemonize yes/' /etc/redis/redis.conf"]
]
}
},
"start_redis": {
"command": { "Fn::Join": ["", ["systemctl start redis"]] }
}
}
}
}
},
"Properties": {
"ImageId": {
"Ref": "AMI"
},
"KeyName": {
"Ref": "Key"
},
"InstanceType": {
"Ref": "InstanceTypeParameter"
},
"SubnetId": {
"Ref": "SubnetIdParameter"
},
"SecurityGroupIds": {
"Ref": "SecurityGroupIdsParameter"
},
"PrivateIpAddress": "10.0.0.57",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe",
"\n",
"yum update -y",
"\n",
"/opt/aws/bin/cfn-init -v",
" --stack ",
{ "Ref": "AWS::StackName" },
" --resource DevInstance",
" --configsets setup",
" --region ",
{ "Ref": "AWS::Region" },
"\n",
"/opt/aws/bin/cfn-signal -e $?",
" --stack ",
{ "Ref": "AWS::StackName" },
" --resource DevInstance",
" --region ",
{ "Ref": "AWS::Region" },
"\n"
]
]
}
}
}
}
Commands are processed in alphabetical order by name. So in your original configuration, they were going to be executed in this order:
change_redis_6379_daemonize
change_redis_6379_protected_mode
install_redis
start_redis
Separating out the installation part to make it execute first is the correct solution.
Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html#aws-resource-init-commands

Error in Userdata Scripts in Cloud-init of EC2 Cloudformation Template

I am executing the below code to create stack. Instance got created but received this error:
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
No Match for argument: aws-cfn-bootstrap
No package aws-cfn-bootstrap available.
No packages marked for update
/var/lib/cloud/instance/scripts/part-001: line 4: cd: /opt/aws/bin/cfn-init: No such file or directory
My CloudFormation template is:
{
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "ami-011b3ccf1bd6db744",
"InstanceType": "t2.micro",
"KeyName": "EC2KeyPair",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash",
"\n",
"exec > /tmp/user_data.log 2>&1 \n",
"yum update -y\n",
"yum install -y epel-release\n",
"yum install -y https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.amzn1.noarch.rpm \n",
"ln -s /usr/local/lib/python2.7/site-packages/cfnbootstrap /usr/lib/python2.7/site-packages/cfnbootstrap\n",
"/opt/aws/bin/cfn-init -v",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource MyInstance",
" --configsets scripts ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"scripts": [
"configure_cfn",
"pythonInstallation"
]
},
"configure_cfn": {
"files": {
"/etc/cfn/hooks.d/cfn-auto-reloader.conf": {
"content": {
"Fn::Join": [
"",
[
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.EC2.Metadata.AWS::CloudFormation::Init\n",
"action=/opt/aws/bin/cfn-init -v",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource EC2",
" --configsets wordpress",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
},
"mode": "000400",
"owner": "root",
"group": "root"
},
"/etc/cfn/cfn-hup.conf": {
"content": {
"Fn::Join": [
"",
[
"[main]\n",
"stack=",
{
"Ref": "AWS::StackId"
},
"\n",
"region=",
{
"Ref": "AWS::Region"
},
"\n",
"verbose=true\n",
"interval=5\n"
]
]
},
"mode": "000400",
"owner": "root",
"group": "root"
},
"/var/www/html/index2.html": {
"content": "Hi"
}
},
"services": {
"sysvinit": {
"cfn-hup": {
"enabled": "true",
"ensureRunning": "true",
"files": [
"/etc/cfn/cfn-hup.conf",
"/etc/cfn/hooks.d/cfn-auto-reloader.conf"
]
}
}
}
},
"pythonInstallation": {
"packages": {
"yum": {
"wget": [],
"unzip": [],
"gcc-c++": [],
"zlib-devel": [],
"libffi-devel": [],
"httpd": []
}
},
"sources": {
"usr/src/": "https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz"
},
"commands": {
"python1": {
"command": "tar xzf Python-3.7.2.tgz",
"cwd": "/usr/src/"
},
"python2": {
"command": {
"comm1": "./configure --enable-optimizations",
"comm2": "make altinstall",
"comm3": "rm /usr/src/Python-3.7.2.tgz"
},
"cwd": "/usr/src/Python-3.7.2"
}
},
"files": {
"/var/www/html/index.php": {
"content": {
"Fn::Join": [
"",
[
"<html>\n",
" <head>\n",
" <title>AWS CloudFormation PHP Sample</title>\n",
" <meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\">\n",
" </head>\n",
" <body>\n",
" <h1>Welcome to the AWS CloudFormation PHP Sample</h1>\n",
" </body>\n",
"</html>\n"
]
]
}
}
}
}
}
}
}
}
}
It would appear that Redhat does not have cfn-init pre-loaded.
You could either use an Amazon Linux AMI instead, or see How to install aws-cfn-bootstrap/cfn-init package in Redhat using CloudFormation? - Stack Overflow for tips on how to load cfn-init as part of the boot process.

CloudFormation is not executing my command?

I am executing command through CloudFormation but it is not working. Also I am not getting any error in logs. Able to create instance & EBS with this template but command is not being executed.
This the template which I am using:
{
"Description" : "AWS CloudFormation Sample Template
EC2InstanceWithSecurityGroupSampleAndEBS",
"Resources": {
"Ec2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"KeyName": "123",
"ImageId": "280a1kk",
"InstanceType": "t1.micro",
"NetworkInterfaces" : [{
"GroupSet" : ["ds"],
"AssociatePublicIpAddress" : "true",
"DeviceIndex" : "0",
"DeleteOnTermination" : "true",
"SubnetId" : "7f"
}
],
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sdm",
"Ebs": {
"VolumeType": "io1",
"Iops": 300,
"DeleteOnTermination": true,
"VolumeSize": 100
}
}
],
},
"Metadata": {
"AWS::CloudFormation::Init": {
"config": {
"commands": {
"1_mkfs": {
"command" : "mkfs -t ext4 /dev/xvdm\n"
},
"2_mkdir": {
"command" : "mkdir /mntfirm\n"
},
"3_mount": {
"command" : "mount /dev/xvdm /mntfirm"
}
}
}
}
}
}
}
}
Please help me out.
you need to install cfn bootstrap and execute the commands, try this
{
"Description":"AWS CloudFormation Sample Template EC2InstanceWithSecurityGroupSampleAndEBS",
"Resources":{
"Ec2Instance":{
"Metadata":{
"AWS::CloudFormation::Init":{
"configSets" : {
"InstallAndRun" : [ "config" ]
},
"config":{
"commands":{
"1_mkfs":{
"command":"mkfs -t ext4 /dev/xvdm\n"
},
"2_mkdir":{
"command":"mkdir /mntfirm\n"
},
"3_mount":{
"command":"mount /dev/xvdm /mntfirm"
}
}
}
}
},
"Type":"AWS::EC2::Instance",
"Properties":{
"ImageId":"AMI",
"InstanceType":"t1.micro",
"KeyName": "KEY",
"NetworkInterfaces":[
{
"GroupSet":[
"sg-xxx"
],
"AssociatePublicIpAddress":"true",
"DeviceIndex":"0",
"DeleteOnTermination":"true",
"SubnetId":"subnet-xxx"
}
],
"BlockDeviceMappings":[
{
"DeviceName":"/dev/sdm",
"Ebs":{
"VolumeType":"io1",
"Iops":300,
"DeleteOnTermination":true,
"VolumeSize":100
}
}
],
"UserData":{
"Fn::Base64":{
"Fn::Join":[
"",
[
"#!/bin/bash -xe\n",
"apt-get install -y python-setuptools\n",
"mkdir -p /opt/aws/bin\n",
"wget https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
"easy_install --script-dir /opt/aws/bin aws-cfn-bootstrap-latest.tar.gz\n",
"apt-get update\n",
"# Install the files and packages from the metadata\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ",
{
"Ref":"AWS::StackName"
},
" --resource Ec2Instance ",
" --config InstallAndRun ",
" --region ",
{
"Ref":"AWS::Region"
},
"\n",
"# Signal the status from cfn-init\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ",
{
"Ref":"AWS::StackName"
},
" --resource Ec2Instance ",
" --region ",
{
"Ref":"AWS::Region"
},
"\n"
]
]
}
}
}
}
}
}
and all the events will log to syslog

CloudFormation template failing to create "files" under MetaData Section

Could somebody help me point what wrong am I doing here. I am failing to understand how the meta data part uses the authentication resource, In the AWS::CloudFormation::Authentication part I've mentioned the role same as the one attached to the instance, Yet I'm unable to create the file "some.txt"
{
"Parameters": {
"SecurityGroupId": {
"Description": "Security group for instance",
"Type": "AWS::EC2::SecurityGroup::Id"
}
},
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"configsets": {
"InstallIt": ["config1"]
},
"config1": {
"files": {
"/home/ec2-user/some.txt": {
"content": "This is my name ",
"encoding": "base64",
"mode": "000644",
"owner": "root",
"group": "root"
}
}
}
},
"AWS::CloudFormation::Authentication": {
"HelpMe": {
"type": "S3",
"buckets": "poc-bucket",
"roleName": "EC2andS3"
}
}
},
"Properties": {
"KeyName": "GoldenImage-NV-Anant",
"DisableApiTermination": "false",
"ImageId": "ami-0b33d91d",
"InstanceType": "t2.micro",
"Monitoring": "false",
"SubnetId": "subnet-73487a59",
"SecurityGroupIds": [{
"Ref": "SecurityGroupId"
}],
"IamInstanceProfile": {
"Ref": "MyInstanceProfile"
},
"Tags": [{
"Key": "Name",
"Value": "GeicoUserDataPocInstance"
}],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"", [
"#!/bin/bash -ex \n",
"echo \"hello dudes\" > /home/ec2-user/hello.txt \n",
"yum update -y aws-cfn-bootstrap\n",
"/opt/aws/bin/cfn-init -v",
" --stack ", {
"Ref": "AWS::StackId"
},
" --resource MyInstance ",
" --configsets InstallIt ",
" --region ", {
"Ref": "AWS::Region"
}, "\n",
"echo \"bye dudes\" > /home/ec2-user/bye.txt", "\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ", {
"Ref": "AWS::StackId"
},
" --resource MyInstance ",
" --region ", {
"Ref": "AWS::Region"
}, "\n"
]
]
}
}
},
"CreationPolicy": {
"ResourceSignal": {
"Timeout": "PT90M",
"Count": "1"
}
}
},
"MyInstanceProfile": {
"Description": "Instance profile for the instance",
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": ["EC2andS3"]
}
}
}
}
configsets should be configSets with capital S:
"configSets": {
"InstallIt": ["config1"]
},
buckets property needs to be a list of strings (this might not be necessary, the documentation is a bit unclear):
"buckets": ["poc-bucket"]
AWS::CloudFormation::Authentication resource shouldn't be necessary unless the source of your file is an S3 bucket. Even then, it still shouldn't be necessary when using an attached instance profile, since it will use the instance profile for authentication by default.

Circular dependency in aws stack because of private ip [AWS CloudFormation]

I am using a cloudformation template to create my EC2 instance. In userdata section i need to run a shell file that i have created in metadata. For that shell file i am passing private ip of the instance as a parameter.
To get the private ip i am using this :
{
"Fn::GetAtt" : [ "ConsoleServer", "PrivateIp" ]
},
i ask the wait handler to wait while my user data gets executed but the wait handeler is dependent on the EC2 that i am trying to configure.
This is causing cicular dependency but i am unable to understand how to get private ip of the instance using some other way ?
Below are the part that matter : Metadata
"Resources": {
"ConsoleServer": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"config": {
"files": {
"/usr/local/share/deployment-script.sh": {
"mode": "755",
"owner": "ec2-user",
"group": "ec2-user",
"content": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"sh master.sh ",
{
"Ref": "S3ConsoleZip"
}, " ",
{
"Fn::GetAtt" : [ "ConsoleServer", "PrivateIp" ]
},
and this is my userdata section followed by waithandler
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -v\n",
"sudo su",
"\n",
"chmod -R 775 /usr/local/share\n",
"yum update -y aws-cfn-bootstrap\n",
"## Error reporting helper function\n",
"function error_exit\n",
"{\n",
" /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '",
{
"Ref": "WaitHandleServer"
},
"'\n",
" exit 1\n",
"}\n",
"## Initialize CloudFormation bits\n",
"/opt/aws/bin/cfn-init -v -s ",
{
"Ref": "AWS::StackName"
},
" -r ConsoleServer",
" --region ",
{
"Ref": "AWS::Region"
},
" > /tmp/cfn-init.log 2>&1 || error_exit $(</tmp/cfn-init.log)\n",
"cd /usr/local/share\n",
********* "sh deployment-script.sh >> /home/ec2-user/deployment-script.log\n",
"/opt/aws/bin/cfn-signal",
" -e 0",
" '",
{
"Ref": "WaitHandleServer"
},
"'",
"\n",
"date > /home/ec2-user/stoptime"
]
]
}
}
}
},
"WaitHandleServer": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"WaitConditionServer": {
"Type": "AWS::CloudFormation::WaitCondition",
"DependsOn": "ConsoleServer",
"Properties": {
"Handle": {
"Ref": "WaitHandleServer"
},
"Timeout": "1200"
}
}
},
i have added ********* where call is being made from user data section
To get EC2 IP address, you can get it by call this API using curl inside your script:
curl http://169.254.169.254/latest/meta-data/local-ipv4
Read more about Instance Metadata and User Data.