Circular dependency in aws stack because of private ip [AWS CloudFormation] - amazon-web-services

I am using a cloudformation template to create my EC2 instance. In userdata section i need to run a shell file that i have created in metadata. For that shell file i am passing private ip of the instance as a parameter.
To get the private ip i am using this :
{
"Fn::GetAtt" : [ "ConsoleServer", "PrivateIp" ]
},
i ask the wait handler to wait while my user data gets executed but the wait handeler is dependent on the EC2 that i am trying to configure.
This is causing cicular dependency but i am unable to understand how to get private ip of the instance using some other way ?
Below are the part that matter : Metadata
"Resources": {
"ConsoleServer": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"config": {
"files": {
"/usr/local/share/deployment-script.sh": {
"mode": "755",
"owner": "ec2-user",
"group": "ec2-user",
"content": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"sh master.sh ",
{
"Ref": "S3ConsoleZip"
}, " ",
{
"Fn::GetAtt" : [ "ConsoleServer", "PrivateIp" ]
},
and this is my userdata section followed by waithandler
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -v\n",
"sudo su",
"\n",
"chmod -R 775 /usr/local/share\n",
"yum update -y aws-cfn-bootstrap\n",
"## Error reporting helper function\n",
"function error_exit\n",
"{\n",
" /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '",
{
"Ref": "WaitHandleServer"
},
"'\n",
" exit 1\n",
"}\n",
"## Initialize CloudFormation bits\n",
"/opt/aws/bin/cfn-init -v -s ",
{
"Ref": "AWS::StackName"
},
" -r ConsoleServer",
" --region ",
{
"Ref": "AWS::Region"
},
" > /tmp/cfn-init.log 2>&1 || error_exit $(</tmp/cfn-init.log)\n",
"cd /usr/local/share\n",
********* "sh deployment-script.sh >> /home/ec2-user/deployment-script.log\n",
"/opt/aws/bin/cfn-signal",
" -e 0",
" '",
{
"Ref": "WaitHandleServer"
},
"'",
"\n",
"date > /home/ec2-user/stoptime"
]
]
}
}
}
},
"WaitHandleServer": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"WaitConditionServer": {
"Type": "AWS::CloudFormation::WaitCondition",
"DependsOn": "ConsoleServer",
"Properties": {
"Handle": {
"Ref": "WaitHandleServer"
},
"Timeout": "1200"
}
}
},
i have added ********* where call is being made from user data section

To get EC2 IP address, you can get it by call this API using curl inside your script:
curl http://169.254.169.254/latest/meta-data/local-ipv4
Read more about Instance Metadata and User Data.

Related

Cloudformation - Using parameters in EC2 server

Below is my cloudformation template. Basically I want to use a parameter called "User" in a command in EC2 ubuntu server.
I have tried many solutions, but the mkdir command is not fails, rest of the template is working.
In the below example I have tried to set the "User" value as a tag as well as a property. Both failed. Suggest if you have seen any such examples.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"UserName": {
"Type": "String",
"Description": "Enter the username",
"AllowedPattern": "[a-zA-Z0-9_\\-]+",
"ConstraintDescription": "It should be a combination of characters from [A-Z],[a-z],[0-9],- and _."
},
"KeyName": {
"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
"Type": "AWS::EC2::KeyPair::KeyName",
"ConstraintDescription" : "must be the name of an existing EC2 KeyPair."
}
},
"Resources": {
"EC2Ansiblemaster2743": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"User" : { "Ref": "UserName" },
"AWS::CloudFormation::Init" : {
"configSets" : {
"CreateFile" : [
"create_file"
]
},
"create_file" : {
"files" : {
"/home/ubuntu/healthy.txt": {
"content": "Hello"
}
},
"commands" : {
"mkdir" : {
"command" : "sudo mkdir /home/$User"
}
}
}}
},
"Properties": {
"AvailabilityZone": "us-east-2a",
"ImageId": "ami-05c1fa8df71875112",
"InstanceType": "t2.micro",
"SourceDestCheck": false,
"KeyName" : { "Ref" : "KeyName" },
"Tags" : [
{
"Key" : "User",
"Value" : { "Ref": "UserName" }
}],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"sudo apt-get update\n",
"sudo apt-get -y install python-pip\n",
"sudo pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
"# Install the files and packages from the metadata\n",
"sudo /usr/local/bin/cfn-init ",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource EC2Ansiblemaster2743 ",
" --configsets CreateFile ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n",
"# Signal the status from cfn-init\n",
"sudo /usr/local/bin/cfn-signal -e $? ",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource EC2Ansiblemaster2743 ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
}
},
"SecurityGroupIds": [
{
"Ref": "EC2SG2AnsM0012743"
}
]
}
},
"EC2SG2AnsM0012743": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupName": "SGAnsibleMas2743",
"GroupDescription": "First",
"SecurityGroupIngress": [
{
"CidrIp": "0.0.0.0/0",
"FromPort": "0",
"ToPort": "65535",
"IpProtocol": "tcp",
"Description": "From Anywhere"
},
{
"CidrIp": "0.0.0.0/0",
"FromPort": "22",
"ToPort": "22",
"IpProtocol": "tcp",
"Description": "SSH from anywhere , from and to port will be same - 22"
}
]
},
}
}
}
I used the Fn:Sub to get the parameter to Userdata.
{
"Fn::Sub":
[
"UserName=${variable}\n",
{
"variable": {
"Ref": "UserName"
}
}]
Later I curl 169.254.169.254/latest/user-data to extract the Username string.
Please let me know if any easier solution is there for this issue.
Use the join intrinsic function to properly form the command string by referencing the username in the template.
"command" : "Fn::Join": [ "", "mkdir /home/", { "Ref": "UserName" }]
The issue is that your mkdir command does not resolve as you have it. There is no linkage between the username parameter and the command in the init section. By using the Join function I am forming the command with the correct parameter value.

Error in Userdata Scripts in Cloud-init of EC2 Cloudformation Template

I am executing the below code to create stack. Instance got created but received this error:
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
No Match for argument: aws-cfn-bootstrap
No package aws-cfn-bootstrap available.
No packages marked for update
/var/lib/cloud/instance/scripts/part-001: line 4: cd: /opt/aws/bin/cfn-init: No such file or directory
My CloudFormation template is:
{
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "ami-011b3ccf1bd6db744",
"InstanceType": "t2.micro",
"KeyName": "EC2KeyPair",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash",
"\n",
"exec > /tmp/user_data.log 2>&1 \n",
"yum update -y\n",
"yum install -y epel-release\n",
"yum install -y https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.amzn1.noarch.rpm \n",
"ln -s /usr/local/lib/python2.7/site-packages/cfnbootstrap /usr/lib/python2.7/site-packages/cfnbootstrap\n",
"/opt/aws/bin/cfn-init -v",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource MyInstance",
" --configsets scripts ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets": {
"scripts": [
"configure_cfn",
"pythonInstallation"
]
},
"configure_cfn": {
"files": {
"/etc/cfn/hooks.d/cfn-auto-reloader.conf": {
"content": {
"Fn::Join": [
"",
[
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.EC2.Metadata.AWS::CloudFormation::Init\n",
"action=/opt/aws/bin/cfn-init -v",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource EC2",
" --configsets wordpress",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
},
"mode": "000400",
"owner": "root",
"group": "root"
},
"/etc/cfn/cfn-hup.conf": {
"content": {
"Fn::Join": [
"",
[
"[main]\n",
"stack=",
{
"Ref": "AWS::StackId"
},
"\n",
"region=",
{
"Ref": "AWS::Region"
},
"\n",
"verbose=true\n",
"interval=5\n"
]
]
},
"mode": "000400",
"owner": "root",
"group": "root"
},
"/var/www/html/index2.html": {
"content": "Hi"
}
},
"services": {
"sysvinit": {
"cfn-hup": {
"enabled": "true",
"ensureRunning": "true",
"files": [
"/etc/cfn/cfn-hup.conf",
"/etc/cfn/hooks.d/cfn-auto-reloader.conf"
]
}
}
}
},
"pythonInstallation": {
"packages": {
"yum": {
"wget": [],
"unzip": [],
"gcc-c++": [],
"zlib-devel": [],
"libffi-devel": [],
"httpd": []
}
},
"sources": {
"usr/src/": "https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz"
},
"commands": {
"python1": {
"command": "tar xzf Python-3.7.2.tgz",
"cwd": "/usr/src/"
},
"python2": {
"command": {
"comm1": "./configure --enable-optimizations",
"comm2": "make altinstall",
"comm3": "rm /usr/src/Python-3.7.2.tgz"
},
"cwd": "/usr/src/Python-3.7.2"
}
},
"files": {
"/var/www/html/index.php": {
"content": {
"Fn::Join": [
"",
[
"<html>\n",
" <head>\n",
" <title>AWS CloudFormation PHP Sample</title>\n",
" <meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\">\n",
" </head>\n",
" <body>\n",
" <h1>Welcome to the AWS CloudFormation PHP Sample</h1>\n",
" </body>\n",
"</html>\n"
]
]
}
}
}
}
}
}
}
}
}
It would appear that Redhat does not have cfn-init pre-loaded.
You could either use an Amazon Linux AMI instead, or see How to install aws-cfn-bootstrap/cfn-init package in Redhat using CloudFormation? - Stack Overflow for tips on how to load cfn-init as part of the boot process.

CloudFormation template failing to create "files" under MetaData Section

Could somebody help me point what wrong am I doing here. I am failing to understand how the meta data part uses the authentication resource, In the AWS::CloudFormation::Authentication part I've mentioned the role same as the one attached to the instance, Yet I'm unable to create the file "some.txt"
{
"Parameters": {
"SecurityGroupId": {
"Description": "Security group for instance",
"Type": "AWS::EC2::SecurityGroup::Id"
}
},
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"configsets": {
"InstallIt": ["config1"]
},
"config1": {
"files": {
"/home/ec2-user/some.txt": {
"content": "This is my name ",
"encoding": "base64",
"mode": "000644",
"owner": "root",
"group": "root"
}
}
}
},
"AWS::CloudFormation::Authentication": {
"HelpMe": {
"type": "S3",
"buckets": "poc-bucket",
"roleName": "EC2andS3"
}
}
},
"Properties": {
"KeyName": "GoldenImage-NV-Anant",
"DisableApiTermination": "false",
"ImageId": "ami-0b33d91d",
"InstanceType": "t2.micro",
"Monitoring": "false",
"SubnetId": "subnet-73487a59",
"SecurityGroupIds": [{
"Ref": "SecurityGroupId"
}],
"IamInstanceProfile": {
"Ref": "MyInstanceProfile"
},
"Tags": [{
"Key": "Name",
"Value": "GeicoUserDataPocInstance"
}],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"", [
"#!/bin/bash -ex \n",
"echo \"hello dudes\" > /home/ec2-user/hello.txt \n",
"yum update -y aws-cfn-bootstrap\n",
"/opt/aws/bin/cfn-init -v",
" --stack ", {
"Ref": "AWS::StackId"
},
" --resource MyInstance ",
" --configsets InstallIt ",
" --region ", {
"Ref": "AWS::Region"
}, "\n",
"echo \"bye dudes\" > /home/ec2-user/bye.txt", "\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ", {
"Ref": "AWS::StackId"
},
" --resource MyInstance ",
" --region ", {
"Ref": "AWS::Region"
}, "\n"
]
]
}
}
},
"CreationPolicy": {
"ResourceSignal": {
"Timeout": "PT90M",
"Count": "1"
}
}
},
"MyInstanceProfile": {
"Description": "Instance profile for the instance",
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": ["EC2andS3"]
}
}
}
}
configsets should be configSets with capital S:
"configSets": {
"InstallIt": ["config1"]
},
buckets property needs to be a list of strings (this might not be necessary, the documentation is a bit unclear):
"buckets": ["poc-bucket"]
AWS::CloudFormation::Authentication resource shouldn't be necessary unless the source of your file is an S3 bucket. Even then, it still shouldn't be necessary when using an attached instance profile, since it will use the instance profile for authentication by default.

AWS::CloudFormation::Init not executing commands in metadata

This is my template with a test command in the metadata. I can't figure out why the command doesn't get executed. It's supposed to save a string to a file in /tmp.
The machine is Ubuntu 16.04 with cloud-init installed. In UserData I install the helper scripts and execute cfn-init.
Thanks for your help.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Metadata": {
"AWS::CloudFormation::Designer": {
"2af5b799-f6bf-7f20-a6eb-943274f18373": {
"size": {
"width": 60,
"height": 60
},
"position": {
"x": 326,
"y": 118
},
"z": 0,
"embeds": []
}
}
},
"Resources": {
"EC2I3WADD": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": "ami-c60b90d1",
"KeyName": "CF-KEY",
"InstanceType": "t2.micro",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"apt-get -y install python-setuptools\n",
"mkdir aws-cfn-bootstrap-latest\n",
"curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1\n",
"easy_install aws-cfn-bootstrap-latest\n",
"/usr/local/bin/cfn-init --stack ",
{
"Ref": "AWS::StackName"
},
" --resource WebServer",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
}
}
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "2af5b799-f6bf-7f20-a6eb-943274f18373"
},
"AWS::CloudFormation::Init": {
"config": {
"commands": {
"test": {
"command": "echo \"$MAGIC\" > /tmp/test.txt",
"env": {
"MAGIC": "I come from the environment!"
}
}
}
}
}
}
}
}
}
The issue is likely being cause by an incorrect --resource argument being passed to the cfn-init command within your UserData. This argument should match the logical resource name of the resource containing the MetaData, in your case EC2I3WADD.
"/usr/local/bin/cfn-init --stack ", { "Ref": "AWS::StackName" }, " --resource EC2I3WADD", " --region ", { "Ref": "AWS::Region" }
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html

CloudFormation WaitCondition requires public access

I have a CloudFormation script which deploys a single instance in the VPC.
The script only works when the instance has a public IP. Without a public IP address, the template deployment fails at WaitCondition stage. I guess thats because the WaitCondition requires instance has access to the Internet?
Instead of assigning a public IP, will it work if I set the proxy address on the server? So the instance access the Internet via web proxy. If that's the case, what's the web URLs need to be whitelisted on the proxy? and can I add the add proxy address command in the userdata?
Below is my template:
"Mappings": {
"AWSRegion2AMI": {
"ap-southeast-2": {
"Windows2008r2": "ami-27b39a44",
"Windows2012r2": "ami-83b198e0"
}
}
},
"Resources": {
"DomainController": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"config": {
"files": {
"c:\\cfn\\cfn-hup.conf": {
"content": {
"Fn::Join": ["", [
"[main]\n",
"stack=", {
"Ref": "AWS::StackId"
}, "\n",
"region=", {
"Ref": "AWS::Region"
}, "\n"
]]
}
},
"c:\\cfn\\hooks.d\\cfn-auto-reloader.conf": {
"content": {
"Fn::Join": ["", [
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.DomainController.Metadata.AWS::CloudFormation::Init\n",
"action=cfn-init.exe -v -s ", {
"Ref": "AWS::StackId"
},
" -r DomainController",
" --region ", {
"Ref": "AWS::Region"
}, "\n"
]]
}
},
"C:\\cfn\\RunCommand.bat": {
"content": "%~1\nIF %ERRORLEVEL% GTR 10 ( exit /b 1 ) else ( exit /b 0 )"
}
},
"commands": {
"1-run-dcpromo": {
"command": {
"Fn::Join": ["", [
"C:\\cfn\\RunCommand.bat \"dcpromo /unattend /ReplicaOrNewDomain:Domain /NewDomain:Forest /NewDomainDNSName:", {
"Ref": "DomainDNSName"
},
" /ForestLevel:4 /DomainNetbiosName:", {
"Ref": "DomainNetBIOSName"
},
" /DomainLevel:4 /InstallDNS:Yes /ConfirmGc:Yes /CreateDNSDelegation:No /DatabasePath:\"C:\\Windows\\NTDS\" /LogPath:\"C:\\Windows\\NTDS\" /SYSVOLPath:\"C:\\Windows\\SYSVOL\" /SafeModeAdminPassword=", {
"Ref": "RestoreModePassword"
},
" /RebootOnCompletion:Yes\""
]]
},
"waitAfterCompletion": "forever"
},
"2-signal-success": {
"command": {
"Fn::Join": ["", [
"cfn-signal.exe -e 0 \"", {
"Fn::Base64": {
"Ref": "DomainControllerWaitHandle"
}
}, "\""
]]
}
}
},
"services": {
"windows": {
"cfn-hup": {
"enabled": "true",
"ensureRunning": "true",
"files": ["c:\\cfn\\cfn-hup.conf", "c:\\cfn\\hooks.d\\cfn-auto-reloader.conf"]
}
}
}
}
}
},
"Properties": {
"ImageId": {
"Fn::FindInMap": ["AWSRegion2AMI", {
"Ref": "AWS::Region"
}, "Windows2008r2"]
},
"InstanceType": {
"Ref": "InstanceType"
},
"NetworkInterfaces": [{
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"SubnetId": {
"Ref": "SubnetId"
}
}],
"KeyName": {
"Ref": "KeyName"
},
"UserData": {
"Fn::Base64": {
"Fn::Join": ["", [
"<script>\n",
"cfn-init.exe -v -s ", {
"Ref": "AWS::StackId"
},
" -r DomainController ",
" --region ", {
"Ref": "AWS::Region"
}, "\n",
"</script>"
]]
}
}
}
},
"DomainControllerWaitCondition": {
"Type": "AWS::CloudFormation::WaitCondition",
"DependsOn": "DomainController",
"Properties": {
"Handle": {
"Ref": "DomainControllerWaitHandle"
},
"Timeout": "1500"
}
},
"DomainControllerWaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
}
Pass in the proxy by adding on of those parameters to cfn-signal.exe:
--http-proxy
An HTTP proxy (non-SSL). Use the following format: http://user:password#host:port
--https-proxy
An HTTPS proxy. Use the following format: https://user:password#host:port
The signal is sent to an S3 Bucket. You can whitelist something like:
https://cloudformation-waitcondition-*.s3.amazonaws.com
Note: You can also use the S3 VPC endpoint feature to allow resources inside private subnets to access S3. http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html