I've created a cloudformation script that creates an ec2 instance, a few volumes upon other features. When the ec2 instance has been initialised I want it to run a bash script automatically to mount a volume created in the script (not the root volume) to /var/www/vhosts. But at the moment it creates the directory so I know its running the bash script but doesn't successfully execute the other commands. Here is my cloudformation script. When I run the 3 commands listed below separately when ssh to the ec2 instance they work fine and the volume has been mounted to the correct location.
"Ec2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {
"Ref": "ImageIdentification"
},
"AvailabilityZone" : "eu-west-1a",
"InstanceType": {
"Ref": "InstanceType"
},
"SecurityGroupIds" : [
{
"Ref": "SecurityGroup"
}
],
"DisableApiTermination" : "true",
"KeyName": {
"Ref": "keyPairName"
},
"Tags" : [
{"Key" : "Name", "Value" : { "Ref" : "EC2InstanceName"}}
],
"UserData" : {"Fn::Base64" : {"Fn::Join" : ["", [
"#!/bin/bash -v\n",
"sudo mkdir -p /var/www/vhosts\n",
"sudo mkfs -t ext4 /dev/xvdh\n",
"sudo mount /dev/xvdh /var/www/vhosts\n"
]]}}
}
},
Thanks for your help.
To run the bash script in CloudFormation you need to specify the remote link in the user data section or write down the commands one by one in the user data.
Tags:
- Key: Name
Value: test
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
cd /var/tmp
wget https://raw.githubusercontent.com/installer/cloudformation/install.sh
sudo bash /var/tmp/install.sh 4 '${CORE.PrivateIp}'
Related
"AppLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"AssociatePublicIpAddress": true,
"EbsOptimized": false,
"ImageId": {
"Ref": "amiID"
},
"InstanceType": "t3.small",
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash -xe\n",
"har-extractor /home/ubuntu/work/git.codavel.com.har --output /home/ubuntu/extract/\n"
]]
}
},
"SecurityGroups": [
{
"Ref": "InstanceSecGroup"
}
],
}
},
Hi Team,
This is my cloud-formation template for auto-scaling and it is working properly but the thing is I am running one command in Userdata that is not working. I have tried every possible thing but didn't work at all. And If I am running this manually this command is working.
So please help me out in this how will I resolve this issue. I am running this command on Ubuntu machine.
You can debug this by taking a look at the /var/log/cloud-init-output.log file which contains the output for your Linux Userdata commands.
If this does not provide any useful debug, the next necessary step would be try running the command as root which mimics exactly what the functionality carries out.
I created a Windows 2012 AMI and created an instance of that AMI using the CloudFormation template shown below.
In that JSON script I want to call a PowerShell script to disable a service (simple one). The EC2 Windows 2012 instance gets created. I made sure EC2Config service was running before I took AMI. It works now. Following is the code that works fine. But the question is, I don't clearly understand the interaction between cfn-hup, cfn-signal and cfn-init. Honestly I read about all the 4 helper scripts. But I am not wrap my brain around these helper scripts.
Are there any blogs or documentation about how these 4 helper scripts work together?
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"files" : {
"c:\\cfn\\cfn-hup.conf" : {
"content" : { "Fn::Join" : ["", [
"[main]\n",
"stack=", { "Ref" : "AWS::StackId" }, "\n",
"region=", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\cfn\\hooks.d\\cfn-auto-reloader.conf" : {
"content": { "Fn::Join" : ["", [
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.MyInstance.Metadata.AWS::CloudFormation::Init\n",
"action=cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\scripts\\test.ps1" : {
"content": { "Fn::Join" : ["", [
"Write-Host Hello World!\n"
]]}
}
},
"commands" : {
"1-run-script" : {
"command" : { "Fn::Join" : [ "", [
"Powershell.exe Set-ExecutionPolicy Unrestricted -force;Unblock-File C:\\PowershellScripts\\WindowsServiceManager.ps1;. C:\\PowershellScripts\\WindowsServiceManager.ps1;SetWindowsServiceStartupType Dnscache Manual;StopWindowsService Dnscache"
]]}}
},
"services": {
"windows": {
"cfn-hup": {
"enabled": "true",
"ensureRunning": "true",
"files": ["c:\\cfn\\cfn-hup.conf", "c:\\cfn\\hooks.d\\cfn-auto-reloader.conf"]
}
}
}
}
}
},
"Properties": {
"DisableApiTermination": "FALSE",
"ImageId": "ami-3723c04f",
"InstanceType": "t2.micro",
"KeyName": "EC2Instances",
"Monitoring": "false",
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"<script>\n",
"cfn-init.exe -v -s ", { "Ref" : "AWS::StackName" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"cfn-signal.exe -e 0 ", { "Fn::Base64" : { "Ref" : "WindowsServerWaitHandle" }}, "\n",
"</script>\n"
]]}}
}
},
"WindowsServerWaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"WindowsServerWaitCondition": {
"Type": "AWS::CloudFormation::WaitCondition",
"DependsOn": "MyInstance",
"Properties": {
"Handle": { "Ref": "WindowsServerWaitHandle" },
"Timeout": "1800"
}
}
}
}
Found a decent explanation here:
https://aws.amazon.com/blogs/devops/best-practices-for-deploying-applications-on-aws-cloudformation-stacks/
Sequence of how AWS::CloudFormation::Init works:
You specify application configuration using the AWS::CloudFormation::Init section for an EC2 instance in your CloudFormation template.
You kick-off a CloudFormation stack creation using the template.
The AWS CloudFormation service starts creating a stack, including the EC2 instance.
After the EC2 instance is up and running, a CloudFormation helper script, cfn-init, is executed on the instance to configure the instance in accordance with your AWS::CloudFormation::Init template specification.*
Another CloudFormation helper script, cfn-signal, is executed on the instance to let the remote AWS CloudFormation service know the result (success/failure) of the configuration.* You can optionally have the CloudFormation service hold off on marking the EC2 instance state and the stack state “CREATE_COMPLETE” until the CloudFormation service hears a success signal for the instance. The holding-off period is specified in the template using a CreationPolicy.
*You can download the CloudFormation helper scripts for both Linux and Windows. These come preinstalled on the Linux and Windows AMIs provided by Amazon. You need to specify the commands to trigger cfn-init and cfn-signal in the EC2 user data script. Once an instance is up and running, the EC2 user data script is executed automatically for most Linux distributions and Windows.
Once your application stack is up and running, chances are that you will update the application, apply an OS patch, or perform some other configuration update in a stack’s lifecycle. You just update the AWS::CloudFormation::Init section in your template (for example, specify a newer version of an application package), and call UpdateStack. When you do, CloudFormation updates the instance metadata in accordance with the updated template. Then the cfn-hup daemon running on the instance detects the updated metadata and reruns cfn-init to update the instance in accordance with the updated configuration. cfn-hup is one of the CloudFormation helper scripts available on both Linux and Windows.
I am having cloudformation template which contains two instances with userdata property.
I need to fetch the data from one instance's userdata and pass to another instances userdata.
For example(from below), need to fetch "test" from instance1, and pass to instance2 userdata.
Sample Template:
"instance1": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"set -x\n",
"echo test\n",
]]}}}},
"instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"set -x\n",
//fetch the value
]]}}}},
Basically you have 2 options:
Use some sort of a shared memory ( ElastiCache, RDS, S3.... )
scp/ssh directly to another instance and fetch what ever info you need.
I would suggest using shared memory for example S3:
On instance 1:
echo "test" > /tmp/myfile
aws s3 cp /tmp/myfile s3://<bucket>/myfile
On instance 2:
aws s3 cp s3://<bucket>/myfile /tmp/myfile
cat /tmp/myfile
I want to assign one of my reserved Elastic IP's(ec2 classic ip) to Autoscaling group in VPC. Using AWS Cli I moved ip to vpc:
$ aws ec2 move-address-to-vpc --public-ip 23.23.23.23
And saw in aws concole, that this IP passed to VPC.
And Assigned in tags of AutoscalingGroup in Cloudformation template in Resources:
"Process": {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"LaunchConfigurationName": {"Ref": "PreprocessorLC"},
"LoadBalancerNames": [{"Ref": "ProcessELB"}],
"VPCZoneIdentifier" : [{ "Fn::Join" : [",", [ { "Ref" : "PublicSubnet1"}, { "Ref" : "PublicSubnet2"} ]]}],
"AvailabilityZones": {"Ref": "AZs"},
"MinSize" : "1",
"MaxSize" : "1",
"HealthCheckGracePeriod": 300,
"Tags" : [
{"Key": "Name", "Value": {"Fn::Join": ["", [{"Ref": "Env"}, "-Process"]]}, "PropagateAtLaunch": true},
{"Key": "WorkersScalingGroup", "Value": {"Fn::Join": ["", ["Offering-", {"Ref": "Env"}, "-Process-Worker"]]}, "PropagateAtLaunch": true},
{"Key": "EIP", "Value": {"Ref": "ProcessIP"}, "PropagateAtLaunch": true},
{"Key": "Environment", "Value": {"Ref": "Env"}, "PropagateAtLaunch": true}
]
}
}
And added value of "ProcessIP" in Parameters:
"ProcessIP":{
"Description": "DEV: 23.23.23.23",
"Type": "String",
"Default": "23.23.23.23",
"AllowedValues": ["23.23.23.23"]
}
And it doesn't worked. Still get random IP.
If someone can tell where I'm wrong or what should to add for make it work?
Thanks!
In my case, I needed to keep a bank of unassigned EIPs and randomly assign them to the EC2 when they boot. That way I always know my servers will be using a specific list of IPs that I can whitelist in other places.
If you create several EIPs named "prod-pool" you can then use this script.
apt install -y jq awscli
ALLOCATION_ID=`aws ec2 describe-addresses --filters="Name=tag:Name,Values=prod-pool" | jq -r '.Addresses[] | "\(.InstanceId) \(.AllocationId)"' | grep null | awk '{print $2}' | xargs shuf -n1 -e`
if [ ! -z $ALLOCATION_ID ]; then
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $ALLOCATION_ID --allow-reassociation
fi
You can attached this policy to your IAM user
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowEIPAttachment",
"Effect": "Allow",
"Resource": [
"*"
],
"Action": [
"ec2:AssociateAddress",
"ec2:DisassociateAddress"
]
}
]
}
Here is simple bash script:
#!/bin/sh
# Region in Which instance is running
EC2_REGION='us-east-1'
AWS_ACCESS_KEY='xxxxxxxxxxxx'
AWS_SECRET_ACCESS_KEY='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
#Instance ID captured through Instance meta data
InstanceID=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id`
#Elastic IP captured through the EIP instance tag
Elastic_IP=`/opt/aws/apitools/ec2/bin/ec2-describe-tags -O $AWS_ACCESS_KEY -W $AWS_SECRET_ACCESS_KEY --filter resource-id=$InstanceID --filter key='EIP' | cut -f5`
Allocate_ID=`/opt/aws/apitools/ec2/bin/ec2-describe-tags -O $AWS_ACCESS_KEY -W $AWS_SECRET_ACCESS_KEY --filter resource-id=$InstanceID --filter key="AllocationID" | cut -f5`
#Assigning Elastic IP to Instance
aws ec2 associate-address --instance-id $InstanceID --allocation-id $Allocate_ID
You need to explicitly associate the Elastic IP address with your desired EC2 instance. You can do this in a userdata script at launch time, or externally through other scripting or Configuration Management.
PropagateAtLaunch simply propagates tags from the Auto Scaling Group to any instances that are launched as a result of Auto Scaling actions. I'm not aware of any magic that would cause a tagged Elastic IP address to be associated with a launched instance.
See more discussion and examples of launch time scripting with EIPs here.
I created a AWS Lambda function which will automatically bind an Elastic IP address from a pool to instance of an autoscaling group. This alleviates the need to grab an EIP address in the bootscript of the instances. For a complete description check out
https://binx.io/blog/2019/09/02/how-to-dynamically-bind-elastic-ip-addresses-to-an-auto-scaling-group/
While creating along my AWS CloudFormation template I hit the 16KB limitation of user data.... and then I found out I can put the script in S3 (with all my user data) and copy that file over as part of user data and run that script... but my question is how can I take the parameters that I am passing into CloudFormation like below and pass that into the file/script/userdata that I download from S3 that I will run? So how can I pass the parameters from CloudFormation into /root/usr.sh script?
Here is my user data:
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -x\n\n",
"yum -y install tcsh lvm2 sysstat\n\n\n",
"# AWS CLI download and Installation\n",
"curl \"https://s3.amazonaws.com/aws-cli/awscli-bundle.zip\" -o \"/usr/awscli-bundle.zip\"\n",
"unzip /usr/awscli-bundle.zip -d /usr/awscmdline/\n",
"/usr/awscmdline/awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws\n",
"/usr/local/aws/bin/aws configure set region ",
{
"Ref": "AWS::Region"
},
"\n",
"/usr/local/bin/aws s3 cp s3://test123/usr.sh /root/usr.sh \n",
"chmod 744 /root/usr.sh \n",
"/root/usr.sh"
]
]
}
}
and here are the sample parameters:
"Parameters": {
"SelectInstanceType": {
"Description": "EC2 instance type",
"Type": "String",
"Default": "r3.8xlarge",
"AllowedValues": [
"r3.large",
"r3.xlarge",
"r3.2xlarge",
"r3.4xlarge",
"r3.8xlarge",
"c4.large",
"c4.xlarge",
"c4.2xlarge",
"c4.4xlarge",
"c4.8xlarge"
],
"ConstraintDescription": "Must be a valid EC2 instance type."
},
"Keyname": {
"Description": "Keypair to use to launch the instance",
"Type": "AWS::EC2::KeyPair::KeyName"
},
"IPAddress": {
"Description": "Private IP",
"Type": "String",
"Default": "10.10.10.X"
},
There's a few ways you could do it...
Configurations in a file
You could create a file with your configurations and then read the file from your script. For an example see: Setting environment variables with user-data
Set environment variables
As part of your User Data script, before you download and call a script, set environment variables (also in the above example file).
Pass parameters when executing your script
When downloading a script from Amazon S3 and the calling it, append parameters in the same way that your script is currently inserting AWS::Region. Your script will then need to read those parameters from the command-line.
Refer to parameters like this: { "Ref" : "InstanceTypeParameter" }
See: CloudFormation Parameters documentation