I am having cloudformation template which contains two instances with userdata property.
I need to fetch the data from one instance's userdata and pass to another instances userdata.
For example(from below), need to fetch "test" from instance1, and pass to instance2 userdata.
Sample Template:
"instance1": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"set -x\n",
"echo test\n",
]]}}}},
"instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"set -x\n",
//fetch the value
]]}}}},
Basically you have 2 options:
Use some sort of a shared memory ( ElastiCache, RDS, S3.... )
scp/ssh directly to another instance and fetch what ever info you need.
I would suggest using shared memory for example S3:
On instance 1:
echo "test" > /tmp/myfile
aws s3 cp /tmp/myfile s3://<bucket>/myfile
On instance 2:
aws s3 cp s3://<bucket>/myfile /tmp/myfile
cat /tmp/myfile
Related
I am new to CloudFormation templates. I have basic template in yaml that creates an EC2 Instance. Every time I create a stack and use this template, the EC2 Instance is ALWAYS created on US East N. Virginia region. I am trying to change this so that the EC2 Instance resides in US-WEST-2 region. After some research, it appears that this is something that is not specified within the template. Instead, I need to change the region to us-west-2 in AWS console and then create a new stack. Is my understanding correct?
Unfortunately, you can't specify the region in a cloudformation template.
You should either pass region as a command line argument
aws --region eu-west-1 cloudformation create-stack --stack-name ...
or, specify the default region in aws cli config file ~/.aws/config
[default]
region=eu-west-1
What am I missing here? I am sure we can specify region where the stack is created in CFN template using parameters and we do have active templates which creates our stack in respective region based on the parameter value.
The AWS::Region pseudo parameter is a value that AWS CloudFormation resolves as the region where the stack is created.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/gettingstarted.templatebasics.html
Here is a sub-section of sample template
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"InstanceType": {
"Description": "Instance Type",
"Type": "String",
"Default": "t2.xlarge"
},
"SubnetUSEAST1": {
"Description": "Subnet on which Ec2 instance needs to be created",
"Type": "String",
"Default": "subnet-xxxxxxxx"
},
"SubnetUSWEST2": {
"Description": "Subnet on which Ec2 instance needs to be created",
"Type": "String",
"Default": "subnet-yyyyyyyy"
}
},
"Conditions": {
"useast1": {
"Fn::Equals": [
{
"Ref": "AWS::Region"
},
"us-east-1"
]
},
"uswest2": {
"Fn::Equals": [
{
"Ref": "AWS::Region"
},
"us-west-2"
]
}
},
"Resources": {
"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"InstanceType": {
"Ref": "InstanceType"
},
"NetworkInterfaces": [
{
"SubnetId": {
"Fn::If": [
"useast1",
{
"Ref": "SubnetUSEAST1"
},
{
"Ref": "SubnetUSWEST2"
}
]
},
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0"
}
]
}
}
}
}
If you are able to split your template into parts, you could deploy to different regions at once via some orchestration and StackSets.
I've created a cloudformation script that creates an ec2 instance, a few volumes upon other features. When the ec2 instance has been initialised I want it to run a bash script automatically to mount a volume created in the script (not the root volume) to /var/www/vhosts. But at the moment it creates the directory so I know its running the bash script but doesn't successfully execute the other commands. Here is my cloudformation script. When I run the 3 commands listed below separately when ssh to the ec2 instance they work fine and the volume has been mounted to the correct location.
"Ec2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {
"Ref": "ImageIdentification"
},
"AvailabilityZone" : "eu-west-1a",
"InstanceType": {
"Ref": "InstanceType"
},
"SecurityGroupIds" : [
{
"Ref": "SecurityGroup"
}
],
"DisableApiTermination" : "true",
"KeyName": {
"Ref": "keyPairName"
},
"Tags" : [
{"Key" : "Name", "Value" : { "Ref" : "EC2InstanceName"}}
],
"UserData" : {"Fn::Base64" : {"Fn::Join" : ["", [
"#!/bin/bash -v\n",
"sudo mkdir -p /var/www/vhosts\n",
"sudo mkfs -t ext4 /dev/xvdh\n",
"sudo mount /dev/xvdh /var/www/vhosts\n"
]]}}
}
},
Thanks for your help.
To run the bash script in CloudFormation you need to specify the remote link in the user data section or write down the commands one by one in the user data.
Tags:
- Key: Name
Value: test
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
cd /var/tmp
wget https://raw.githubusercontent.com/installer/cloudformation/install.sh
sudo bash /var/tmp/install.sh 4 '${CORE.PrivateIp}'
I created a Windows 2012 AMI and created an instance of that AMI using the CloudFormation template shown below.
In that JSON script I want to call a PowerShell script to disable a service (simple one). The EC2 Windows 2012 instance gets created. I made sure EC2Config service was running before I took AMI. It works now. Following is the code that works fine. But the question is, I don't clearly understand the interaction between cfn-hup, cfn-signal and cfn-init. Honestly I read about all the 4 helper scripts. But I am not wrap my brain around these helper scripts.
Are there any blogs or documentation about how these 4 helper scripts work together?
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"files" : {
"c:\\cfn\\cfn-hup.conf" : {
"content" : { "Fn::Join" : ["", [
"[main]\n",
"stack=", { "Ref" : "AWS::StackId" }, "\n",
"region=", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\cfn\\hooks.d\\cfn-auto-reloader.conf" : {
"content": { "Fn::Join" : ["", [
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.MyInstance.Metadata.AWS::CloudFormation::Init\n",
"action=cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\scripts\\test.ps1" : {
"content": { "Fn::Join" : ["", [
"Write-Host Hello World!\n"
]]}
}
},
"commands" : {
"1-run-script" : {
"command" : { "Fn::Join" : [ "", [
"Powershell.exe Set-ExecutionPolicy Unrestricted -force;Unblock-File C:\\PowershellScripts\\WindowsServiceManager.ps1;. C:\\PowershellScripts\\WindowsServiceManager.ps1;SetWindowsServiceStartupType Dnscache Manual;StopWindowsService Dnscache"
]]}}
},
"services": {
"windows": {
"cfn-hup": {
"enabled": "true",
"ensureRunning": "true",
"files": ["c:\\cfn\\cfn-hup.conf", "c:\\cfn\\hooks.d\\cfn-auto-reloader.conf"]
}
}
}
}
}
},
"Properties": {
"DisableApiTermination": "FALSE",
"ImageId": "ami-3723c04f",
"InstanceType": "t2.micro",
"KeyName": "EC2Instances",
"Monitoring": "false",
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"<script>\n",
"cfn-init.exe -v -s ", { "Ref" : "AWS::StackName" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"cfn-signal.exe -e 0 ", { "Fn::Base64" : { "Ref" : "WindowsServerWaitHandle" }}, "\n",
"</script>\n"
]]}}
}
},
"WindowsServerWaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"WindowsServerWaitCondition": {
"Type": "AWS::CloudFormation::WaitCondition",
"DependsOn": "MyInstance",
"Properties": {
"Handle": { "Ref": "WindowsServerWaitHandle" },
"Timeout": "1800"
}
}
}
}
Found a decent explanation here:
https://aws.amazon.com/blogs/devops/best-practices-for-deploying-applications-on-aws-cloudformation-stacks/
Sequence of how AWS::CloudFormation::Init works:
You specify application configuration using the AWS::CloudFormation::Init section for an EC2 instance in your CloudFormation template.
You kick-off a CloudFormation stack creation using the template.
The AWS CloudFormation service starts creating a stack, including the EC2 instance.
After the EC2 instance is up and running, a CloudFormation helper script, cfn-init, is executed on the instance to configure the instance in accordance with your AWS::CloudFormation::Init template specification.*
Another CloudFormation helper script, cfn-signal, is executed on the instance to let the remote AWS CloudFormation service know the result (success/failure) of the configuration.* You can optionally have the CloudFormation service hold off on marking the EC2 instance state and the stack state “CREATE_COMPLETE” until the CloudFormation service hears a success signal for the instance. The holding-off period is specified in the template using a CreationPolicy.
*You can download the CloudFormation helper scripts for both Linux and Windows. These come preinstalled on the Linux and Windows AMIs provided by Amazon. You need to specify the commands to trigger cfn-init and cfn-signal in the EC2 user data script. Once an instance is up and running, the EC2 user data script is executed automatically for most Linux distributions and Windows.
Once your application stack is up and running, chances are that you will update the application, apply an OS patch, or perform some other configuration update in a stack’s lifecycle. You just update the AWS::CloudFormation::Init section in your template (for example, specify a newer version of an application package), and call UpdateStack. When you do, CloudFormation updates the instance metadata in accordance with the updated template. Then the cfn-hup daemon running on the instance detects the updated metadata and reruns cfn-init to update the instance in accordance with the updated configuration. cfn-hup is one of the CloudFormation helper scripts available on both Linux and Windows.
I have existing infrastructure in us-east-1 region which needed to be cloned exactly to us-east-2 region. Used AWS CloudFormer to generate the JSON template from existing us-east-1 region, replaced all the us-east-1 with us-east-2 and started creating the stack but getting errors saying "Resource creation cancelled", specifically for all the EC2 instances
A snapshot of the template (only EC2 instance):
"instancei071dd59b": {
"Type": "AWS::EC2::Instance",
"Properties": {
"DisableApiTermination": "false",
"InstanceInitiatedShutdownBehavior": "stop",
"ImageId": "ami-1a41b377",
"InstanceType": "t2.medium",
"KeyName": "MyServer",
"Monitoring": "false",
"Tags": [
{
"Key": "MyServer OS",
"Value": "Windows Server"
},
{
"Key": "Name",
"Value": "MyServer_WEB_TEST_2"
}
],
"Volumes": [
{
"Device": "xvdb",
"VolumeId": {
"Ref": "volumevol9124b841"
}
}
],
"NetworkInterfaces": [
{
"DeleteOnTermination": "true",
"DeviceIndex": 0,
"SubnetId": {
"Ref": "subnet24031c0f"
},
"PrivateIpAddresses": [
{
"PrivateIpAddress": "172.31.53.184",
"Primary": "true"
}
],
"GroupSet": [
{
"Ref": "sgMyServerWEB"
}
],
"AssociatePublicIpAddress": "true"
}
]
}
},
"volumevol9124b841": {
"Type": "AWS::EC2::Volume",
"Properties": {
"AvailabilityZone": "us-east-2b",
"Size": "30",
"SnapshotId": "snap-95288b92",
"VolumeType": "gp2"
}
}
Before going with cloudformation template you will need to make sure you have following things in place :
Move your instance AMI to us-east-2 region then replace the snapshot id and AMI id in your template
Create a security group replace the security group id in your template
Replace subnet ID in your CF template with the one in us-east-2 region
The reason you will have to do this is every resource on AWS has unique IDs which cannot be replicated, if you want to replicate same you will need different Ids for that you need to create seperate resources and use them in your template.
If your doing this for a single instance only then you might do it manually by exporting AMI to us-east-2 region.
For collecting AMI ID in different region, I'd recommend to use the image name instead the AMI ID as the key.
To build resources to be placed in different regions, definitely is better to use CloudFormation. In this case you can use the lambda cli2cloudformation (https://github.com/lucioveloso/cli2cloudformation).
Using it, you can get the AMI ID across all regions and whatever other information that you are able to get using CLI.
To collect the AMI ID, create a lambda with cli2cloudformation and inside your template, create a custom resource as bellow:
"imageIdNameBased": {
"Type": "Custom::cli2cfnLambda",
"Properties": {
"ServiceToken": "arn:aws:lambda:eu-west-1:123456789012:function:cli2cfnLambda",
"CliCommandCreate": "ec2 describe-images --filters 'Name=name,Values=amzn-ami-hvm-2017.03.0.20170417-x86_64-gp2' --query 'Images[0]'"
}
}
In this case, I'm getting the AMI ID to the Image named 'amzn-ami-hvm-2017.03.0.20170417-x86_64-gp2'. You can change to your image name.
After that, you can retrieve it in any point of your CloudFormation stack.
"Fn::GetAtt" : ["imageIdNameBased", "ImageId"]
While creating along my AWS CloudFormation template I hit the 16KB limitation of user data.... and then I found out I can put the script in S3 (with all my user data) and copy that file over as part of user data and run that script... but my question is how can I take the parameters that I am passing into CloudFormation like below and pass that into the file/script/userdata that I download from S3 that I will run? So how can I pass the parameters from CloudFormation into /root/usr.sh script?
Here is my user data:
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -x\n\n",
"yum -y install tcsh lvm2 sysstat\n\n\n",
"# AWS CLI download and Installation\n",
"curl \"https://s3.amazonaws.com/aws-cli/awscli-bundle.zip\" -o \"/usr/awscli-bundle.zip\"\n",
"unzip /usr/awscli-bundle.zip -d /usr/awscmdline/\n",
"/usr/awscmdline/awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws\n",
"/usr/local/aws/bin/aws configure set region ",
{
"Ref": "AWS::Region"
},
"\n",
"/usr/local/bin/aws s3 cp s3://test123/usr.sh /root/usr.sh \n",
"chmod 744 /root/usr.sh \n",
"/root/usr.sh"
]
]
}
}
and here are the sample parameters:
"Parameters": {
"SelectInstanceType": {
"Description": "EC2 instance type",
"Type": "String",
"Default": "r3.8xlarge",
"AllowedValues": [
"r3.large",
"r3.xlarge",
"r3.2xlarge",
"r3.4xlarge",
"r3.8xlarge",
"c4.large",
"c4.xlarge",
"c4.2xlarge",
"c4.4xlarge",
"c4.8xlarge"
],
"ConstraintDescription": "Must be a valid EC2 instance type."
},
"Keyname": {
"Description": "Keypair to use to launch the instance",
"Type": "AWS::EC2::KeyPair::KeyName"
},
"IPAddress": {
"Description": "Private IP",
"Type": "String",
"Default": "10.10.10.X"
},
There's a few ways you could do it...
Configurations in a file
You could create a file with your configurations and then read the file from your script. For an example see: Setting environment variables with user-data
Set environment variables
As part of your User Data script, before you download and call a script, set environment variables (also in the above example file).
Pass parameters when executing your script
When downloading a script from Amazon S3 and the calling it, append parameters in the same way that your script is currently inserting AWS::Region. Your script will then need to read those parameters from the command-line.
Refer to parameters like this: { "Ref" : "InstanceTypeParameter" }
See: CloudFormation Parameters documentation