"AppLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"AssociatePublicIpAddress": true,
"EbsOptimized": false,
"ImageId": {
"Ref": "amiID"
},
"InstanceType": "t3.small",
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash -xe\n",
"har-extractor /home/ubuntu/work/git.codavel.com.har --output /home/ubuntu/extract/\n"
]]
}
},
"SecurityGroups": [
{
"Ref": "InstanceSecGroup"
}
],
}
},
Hi Team,
This is my cloud-formation template for auto-scaling and it is working properly but the thing is I am running one command in Userdata that is not working. I have tried every possible thing but didn't work at all. And If I am running this manually this command is working.
So please help me out in this how will I resolve this issue. I am running this command on Ubuntu machine.
You can debug this by taking a look at the /var/log/cloud-init-output.log file which contains the output for your Linux Userdata commands.
If this does not provide any useful debug, the next necessary step would be try running the command as root which mimics exactly what the functionality carries out.
Related
Context
We have an automated AMI building process done by Packer in order to setup our instance images after code changes, and assign them to our Load Balancer launch configuration for faster autoscaling.
Problem
Recently the instances launched with the development environment AMI build started refusing the corresponding private key. After attaching an instance with that same AMI in a public subnet, I connected through EC2 connect and noticed that the public key was not present in the /home/ec2-user/.ssh/authorized_keys file, even though it was added at launch time through the launch configuration or even manually through the console.
The only one present being the Packed temporary SSH key created during the AMI packaging.
Additional info
Note that the key pair is mentionned in the instance details as though it was present but it is NOT, which was tricky to debug because we tend to trust what the console tells us.
What's even more puzzling is that the same AMI build for QA environment (which is exactly the same apart from some application variables) does include the EC2 SSH key correctly, and we are using the same key for DEV & QA environments.
We're using the same Packer version (1.5.1) than ever to avoid inconsistencies, so it most likely cannot come from that, but I suspect it does come from the build since it does not happen with other AMIs.
If someone has a clue about what's going on, I'd be glad to know.
Thanks !
Edit
Since it was requested in the comment, I can show you the "anonymized" packer template, for confidentiality reasons I can't show you the ansible tasks or other details. However, note that this template is IDENTICAL to the one used for QA (same source code) which does not gives the error.
"variables": {
"aws_region": "{{env `AWS_REGION`}}",
"inventory_file": "{{env `Environment`}}",
"instance_type": "{{env `INSTANCE_TYPE`}}"
},
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"vpc_id": "vpc-xxxxxxxxxxxxxxx",
"subnet_id": "subnet-xxxxxxxxxxxxxxxxxx",
"security_group_id": "sg-xxxxxxxxxxxxxxxxxxxxx",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "amzn2-ami-hvm-2.0.????????.?-x86_64-gp2",
"root-device-type": "ebs",
"state": "available"
},
"owners": "amazon",
"most_recent": true
},
"instance_type": "t3a.{{ user `instance_type` }}",
"ssh_username": "ec2-user",
"ami_name": "APP {{user `inventory_file`}}",
"force_deregister": "true",
"force_delete_snapshot": "true",
"ami_users": [
"xxxxxxxxxxxxxxxxx",
"xxxxxxxxxxxxxxxxx",
"xxxxxxxxxxxxxxxxx"
]
}
],
"provisioners": [
{
"type": "shell",
"inline": [
"sudo amazon-linux-extras install ansible2",
"mkdir -p /tmp/libs"
]
},
{
"type": "file",
"source": "../frontend",
"destination": "/tmp"
},
{
"type": "file",
"source": "../backend/target/",
"destination": "/tmp"
},
{
"type": "ansible-local",
"playbook_file": "./app/instance-setup/playbook/setup-instance.yml",
"inventory_file": "./app/instance-setup/playbook/{{user `inventory_file`}}",
"playbook_dir": "./app/instance-setup/playbook",
"extra_arguments": [
"--extra-vars",
"ENV={{user `inventory_file`}}",
"--limit",
"{{user `inventory_file`}}",
"-v"
]
},
{
"type": "shell",
"inline": [
"sudo rm -rf /tmp/*"
]
}
]
}
I've created a cloudformation script that creates an ec2 instance, a few volumes upon other features. When the ec2 instance has been initialised I want it to run a bash script automatically to mount a volume created in the script (not the root volume) to /var/www/vhosts. But at the moment it creates the directory so I know its running the bash script but doesn't successfully execute the other commands. Here is my cloudformation script. When I run the 3 commands listed below separately when ssh to the ec2 instance they work fine and the volume has been mounted to the correct location.
"Ec2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {
"Ref": "ImageIdentification"
},
"AvailabilityZone" : "eu-west-1a",
"InstanceType": {
"Ref": "InstanceType"
},
"SecurityGroupIds" : [
{
"Ref": "SecurityGroup"
}
],
"DisableApiTermination" : "true",
"KeyName": {
"Ref": "keyPairName"
},
"Tags" : [
{"Key" : "Name", "Value" : { "Ref" : "EC2InstanceName"}}
],
"UserData" : {"Fn::Base64" : {"Fn::Join" : ["", [
"#!/bin/bash -v\n",
"sudo mkdir -p /var/www/vhosts\n",
"sudo mkfs -t ext4 /dev/xvdh\n",
"sudo mount /dev/xvdh /var/www/vhosts\n"
]]}}
}
},
Thanks for your help.
To run the bash script in CloudFormation you need to specify the remote link in the user data section or write down the commands one by one in the user data.
Tags:
- Key: Name
Value: test
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
cd /var/tmp
wget https://raw.githubusercontent.com/installer/cloudformation/install.sh
sudo bash /var/tmp/install.sh 4 '${CORE.PrivateIp}'
I have added an EMR cluster to a stack. After updating the stack successfully (CloudFormation), I can see the master and slave nodes in EC2 console and I can SSH into the master node. But AWS console does not show the new cluster. Even aws emr list-clusters doesn't show the cluster. I have triple checked the region and I am certain I'm looking at the right region.
Relevant CloudFormation JSON:
"Spark01EmrCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Name": "Spark01EmrCluster",
"Applications": [
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"Instances": {
"Ec2KeyName": {"Ref": "KeyName"},
"Ec2SubnetId": {"Ref": "PublicSubnetId"},
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Master"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Core"
}
},
"Configurations": [
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"ConfigurationProperties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
}
],
"BootstrapActions": [
{
"Name": "InstallPipPackages",
"ScriptBootstrapAction": {
"Path": "[S3 PATH]"
}
}
],
"JobFlowRole": {"Ref": "Spark01InstanceProfile"},
"ServiceRole": "MyStackEmrDefaultRole",
"ReleaseLabel": "emr-5.13.0"
}
}
The reason is missing VisibleToAllUsers property, which defaults to false. Since I'm using AWS Vault (i.e. using STS AssumeRole API to authenticate), I'm basically a different user every time, so I couldn't see the cluster. I couldn't update the stack to add VisibleToAllUsers either as I was getting Job flow ID does not exist.
The solution was to login as root user and fix things from there (I had to delete the cluster manually, but removing it from the stack template JSON and updating the stack would probably have worked if I hadn't messed things up already).
I then added the cluster back to the template (with VisibleToAllUsers set to true) and updated the stack as usual (AWS Vault).
I created a Windows 2012 AMI and created an instance of that AMI using the CloudFormation template shown below.
In that JSON script I want to call a PowerShell script to disable a service (simple one). The EC2 Windows 2012 instance gets created. I made sure EC2Config service was running before I took AMI. It works now. Following is the code that works fine. But the question is, I don't clearly understand the interaction between cfn-hup, cfn-signal and cfn-init. Honestly I read about all the 4 helper scripts. But I am not wrap my brain around these helper scripts.
Are there any blogs or documentation about how these 4 helper scripts work together?
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"files" : {
"c:\\cfn\\cfn-hup.conf" : {
"content" : { "Fn::Join" : ["", [
"[main]\n",
"stack=", { "Ref" : "AWS::StackId" }, "\n",
"region=", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\cfn\\hooks.d\\cfn-auto-reloader.conf" : {
"content": { "Fn::Join" : ["", [
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.MyInstance.Metadata.AWS::CloudFormation::Init\n",
"action=cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\scripts\\test.ps1" : {
"content": { "Fn::Join" : ["", [
"Write-Host Hello World!\n"
]]}
}
},
"commands" : {
"1-run-script" : {
"command" : { "Fn::Join" : [ "", [
"Powershell.exe Set-ExecutionPolicy Unrestricted -force;Unblock-File C:\\PowershellScripts\\WindowsServiceManager.ps1;. C:\\PowershellScripts\\WindowsServiceManager.ps1;SetWindowsServiceStartupType Dnscache Manual;StopWindowsService Dnscache"
]]}}
},
"services": {
"windows": {
"cfn-hup": {
"enabled": "true",
"ensureRunning": "true",
"files": ["c:\\cfn\\cfn-hup.conf", "c:\\cfn\\hooks.d\\cfn-auto-reloader.conf"]
}
}
}
}
}
},
"Properties": {
"DisableApiTermination": "FALSE",
"ImageId": "ami-3723c04f",
"InstanceType": "t2.micro",
"KeyName": "EC2Instances",
"Monitoring": "false",
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"<script>\n",
"cfn-init.exe -v -s ", { "Ref" : "AWS::StackName" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"cfn-signal.exe -e 0 ", { "Fn::Base64" : { "Ref" : "WindowsServerWaitHandle" }}, "\n",
"</script>\n"
]]}}
}
},
"WindowsServerWaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"WindowsServerWaitCondition": {
"Type": "AWS::CloudFormation::WaitCondition",
"DependsOn": "MyInstance",
"Properties": {
"Handle": { "Ref": "WindowsServerWaitHandle" },
"Timeout": "1800"
}
}
}
}
Found a decent explanation here:
https://aws.amazon.com/blogs/devops/best-practices-for-deploying-applications-on-aws-cloudformation-stacks/
Sequence of how AWS::CloudFormation::Init works:
You specify application configuration using the AWS::CloudFormation::Init section for an EC2 instance in your CloudFormation template.
You kick-off a CloudFormation stack creation using the template.
The AWS CloudFormation service starts creating a stack, including the EC2 instance.
After the EC2 instance is up and running, a CloudFormation helper script, cfn-init, is executed on the instance to configure the instance in accordance with your AWS::CloudFormation::Init template specification.*
Another CloudFormation helper script, cfn-signal, is executed on the instance to let the remote AWS CloudFormation service know the result (success/failure) of the configuration.* You can optionally have the CloudFormation service hold off on marking the EC2 instance state and the stack state “CREATE_COMPLETE” until the CloudFormation service hears a success signal for the instance. The holding-off period is specified in the template using a CreationPolicy.
*You can download the CloudFormation helper scripts for both Linux and Windows. These come preinstalled on the Linux and Windows AMIs provided by Amazon. You need to specify the commands to trigger cfn-init and cfn-signal in the EC2 user data script. Once an instance is up and running, the EC2 user data script is executed automatically for most Linux distributions and Windows.
Once your application stack is up and running, chances are that you will update the application, apply an OS patch, or perform some other configuration update in a stack’s lifecycle. You just update the AWS::CloudFormation::Init section in your template (for example, specify a newer version of an application package), and call UpdateStack. When you do, CloudFormation updates the instance metadata in accordance with the updated template. Then the cfn-hup daemon running on the instance detects the updated metadata and reruns cfn-init to update the instance in accordance with the updated configuration. cfn-hup is one of the CloudFormation helper scripts available on both Linux and Windows.
I am creating an AWS EC2 instance in a VPC with internet access using cloudformation. I am able to create the EC2 as expected based on the JSON. But it seems like the instance state goes to stopped soon after creating the EC2. I was expecting the EC2 to be up and in running state as soon as created.
Has anyone faced this problem?
I am able to go to the AWS console and manually make the instance to running state successfully though.
Here is the JSON for EC2
"PublicEC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {
"Fn::FindInMap": ["AWSRegionArch2AMI", {
"Ref": "AWS::Region"
},
"64"
]
},
"InstanceType": {
"Ref": "InstanceType"
},
"KeyName": {
"Ref": "KeyPair"
},
"BlockDeviceMappings": [{
"DeviceName": "/dev/sda1",
"Ebs": {
"VolumeSize": "8"
}
}, {
"DeviceName": "/dev/sdm",
"Ebs": {
"VolumeSize": "8"
}
}],
"Tags": [{
"Key": "Name",
"Value": "Sample-PublicEC2"
}],
"UserData": {
"Fn::Base64": {
"Ref": "WebServerPort"
}
},
"NetworkInterfaces": [{
"AssociatePublicIpAddress": "true",
"DeleteOnTermination": "true",
"DeviceIndex": "0",
"SubnetId": {
"Ref": "PublicSubnet"
},
"GroupSet": [{
"Ref": "PublicSecurityGroup"
}]
}]
}
}
The UserData in your template looks invalid. It's possible that the instance startup aborts on invalid data. Try removing this property and creating the stack again.
If this doesn't solve the problem, you can try looking at the console output of the stopped instance for more information. See Getting Console Output and Rebooting Instances for instructions on how to do this using the AWS Management Console.