AWS cloudformation - how to attach EBS volume to EC2 windows instance - amazon-web-services

Here is a working Amazon CloudFormation JSON template that creates an Amazon EC2 Windows 2016 instance.
I want to attach an EBS volume that is backed-up on an S3 bucket. How can I do this? Any pointers please?
{
"Parameters" :{
"KeyName" : {
"Description" : "Name of the existing EC2 KeyPair",
"Type" : "String"
}
},
"Mappings" : {
"RegionMap" : {
"us-east-1" : {
"AMI" : "ami-48b4bf31"
},
"us-west-1": {
"AMI" : "ami-48b4bf31"
},
"us-west-2":{
"AMI" : "ami-48b4bf31"
}
}
},
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties": {
"KeyName" : {"Ref" : "KeyName"},
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}
}
}
},
"Outputs" : {
"AvailablityZone" : {
"Description" : "Availability Zone of the newly created EC2 instance",
"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] }
},
"PublicIp" :{
"Description" : "Public IP is",
"Value": {"Fn::GetAtt": ["Ec2Instance", "PublicIp"] }
}
}
}

There are probably two parts to this question:
1) How do I specify details about the ebs volumes of my windows host in cloudformation
I think you are looking for something like this block device mappings (ephemeral disk or ebs) or Volumes (EBS only)
"BlockDeviceMappings" : [
{
"DeviceName" : "/dev/sdc",
"Ebs" : { "VolumeSize" : "50" }
},
{
"DeviceName" : "/dev/sdd",
"Ebs" : { "VolumeSize" : "100" }
}
]
or volumes
"Volumes" : [
{ "VolumeId" :
{ "Ref" : "NewVolume" }, "Device" : "/dev/sdk" }
]
and secondly
2) How do I back up my windows host.
The easiest way is probably with the aws cli snapshot command.
aws ec2 create-snapshot --volume-id vol-1234567890abcdef0 --description "Backup of my windows server"
Sometimes people drop that in cron on a linux box, wrap it with a little logic and loop it over the whole infrastructure. Nice, easy, cheap backups of AWS infrastructure. If you ever need to restore you can create a new image from snapshot.

Backup: Your best strategy is to create an AMI on a regular basis. An AMI is a snapshot of the volumes attached to an Amazon EC2 instance -- yes, it can include ALL the disks attached to an instance. An AMI is actually just a collection of EBS snapshots, plus some metadata.
Restore: Launch a new Amazon EC2 instance from that AMI. It will contain all data, on all disks, that was present when the AMI was created. (It creates new EBS volumes, but they will contain the same data as when the AMI was created.)
Each time you create a new AMI, it will receive a new AMI-ID. Therefore, I suggest that your CloudFormation template accepts the AMI-ID as a parameter that can be entered when the stack is created. You would simply paste in the AMI-ID of the latest AMI and the instance would use that AMI.
Also, please note that AMIs are created in one region only. You can copy the AMI to a different region, but it will receive a different AMI-ID in the new region.
Frankly, your CloudFormation templates appears to be merely launching the EC2 instance, which you could do just as easily in the console without using CloudFormation.

Related

attribute publicip was not found for resource during aws cloudformation

I'm very new to aws cloudformation, I try to launch a EC2 with Neo4j install in a private VPC, I have found someone who has already created a cloudformation template for instance with Neo4j, but that instance is for public VPC, so I have modified the template to suit my purpose, but I got this problem when I launch it: 'attribute publicip was not found for resource'
Here is some part of the script (without the neo4j bash script and EBS volume setup):
"Mappings" : {
"AWSRegionArch2AMI" : {
"eu-west-1" : { "64" : "ami-58d7e821" }
}
},
"Parameters": {
"InstanceType" : {
"Description" : "EC2 instance type",
"Type" : "String",
"Default" : "m5.large",
"ConstraintDescription" : "Must be a valid EC2 instance type."
},
"SSHKeyName": {
"Description": "Name of the SSH key that you will use to access the server (must be on AWS Availability Zone already)",
"Type": "String"
},
"NetworkWhitelist": {
"Description": " The IP address range that can be used to connect to the Neo4j server (by REST or SSH)",
"Type": "String",
"MinLength": "9",
"MaxLength": "18",
"Default": "",
"AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
"ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x."
},
"SubnetId" : {
"Type" : "AWS::EC2::Subnet::Id",
"Description" : "SubnetId of an existing subnet (for the primary network) in your Virtual Private Cloud (VPC)"
},
"SecurityGroupIds": {
"Type": "AWS::EC2::SecurityGroup::Id",
"Description" : "Existing SecurityGroups ID"
},
"AvailabilityZone": {
"Type" : "AWS::EC2::AvailabilityZone::Name",
"Description" : "Select the Availability Zone"
}
},
"Resources": {
"Server": {
"Type": "AWS::EC2::Instance",
"Properties": {
"AvailabilityZone": {
"Ref": "AvailabilityZone"
},
"DisableApiTermination": "FALSE",
"ImageId": {
"Fn::FindInMap": [ "AWSRegionArch2AMI", {
"Ref": "AWS::Region"
}, "64"]
},
"InstanceType": {
"Ref": "InstanceType"
},
"KeyName": {"Ref": "SSHKeyName"},
"Monitoring": "false",
"NetworkInterfaces" : [
{
"AssociatePublicIpAddress": false,
"DeleteOnTermination": true,
"DeviceIndex": "0",
"SubnetId": {"Ref": "SubnetId"},
"GroupSet": [ {"Ref": "SecurityGroupIds"} ]
}
],
Can't I lanuch an instance without public ip address just like in 'Configure Instance Details' in 'Launch instance wizard'?
thank you
Do you have the "Auto-assign Public IP" option enabled for the subnet you're trying to create the instance in? Because you're explicitly not associating a public IP address, it might be failing because the resource is expecting to have a public IP address assigned. A surefire way to test this would be to set the SubnetId parameter to the ID of a subnet that does not automatically assign public IP addresses when you deploy the stack.
Perhaps you are tying to do Fn::GetAtt on the Instance logical Id to get the PublicIp somewhere in your code and the Instance doesn't have PublicIp assigned to it.
I experienced this error just a couple of weeks ago, while getting my feet wet with CloudFormation. In my case, I'd dropped the public IP for the interface, in favor of only a private IP, but I still had an output configured in the CloudFormation template that referenced the now non-existent publicid attribute. Removing that output from the template fixed my issue.

Cloudformation - Confused with the interaction between 4 cfn helper scripts ..this is what I did and it works

I created a Windows 2012 AMI and created an instance of that AMI using the CloudFormation template shown below.
In that JSON script I want to call a PowerShell script to disable a service (simple one). The EC2 Windows 2012 instance gets created. I made sure EC2Config service was running before I took AMI. It works now. Following is the code that works fine. But the question is, I don't clearly understand the interaction between cfn-hup, cfn-signal and cfn-init. Honestly I read about all the 4 helper scripts. But I am not wrap my brain around these helper scripts.
Are there any blogs or documentation about how these 4 helper scripts work together?
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"files" : {
"c:\\cfn\\cfn-hup.conf" : {
"content" : { "Fn::Join" : ["", [
"[main]\n",
"stack=", { "Ref" : "AWS::StackId" }, "\n",
"region=", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\cfn\\hooks.d\\cfn-auto-reloader.conf" : {
"content": { "Fn::Join" : ["", [
"[cfn-auto-reloader-hook]\n",
"triggers=post.update\n",
"path=Resources.MyInstance.Metadata.AWS::CloudFormation::Init\n",
"action=cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n"
]]}
},
"c:\\scripts\\test.ps1" : {
"content": { "Fn::Join" : ["", [
"Write-Host Hello World!\n"
]]}
}
},
"commands" : {
"1-run-script" : {
"command" : { "Fn::Join" : [ "", [
"Powershell.exe Set-ExecutionPolicy Unrestricted -force;Unblock-File C:\\PowershellScripts\\WindowsServiceManager.ps1;. C:\\PowershellScripts\\WindowsServiceManager.ps1;SetWindowsServiceStartupType Dnscache Manual;StopWindowsService Dnscache"
]]}}
},
"services": {
"windows": {
"cfn-hup": {
"enabled": "true",
"ensureRunning": "true",
"files": ["c:\\cfn\\cfn-hup.conf", "c:\\cfn\\hooks.d\\cfn-auto-reloader.conf"]
}
}
}
}
}
},
"Properties": {
"DisableApiTermination": "FALSE",
"ImageId": "ami-3723c04f",
"InstanceType": "t2.micro",
"KeyName": "EC2Instances",
"Monitoring": "false",
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"<script>\n",
"cfn-init.exe -v -s ", { "Ref" : "AWS::StackName" },
" -r MyInstance",
" --region ", { "Ref" : "AWS::Region" }, "\n",
"cfn-signal.exe -e 0 ", { "Fn::Base64" : { "Ref" : "WindowsServerWaitHandle" }}, "\n",
"</script>\n"
]]}}
}
},
"WindowsServerWaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
},
"WindowsServerWaitCondition": {
"Type": "AWS::CloudFormation::WaitCondition",
"DependsOn": "MyInstance",
"Properties": {
"Handle": { "Ref": "WindowsServerWaitHandle" },
"Timeout": "1800"
}
}
}
}
Found a decent explanation here:
https://aws.amazon.com/blogs/devops/best-practices-for-deploying-applications-on-aws-cloudformation-stacks/
Sequence of how AWS::CloudFormation::Init works:
You specify application configuration using the AWS::CloudFormation::Init section for an EC2 instance in your CloudFormation template.
You kick-off a CloudFormation stack creation using the template.
The AWS CloudFormation service starts creating a stack, including the EC2 instance.
After the EC2 instance is up and running, a CloudFormation helper script, cfn-init, is executed on the instance to configure the instance in accordance with your AWS::CloudFormation::Init template specification.*
Another CloudFormation helper script, cfn-signal, is executed on the instance to let the remote AWS CloudFormation service know the result (success/failure) of the configuration.* You can optionally have the CloudFormation service hold off on marking the EC2 instance state and the stack state “CREATE_COMPLETE” until the CloudFormation service hears a success signal for the instance. The holding-off period is specified in the template using a CreationPolicy.
*You can download the CloudFormation helper scripts for both Linux and Windows. These come preinstalled on the Linux and Windows AMIs provided by Amazon. You need to specify the commands to trigger cfn-init and cfn-signal in the EC2 user data script. Once an instance is up and running, the EC2 user data script is executed automatically for most Linux distributions and Windows.
Once your application stack is up and running, chances are that you will update the application, apply an OS patch, or perform some other configuration update in a stack’s lifecycle. You just update the AWS::CloudFormation::Init section in your template (for example, specify a newer version of an application package), and call UpdateStack. When you do, CloudFormation updates the instance metadata in accordance with the updated template. Then the cfn-hup daemon running on the instance detects the updated metadata and reruns cfn-init to update the instance in accordance with the updated configuration. cfn-hup is one of the CloudFormation helper scripts available on both Linux and Windows.

DeletionPolicy:Snapshot cannot be specified for a cluster instance, use deletion policy on the cluster instead

I am trying to create RDS cluster and aurora instance using the cloudoformation template below:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "example setup",
"Parameters" : {
"DBInstanceIdentifier" : {
"Type": "String",
"Description": "Name for the DB instance."
},
"DBUser" : {
"Type": "String",
"Description": "Master user"
},
"DBPassword" : {
"Type": "String",
"Description": "Pass"
},
"DBModel" : {
"Type": "String",
"Description": "Instance model to be used for the DB."
}
},
"Resources": {
"RDSCluster": {
"Type": "AWS::RDS::DBCluster",
"Properties": {
"MasterUsername": { "Ref" : "DBUser" },
"MasterUserPassword": { "Ref" : "DBPassword" },
"Engine": "aurora",
"DBClusterParameterGroupName": "default.aurora5.6",
"VpcSecurityGroupIds": [{"Fn::GetAtt" : [ "DBFromSiteSecurityGroup" , "GroupId" ]}]
}
},
"AuroraInstance": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBInstanceIdentifier": { "Ref" : "DBInstanceIdentifier" },
"DBParameterGroupName": "default.aurora5.6",
"Engine": "aurora",
"DBClusterIdentifier": {
"Ref": "RDSCluster"
},
"PubliclyAccessible": "true",
"DBInstanceClass": { "Ref" : "DBModel" }
}
},
"DBFromSiteSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable MySQL",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "3306", "ToPort" : "3306", "CidrIp" : "195.171.102.98/32"}
]
}
},
"DBFromSiteSecurityGroupIngress1" : {
"Type" : "AWS::EC2::SecurityGroupIngress",
"Properties" : {
"GroupName" : { "Ref" : "DBFromSiteSecurityGroup" },
"IpProtocol" : "tcp",
"ToPort" : "3306",
"FromPort" : "3306",
"SourceSecurityGroupName" : { "Ref" : "DBFromSiteSecurityGroup" }
}
}
}
}
The db_model parameter I am passing is "db.t2.medium". The cluster gets created successfully in the cloudformation console however the AWS::RDS::DBInstance creation fails with the following error
"DeletionPolicy:Snapshot cannot be specified for a cluster instance, use deletion policy on the cluster instead."
What's more weird that when I try to run the same CF template in say eu london region, it works fine!!! Is there something wrong with the EU ireland region and aurora?
From AWS Support
This is a known issue and has been reported by other customers as well. The service team is currently working on the fix for this but there is no ETA as to when that would be pushed.
The work-around in the meanwhile is to specify a DeletionPolicy inside the DB instance resource definition that is failing to create, with the value of 'Delete'. [1]
An example below:
"Resources": {
"Database1": {
"DeletionPolicy": "Delete",
"Properties": {...},
"Type": "AWS::RDS::DBInstance"
}
}
References:
[1] DeletionPolicy - http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html#w2ab2c19c23c11c17
An update from AWS Support:
When creating an Amazon Aurora DBInstance in a DB Cluster using AWS
CloudFormation, CloudFormation applies a default Deletion policy of
“Delete”, if a deletion policy is not specified. If a Deletion Policy
of “Snapshot” is specified for an Amazon Aurora DBInstance,
CloudFormation returns an error, because instances in a DB Cluster
cannot be snapshotted individually; Snapshotting must be done at the
DB Cluster level.
As part of a recent deployment, we inadvertently changed the default
deletion policy for an Amazon Aurora DBInstance to “Snapshot”. This
caused our template validation to fail. To remedy this, CloudFormation
is reverting the value of default DeletionPolicy for Amazon Aurora
DBInstances to “Delete”. This fix will be completed by 21st July,
2017. Until this fix is completely rolled out, customers can explicitly override our incorrect defaults by specifying a deletion
policy of “Delete” for Amazon Aurora DBInstances.
We have corrected the gap in our testing that led to this situation,
and will continue to improve our testing to prevent recurrences. We
recognize how critical it is for us to preserve existing behavior for
our customers, and apologize for this inconvenience.

Cloning infrastructure from one region to another: AWS CloudFormation

I have existing infrastructure in us-east-1 region which needed to be cloned exactly to us-east-2 region. Used AWS CloudFormer to generate the JSON template from existing us-east-1 region, replaced all the us-east-1 with us-east-2 and started creating the stack but getting errors saying "Resource creation cancelled", specifically for all the EC2 instances
A snapshot of the template (only EC2 instance):
"instancei071dd59b": {
"Type": "AWS::EC2::Instance",
"Properties": {
"DisableApiTermination": "false",
"InstanceInitiatedShutdownBehavior": "stop",
"ImageId": "ami-1a41b377",
"InstanceType": "t2.medium",
"KeyName": "MyServer",
"Monitoring": "false",
"Tags": [
{
"Key": "MyServer OS",
"Value": "Windows Server"
},
{
"Key": "Name",
"Value": "MyServer_WEB_TEST_2"
}
],
"Volumes": [
{
"Device": "xvdb",
"VolumeId": {
"Ref": "volumevol9124b841"
}
}
],
"NetworkInterfaces": [
{
"DeleteOnTermination": "true",
"DeviceIndex": 0,
"SubnetId": {
"Ref": "subnet24031c0f"
},
"PrivateIpAddresses": [
{
"PrivateIpAddress": "172.31.53.184",
"Primary": "true"
}
],
"GroupSet": [
{
"Ref": "sgMyServerWEB"
}
],
"AssociatePublicIpAddress": "true"
}
]
}
},
"volumevol9124b841": {
"Type": "AWS::EC2::Volume",
"Properties": {
"AvailabilityZone": "us-east-2b",
"Size": "30",
"SnapshotId": "snap-95288b92",
"VolumeType": "gp2"
}
}
Before going with cloudformation template you will need to make sure you have following things in place :
Move your instance AMI to us-east-2 region then replace the snapshot id and AMI id in your template
Create a security group replace the security group id in your template
Replace subnet ID in your CF template with the one in us-east-2 region
The reason you will have to do this is every resource on AWS has unique IDs which cannot be replicated, if you want to replicate same you will need different Ids for that you need to create seperate resources and use them in your template.
If your doing this for a single instance only then you might do it manually by exporting AMI to us-east-2 region.
For collecting AMI ID in different region, I'd recommend to use the image name instead the AMI ID as the key.
To build resources to be placed in different regions, definitely is better to use CloudFormation. In this case you can use the lambda cli2cloudformation (https://github.com/lucioveloso/cli2cloudformation).
Using it, you can get the AMI ID across all regions and whatever other information that you are able to get using CLI.
To collect the AMI ID, create a lambda with cli2cloudformation and inside your template, create a custom resource as bellow:
"imageIdNameBased": {
"Type": "Custom::cli2cfnLambda",
"Properties": {
"ServiceToken": "arn:aws:lambda:eu-west-1:123456789012:function:cli2cfnLambda",
"CliCommandCreate": "ec2 describe-images --filters 'Name=name,Values=amzn-ami-hvm-2017.03.0.20170417-x86_64-gp2' --query 'Images[0]'"
}
}
In this case, I'm getting the AMI ID to the Image named 'amzn-ami-hvm-2017.03.0.20170417-x86_64-gp2'. You can change to your image name.
After that, you can retrieve it in any point of your CloudFormation stack.
"Fn::GetAtt" : ["imageIdNameBased", "ImageId"]

Creating an ec2 instance along with IAM roles using cloud formation

I'm very new to Amazon cloudformation technique. I'm trying to launch an ec2 instance along with the IAM roles.
I have cloudformation script for this. But the problem I face is the IAM roles and Ec2 instances are created, but they aren't tied with each other.
I did create the IAM-roles using AWS::IAM::Role and AWS::IAM::InstanceProfile.
Is there any other command that I should use?
Thanks in advance.
Had to dig to get the final result, but here's an example of
Defining an access role (this will allow the EC2 instance to step into / assume to role),
Defining a policy for the role (i.e. when the EC2 assumes the role, what resources does it have access to),
Defining the instance profile (that is referenced by the EC2 instance, and has the access role mapped in)
"S3AccessRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"AssumeRolePolicyDocument" : {
"Statement" : [ {
"Effect" : "Allow",
"Principal" : {
"Service" : [ "ec2.amazonaws.com" ]
},
"Action" : [ "sts:AssumeRole" ]
} ]
},
"Path" : "/"
}
},
"S3RolePolicies" : {
"Type" : "AWS::IAM::Policy",
"Properties" : {
"PolicyName" : "s3access",
"PolicyDocument" : {
"Statement" : [ {
"Effect" : "Allow",
"Action" : "s3:*",
"Resource" : "*"
}]
},
"Roles" : [ { "Ref" : "S3AccessRole" } ]
}
},
"S3InstanceProfile" : {
"Type" : "AWS::IAM::InstanceProfile",
"Properties" : {
"Path" : "/",
"Roles" : [ { "Ref" : "S3AccessRole" } ]
}
}
The policy above allows all access to s3 resources. Adjust according to your needs. The IamInstanceProfile reference in the EC2 instance properties would refer be { "Ref" : "S3InstanceProfile" }
Note that as of May 2015, when you creating a stack that creates IAM roles, you need to check a box acknowledging such creation, otherwise you'll get a "Stack creation error: Requires capabilities : [CAPABILITY_IAM]" error.
The easiest way to solve such problems is to use CloudFormer. CloudFormer is a tool that creates a starting point template from the AWS resources you already have running in your environment.
The CloudFormer tool is packaged as a standalone application that you
can launch inside your AWS environment. The application is started on
a t1.micro Amazon EC2 instance via AWS CloudFormation.
Once you have launched Cloud Former, you will get a web interface (check the URL in the Output section of the launched stack), that will be able to describe all your resources in a specific region. It will lead you through which resources you wish in each category (DNS, Network, Compute...). At the end you can see the template and copy it, or save it in S3.
But if you wish to do it manually, you need to add the AWS::IAM::InstanceProfile you created to the Properties of AWS::EC2::Instance as IamInstanceProfile
{
"Type" : "AWS::EC2::Instance",
"Properties" : {
"AvailabilityZone" : String,
"BlockDeviceMappings" : [ EC2 Block Device Mapping, ... ],
"DisableApiTermination" : Boolean,
"EbsOptimized" : Boolean,
"IamInstanceProfile" : String,
"ImageId" : String,
"InstanceType" : String,
...
"UserData" : String,
"Volumes" : [ EC2 MountPoint, ... ]
}
}
See more details on AWS::EC2::Instance here
Suppose the AWS::IAM::InstanceProfile resource you create is called MyNewRole. To create an instance with that role (in the same CloudFormation template) set the EC2 resource's IamInstanceProfile property to a Ref to that resource. Here's an example (with lots of other details left out):
"Resources": {
"MyNewRole": {
"Type": "AWS::IAM::InstanceProfile",
... more stuff here
},
"MyNewEc2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"IamInstanceProfile": { "Ref": "MyNewRole" },
... more stuff here
}
}
}