AWS cloudformation fail instance not running - amazon-web-services

I am trying to create an Amazon EC2 instance then create an Amazon EBS volume and attach it to the instance. I am using a CloudFormation template for this. Unfortunately the stack creation is failing when attaching newly created volume to the instance with the following error:
Instance 'i-01eebc8c9c492c035' is not 'running'. (Service: AmazonEC2; Status Code: 400; Error Code: IncorrectState; Request ID: 635572fd-dd25-4a02-9306-6e22f88e13dc)
What I do not understand is, when the instance creation is complete, that means the instance is up and running. How can this error be possible?
I am using the following CloudFormation template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "single instance template",
"Parameters": {
"InstanceType": {
"Type": "String",
"Default": "t2.micro"
},
"InstanceName": {
"Type": "String",
"Default": "test_CFT"
},
"RootVolumeSize": {
"Type": "String",
"Default": "50"
},
"Volume1Size": {
"Type": "String",
"Default": "8"
},
"Region": {
"Type": "String",
"Default": "us-east-2"
},
"AMIID": {
"Type": "String",
"Default": "ami-8c122be9"
},
"SubnetIds": {
"Type": "CommaDelimitedList",
"Default": "subnet-595e7422"
},
"SecurityGroupIDs": {
"Type": "CommaDelimitedList",
"Default": "sg-082faee8335351537"
}
},
"Resources": {
"Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {
"Ref": "AMIID"
},
"InstanceType": {
"Ref": "InstanceType"
},
"KeyName": "thehope",
"NetworkInterfaces": [
{
"AssociatePublicIpAddress": "false",
"DeviceIndex": "0",
"SubnetId": {
"Fn::Select": [
0,
{
"Ref": "SubnetIds"
}
]
},
"GroupSet": {
"Ref": "SecurityGroupIDs"
}
}
],
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"VolumeSize": {
"Ref": "RootVolumeSize"
},
"DeleteOnTermination": "true",
"VolumeType": "gp2"
}
}
],
"Tags": [
{
"Key": "Name",
"Value": {
"Ref": "InstanceName"
}
}
]
}
},
"Volume1": {
"DeletionPolicy": "Delete",
"Properties": {
"AvailabilityZone": {
"Fn::GetAtt": [
"Instance",
"AvailabilityZone"
]
},
"Encrypted": "False",
"Size": {
"Ref": "Volume1Size"
},
"Tags": [
{
"Key": "Name",
"Value": "New_volume"
}
],
"VolumeType": "gp2"
},
"Type": "AWS::EC2::Volume"
},
"VolumeAttachment1": {
"Properties": {
"Device": "/dev/xvdb",
"InstanceId": {
"Ref": "Instance"
},
"VolumeId": {
"Ref": "Volume1"
}
},
"Type": "AWS::EC2::VolumeAttachment"
}
},
"Outputs": {
"InstanceId": {
"Description": "InstanceId of the instance",
"Value": {
"Ref": "Instance"
}
},
"AZ": {
"Description": "Availability Zone of the instance",
"Value": {
"Fn::GetAtt": [
"Instance",
"AvailabilityZone"
]
}
},
"PrivateIP": {
"Description": "PrivateIP of the instance",
"Value": {
"Fn::GetAtt": [
"Instance",
"PrivateIp"
]
}
}
}
}
What am I doing wrong?

Since you are creating new volumes, it would be easier to simply specify the volumes as part of the instance rather than specifying an Amazon EBS volume and then attaching it to the instance.
From Amazon EC2 Block Device Mapping Property - AWS CloudFormation:
This example sets the EBS-backed root device (/dev/sda1) size to 50 GiB, and another EBS-backed device mapped to /dev/sdm that is 100 GiB in size.
"BlockDeviceMappings" : [
{
"DeviceName" : "/dev/sda1",
"Ebs" : { "VolumeSize" : "50" }
},
{
"DeviceName" : "/dev/sdm",
"Ebs" : { "VolumeSize" : "100" }
}
]

That was quite fascinating, seeing how the instance stops!
When using Amazon Linux 2, it can be fixed by changing:
"DeviceName": "/dev/sda1",
into:
"DeviceName": "/dev/xvda",
Or, it can be fixed by using Amazon Linux (version 1) with /dev/sda1.
However, this doesn't fix your VolumeAttachment issue.

I was facing the same issue until I changed the AMI in my template. Initially, I was testing with Linux AMI in the N.Virginia region where it failed but when I used a CENTOS AMI that I had subscribed to it works.

Related

Why does my subnet and VPC show side by side on CloudFormation

I've been trying to figure out why my VPC and subnet show side by side instead of the subnet inside of the VPC? (I used Atom to generate this.)
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "vpc",
"Metadata": {
},
"Parameters": {"siggyVpcCidr": {
"Description": "vpc cidr",
"Type": "String",
"Default": "10.0.0.0/16"
},
"siggySubnetCidr": {
"Description": "cidr for the subnet",
"Type": "String",
"Default": "10.0.1.0/2"
},
"Subnet1Az": {
"Description": "AZ for siggySubnetCidr",
"Type": "AWS::EC2::AvailabilityZone::Name"
}
},
"Mappings": {
},
"Conditions": {
},
"Resources": {
"siggyVpc": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": { "Ref": "siggyVpcCidr" },
"Tags": [{ "Key": "Name", "Value": "siggyVpc" }]
}
},
"siggyIgw": {
"Type": "AWS::EC2::InternetGateway",
"Properties": {
"Tags": [{ "Key": "Name", "Value": "siggyIgw1" }]
}
},
"AttachGateway": {
"Type": "AWS::EC2::VPCGatewayAttachment",
"Properties": {
"VpcId": { "Ref": "siggyVpc" },
"InternetGatewayId": { "Ref": "siggyIgw" }
}
},
"SubnetSiggy": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"AvailabilityZone": { "Ref": "Subnet1Az" },
"VpcId": { "Ref": "siggyVpc" },
"CidrBlock": { "Ref": "siggySubnetCidr" },
"Tags": [{ "Key": "Name", "Value": "siggySubnetCidr" }]
}
}
},
"Outputs": {
}
}
They are separate resources. CloudFormation templates arrange resources in a flat array. This is pretty much true of most resources. Some resources can be implicitly defined when creating resources, but that probably won't be reflected with an export where you create a template from existing resources.
You would need to inspect the VpcId property to determine the VPC to which the subnet belongs.

AWS CDK: A load balancer cannot be attached to multiple subnets in the same Availability Zone

I'm using AWS CDK and it is failing at App-Load-Balancer level and surprisingly it works for Web-Load-balancer. Looking at generated CloudFormation, it is clear that all "Private Subnets" are getting created in separated AZ and associated Auto-Scaling Group too is creating the instance across multiple-AZs. However, when the CDK is executed, it is failing with the error message - A load balancer cannot be attached to multiple subnets in the same Availability Zone (Service: AmazonElasticLoadBalancing; Status Co
de: 400; Error Code: InvalidConfigurationRequest; Request ID: 62c554cb-34ab-43ef-bac0-be2f0d6fc742; Proxy: null)
APP Server characteristics
AUTOSCALING CF Snippet:
"InstaLendaappASGapp1appsvrASG950CF7C4": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"MaxSize": "3",
"MinSize": "1",
"DesiredCapacity": "2",
"LaunchConfigurationName": {
"Ref": "InstaLendaappASGapp1appsvrLaunchConfig18DAF6BB"
},
"Tags": [
{
"Key": "Name",
"PropagateAtLaunch": true,
"Value": "webapp-dc-3-tier-ha/InstaLend-a-appASG-app-1-appsvr-"
}
],
"TargetGroupARNs": [
{
"Ref": "InstaLendaapplbInstaLendalstnrPrivate80InstaLendatgtprivateGroup8D2C8D01"
}
],
"VPCZoneIdentifier": [
{
"Ref": "InstaLendavpcInstaLendaprivateSNSubnet1Subnet35AF6769"
},
{
"Ref": "InstaLendavpcInstaLendaprivateSNSubnet2SubnetD8513C5D"
},
{
"Ref": "InstaLendavpcInstaLendaprivateSNSubnet3SubnetB7B2D12C"
}
]
}
LOADBALANCER CF Snippet:
"InstaLendaapplbCC4F6682": {
"Type": "AWS::ElasticLoadBalancingV2::LoadBalancer",
"Properties": {
"LoadBalancerAttributes": [
{
"Key": "deletion_protection.enabled",
"Value": "false"
}
],
"Name": "InstaLend-a-app-lb",
"Scheme": "internal",
"SecurityGroups": [
{
"Fn::GetAtt": [
"InstaLendasginternal8649CE7C",
"GroupId"
]
}
],
"Subnets": [
{
"Ref": "InstaLendavpcInstaLendaprivateSNSubnet1Subnet35AF6769"
},
{
"Ref": "InstaLendavpcInstaLendaprivateSNSubnet2SubnetD8513C5D"
},
{
"Ref": "InstaLendavpcInstaLendaprivateSNSubnet3SubnetB7B2D12C"
},
{
"Ref": "InstaLendavpcInstaLendaprivateSNDBSubnet1Subnet2DD722D8"
},
{
"Ref": "InstaLendavpcInstaLendaprivateSNDBSubnet2Subnet59278CD3"
},
{
"Ref": "InstaLendavpcInstaLendaprivateSNDBSubnet3SubnetCC805230"
}
],
"Type": "application"
},
"UpdateReplacePolicy": "Delete",
"DeletionPolicy": "Delete",
"Metadata": {
"aws:cdk:path": "webapp-dc-3-tier-ha/InstaLend-a-app-lb/Resource"
}
}
While LB is selecting 6 subnets, ASG associated is selecting only 3 subnets. The details of 3 ASG Subnets (i.e. Private Subnets) had been pasted below:
1st SUBNET
"InstaLendavpcInstaLendaprivateSNSubnet1Subnet35AF6769": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.2.3.0/24",
"VpcId": {
"Ref": "InstaLendavpcE5C8A638"
},
"AvailabilityZone": "us-east-2a",
"MapPublicIpOnLaunch": false,
"Tags": [
{
"Key": "aws-cdk:subnet-name",
"Value": "InstaLend-a-privateSN"
....
2nd SUBNET
"InstaLendavpcInstaLendaprivateSNSubnet2SubnetD8513C5D": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.2.4.0/24",
"VpcId": {
"Ref": "InstaLendavpcE5C8A638"
},
"AvailabilityZone": "us-east-2b",
"MapPublicIpOnLaunch": false,
"Tags": [
{
"Key": "aws-cdk:subnet-name",
"Value": "InstaLend-a-privateSN"
},
{
"Key": "aws-cdk:subnet-type",
"Value": "Private"
},
....
3rd SUBNET
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.2.5.0/24",
"VpcId": {
"Ref": "InstaLendavpcE5C8A638"
},
"AvailabilityZone": "us-east-2c",
"MapPublicIpOnLaunch": false,
"Tags": [
{
"Key": "aws-cdk:subnet-name",
"Value": "InstaLend-a-privateSN"
},
{
"Key": "aws-cdk:subnet-type",
"Value": "Private"
},

AWS Cloudfromation and autoscaling : The requested configuration is currently not supported. Launching EC2 instance failed

I want to replicate the infrastructure from one region(us-east-1) to another(us-east-2). so,I have generated a cloudfromation template of an existing infrastructure with the help of cloudformer tool.
"asgamazonecsclisetupapijoulebugprodEcsInstanceAsg1EIBNOXSXJ7HD": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": [
"us-east-2b",
"us-east-2c"
],
"Cooldown": "300",
"DesiredCapacity": "3",
"HealthCheckGracePeriod": "300",
"HealthCheckType": "ELB",
"MaxSize": "16",
"MinSize": "3",
"VPCZoneIdentifier": [
{
"Ref": "subnet81c8ebab"
},
{
"Ref": "subnet5df40214"
}
],
"LaunchConfigurationName": {
"Ref": "lcamazonecsclisetupapijoulebugprodAMI2017d"
},
"LoadBalancerNames": [
{
"Ref": "elbJBAPILiveCleanbit2016"
}
],
"Tags": [
{
"Key": "Name",
"Value": "Live - Cleanbit2016 - joulebug-api",
"PropagateAtLaunch": true
}
],
"TerminationPolicies": [
"Default"
]
}
},
"lcamazonecsclisetupapijoulebugprodAMI2017d": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"AssociatePublicIpAddress": true,
"ImageId": "ami-0e6d83442546978bc",
"InstanceType": "c3.large",
"KeyName": "cleanbit2016_vpc",
"IamInstanceProfile": "amazon-ecs-cli-setup-api-joulebug-prod-EcsInstanceProfile-1M4GOHBP3FP5L",
"InstanceMonitoring": "true",
"SecurityGroups": [
{
"Ref": "sgCleanbit2016WebServerSG"
}
],
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"SnapshotId": "snap-0b2477be9c863d014",
"VolumeSize": 8
}
},
{
"DeviceName": "/dev/xvdcz",
"Ebs": {
"VolumeSize": 22
}
}
]
}
},
"elbJBAPILiveCleanbit2016": {
"Type": "AWS::ElasticLoadBalancing::LoadBalancer",
"Properties": {
"Policies": [
{
"PolicyName": "AWSConsole-SSLNegotiationPolicy-JB-API-Live-Cleanbit2016-1467998170471",
"PolicyType": "SSLNegotiationPolicyType",
}
],
}
}
"subnet81c8ebab": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.0.3.0/24",
"AvailabilityZone": "us-east-2b",
"VpcId": {
"Ref": "vpcdcbd08bb"
},
"Tags": [
{
"Key": "Name",
"Value": "Cleanbit2016 - Public 1b"
}
]
}
},
"sgCleanbit2016WebServerSG": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Web server security group for public subnet in vpc.",
"VpcId": {
"Ref": "vpcdcbd08bb"
},
"Tags": [
{
"Key": "Name",
"Value": "Cleanbit2016_ WebServerSG"
}
]
}
},
While launching the template in other region(us-east-2), it is throwing following error:
The requested configuration is currently not supported. Please check the documentation for supported configurations. Launching EC2 instance failed.
You don't have details of the regions you're using, but if you are trying to do this outside of us-east-1 the Availablility Zones won't work. It also looks like you have a number of other parts of the stack hard-coded, which may not work in another region.
And if you are trying to do this in us-east-1, there is the possibility that one of the AZs is unavailable to you - see this question for more details.
You have not provided enough information to be able to diagnose the situation.
I took your template, removed the portions that were incomplete (eg removed the Load Balancer because it was missing Listeners), simplified a few things and it works fine:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"VPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": "10.0.0.0/16",
"Tags": [
{
"Key": "Name",
"Value": "Lab VPC"
}
]
}
},
"ASG": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": [
"us-east-2b",
"us-east-2c"
],
"Cooldown": "300",
"DesiredCapacity": "1",
"HealthCheckGracePeriod": "300",
"MaxSize": "16",
"MinSize": "1",
"VPCZoneIdentifier": [
{
"Ref": "Subnet1"
},
{
"Ref": "Subnet2"
}
],
"LaunchConfigurationName": {
"Ref": "LaunchConfig"
}
}
},
"LaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"AssociatePublicIpAddress": true,
"ImageId": "ami-0b59bfac6be064b78",
"InstanceType": "t2.micro",
"InstanceMonitoring": "true",
"SecurityGroups": [
{
"Ref": "WebServerSG"
}
]
}
},
"Subnet1": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.0.0.0/24",
"AvailabilityZone": "us-east-2b",
"VpcId": {
"Ref": "VPC"
},
"Tags": [
{
"Key": "Name",
"Value": "Public 1"
}
]
}
},
"Subnet2": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"CidrBlock": "10.0.1.0/24",
"AvailabilityZone": "us-east-2c",
"VpcId": {
"Ref": "VPC"
},
"Tags": [
{
"Key": "Name",
"Value": "Public 2"
}
]
}
},
"WebServerSG": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Web server security group for public subnet in vpc.",
"VpcId": {
"Ref": "VPC"
},
"Tags": [
{
"Key": "Name",
"Value": "WebServerSG"
}
]
}
}
}
}
Therefore, your problem lies in part of the template you did not provide.
You could start with this version, then progressively add back parts of your template until you discover what is causing the error.
If you wish to create a CloudFormation template that can run in multiple regions, you should not refer to specific Availability Zones (eg us-east-2b).
You can use Fn::GetAZs - AWS CloudFormation to obtain a list of AZs in region.
After a lot of debugging, when I started launching the things manually, I found the same error and I got to know that c3.large is causing the error. When I launch the template with c4.large it successfully launched the template from us-east-1 to us-east-2.

AWS Cloudformation Random Build Fail

I'm using Cloudformation to automate instance creation. I'm using an auto scaling group to build a variable number dependent on what i need. However there is a reoccurring issue where at least 1 instance fails to run when i build 2+ instances from this script.
For Example: I state 7 instances for cloudformation to build, 6 will work perfectly but there is always 1 that will not work. There is no error, it just seems to ignore commands. In the userdata section of this script i launch dockers to run in the instance.
I get this error: rpc error code = 13 desc = transport is closing
Can someone take a look to see if doing something wrong or I'm missing a step?
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Test",
"Parameters": {
"InstanceType": {
"Type": "String",
"Default": "t2.large"
},
"NoOfInstances": {
"Type": "String",
"ConstraintDescription": ""
},
"RoleName": {
"Type": "String",
"Default": "**",
"ConstraintDescription": ""
},
"VPCParameter": {
"Type": "AWS::EC2::VPC::Id",
"Default": "**"
},
"SubnetsParameter": {
"Type": "List<AWS::EC2::Subnet::Id>",
"Default": "**"
},
"KeyName": {
"Type": "AWS::EC2::KeyPair::KeyName",
"Default": "**",
"ConstraintDescription": ""
}
},
"Mappings": {
"AWSInstanceType2Arch": {
"t2.large": {
"Arch": "HVM64"
}
},
"AWSRegionArch2AMI": {
"**": {
"HVM64": "**"
}
}
},
"Resources": {
"LaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"KeyName": {
"Ref": "KeyName"
},
"IamInstanceProfile": {
"Ref": "RoleName"
},
"SecurityGroups": [{
"Ref": "WebServerSecurityGroup"
}],
"ImageId": {
"Fn::FindInMap": ["AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
}, {
"Fn::FindInMap": ["AWSInstanceType2Arch", {
"Ref": "InstanceType"
}, "Arch"]
}
]
},
"InstanceType": {
"Ref": "InstanceType"
},
"UserData": {
"Fn::Base64": {
"Fn::Join": ["", [ ** ]]
}
}
}
},
"AutoScalingServerGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"LaunchConfigurationName": {
"Ref": "LaunchConfig"
},
"MinSize": "1",
"MaxSize": "30",
"Cooldown": "300",
"VPCZoneIdentifier": {
"Ref": "SubnetsParameter"
},
"DesiredCapacity": {
"Ref": "NoOfInstances"
},
"Tags": [ ** ]
}
},
"WebServerSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "**",
"VpcId": {
"Ref": "VPCParameter"
},
"SecurityGroupIngress": [{
"IpProtocol": "tcp",
"CidrIp": "**",
"FromPort": "**",
"ToPort": "**"
}, ]
}
},
"WebServerScaleUpPolicy": {
"Type": "AWS::AutoScaling::ScalingPolicy",
"Properties": {
"AdjustmentType": "ChangeInCapacity",
"AutoScalingGroupName": {
"Ref": "AutoScalingServerGroup"
},
"Cooldown": "60",
"ScalingAdjustment": "1"
}
},
"WebServerScaleDownPolicy": {
"Type": "AWS::AutoScaling::ScalingPolicy",
"Properties": {
"AdjustmentType": "ChangeInCapacity",
"AutoScalingGroupName": {
"Ref": "AutoScalingServerGroup"
},
"Cooldown": "60",
"ScalingAdjustment": "-1"
}
}
}
}
You could be hitting EC2 instance limits - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html
You could also look at the Activity History tab for the Auto Scaling Group. See if it has any useful information.

AWS::AutoScaling::LaunchConfiguration You are not authorized to perform this operation

While cloudformation is building the stack, I get the following error:
AWS::AutoScaling::LaunchConfiguration N1ClusterServerLaunchConfig You are not authorized to perform this operation
I have admin full access user privileges. But, still this fails. Is the authorization due to the IAMs defined inside the template?
"N1ClusterServerAutoScale": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": {
"Fn::GetAZs": ""
},
"LaunchConfigurationName": {
"Ref": "N1ClusterServerLaunchConfig"
},
"MinSize": "2",
"MaxSize": "64",
"DesiredCapacity": {
"Ref": "ClusterSize"
},
"Tags": [
{
"Key": "Name",
"Value": {
"Ref": "AWS::StackName"
},
"PropagateAtLaunch": true
}
]
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "a715af00-ebba-4fab-a817-d5ee1986dfe7"
}
}
},
"N1ClusterServerLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"RegionMap",
{
"Ref": "AWS::Region"
},
"hvm"
]
},
"InstanceType": {
"Ref": "InstanceType"
},
"KeyName": {
"Ref": "KeyPair"
},
"SecurityGroups": [
{
"Ref": "N1ClusterSecurityGroup"
},
{
"Ref": "N1NodeSecurityGroup"
}
],
"IamInstanceProfile": {
"Ref": "IamInstanceProfile"
},
IAM
"AllowComputeFrom": {
"Description": "The net block (CIDR) that N1-COMPUTE is available to.",
"Default": "0.0.0.0/0",
"Type": "String"
},
"IamInstanceProfile": {
"Description": "The name of an IAM Profile which can access required S3 buckets and instances.",
"Default": "arn:aws:iam::247256189695:instance-profile/n1-compute-instance",
"Type": "String"
},
"IamInstanceProfileShort": {
"Description": "The last part of the name of an IAM Profile which can create instances.",
"Default": "n1-compute-instance",
"Type": "String"
},
ec2:RunInstances is required to use a Launch template