I am going through the AWS well-architected labs guides and am on the automated deployment of the VPC (https://www.wellarchitectedlabs.com/Security/200_Automated_Deployment_of_VPC/Lab_Guide.html).
The instructions are pretty straight forward and I upload the template provided (tried both the one linked directly in the instructions and the one from their git repo I found here https://github.com/awslabs/aws-well-architected-labs/blob/master/Security/200_Automated_Deployment_of_VPC/Code/vpc-alb-app-db.yaml). But when it runs, I keep getting failure messages. These errors saying unable to spin up things in us-west-1c:
And this for the VPC:
I am logged in with an IAM user that had admin access and tried cleaning up the YAML as there was some duplicate fields I found, but nothing worked. Is it just that my account can't create anything in us-west-1c because it is overloaded or something, or is it some other issue?
The answer is in the picture: CloudFormation could not create ALB1Subnet3 because the availability zone us-west-1c doesn't exist. When anything in a new stack fails, the entire stack is rolled back.
As described here, you can get the list of availability zones for a region with the CLI:
aws ec2 describe-availability-zones --region us-west-1
You'll see that this region only has two availability zones:
{
"AvailabilityZones": [
{
"State": "available",
"OptInStatus": "opt-in-not-required",
"Messages": [],
"RegionName": "us-west-1",
"ZoneName": "us-west-1b",
"ZoneId": "usw1-az3",
"GroupName": "us-west-1",
"NetworkBorderGroup": "us-west-1"
},
{
"State": "available",
"OptInStatus": "opt-in-not-required",
"Messages": [],
"RegionName": "us-west-1",
"ZoneName": "us-west-1c",
"ZoneId": "usw1-az1",
"GroupName": "us-west-1",
"NetworkBorderGroup": "us-west-1"
}
]
}
Update
It's been pointed out that us-west-1c appears in my availability zone listing. This happens because my availability zones are not the same as the OP's availability zones. To quote the docs:
To ensure that resources are distributed across the Availability Zones for a Region, we independently map Availability Zones to names for each AWS account. For example, the Availability Zone us-east-1a for your AWS account might not be the same location as us-east-1a for another AWS account.
So for the OP, the error message was correct: there is no us-west-1c. And my explanation -- that us-west-1 only has two zones -- was also correct.
However, to take this to the bitter end (and head off any more comments), if you look at the AWS Global Infrastructure map (about halfway down the page here), and mouse over the us-west-1 region, you'll see that it has three AZs. However, any given account may not have access to all of the AZs in a region.
Related
I'm trying to create an AMI by packer in a AWS codebuild project.
This AMI will be used to launch template
and the launch template will be used to ASG.
and when the ASG get an instance by this launch template, it should work with an existing target group for ALB.
for clarification, my expectation is...
generate AMI in a code build project by packer
create launch template with the #1 AMI
use the #2 launch template to ASG
ASG launch a new instance
existing target group do health check #4 instance.
In the step 5, my existing target group failed to do health check well for the new instance because it had different vpc.
(existing target group is using a custom VPC and the #4 instance had default vpc)
So, I backed to #1 to set the same VPC during the AMI generation.
But the codebuild project failed when it called the packer template in it.
it returned below
==> amazon-ebs: Prevalidating AMI Name...
amazon-ebs: Found Image ID: ami-12345678
==> amazon-ebs: Creating temporary keypair: packer_6242d99f-6cdb-72db-3299-12345678
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Error launching source instance: UnauthorizedOperation: You are not authorized to perform this operation.
Before this update, there were no vpc and subnet related settings in the packer template, and they worked.
I added some vpc related permissions for this code build project but no lucks yet.
Below is my builders configuration on the packer-template.json
"builders": [
{
"type": "amazon-ebs",
"region": "{{user `aws_region`}}",
"instance_type": "t2.micro",
"ssh_username": "ubuntu",
"associate_public_ip_address": true,
"subnet_id": "subnet-12345678",
"vpc_id": "vpc-12345678",
"iam_instance_profile": "blah-profile-12345678",
"security_group_id": "sg-12345678",
"ami_name": "{{user `new_ami_name`}}",
"ami_description": "AMI from Packer {{isotime \"20060102-030405\"}}",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "{{user `source_ami_name`}}",
"root-device-type": "ebs"
},
"owners": ["************"],
"most_recent": true
},
"tags": {
"Name": "{{user `new_ami_name`}}"
}
}
],
Added on this step (not exist before)
subnet_id
vpc_id
iam_instance_profile
security_group_id
Q1. Is this correct configuration to use VPC on here?
Q1-1. If yes, which permissions are required to allow this task?
Q1-2. If not, could you let me know the correct format of this?
Q2. Or... Is this correct way to get some instances which are able to communicate with my existing target groups...?
Thanks in advance. Your any kind of mentions will be helpful to me.
I got some helps from a local community.
And now I see I wrote too much wide and not good question without enough informations. There were several issues.
I should have used CloudTrail instead of CloudWatch to know which role and actions are making problems. My codebuild project had not ec2.RunInstances permission.
After I saw this on CloudTrail, I updated the role policy for the codebuild project and it was passed. But there was another issue.
After launch the instance by packer, it failed to connect with ssh. I got some answers from Stack overflow by searching about packer's timeout issue by ssh. and update the security group to allow ssh for packer.
Will remove this question if it is required.
Thanks for my local community and the previous answers & questioners on Stack overflow.
While creating a spot instance using console, it is not compulsory to choose a subnet. I can...
Subnet: No preference (default subnet in any Availability Zone)
But while creating a spot instance using cloudformation template, I need to choose a subnet and I am not sure which one to choose for best price.
Does it mean that using web interface is better than cloudformation template (for spot instances)?
Requirement and corresponding allocation strategy
I interpreted your requirement "best price" as obtaining a SpotFleet [1] which is configured for cost optimization. AWS describes how to create a cost-optimized spot fleet in the docs [2]:
To optimize the costs for your use of Spot Instances, specify the lowestPrice allocation strategy so that Spot Fleet automatically deploys the cheapest combination of instance types and Availability Zones based on the current Spot price.
You already specified the correct AllocationStrategy lowestPrice for this case in your CloudFormation template provided in the comments above. AWS describes this allocation strategy as:
The Spot Instances come from the pool with the lowest price. This is the default strategy.
What is a spot instance pool?
A Spot Instance pool is a set of unused EC2 instances with the same instance type (for example, m5.large), operating system, Availability Zone, and network platform [i.e. EC2-Classic or EC2-VPC, see [3]].
The Spot Fleet selects the Spot Instance pools that are used to fulfill the request, based on the launch specifications included in your Spot Fleet request, and the configuration of the Spot Fleet request [which is lowestPrice in our case]. [2]
That answers one part of your question:
No, the subnet does not directly determine the spot price. It is the availability zone which determines the price, i.e. instances of the same instance type, operating system and network platform always have the same spot price if they reside in the same availability zone. Since a subnet cannot span multiple availability zones, the spot price for instances in a particular subnet is always identical - i.e. the price for the (AZ, Instance Type, OS, Network Platform) combination.
Automatic subnet and AZ selection
You state in your question that you have to choose a subnet. That is not true. The NetworkInterfaces attribute of your LaunchSpecification [4] is optional.
If you omit it, CloudFormation behaves exactly the same as the AWS Management Console.
You can read about it in the Spot Fleet Examples section of the docs [5]:
Example 1: Launch Spot Instances Using the Lowest-Priced Availability Zone or Subnet in the Region
The following example specifies a single launch specification without an Availability Zone or subnet. The Spot Fleet launches the instances in the lowest-priced Availability Zone that has a default subnet. The price you pay does not exceed the On-Demand price.
{
"TargetCapacity": 20,
"IamFleetRole": "arn:aws:iam::123456789012:role/aws-ec2-spot-fleet-tagging-role",
"LaunchSpecifications": [
{
"ImageId": "ami-1a2b3c4d",
"KeyName": "my-key-pair",
"SecurityGroups": [
{
"GroupId": "sg-1a2b3c4d"
}
],
"InstanceType": "m3.medium",
"IamInstanceProfile": {
"Arn": "arn:aws:iam::123456789012:instance-profile/my-iam-role"
}
}
]
}
If you really need to assign a NetworkInterface to your instance (e.g. because you require a public IP address), there is also a solution listed in [5]:
{
"ImageId": "ami-1a2b3c4d",
"KeyName": "my-key-pair",
"InstanceType": "m3.medium",
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-1a2b3c4d",
"Groups": [ "sg-1a2b3c4d" ],
"AssociatePublicIpAddress": true
}
],
"IamInstanceProfile": {
"Arn": "arn:aws:iam::880185128111:instance-profile/my-iam-role"
}
}
That is exactly what you implemented in your CloudFormation template as far as I see.
AWS states for this approach:
The Spot service launches the instances in whichever subnet is in the lowest-priced Availability Zone.
and
You can't specify different subnets from the same Availability Zone in a Spot Fleet request.
That means, you should create one launch specification for each availability zone you want to distribute your instances across. Each of those launch specifications should differ by SubnetId:
[...]
LaunchSpecifications: [
{
[...]
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-1a2b3c4d", // choose a subnet from AZ a
"Groups": [ "sg-1a2b3c4d" ],
"AssociatePublicIpAddress": true
}
],
[...]
},
{
[...]
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-xxxxxxxx", // choose a subnet from AZ b
"Groups": [ "sg-1a2b3c4d" ],
"AssociatePublicIpAddress": true
}
],
[...]
},
[... etc. for each AZ]
],
[...]
References
[1] https://docs.aws.amazon.com/de_de/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-spotfleet.html
[2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html#spot-fleet-allocation-strategy
[3] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-classic-platform.html
[4] https://docs.aws.amazon.com/de_de/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-spotfleet-spotfleetrequestconfigdata-launchspecifications.html#cfn-ec2-spotfleet-spotfleetrequestconfigdata-launchspecifications-networkinterfaces
[5] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-examples.html#fleet-config1
Spot instance pricing is determined by long term trends of supply and demand for EC2 instances, not which subnet you chose to launch the spot instance into.
Its always beneficial to develop a cloudformation template for each of your AWS solutions. This keeps with the "Infrastructure as Code" mantra which carries many positive benefits such as reuse, ability to track changes with revision control, cattle not pets, etc.
Regarding the default subnets: For your account, each Region has a default VPC which contains default subnets per Availability Zone. This is a convenience feature provided by AWS so you can start launching instances quickly. These default subnets are public subnets and if you launch instances into them, these instances get public ips and are exposed to the internet. For better security, develop a private subnet and launch instances into the private subnet and access them via bastion host.
I have two Elasticsearch clusters set up on AWS using EC2 servers, sat behind a load balancer. I want to configure AWS to start up new instances when the load hits a certain threshold on the load balancer. Currently I have the Elasticsearch instances speaking to each other with the following config in config/elasticsearch.yml:
discover.zen.ping.unicast.hosts: ["172.x.x.x","172.x.x.x"]
Since the new instance will start up on an indeterminable private IP in the subnet, I was wondering if it was possible to point the config at an IP range so that new clusters could be discovered as they start up. Are there security implications to doing this?
If not, is there another way to achieve the same outcome? I am aware that Elasticsearch is meant to handle load balancing and scaling by itself, however this is part of a larger solution and specified in security specifications of my partner on the project.
The Question
When hosting Elasticsearch on AWS EC2:
Can you set up Elasticsearch discovery to use an IP range?
Are there security implications to doing this when using AWS?
If not, what is the appropriate way to handle this scenario?
I found this out myself. For anyone else who happens upon this, here is what I did.
I set up a basic instance with Elasticsearch, Apache and Java installed, and made a simple Elasticsearch startup script so that it would start on system boot.
I set the node name on my ES instances to be node.name: ${HOSTNAME} so that each node would be identifiable in the cluster and has a unique name in case I ever need to manually log into the server to check anything.
The big gotcha is that you have to use Elasticsearch's cloud-aws plugin for the relevant version of Elasticsearch that you're using. It can be found here
Using elasticsearch.yml you can configure your installations using cloud-aws specific config, you can make it so that ES isn't dependent on an IP address to join a cluster, but can instead discover a cluster based on instance tags, subnets, or anything else that is unique but determinable about your new instances in AWS.
I found this article very helpful. The main points from the article that are helpful are below.
AWS IAM Role
An IAM role needs to be set up and allocated to your new instances to allow them to communicate with AWS and check the new instances for certain values. The role should have these permissions.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeTags",
"ec2:DescribeRegions",
"ec2:DescribeSecurityGroups"
],
"Resource": "*"
}
]
Elasticsearch config
After installing cloud-aws, you will be able to use new config in your elasticsearch.yml file. First, remove discover.zen.ping.unicast.hosts: as this is no longer needed. The most basic setup to get this to work is:
cloud:
aws:
access_key: <YOUR AWS KEY>
secret_key: <YOUR AWS SECRET>
discovery:
type: ec2
But a more tuned configuration is below. It will look for instances with a tag of esdiscovery:enabled:
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.esdiscovery: enabled
cloud:
aws:
access_key: <YOUR AWS KEY>
secret_key: <YOUR AWS SECRET>
region: <YOUR AWS REGION>
discovery:
type: ec2
If you're using EU West, then you need to replace the region with the following config:
cloud:
aws:
ec2:
endpoint: ec2.eu-west-2.amazonaws.com
I have 2 VPCs in my account:
aws ec2 describe-vpcs --query 'Vpcs[*].VpcId'
[
"vpc-654bf20c",
"vpc-184bf271"
]
But when I check my account-attributes, I don't see any default VPC tagged:
aws ec2 describe-account-attributes --query 'AccountAttributes[5]'
{
"AttributeName": "default-vpc",
"AttributeValues": [
{
"AttributeValue": "none"
}
]
}
How do I make any of the VPCs as a default VPC so that when I push stacks using CloudFormation, I don't have to mention VPCs?
You wont be able to convert a non-default VPC into default on your own, you need to get in touch with Amazon support for that.
If you have UI access to amazon console can you cross check if "Default VPC" for either of your VPC is Yes. See reference image.
If there is none, this means that you have intentionally/mitakenly deleted this VPC in the past. You need to raise a support ticket for getting a new default VPC.
Other work around could be that you move all configurations to some other region, if that is possible, you will get a default VPC there.
Thanks
Hope it helps!
I have setup a few of the amazon AWS CLI tools (EC2, Auto Scaling, MOnitoring and ELB). The tools are setup correctly and work perfectly. My environment vars are all set, the relevant ones to this Q being:
export EC2_REGION=eu-west-1
export EC2_URL=https://ec2.$EC2_REGION.amazonaws.com
export AWS_ELB_URL=https://elasticloadbalancing.$EC2_REGION.amazonaws.com
When I run ec2-describe-instance-status i-XXXXXXXX for ANY of my instances, I get:
Client.InvalidInstanceID.NotFound: The instance ID 'i-XXXXXXXX' does not exist
I KNOW the instance ID exists, I copied it out of the AWS web console, and it is in the eu-west-1 region, and my env vars are set to this region.
For the life of me I can't figure out why it will not find my instances. Is there anything glaringly obvious that I am doing incorrectly?
UPDATE: recreating x509 cert/pk solved this... for some reason.
I had the same problem. It was because I wasn't defining a region for my commands. I assumed it would list all instances across all regions but it defaults to us-west-1 and I don't have any instances there.
To describe my machines in Ireland I use the following:
ec2-describe-instances --region eu-west-1
NB: I'm defining my AWS access key and secret elsewhere.
To avoid this problem going forward, I've now set my region via an environment variable on my linux and windows machines: EC2_URL=https://ec2.eu-west-1.amazonaws.com
so that I don't have to be explicit on the command line.
Update May 2014 You can also set the region by adding the following lines to the ~/.aws/config file in your home folder (not tested on Windows). This is my preferred method now, especially on my VM's and containers:
[default]
region = eu-west-1
For more information see the offical docs here.
Update May 2021
Since I work across so many regions now I use Implicit and ephemeral environment variables to define my region for that command and NOT have a default in my .aws/config which can be dangerous. This also makes bash scripting easier as I can define it for the whole script/utility. It's a tiny bit more typing but far safer, more flexible and transparent e.g.:
AWS_DEFAULT_REGION=eu-central-1 aws ec2 describe-instances
# or for a script/utility
AWS_DEFAULT_REGION=us-east-1 ./tagInstances.sh
In my case, I had two sets of credentials in ~/.aws/credentials. To specify the credentials tag, use
aws ec2 describe-instances --instance-id <your-instance-id> --profile <your-profile-name> --region <your-region>
Weird issue - as usual when encountering something weird in software development, one should first question the assumptions:
I KNOW the instance ID exists, I copied it out of the AWS web console,
and it is in the eu-west-1 region, and my env vars are set to this
region.
So the instance ID stems from a different environment than the one you want to use it in - I would try to derive the instance ID via the same environment instead, i.e.:
ec2-describe-instances
I venture the guess that the list won't return the instances you are expecting. This would indicate that you are either using AWS credentials that belong to another account or that these credentials do not have the required Amazon EC2 read permissions assigned via IAM policies for example.
I had a similar issue and I write here the solution for anybody who can find it helpful.
I was stucked with this error during some hours.
Client.InvalidInstanceID.NotFound: The instance ID 'i-XXXXXXXX' does not exist
Finally I found what was happening: I had my instance in a different region than the default region (US East (Northern Virginia)) and I had to update this information. By default the commands look only for instances in the default region!
It is explained in the docs, section (Optional): Set the region http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html
It's very simple problem. If you are getting this error
Client.InvalidInstanceID.NotFound: The instance ID 'i-XXXXXXXX' does not exist
Then follow the steps.
Check in which country region your instance is image here
Now enter root#Indian:~# aws configure
Enter
AWS Access Key ID [****************D7M2]:
AWS Secret Access Key [****************2h3r]:
Default region name [us-east-1]:
When asked for default region, Change the region to which the instances is residing. Eg: us-east-2. Then press Enter.
Acutally these are the list of available Region names "RegionNames"
"Regions": [
{
"RegionName": "ap-south-1",
"Endpoint": "ec2.ap-south-1.amazonaws.com"
},
{
"RegionName": "eu-west-2",
"Endpoint": "ec2.eu-west-2.amazonaws.com"
},
{
"RegionName": "eu-west-1",
"Endpoint": "ec2.eu-west-1.amazonaws.com"
},
{
"RegionName": "ap-northeast-2",
"Endpoint": "ec2.ap-northeast-2.amazonaws.com"
},
{
"RegionName": "ap-northeast-1",
"Endpoint": "ec2.ap-northeast-1.amazonaws.com"
},
{
"RegionName": "sa-east-1",
"Endpoint": "ec2.sa-east-1.amazonaws.com"
},
{
"RegionName": "ca-central-1",
"Endpoint": "ec2.ca-central-1.amazonaws.com"
},
{
"RegionName": "ap-southeast-1",
"Endpoint": "ec2.ap-southeast-1.amazonaws.com"
},
{
"RegionName": "ap-southeast-2",
"Endpoint": "ec2.ap-southeast-2.amazonaws.com"
},
{
"RegionName": "eu-central-1",
"Endpoint": "ec2.eu-central-1.amazonaws.com"
},
{
"RegionName": "us-east-1",
"Endpoint": "ec2.us-east-1.amazonaws.com"
},
{
"RegionName": "us-east-2",
"Endpoint": "ec2.us-east-2.amazonaws.com"
},
{
"RegionName": "us-west-1",
"Endpoint": "ec2.us-west-1.amazonaws.com"
},
{
"RegionName": "us-west-2",
"Endpoint": "ec2.us-west-2.amazonaws.com"
}
]
}
Default output format [None]:
Leave the output format blank and press Enter. Now You are Done
Now in the console just type
root#Indian-3543:~# aws ec2 describe-instance --instance-id i-06343434322t
MAKE HAPPY BE HAPPY
I got this fixed by changing EC2_URL from 'https://ec2.ap-southeast-1.amazonaws.com' to 'ec2.ap-southeast-1.amazonaws.com'