When I manually create an AWS ECS cluster in the UI, by default it shows the default VPC with the default subnets selected, one in each availability region, which I infer to be best practice.
I'm creating a new ECS Fargate deployment using CloudFormation, and I believe the best practice is to create a new VPC with its own subnets, similarly each in a different availability zone.
Rather than hard-coding them in CloudFormation, how can I enumerate the availability zones for the current region and create subnets for each of them?
I'm new to CloudFormation. I've seen that there is a built-in function to enumerate availability zones, so with some thought I'm sure I can figure this out on my own. And I'll be sure to add that as an answer here. But being what surely is a common use case, perhaps there is an existing best-practices template somewhere that already shows how to do that so I don't have to reinvent it. (I haven't been able to find it immediately by searching.)
Related
Does anyone have any guidance on what the most efficient way to launch an EMR cluster using EC2 spot instances? I'm using a 10-node r5d.8xlarge (1 master 9 core) and usually I will clone a previous instance. What's been happening more often than not is that I'll get an "out of capacity" error. When that happens I change the subnet and try again. It's really a guessing game at this point and I'm sure there's a much better way to do this, unfortunately I haven't found one.
There's no need for it to be a guessing game. You can use Instance Fleets - Amazon EMR to specify multiple:
Instance Types
For each fleet, you specify up to five EC2 instance types. Amazon EMR
chooses any combination of these EC2 instance types to fulfill your
target capacities.
Availability Zones
When you use instance fleets, you can specify multiple EC2 subnets
within a VPC, each corresponding to a different Availability Zone.
Instances are always provisioned in only one Availability Zone.
See also: New – Amazon EMR Instance Fleets | AWS News Blog
What are the advantages of creating ECS cluster inside a VPC? I see that in the console page, it asks for whether one wants to create the ECS cluster inside the VPC or not?
Of course the right answer for your question is: it depends.
There could be many reasons to have you ECS cluster in an existing VPC, e.g. because that VPC is reserved to your development resources and you are creating your ECS cluster for dev or the VPC is paired with another one and you want to allow the ECS pods to be able to access the paired network resources.
For the opposite reasons maybe you want to have the ECS in a separated VPC. Of course the documentation can help you to make a decision.
The question in the console page is more about do you have already a VPC for this cluster or do you want me to create one for you? Let's say you are creating a new ECS cluster and you need a separated VPC, but you don't want to bother creating a new one, you can just leave this task to the AWS console.
I noticed there is already a
-subnet=... The AWS subnet to deploy to (single instance apps only)
parameter but not for ELB-based deployments, yet. Is it already possible to define which vpc/subnets should be used for a deployment?
Currently all subnets of the default VPC are associated with the automatically created auto-scaling groups. The auto-scaling groups will then launch instances across them in a balanced fashion.
Please file a feature request if you need more control: https://github.com/cloudcaptainsh/cloudcaptain/issues
I have one VPC with an RDS instance in it. They are both located in the same region.
I want to use the RDS instance in another VPC, that is in another region on another AWS account (we have multiple AWS accounts). If that's not complicated enough the 2nd VPC comes up via CloudFormation (i.e. dynamic). Whenever I am bringing up a CloudFormation stack I want to attach the RDS instance automatically.
I have looked at:
exposing the RDS instance on the public internet :(
an ELB w/ TCP transport to put the database instance behind
VPC peering but the different regions and the approval workflow in the AWS console make little sense in the case we are using CloudFormation
All of these seem suboptimal to me and was wondering if somebody already did this before. If yes, please share what you did and what the though process behind it was.
Use a VPN tunnel from one VPC to the other. You could build your own or look at Vyatta. Ideally the two VPCs do not have overlapping CIDRs. Note that you cannot use VPC peering inter-region.
For anyone who stumbles around here, it looks like AWS VPC Peering can now be done cross region: https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html
I have a pretty standard stack, RDS, 2 EC2 instances using ELB. Because I wanted the ELB to be restricted to a particular IP range I've launched the stack in VPC, for DR reasons across 2 subnets.
I use several ephemeral ec2 machines, which when not in VPC I allowed to startup in any availability zone. Which (as far as I understand) would mean that if an AZ were unavailable then machines would be started in other zones.
Is there a way to emulate this in VPC? Is there a way of saying launch a machine in any subnet in a VPC?
If not its fairly easy to workaround by picking a subnet at random, and if it fails trying another. Just wondered if there was a supported method that's cleaner?
I'm using python and boto.
thanks
Which (as far as I understand) would mean that if an AZ were unavailable then machines would be started in other zones.
That's correct and it would indeed be nice to have this option available within an Amazon VPC as well when running instances directly via the available Amazon EC2 API actions.
Unfortunately both the RunInstances and the RequestSpotInstances API actions only allow to specify the optional parameters SubnetId or LaunchSpecification.SubnetId respectively (The ID of the subnet in which to launch the [Spot] Instance), thus won't have any information into which VPC you would want to launch the instance if no subnet is specified.
Workaround
You can achieve the desired behavior indirectly via Auto Scaling by means of its CreateAutoScalingGroup API action, see parameter VPCZoneIdentifier:
A comma-separated list of subnet identifiers of Amazon Virtual Private Clouds (Amazon VPCs).
This feature is also available via the AutoScalingGroup resource type within AWS CloudFormation.