I'm trying to create a deployment where as many tasks as possible are on each EC2 instance, but ECS still places one task per instance which then makes them heavily underutilized.
Here's all the settings I think are relevant:
ASG capacity provider uses managed scaling, target capacity is set to 100%, instances are t3a.micro (2048 CPU, 960 memory)
Task uses 480 memory, 1024 CPU but for testing I tried to go to 200 memory and 500 CPU and nothing changed
The service placement strategies are binpack(cpu) and binpack(memory)
The tasks are in a public VPC within subnets that have access to nat gateway, public ip assign is disabled, the networking mode is awsvpc
I'm changing the desired count to test this
On every deploy tasks are placed into empty instances and if there aren't any ASG creates new ones, there's never more than one task per instance.
It seems that the micro EC2 instance you require doesn't have enough ENI capacity to scale more than 1 task per container instance.
According to the documentation:
With the awsvpc network mode, Amazon ECS creates and manages an
Elastic Network Interface (ENI) for each task and each task receives
its own private IP address within the VPC. This ENI is separate from
the underlying hosts ENI. If an Amazon EC2 instance is running
multiple tasks, then each task’s ENI is separate as well.
Since "this ENI is separate from the underlying hosts ENI", running 1 ECS task requires at least 2 interfaces. In case of running 2 ECS tasks you would need 3 ENI and so on.
When I get a description of EC2 instance types (you can just run aws ec2 describe-instance-types --filters "Name=instance-type,Values=t3a.micro") I see that t3a.micro has a maximum limit of only 2 available network interfaces ("MaximumNetworkInterfaces": 2). So, you need to make better use of the existing maximum number of network interfaces or to get a container instance with capacity to attach more network interfaces.
A solution might be to use an instance type with more ENI capability or to increase task density with ENI trunking. Please read about ENI trunking considerations before. ENI trunking might look something like this:
Related
My AWS solution spans over 3 availability zones. In my backend the user is able to trigger a heavy compute job with beefy px instances. Therefore I wrote a CFN template, which provision all resorucess to execute the compute job (secret store, IAM Role, EC2 instance, log group). However when I try to create the template, it returns with a 500 and states that no capacity for my instance type is available for the availability zone i choose. My template provides a subnet for the EC2 instance and an availability zone for the attached volume. In the end I don't care in which availability zone the ec2 is provisioned as long it is in one of my subnets. Does someone know a way to provision an EC2 instance and it's volume (with cloudforamtion) by not specifically choosing one availability zone, but rather provide a range of subnets/availability zones ?
TLDR:
Does someone know a way to provision an EC2 instance and it's volume (with cloudforamtion) by not specifically choosing one availability zone, but rather provide a range of subnets/availability zones ?
I have 3 containerized applications. All these applications are a simple python script that calls an api at different times. For example, App1: calls every hour, App2: calls every 30 mins, and App3: calls every minute. Right now we have an ec2 instance for each application, which is ok because I need the applications to run on different AWS EC2 Instances because I don’t want my IP Blacklisted. But now how would I go about scaling an application? For example, I need 3 instances of app 1.
Im currently using Terraform to create the EC2 instances and while that can work I don't know how to go about automating updating the images on each EC2 instance.
I tried using AWS ECS to scale the container but this requires that it shares IP address ( I would like for these applications to live in different VMS(EC2 Instances.), so that the IP does not have a problem getting blacklisted.) with the other applications and I don’t want that.
What tools or architecture can best help me accomplish the scaling of these applications?
For your specific case, I would suggest AWS Fargate ECS + Cloudwatch events cron triggers (Scheduled Fargate ECS tasks).
Fargate instances have awsvpc network mode by default, which means if you run them in public subnet with public IP allocation, each container instance will retrieve unique public IP (via dedicated automatically attached ENI - Elastic Network Interface).
Additionally, running tasks on Fargate will allow you to utilize much more convenient docker packaging of your applications while keeping management & operating requirements low - as Fargate is a Server-less container scheduler, and it requires less effort than operating AWS EC2 instances.
You will need 3 ECS resources :
cluster
template resource for container definition
task definition resource
and 3 CloudWatch resources:
aws_cloudwatch_event_rule
aws_cloudwatch_event_target
aws_cloudwatch_log_group
in Terraform to make it work, + ECR repo/s for storing image artifacts.
I want to utilize AWS ECS fargate but I have a requirement from the security team to restrict the Private IP range of my fargate instances within a subnet.
From the AWS networking docs, It seems like it is possible to use a seperate non-managed ENI to specify the private IP address. Is there a method to use a static IP range for the fargate instances?
For tasks that use the Fargate launch type, the task ENI that is
created is fully managed by AWS Fargate. Depending on which Fargate
platform version your task uses, the network traffic for your task may
also use a separate Fargate-owned ENI.
The fixed private IP range is decided based on the subnet(s) that you will deploy your Fargate tasks into.
If you create seperate subnet(s) for your Fargate tasks, and then ensure that only Fargate tasks are launched in these subnets you could then provide this range to your security team.
Fargate creates the ENI when it launches the task, with a random private IP address form the pool of available IPs in your subnet(s) range.
We have a 3rd party integration which needs the EC2 instance IP to be whitelisted. The 3rd party whitelists the IP on their server and then only the EC2 instance can communicate with them.
In the case of single instance this works.
However when auto scaling kicks in, we would end up in more than 1 instance. These new instances automatically get new IPs for every autoscale action.
Is it possible for us to ask AWS to assign IPs from a say a set of 4 predefined Elastic IPs? ( Assumption is that autoscaling is restricted to say 4 and we have 4 floating EIPs )
I'm trying to avoid gateway NAT since there is a big cost associated with it.
Any ideas?
With autoscaling this is not directly possible to assign an Elastic IP to autoscaled instances. However there are couple of options you can consider.
After instance autoscales, having a boot up script(e.g UserData in Linux) with AWS EC2 CLI commands to associate an Elastic IP address you have allocated to your account writing a command line script. Note that you need to handle the health checks accordingly for the transition to happen smoothly.
Having a CloudWatch alarm trigger to execute an Lambda function which will associate an Elastic IP address to the instance newly started. For this you can use AWS SDK and code to check the instance without EIP and Associate an available EIP to it.
Auto Scaling will not automatically assign an Elastic IP address to an instance.
You could write some code to do this and include it as part of the User Data that is executed when an instance starts. It would:
Retrieve a list of Elastic IP addresses
Find one that is not currently associated with an EC2 instance
Associate it with itself (that is, with the EC2 instance that is running the User Data script)
Use a NAT instance. There's only a small cost associated with a t2.nano and you should find that more than adequate for the purpose.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html
While not as reliable as a NAT Gateway (you're paying for hands-off reliability and virtually infinite scalability), it's unlikely you'll have trouble with a NAT instance unless the underlying hardware fails, and you can help mitigate this by configuring Instance Recovery:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html
Sorry, I had a few basic questions. I'm planning to use an AWS EC2 instance.
1) Is an EC2 instance a single virtual machine image or is it a
single physical machine? Documentation from Amazon states that it is
a "virtual server", but I wanted to clarify if it is an image
embedded inside one server or if it is an single physical server
itself.
2) Is an Elastic Load Balancer a single EC2 instance that handles
all requests from all users and simply forwards the request to the
least loaded EC2 instances?
3) When Auto-Scaling is enabled for an EC2 instance, does it simply
exactly replicate the original EC2 instance when it needs to scale
up?
An EC2 instance is a VM that gets some percentage of the underlying physical host's RAM, CPU, disk, and network i/o. That percentage could theoretically be 100% for certain instance types, including bare-metal instances, but is typically some fraction depending on which instance type you choose.
ELB is a service, not a single EC2 instance. It will scale on your behalf. It routes by round robin for TCP, and routes on fewest outstanding requests for HTTP and HTTPS.
Auto Scaling is "scale out" (it adds new EC2 instances), not "scale up" (resizing an existing EC2 instance). It launches a new instance from a template called an AMI.
It is a virtual server, a VM, as stated in the documentation.
It's a little more complicated that that, based on the way AWS might scale the load balancer, or create a version in each availability zone, etc. It also provides more features such as auto-scaling integration, health checks, SSL termination. I suggest you read the documentation.
It uses a machine image that you specify when you create the auto-scaling group (when you create the Launch Configuration used by the Auto-scaling group to be more precise). A common practice is to configure a machine image that will download any updates and launch the latest version of your application on startup.
You might also be interested in Elastic Beanstalk which is a PaaS that manages much of the AWS infrastructure for you. There are also third-party PaaS offerings such as OpenShift and Heroku that also manage AWS resources for you.