We are having ECS cluster where we are having 1 container namely A (which has soft limit of 412 mb at container level) and ECS agent and another container(Running via userdata) which is hard limited to 512 mb and our instance type is t3a.micro? My doubt is How much memory can ECS agent use?
From documentation, soft limit means to be memory reservation,In this case does that mean 412 mb of memory is reserved only for container A?
Related
In the AWS console I setup a ECS cluster. I registered a EC2 container instance on a m3.medium, which has 2vCPUs. In the ECS console it says only 1024 CPU units are available.
Is this expected behavior?
Should the m3.medium instance not make 2048 CPU units available for the cluster?
I have been searching the documentation. I find a lot of explanation of how tasks consume and reserve CPU, but nothing about how the container agent contributes to the available CPU.
ECS Screenshot
tldr
In the AWS console I setup a ECS cluster. I registered a EC2 container > instance on a m3.medium, which has 2vCPUs. In the ECS console it says > only 1024 CPU units are available. > > Is this expected behavior?
yes
Should the m3.medium instance not make 2048 CPU units available for
the cluster?
No. m3.medium instances only have 1vCPU (1024 CPU units)
I have been searching the documentation. I find a lot of explanation
of how tasks consume and reserve CPU, but nothing about how the
container agent contributes to the available CPU.
There prob isn't going to be much official documentation on the container agent performance specifically, but my best recommendation is to pay attention to issues,releases,changelogs, etc in the github.com/aws/amazon-ecs-agent and github.com/aws/amazon-ecs-init projects
long version
The m3 instance types are essentially deprecated at this point (not even listed on main instance details page), but you'll be able to see the m3.medium only has 1vCPU in the the details table on kinda hard to find ec2/previous-generations page
Instance Family Instance Type Processor Arch vCPU Memory (GiB) Instance Storage (GB) EBS-optimized Available Network Performance
...
General purpose m3.medium 64-bit 1 3.75 1 x 4 - Moderate
...
According to this knowledge-center/ecs-cpu-allocation article, 1 vCPU is equivalent to 1024 CPU Units as described in
Amazon ECS uses a standard unit of measure for CPU resources called
CPU units. 1024 CPU units is the equivalent of 1 vCPU.
For example, 2048 CPU units is equal to 2 vCPU.
Capacity planning for ECS cluster on an EC2 can be... a journey... and will be highly dependent on your specific workload. You're unlikely to find a "one size" fits all documentation/how-to source but I can recommend the following starting points:
The capacity section of the ECS bestpractices guide
The cluster capacity providers section of the ECS Developers guide
Running on an m3.medium is prob a problem in and of itself since the smallest types i've in the documentation almost are c5.large, r5.large and m5.large which have 2vCPU
While launching 2nd instance getting below error
you have requested more vcpu capacity than your current vcpu limit of 1 allows for the instance bucket that the specified instance type belongs to. please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.
First, please note that the AWS Free Tier is a billing discount. It does not impact your ability to use any services. If the resources you use are within the limits imposed by the AWS Free Tier, you simply won't be charged for those resources. Thus, the AWS Free tier is not the cause of this error.
From On-Demand Instances - Amazon Elastic Compute Cloud:
There is a limit on the number of running On-Demand Instances per AWS account per Region. On-Demand Instance limits are managed in terms of the number of virtual central processing units (vCPUs) that your running On-Demand Instances are using, regardless of the instance type.
You did not tell us what Instance Type you are running, or attempted to run, but it sounds like you have exceeded this vCPU limit. This is more likely to happen with large or expensive instance types, or instance types that use GPU resources. You can view the limits in the Amazon EC2 management console by clicking Limits and then searching for vCPU.
If this is impacting your use of AWS, you can request an increase to these default limits.
See: Calculate a vCPU limit increase request for an Amazon EC2 On-Demand Instance
I have configured an AWS ECS Cluster with 3 instances (m5.large), with one instance across each availability zones (A, B, and C). The Service is configured as follows:
Service type: REPLICA
Number of Tasks: 3
Minimum Healthy Percent: 30
Maximum Percent: 100
Placement Templates: AZ Balanced Spread
Service AutoScaling: No.
In the Task Definition, I have used the following:
Network Mode: awsvpc
Task Memory: --
Task CPU: --
At the container level, I have configured only Memory Soft Limit:
Soft Limit: 2048 MB
Hard Limit: --
I have used awslogs for logging. The above configuration works and when I start the service, there is one docker running in each of the instances. The 'docker stats' in one of the instances shows the following:
MEM USAGE / LIMIT
230MiB / 7.501GiB
And the container instance (ECS Console) shows the following:
Resources Registered Available
CPU 2048 2048
Memory 7680 5632
Ports 5 ports
The above results are the same across all the 3 instances -- 2 GB of memory has been reserved (soft limit) and upper memory limit is instance memory of nearly 8 GB (no hard limit set). Everything works as expected so far.
But when I re-deploy the code (using force deploy) from Jenkins, I get the following error in the Jenkins Log:
"message": "(service App-V1-Service) was unable to place a task because no container instance met all of its requirements. The closest matching (container-instance 90d4ba21-4b19-4e31-c42d-d7223b34f17b) has insufficient memory available. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.
In Jenkins, the job shows up as 'Success', but it is the old version of the code that is running. There is sufficient memory available on all the three instances. Also, the I have changed the Minimum Healthy Percent to 30 hoping that ECS can stop the container and re-delpoy the new one. Any solution or pointers to debug this further will be of great help.
As during deployment, the ECS schedule will allocate memory base on soft limit for each container which can be
2048 * 3 = 6144 MB
which is less than the available memory in the instance
5632 (available memory) < 6144 (required memory)
If you running replica in the same ECS container instance then I will recommend to keep minimum soft limit which should be less or equal to 1GB also this is suggested by ECS as well.
So with this configuration, you will be run blue-green deployment as well. As this nothing harm to keep the soft limit minimum as container can scale to use more memory when it's required so applying some big memory for soft limit does not affect the performance.
I will not recommend lowering the Minimum Healthy Percent: 0 as decrease the soft limit to 1GB will resolve the issue.
Or if you want to keep the same memory limit then decrease Minimum Healthy Percent
I have some trouble to understand the memory management on AWS ECS Fargate.
We can set a memoryReservation to our containers, and a "memory" limit on our task.
Why should i define reserverd memory on Fargate? I can understand it for EC2 instance types (to select an instance with enough free memory) but on fargate, aws should put the task on an Instance with enough free memory?
Task Memory is the total memory available for all container ( its hard upper bound)
memoryReservation is a soft lower bound and container can use more memory if required.
This is helpful you have two or more container to in one task definition, to clarify this more we can look into this example
In this example, we allocate 128 MB for WordPress and 128Mb for MySQL, which become 256MB, which is half of the task level memory but we do not want a situation where container halt because of using max memory so we set hard memory limit to 512 and one container will reached to this the agent will the kill container.
deep-dive-into-aws-fargate
I am trying to launch 210 instances of type p2.xlarge. Earlier I am hitting instance limit which I have increased now to 300 but now I am getting 'Client.VolumeLimitExceeded: Volume limit exceeded' error. Can you please help to resolve this issue? Any one knows the
What is Volume limit?
What should I request to increase inorder to run 210 instance (p2.xlarge) and aviod this error.
Your error means: Client.VolumeLimitExceeded: The limit on the number of EBS volumes or total storage was exceeded. Decrease usage or request an increase in your limits.
The AWS Limits page for EBS lists the default limit being 5,000 EBS volumes and 10,000 EBS snapshots. Total storage limit is 20 TB per type (gp2, io12, st1, sc1). Make sure that this is not your issue.
Amazon Elastic Block Store (Amazon EBS) Limits
To request an increase for EBS go to Volume List Increase Request. On the same page your can change the Limit Type for other limit increases.
I would open a support case with Amazon support and ask them to review your requirements to make sure that all your limits are managed for this usage case instead of hitting one limit after the other. Amazon will definitely help you.
The default EBS limits are: Amazon Elastic Block Store (Amazon EBS) Limits
It is the number of EBS volumes an account can use. You have a bigger problem if you are launching 210 p2.xlarge instances. You will hit various resource limits. You may need to request increase in limits for those resources.
Delete unattached/unused EBS volumes in your account
Check your EBS limits in: Amazon EC2 Service Limits
Calculate the number of volumes you will be using for each instance (Check your AMI)
Request for more EBS volume limit.