I want to persist my mongodb data in ecs cluster using EFS. So, I followed the below approach.
created a cluster under EC2 instance
Attached EFS to the EC2 cluster Instance
Created task definitions and services.
Ran services on the EFS mounted EC2 Cluster
Now, everything is going smooth when the containers are getting updated or terminated. But, when the entire ec2 instance is terminated. AWS cloud formation is creating EC2 instance without mounting EFS as expected. Can someone help me how to get EFS attached automatically when the EC2 instance gets terminated and recreated by CloudFormation. I even tried custom ecs images. But, the instances are not getting created as there is no cloud formation template getting created.
Related
I see that the autoscaling group successfully created the EC2 instance but it didn't include it in the cluster? What can I do to fix this issue?
Things I've verified:
I did choose network and ec2 linux
The instance is currently running with 2/2 status check
I did see this question but the answers there were not the issue: EC2 instances not created while creating ECS cluster
I'm trying to create ECS cluster based on EC2 using Terraform and it creates every resources however EC2 instance is not getting registered at ECS and my tasks are failing. I created IAM roles for ECS instance and ECS services and attached respective policies.
Here is the full Terraform code for ECS cluster
https://gist.github.com/billyboar/4c7f1fa5b011896b2d3baa49574977a0
Seems like my VPC didn't make the route table that I created as default. I tried making the route table as default and somehow EC2 instance was registered as ECS container.
I think you have 2 scaling
1 for ec2 instance
1 for ecs task
So I think flow goes like
You get more requests
Your tasks need more resources (cpu/memory)
Your ecs task autoscaling kicks in and creates more task
your ec2 autoscaling also kicks in and creates more ec2
Questions
your ecs needs to know newly launched ec2 (from ec2 autoscaling group) is the host that it can run on (how?)
I want the number of ec2 instance starts from one.
I think one instance is already running in the ecs cluster
Do I make ec2 autoscaling group start from 0?
ec2 autoscaling group has pricing option to use spot instance, but I didn't see such option when I created ecs cluster instances. To be clear, can I make ecs cluster with 0 ecs-instance, and let ec2 autoscaling group starts with 1 instance?
Your ecs task autoscaling kicks in and creates more task
your ec2 autoscaling also kicks in and creates more ec2
Your ECS task scale if there is a memory so fine it will scale and launch new container, if not enough memory then it will not launch an instance, you need to configure cloud watch alarm to scale ECS cluster based on the task scaling.
Tutorial: Scaling Container Instances with CloudWatch Alarms
your ecs needs to know newly launched ec2 (from ec2 autoscaling group)
is the host that it can run on (how?)
ECS agent registers the new instance with the same cluster, and the new task can be placed in any instance that have available resources to handle and launch the container.
I want the number of ec2 instance starts from one. I think one
instance is already running in the ecs cluster Do I make ec2
autoscaling group start from 0?
You should start from 1 otherwise no container will launch and will throw error no instance found in the cluster. if you do not want run container for now then you can start from 0.
ec2 autoscaling group has pricing option to use spot instance, but I
didn't see such option when I created ecs cluster instances. To be
clear, can I make ecs cluster with 0 ecs-instance, and let ec2
autoscaling group starts with 1 instance?
You can read about spot ECS cluster instance here.
Amazon ECS is a highly scalable, high performance, container
management service that supports Docker containers and allows you to
run applications on a managed cluster of Amazon EC2 instances easily.
ECS already handles the placement and scheduling of containers on EC2
instances. When combined with Spot fleet, ECS can deliver significant
savings over EC2 on-demand pricing.
The cluster wizard's dropdown menu only shows t2.micro or larger instances.
I want to use a t2.nano instance to run my docker image.
Can this be achieved while using the ec2 container service?
I know that can manually setup my own t2.nano ec2 instance and run the docker pull and run commands manually from there. Basically use the AWS EC2 Container repository, but not the Clusters or Task Definitions functionality.
They probably missed this instance type in the console wizard. You can launch the EC2 instance yourself (I recommend you to use an AutoScaling Group even if it's one instance) and then run the ECS agent on the instance. You don't have to do much if you pick the ECS-optimized AMI that AWS provides.
You can launch the EC2 instance from the AMI manually, but I recommend you to:
create a Launch Configuration using the latest ECS-optimized AMI and change the user data to be: #!/bin/bash
echo "ECS_CLUSTER=your-cluster-name-here" > /etc/ecs/ecs.config
create an AutoScaling group and set the proper desired capacity
Check if your Cluster got the new instance once the instance is up and healthy
More info:
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI_launch_latest.html
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html
I had to stop my m3.medium EC2 instance from the AWS console to resize it to m3.large. However, after it stopped, it automatically created a new instance.
Any idea why this is happening? It caused some big troubles for me.
Your AutoScaling group with minimum size = 1 spun up a new instance because there were no instances in the 'running' state available to respond to requests, particularly health checks. Your instance was deemed 'unhealthy' and replaced by the ASG.
If your instance storage was ephemeral, I'm afraid it is gone forever unless you recently saved an AMI. If your instance storage was backed by EBS, you can recover it by attaching the EBS volume to a new instance.
In the future, consider configuring your autoscaling group's launch configuration to have everything you need ahead of time, by either bootstrapping the instance or by baking an AMI.
For 'bootstrapping' an instance:
Create a new launch configuration with a standard AMI avaialble from Amazon.
Add user data to the launch configuration to handle installation and configuration of your desired programs.
For 'baking' an AMI:
Install your desired programs and configuration on a new EC2 instance.
Take an image (AMI) of that EC2 instance.
Use that image in your new launch configuration.
Manually working on an instance within an ASG and expecting the instance to persist indefinitely is dangerous, as you've just discovered.
Further Reading:
EC2 Documentation - AutoScaling Health Checks
EC2 Documentation - Amazon Machine Images