I have AWS ECS cluster but spot instance type I selected is too small.
I can't find way to adjust Spot Fleet request ID or change Instance type(s) for Spot Fleet request cluster is using.
Do I have to create a new cluster with a new spot fleet request?
Is there any cli option to adjust cluster?
Do I have manually order EC2 with ECS optimized AMI ?
UPDATE In question How to change instance type in AWS ECS cluster? that sounds similar advised to copy Launch Configuration. But I have no Launch Configuration
There is no way of changing the instance types' requested by a spot fleet after its been created.
If you want to run you ECS workload on another instance type, create a new spot fleet (with instances which are aware of your ECS cluster).
When the spot instances spin up, they will register with your ECS Cluster.
Once they are registered, you can find the old instances (in the ECS Instances tab of the cluster view) and click the checkbox net to them.
Then, go to Actions -> Drain instances
This tells ECS that you no longer wish to use these instances. New tasks will now be scheduled on the new instances.
Once all the tasks are running on the new instances, you can delete the old spot fleet.
On the subject of launch configurations. There are two ways of creating collections of spot instances.
Through a Spot Fleet (which is what you're doing)
Through and Auto Scaling Group (ASG)
ASGs allow you to supply a launch configuration (basically a set of instructions to set up an EC2 instances.
Spot Fleets only allow you to customise the instance on creation via User Data.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
Because you're using Spot Fleets, Launch Configurations are really a consideration for you.
Related
I have created a cluster to run our test environment on Aws ECS everything seems to work fine including zero downtime deploy, But I realised that when I change instance types on Cloudformation for this cluster it brings all the instances down and my ELB starts to fail because there's no instances running to serve this requests.
The cluster is running using spot instances so my question is there by any chance a way to update instance types for spot instances without having the whole cluster down?
Do you have an AutoScaling group? This would allow you to change the launch template or config to have the new instances type. Then you would set the ASG desired and minimum counts to a higher number. Let the new instance type spin up, go into service in the target group. Then just delete the old instance and set your Auto scaling metrics back to normal.
Without an ASG, you could launch a new instance manually, place that instance in the ECS target group. Confirm that it joins the cluster and is running your service and task. Then delete the old instance.
You might want to break this activity in smaller chunks and do it one by one. You can write small cloudformation template as well because by default if you update the instance type then your instances will be restarted and to avoid zero downtime, you might have to do it one at a time.
However, there are two other ways that I can think of here but both will cost you money.
ASG: Create a new autoscaling group or use the existing one and change the launch configuration.
Blue/Green Deployment: Create the exact set of resources but this time with updated instance type and use Route53's weighted routing policy to control the traffic.
It solely depends upon the requirement, if you can pour money then go with above two approaches otherwise stick with the small deployments.
Is there a way to instruct AWS to:
create X (number) spot instances from specific AMI? (I guess this is the easy part)
create something periodic so that when these spot instances are terminated, create X new ones from the same AMI?
When launching Amazon EC2 spot instances, you must specify an AMI. Therefore, choose the desired AMI when launching the spot instances.
Your desire to keep a certain number of spot instances running sounds like an excellent use-case for an Amazon EC2 Spot Fleet. A Spot Fleet allows you to specify the types of EC2 instances you would like to be available (instance type, AZ, price) and any instances terminated by Spot capacity can be automatically replaced with other spot instances of a different instance type, AZ, etc, that is not affected by the capacity issue that terminated the previous instances.
I am using AWS ECS in combination with EC2 instances.
Right now I am setting up Auto Scaling. How can I make sure that when, an EC2 instance gets terminated, all ECS tasks get migrated before the machine gets terminated?
Right now it is not automatically possible to achieve this. The best approach would be to have atleast 2 tasks running of each service, spread on different instances via a placement constraint.
Manually (or scripted) it is possible:
If you want to replace an instance attached to an ECS cluster, you can simply drain the instance. This will do the following
Start a new task of each running service on another instance in the cluster
Wait until the recently started task is 'steady'
shutdown the old task
To drain an instance using the AWS CLI, do the following:
Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.
In the navigation pane, choose Clusters and select the cluster.
Choose ECS Instances and select the check box for the container
instances.
Choose Actions, Drain instances.
After the instances are processed, choose Done.
This can also be done via the command line.
To do it automatically, you will need to add a lifecycle hook on termination.
Call the AWS CLI from the termination lifecycle hook to drain the instance, wait a fixed amount of time and then continue terminating the instance.
I'm launching an EC2 instance in AWS. And there is an auto-created EC2 instance, named Test-env. Every time I terminated it, AWS would auto-created a new one for me. is there any way to terminate the auto-created instance and prevent it from creating a new one?
Shown in the picture above, I terminated the Test-env twice, and AWS just created a new Test-env for me.
Thank a lot.
It seems like you have been using Elastic Beanstalk. If you open up that section of the AWS Management Console, you can delete the application/environment from there. This will bring down the instance as well. When you terminate the Elastic Beanstalk instance manually through the EC2 section, the system thinks that it has failed and will launch a replacement.
Reference: https://forums.aws.amazon.com/thread.jspa?threadID=115913
It seems that AutoScaling group configuration is present for the EC2 instance Test-env.
Can you please check under EC2 -> AutoScaling Groups -> Test-env -> Scaling Policies?
It seems that ScaleUp Policy has been configured for that instance. It means that whenever total number of EC2 instances are less than or equal to 0 (or any limit which was set) then spawn new one.
You need to update these policies or remove it as per your need.
Other than Elastic Beanstalk and AutoScaling group,
another possibility is you have
an on-going Spot Request.
You can cancel it in order to prevent the instance from recreating.
EC2 -> Instances -> Spot Requests
Then, Actions -> Cancel Request
I have a cluster in AWS EC2 Container Service. When I've set it up, I used t2.micro instances because those were sufficient for development. Now I'd like to use more powerful instances, like m4.large.
I would like to know whether it is possible to change the instance types only, so I don't need to recreate the whole cluster. I could not find how to do this.
Yes, you can achieve this in CloudFormation.
Click on the Stack corresponding to your ECS-Cluster.
Click Update Stack
Use radiocurrent template, Next
change EcsInstanceType
Next, Next, Update
Upscale your cluster to 2*n instances
Wait for the n new instances of the new type being created
Downscale your cluster to n
Or you could just drain and terminate the instances 1 by 1
Yes, this is possible.
The instance types in your cluster are determined by the 'Instance Type' setting within your Launch Configuration. To update the instance type without having to recreate the cluster:
Make a copy of the cluster Launch Configuration and update the 'Instance Type'.
Adjust the cluster Auto Scaling Group to point to your new Launch Configuration.
Wait for your new instances to register in your cluster and your services to start.
You can also add multiple instances types to a single cluster by creating multiple Auto Scaling Groups linked to different Launch Configurations. Note however that you can't copy Auto Scaling Groups easily within the console.
To do it without any downtime:
Create a copy of the Launch Configuration used by your Auto Scaling
Group, including any changes you want to make.
Edit the Auto Scaling Group to:
Use the new Launch Configuration
Desired Capacity = Desired Capacity * 2
Min = Desired Capacity
Wait for all new instances to become 'ACTIVE' in the ECS Instances tab of the ECS Cluster
Select the old instances and click Actions -> Drain Instances
Wait until all the old instances are running 0 tasks
Edit the Auto Scaling Group and change Min and Desired back to their original values
Here are the exact steps I took to update the instance type on my cluster:
Go to the cluster service, update Number of tasks to 0
Go to EC2 -> Launch Configurations -> Actions dropdown -> Copy launch configuration and set the new instance type
Go to EC2 -> Auto Scaling Groups -> Edit -> set Launch Configuration to newly created launch configuration
Go to EC2 -> Auto Scaling Groups -> Instances -> Detach instance
Go to EC2 -> Launch Configurations -> Delete old launch configuration
Go to the cluster service, update Number of tasks to your desired count.
Now when tasks start, it'll be running on the updated EC2 instance type.
This can be achieved by modifying EcsInstanceType in the CloudFormation stack for the ECS instance. Any change to the autoscaling group by hand will be overwritten by the next "Scale ECS Instances" operation.
Yes, you can change the instance type in ECS cluster. I believe you have created ECS cluster manually from AWS GUI. Behind the scene, its creating aws cloud formation template as per your inputs from AWS console(ECS) like VPC, instance type, and size, etc. Please follow the below steps for the same.
Find the cloud formation template with the name "EC2ContainerService-{your-ecs-cluster-name}".
Check the existing setting in the Parameters tab(you can check your instance type here).
Now you need to update the cloud formation. Click on-> Update ->use current template ->next->update the EcsInstanceType variable ->next->next->update stack.
Now your cloud formation update. now you can check in EC2 console that there is a new spot fleet with new instance type.
Definitely, there are multiple ways to change the instance type as suggested about using launch configurations.
But beware that, it is a challenge to use multiple launch configuration to attach to ECS cluster that has Container Instances Scaling policies.
For example, If one is running a cluster with t2.medium type of instances using a launch configuration and have a Auto scaling policy attached to ECS cluster then it can signal only Auto scaling group and not more than 1.
The AWS docs has a complete step by step guide covering CloudFormationStack and ECS cluster launched manually.
How do I change my container instance type in Amazon ECS?
From the guide:
To change your container instance type, complete the steps in one of
the following sections:
Update container instances launched in an ECS cluster through the AWS CloudFormation stack
Update container instances launched manually in an ECS cluster