/* My Configuration */
EC2 Console limit : All Standard (A, C, D, H, I, M, R, T, Z) Spot Instance Requests 128vCPU
Region : ap-northeast-2
Current Instance count : 0
Currently, I've no instances running on AWS (all spot instance terminated).
And I using ASG(Auto Scaling Group) with Instance Launch Template (No spot option).
When I try to launch a new instance with Instance Launch Template, it's OK. (instance type : c5.large)
But when I try to launch a new spot instance with Instance Launch Template (Check spot option),
it tells me that "Max spot instance count exceeded".
ASG Activity)
Status : Failed
Description : Launching a new EC2 instance. Status Reason: Max spot instance count exceeded. Launching EC2 instance failed.
Help me please...
Related
I am deploying a queue processing ECS service using EC2 deployment using CDK. Here is my stack
from aws_cdk import core, aws_ecs_patterns, aws_ec2, aws_ecs
class EcsTestStack(core.Stack):
def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
_container_image = aws_ecs.ContainerImage.from_asset(
directory=".",
file='Dockerfile.ECS_Test',
exclude=["cdk.out"]
)
_scaling_steps = [{"upper": 0, "change": -1}, {"lower": 50, "change": +1}, {"lower": 100, "change": +2}]
_vpc = aws_ec2.Vpc(self, "ecs-test-vpc-v4", max_azs=3)
_cluster = aws_ecs.Cluster(self, "ecs-test-cluster-v4", vpc=_vpc)
_cluster.add_capacity("ecs-autoscaling-capacity-v4",
instance_type=aws_ec2.InstanceType("t2.small"),
min_capacity=1,
max_capacity=3)
self.ecs_test = aws_ecs_patterns.QueueProcessingEc2Service(
self,
"ECS_Test_Pattern_v4",
cluster=_cluster,
cpu=512,
scaling_steps=_scaling_steps,
memory_limit_mib=256,
image=_container_image,
min_scaling_capacity=1,
max_scaling_capacity=5,
)
The stack starts out with 1 task in the service and 1 EC2 instance. Based on my _scaling_steps, a new task should be added to the service when the number of messages in the queue > 50 and 2 new tasks should be added to the service. The stack starts out with 1 task in the service and 1 EC2 instance.
But when I add 200 new messages to the queue, I can see 1 new task added to my service and then I get this error message in the even.
service
EcsTestStackV4-ECSTestPatternv4QueueProcessingService5C84D200-c00N6R56hB0p
was unable to place a task because no container instance met all of
its requirements. The closest matching container-instance
81e5380c718c4b558567dc6cc1fb1926 has insufficient CPU units available.
I also notice that no new EC2 instances were added.
Question: how do I get more EC2 instances added to my cluster when the service scales up?
I can see 1 new task added to my service and then I get this error message in the even.
This is because t2.small has 1000 CPU units. So your two tasks take all of them, and there is no other instances to place your extra task on.
I also notice that no new EC2 instances were added.
You set min_capacity=1 so you have only instance. The _scaling_steps are for the tasks only, not for your instances in autoscaling group. If you want more instance you have to set min_capacity=2 or whatever value you want.
I guess you thought that QueueProcessingEc2Service scales both instances and tasks. Sadly this is not the case.
UPDATED
Following the AWS instance scheduler I've been able to setup a scheduler that starts and stops at the beginning and end of the day.
However, the instances keep being terminated and reinstalled.
I have an Amazon Elastic Kubernetes Service (EKS) that returns the following CloudWatch log:
discovered the following log in my CloudWatch
13:05:30
2019-11-21 - 13:05:30.251 - INFO : Handler SchedulerRequestHandler scheduling request for service(s) rds, account(s) 612681954602, region(s) eu-central-1 at 2019-11-21 13:05:30.251936
13:05:30
2019-11-21 - 13:05:30.433 - INFO : Running RDS scheduler for account 612681954602 in region(s) eu-central-1
13:05:31
2019-11-21 - 13:05:31.128 - INFO : Fetching rds Instances for account 612681954602 in region eu-central-1
13:05:31
2019-11-21 - 13:05:31.553 - INFO : Number of fetched rds Instances is 2, number of schedulable resources is 0
13:05:31
2019-11-21 - 13:05:31.553 - INFO : Scheduler result {'612681954602': {'started': {}, 'stopped': {}}}
I don't know if it is my EKS that keeps rebooting my instances, but I really would love to keep them stopped until the next day.
How can I prevent my EC2 instances from automatically rebooting every time one has stopped? Or, even better, how can I deactivate my EKS stack automatically?
Update:
I discovered that EKS has a Cluster Autoscaler. Maybe this could be where the problem lies?
https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html
EKS node group would create an auto scaling group to manage the worker nodes. You need specify the minimum, maximum and desired size of worker nodes. Once any instance is stopped, the auto scaling group would create new instance to match the desired instance size.
Check below doc for details,
https://docs.aws.amazon.com/eks/latest/userguide/launch-workers.html
It is possible to configure an autoscaling group to use Launch Template instead of Launch Configuration. I want to use it because when I configure an autoscaling group using Launch Configuration I can not set mixed instances policy (ondemand + spot instances).
I have created launch template (name 'test') with such configuration:
AMI Ubuntu 16,
some security groups and IAM Role with EC2, AutoScaling and CodeDeploy full access
and all other settings in default *Not specified* state.
Launch Template ID lt-0e1111
Default Version Number 1
Launch Template Name test
Owner arn:aws:iam::11111111:user/itsp
Version 1 (Default)
Name test
Created by arn:aws:iam::179961111910:user/itsp
Description Not specified
Date Created February 8, 2019 at 12:47:54 AM UTC+3
AMI ID ami-0bf1924ed30c48a6a
Availability Zone Not specified
Security Groups Not specified
Instance Type t2.nano
Key pair name itsp
Security Group Ids sg-07e746cf0e4935b84,sg-0a45f7e6ac321614d,sg-27bf6d4b
Advanced Details:
Purchasing option Not specified
Spot request type Not specified
Spot interruption behavior Not specified
IAM Instance Profile Not specified
Stop - Hibernate behavior Not specified
Monitoring Not specified
EBS-optimized instance Not specified
Tenancy Host ID Not specified
RAM Disk ID Not specified
User Data Not specified
Elastic Inference Accelerator Not specified
Maximum Spot price Not specified
Spot expiry date -
Spot block duration Not specified
Shutdown behavior Not specified
Termination Protection Not specified
Placement Group Name Not specified
Tenancy Not specified
Tenancy Affinity Not specified
Kernel ID Not specified
Elastic Graphics Not specified
Then I am creating an AutoScaling group with that launch template. Here is my autoscaling group configuration:
Launch Template test
Launch Template Version 1
Launch Template Description -
Instance Types -
Spot Diversity 0
Optional On-Demand Base 0
On-Demand Percentage 0%
Desired Capacity 1
Min 1
Max 1
Availability Zone(s) eu-central-1a
Subnet(s) subnet-53075e38
Classic Load Balancers -
Target Groups -
Health Check Type EC2
Health Check Grace Period 300
Instance Protection -
Termination Policies Default
Suspended Processes -
Placement Groups -
Default Cooldown 300
Enabled Metrics -
Creation Time Fri Feb 08 00:50:16 GMT+300 2019
Service-Linked Role arn:aws:iam::11111111111:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling
Such configuration appears to be set when I choose
*Fleet Composition - Adhere to the launch template
The launch template determines the instance type and purchase option (On-Demand or Spot).*
option and my lauch template configuration is described above.
So there are no spots in my autoscaling group.
Then I configure a CodeDeploy application 'Application' and DeploymentGroup 'DG_TEST' inside of it with such configuration:
Deployment type Blue/green
Service role ARN arn:aws:iam::11111111111111:role/CodeDeployRole
Deployment configuration CodeDeployDefault.AllAtOnce
Rollback enabled -
Environment configuration: Amazon EC2 Auto Scaling groups -> Name 'test'
Ok, after everything is set up I set desired instances count to 1 in my Autoscaling group and new on-demand instance spawns in my 'test' group.
I am now starting a blue-green deployment with AWS CLI command:
aws deploy create-deployment --application-name Application --s3-location bucket=codedeploybucket,key=application,bundleType=zip --deployment-group-name DG_TEST
And I can see new autoscaling group with same lauch template appears in my green fleet.
So we just checked all our services are working as expected.
Now I want to tweak my autoscaling group configuration a little to mix ondemand and spot instances in it:
Fleet Composition - Combine purchase options and instances
Choose a mix of On-Demand Instances and Spot Instances and multiple instance types. Spot Instances are automatically launched at the lowest price available.
Instance Types:
t2.nano (1vCPUs, 0.5GiB)
t2.small (1vCPUs, 2GiB)
t2.micro (1vCPUs, 1GiB)
Maximum Spot Price - Use default (recommended)
Spot Allocation Strategy - Diversify Spot Instances across your 2 lowest priced instance types per Availability Zone
Optional On-Demand Base - Designate the first 0 instances as On-Demand
On-Demand Percentage 50% On-Demand and 50% Spot
Desired Capacity 1
Min 1
Max 1
I save it and try same AWS CLI command
aws deploy create-deployment --application-name Application --s3-location bucket=codedeploybucket,key=application,bundleType=zip --deployment-group-name DG_TEST
And now I have my deployment failed with such comment:
The following validation error occurred: Valid requests must contain either LaunchTemplate, LaunchConfigurationName, InstanceId or MixedInstancesPolicy parameter. (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError; Request ID: 11111114-2b2b-11e9-93b7-11111111111)
Another strange thing is that if I configure my Launch Templae to use spots only:
Launch Template ID lt-00dc4c187d1111159
Default Version Number 1
Launch Template Name test2
Owner
arn:aws:iam::179961111910:user/itsp
Version 1 (Default)
Name test2
Created by arn:aws:iam::179961111910:user/itsp
Description Not specified
Date Created February 8, 2019 at 1:14:13 AM UTC+3
AMI ID ami-0bf1924ed30c48a6a
Availability Zone Not specified
Security Groups Not specified
Instance Type t2.micro
Key pair name itsp
Security Group Ids sg-0a45f7e6ac321614d,sg-07e746cf0e4935b84,sg-27bf6d4b
Advanced details
Purchasing option spot
Spot request type one-time
Spot interruption behavior Not specified
IAM Instance Profile CodeDeployRole (arn:aws:iam::1111111111:instance-profile/CodeDeployRole)
Stop - Hibernate behavior Not specified
Monitoring Not specified
EBS-optimized instance Not specified
Tenancy Host ID Not specified
RAM Disk ID Not specified
User Data Not specified
Elastic Inference Accelerator Not specified
Maximum Spot price Not specified
Spot expiry date -
Spot block duration Not specified
Shutdown behavior Not specified
Termination Protection Not specified
Placement Group Name Not specified
Tenancy Not specified
Tenancy Affinity Not specified
Kernel ID Not specified
Elastic Graphics Not specified
and use that option in my autoscaling group:
*Fleet Composition - Adhere to the launch template
The launch template determines the instance type and purchase option (On-Demand or Spot).*
Then after same deployment command I receive another error in CodeDeploy (failed deployment reason):
The IAM role arn:aws:iam::179966034910:role/CodeDeployRole does not give you permission to perform operations in the following AWS service: AmazonAutoScaling. Contact your AWS administrator if you need help. If you are an AWS administrator, you can grant permissions to your users or groups by creating IAM policies.
And that is definitely not the case because my role for CodeDeploy have full EC2, Autoscaling and CodeDeploy access.
Am I doing something wrong or that are bugs? How can I configure mixed autoscaling group (ondemand+spot instances) with codedeploy using launch templates? Thanks in advance!
I faced a similar issue when working with CodeDeploy and LaunchTemplates.
It looks like LaunchTemplates are still not supported by CodeDeploy.
We have to stick to LaunchConfigurations. They still don't support versioning, but for the most part, they are pretty much the same :/
I could't find where AWS has documented that little incompatibility issue when migrating LaunchConfigurations to LaunchTemplates... but could find here more info.
Sheers!
I am trying to test auto scale behavior in amazon ECS. Here is how i tested
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
public class LoadHttp {
private static final String GET_URL = "http://tournament-ecs-weather-lb-1526102172.ap-south-1.elb.amazonaws.com/status";
public static void main(String[] args) throws IOException {
System.out.println("GET DONE");
while(true){
sendGET();
}
}
private static void sendGET() throws IOException {
Thread t = new Thread(() -> {
URL obj;
try {
obj = new URL(GET_URL);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setConnectTimeout(5000);
int responseCode = con.getResponseCode();
System.out.println(responseCode);
} catch (Exception e) {
e.printStackTrace();
}
});
t.start();
}
}
and configuration of auto scale of service that i am running
here is my auto scale group configuration
I am using ECS Service to create cluster and my service is running in docker container. Still after executing my test code i found my service is not getting autoscaled. Number of container instance is still one. If you need additional information I will be happy.
If I understood your question, the service is autoscaled when there are not enough resources available in the current instance to handle the requests.
In your particular case, you have set the Min value of the instances as 0 which tells that the system should react fine even there is no instance running. You need to have at least 1 for the min value to make the whole cluster run ok. I believe that for this reason your autoscaling is not launching any instance.
Also, I would suggest you to add the Elastic Load balancer to your autoscaling group which launch asks autoscaling group to launch more instances when the minimum number of instances specified in your autoscaling are not healthy.
Lastly, to test if the autoscaling group is working or not, you check the availability zone of your instance and note that down. Now terminate your instance. Since the running instance is terminated the autoscaling group should automatically launch an instance in different availability zone. If it launches new instance in different availability zone than the terminated instance then it works ok.
Hope this will help.
I want to create an ASG with only 1 instance initially.
I want all the instances of this ASG to be behind an ELB.
So I set load_balancers = ["${aws_elb.Production-Web-ELB.name}"] in the resource "aws_autoscaling_group" "ProductionWeb-ScalingGroup" .
Now, when I write the code for the resource "aws_elb" "Production-Web-ELB", and I set instances = ["${aws_autoscaling_group.ProductionWeb-ScalingGroup.*.id}"], I get the error...
Error configuring: 1 error(s) occurred:
* Cycle: aws_autoscaling_group.ProductionWeb-ScalingGroup, aws_elb.Production-Web-ELB
I understand that this error means that the one resource references the other in a circle. To check it I comment out the load_balancers = ["${aws_elb.Production-Web-ELB.name}"] part and terraform plan without any error.
So my question is: Am I unable using Terraform to create an ASG with an attached ELB and every EC2 that will spawn inside it will be automatically behind the ELB ?
Is there something from the documentation that I missed?
Is there a workaround?
You don't need to explicitly define the instances that will be associated with the ELB in terraform's ELB definition. By using the load_balancers argument, you're associating the ELB with the AutoScaling group, and AutoScaling will know to attach any instances that are created to that ELB when the AutoScaling group launches that instance.
Terraform isn't directly managing the state of the instances in this case -- AWS AutoScaling is, so their state likewise don't need to be defined in terraform beyond defining a launch configuration and associating it to the AutoScaling group.
To tell terraform to launch the AutoScaling group with a single instance, set your min_size argument to 1 and let your scaling policies handle the desired capacity from there. You could alternatively set desired_capacity to 1, but be wary of managing that state in terraform because it will set the desired_capacity to 1 every time you apply your plan.