I am pretty confused on how does auto scaling works with Capacity provider, i do know it creates a target tracking scaling policy in cloud watch which tracks the utilization of instances , but what happens when a Load Balancer health Check fails. Below is our infrastructure
ECS Cluster
ECS Service
ECS Task
ELB
Target Group
ASG(EC2)
We have Target group associated with ELB ( but we have not associated ASG with target group), we are using capacity provider and turned on Managed scaling .
What would be behavior incase of Load Balancer health check failure, how does capacity provider track this health check
Related
I created an NLB and a fargate service.
Then i create a target group with "ip" of my ecs instance.
When i now add a fargate ip to my target group, it works, but how does the scaling work? Suppose ecs has to scale out, i will have to register another ip, but i want it to scale automatically.
Let us say one task is added. How does the network load balancer the new task ip without me manually adding it?
I do not get, how the link is between the nlb and the service of ecs. Does amazon does add targets implicitly?
Instead of manually registering the IP of your Fargate task with the target group, you are supposed to configure the ECS service with knowledge of the load balancer you want to use. The ECS service will then automatically register every task that it creates as part of deployments and auto-scaling.
Is there a way like assigning a specific tag for the EC2 instances to automatically attached to the load balancer on AWS?
I believe I had done that in the past but unable to find that option now.
Since you say you've done it in the past, I believe you're thinking of a feature offered by EC2 auto-scaling groups (ASGs). ASG is a capability of the EC2 infrastructure that scales machine counts up and down based on workload or maintains a set number of healthy instances always running (destroying and replacing failed instances). When an ASG is attached to a load balancer, the instances controlled by the ASG are automatically registered and deregistered from the balancer.
Amazon EC2 Auto Scaling integrates with Elastic Load Balancing to enable you to attach one or more load balancers to an existing Auto Scaling group. After you attach the load balancer, it automatically registers the instances in the group and distributes incoming traffic across the instances.
https://docs.aws.amazon.com/autoscaling/ec2/userguide/attach-load-balancer-asg.html
I have my Beanstalk environment with a "Scaling Trigger" using "CPUUtilization" and it works well.
The problem is that I can not combine this with a system that automatically reboots (or terminate) instances that have been considered "OutOfService" for a certain amount of time.
Into the "Scaling > Scaling Trigger > Trigger measurement" there is the option of "UnHealthyHostCount". But this won't solve my problem optimally, because it will create new instances as far there is one unhealthy, this will provoque my environment to grow until the limit without a real reason. Also, I can not combine 2 "Trigger measurements" and I need the CPU one.
The problem becomes crucial when there is only one instance in the environment, and it becomes OutOfService. The whole environment dies, the Trigger measurement is never triggered.
If you use Classic Load Balancer in your Elastic Beanstalk.
You can go to EC2 -> Auto Scaling Groups.
Then change the Health Check Type of the load balancer from EC2 to ELB.
By doing this, your instances of the Elastic Beanstalk will be terminated once they are not responding. A new instance will be created to replace the terminated instance.
AWS Elastic Beanstalk uses AWS Auto Scaling to manage the creation and termination of instances, including the replacement of unhealthy instances.
AWS Auto Scaling can integrate with the ELB (load balancer), also automatically created by Elastic Beanstalk, for health checks. ELB has a health check functionality. If the ELB detects that an instance is unhealthy, and if Auto Scaling has been configured to rely on ELB health checks (instead of the default EC2-based health checks), then Auto Scaling automatically replaces that instance that was deemed unhealthy by ELB.
So all you have to do is configure the ELB health check properly (you seem to have it correctly configured already, since you mentioned that you can see the instance being marked as OutOfService), and you also have to configure the Auto Scaling Group to use the ELB health check.
For more details on this subject, including the specific steps to configure all this, check these 2 links from the official documentation:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.healthstatus.html#using-features.healthstatus.understanding
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environmentconfig-autoscaling-healthchecktype.html
This should solve the problem. If you have trouble with that, please add a comment with any additional info that you might have after trying this.
Cheers!
You can setup a CloudWatch alarm to reboot the unhealthy instance using StatusCheckFailed_Instance metric.
For detailed information on each step, go through the Adding Reboot Actions to Amazon CloudWatch Alarms section in the following AWS Documentation.
If you want Auto Scaling to replace instances whose application has stopped responding, you can use a configuration file to configure the Auto Scaling group to use Elastic Load Balancing health checks. The following example sets the group to use the load balancer's health checks, in addition to the Amazon EC2 status check, to determine an instance's health.
Example .ebextensions/autoscaling.config
Resources:
AWSEBAutoScalingGroup:
Type: "AWS::AutoScaling::AutoScalingGroup"
Properties:
HealthCheckType: ELB
HealthCheckGracePeriod: 300
See: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environmentconfig-autoscaling-healthchecktype.html
If the autoscaling group's health check type is set to ELB then it will automatically remove any instances that fail the ELB health checks ( set in the healthcheck URL )
As long as the configured health check properly reports than an instance is bad (which sounds like it is the case since you say ELB is marking the instance as unhealthy) this should work, but does this mean other autoscaling triggers like CPU Utilization ( set in Configuration->Scaling->Scaling Trigger) be ignored?
Autoscaling group will not health check to ELB and vice versa.
ELB will check the health status of registered EC2 instances. ELB will continuously ping EC2 instance with specific port and specific page example port 80 and index.html page for every time period say 30 seconds or 60 seconds.
If any one of the registered instance is unhealthy then ELB will not send traffic to those instances and will not terminate or stop EC2 instances. ELB continuously check health status of EC2 instances which is registered in ELB.
If an unhealthy instance become healthy then ELB will send traffic to an instance.
AutoScaling group will health check to EC2 instances same like ELB do. But in AutoScaling group, if an EC2 instance goes to stopped state then it will terminate from the group and launch new instances with same configurations.
If Autoscaling group is integrated with ELB, newly added instance in the group will be added to ELB dashboard.
Health check cannot be done with ELB. You can monitor ELB using AWS CloudWatch logs and upload to target S3 bucket. You can enable monitoring feature in ELB and provide your target S3 bucket to store the logs.
According to the AWS documentation, an ASG can be set to use an ELB to determine the health status of its instances.
If you have attached a load balancer to your Auto Scaling group, you
can optionally have Auto Scaling include the results of Elastic Load
Balancing health checks when determining the health status of an
instance. After you add these health checks, Auto Scaling also marks
an instance as unhealthy if Elastic Load Balancing reports the
instance state as OutOfService. For more information, see Adding
Health Checks to Your Auto Scaling Group.
I have an ASG with one instance with the healthcheck set to ELB. The ASG sizes are set to Desired 1, Min 0, Max 2. The healthcheck grace period is 0. I've configured the instance to fail the healthcheck on purpose so the ELB has marked the instance as unhealthy but the ASG will not replace the instance. It still marks it as in service. What am I missing?