I have an Autoscaling Group behind an Application Load Balancer. Both of them are configured to use 2 subnets for 2 different AZ.
The Autoscaling Group is set to provision only 1 instance (min:1, max: 1, desired: 1)
Now in case of AZ failure the Load balancer will detect that the instance is Unhealthy and stop sending tarfic to it.
But i assume that the Autoscaling Group will consider the instance as Healthy, so it will not create a new instance in the second AZ.
So my question is there a way to make the Loadbalancer to notify the autoscaling group to terminate the instance in the failed AZ and to start a new one in the ok AZ ?
I now that the best practice is to set the Autoscaling group to provision for example 2 instances, one in each AZ. But is there a way to make it work with only 1 instance ?
Thanks.
ASG has two options for health checks. By default, it uses EC2 health checks. But since you are using ALB, you should set ASG to use ALB health checks (in addition to EC2 health checks).
This way if ALB determines that your instance is unhealthy for whatever reason, ASG will also consider is as unhealthy.
There are also custom health checks, but in your case, using ALB health checks should be sufficient.
More is here: https://docs.aws.amazon.com/autoscaling/ec2/userguide/healthcheck.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html
Related
We know that in order to attach an Application/Network load balancer for an auto-scaling group, we need to attach the target group(s) to the existing Auto-scaling group in order to distribute the load. In the case of the Classic Load Balancer, we directly attach the load balancer to the ASG but in case of an ALB, we do it through the target groups consisting of targets which can be either an instance, IP or a lambda function.
My question is, are the instances in these target groups defined for an ALB and the instances in ASG different? Or are they the same? If they are the same, then do we have to create a new target group again with targets defined as the EC2 instances in ASG? I can't understand the mapping of instances in target-groups to the instances in ASG. I went through the documentation for attaching a load balancer to an existing ASG, but all it says is to select an existing target group.
Are the instances in these target groups defined for an ALB and the instances in ASG different? Or are they the same?
They are the same.
ELB -> Target Group -> EC2 (without ASG)
ELB -> Target Group -> ASG -> EC2 (with ASG).
Please read the link below for details.
If they are the same, then do we have to create a new target group
again with targets defined as the EC2 instances in ASG
This quote from the AWS document. We need to add ASG to ELB, then AWS takes care of the rest.
When you use your Elastic Load Balancing load balancer with an Auto
Scaling group, it's not necessary to register your EC2 instances with
the load balancer or target group. With Elastic Load Balancing,
instances that are launched by your Auto Scaling group are
automatically registered with the load balancer or target group, and
instances that are terminated by your Auto Scaling group are
automatically deregistered from the load balancer or target group
Ref: https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-load-balancer.html
Assuming I have two load balancers which point to the same server group but are targeting different services (e.g. web and an internal queue). Both are set to Health Check Type = ELB.
According to the AWS documentation if an instance becomes unhealthy it will be terminated and replaced.
What happens if an instance becomes unhealthy in regards to only one of the two ELBs, will it still be terminated and replaced?
Yes.
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-add-elb-healthcheck.html
Note that if you attach multiple load balancers to an Auto Scaling group, all of them must report that the instance is healthy in order for it to consider the instance healthy. If one load balancer reports an instance as unhealthy, the Auto Scaling group replaces the instance, even if other load balancers report it as healthy.
If the autoscaling group's health check type is set to ELB then it will automatically remove any instances that fail the ELB health checks ( set in the healthcheck URL )
As long as the configured health check properly reports than an instance is bad (which sounds like it is the case since you say ELB is marking the instance as unhealthy) this should work, but does this mean other autoscaling triggers like CPU Utilization ( set in Configuration->Scaling->Scaling Trigger) be ignored?
Autoscaling group will not health check to ELB and vice versa.
ELB will check the health status of registered EC2 instances. ELB will continuously ping EC2 instance with specific port and specific page example port 80 and index.html page for every time period say 30 seconds or 60 seconds.
If any one of the registered instance is unhealthy then ELB will not send traffic to those instances and will not terminate or stop EC2 instances. ELB continuously check health status of EC2 instances which is registered in ELB.
If an unhealthy instance become healthy then ELB will send traffic to an instance.
AutoScaling group will health check to EC2 instances same like ELB do. But in AutoScaling group, if an EC2 instance goes to stopped state then it will terminate from the group and launch new instances with same configurations.
If Autoscaling group is integrated with ELB, newly added instance in the group will be added to ELB dashboard.
Health check cannot be done with ELB. You can monitor ELB using AWS CloudWatch logs and upload to target S3 bucket. You can enable monitoring feature in ELB and provide your target S3 bucket to store the logs.
According to the AWS documentation, an ASG can be set to use an ELB to determine the health status of its instances.
If you have attached a load balancer to your Auto Scaling group, you
can optionally have Auto Scaling include the results of Elastic Load
Balancing health checks when determining the health status of an
instance. After you add these health checks, Auto Scaling also marks
an instance as unhealthy if Elastic Load Balancing reports the
instance state as OutOfService. For more information, see Adding
Health Checks to Your Auto Scaling Group.
I have an ASG with one instance with the healthcheck set to ELB. The ASG sizes are set to Desired 1, Min 0, Max 2. The healthcheck grace period is 0. I've configured the instance to fail the healthcheck on purpose so the ELB has marked the instance as unhealthy but the ASG will not replace the instance. It still marks it as in service. What am I missing?
Is there any way to have either ELB or an EC2 auto-scaling group terminate (or reboot) unhealthy instances from ELB?
There are some specific database failure conditions in our front end which makes it turn unhealthy, so the ELB will stop routing traffic to it. That instance is also part of an auto-scaling group, which scales on the group's CPU Load. So, what ends up happening is that the instance no longer gets traffic from ELB, so it has no CPU load, and skews the group's CPU load, thus screwing up the scaling conditions.
Is there an "easy" way to somehow configure ELB or an autoscaling group to automatically terminate unhealthy instances from the group without actually having to write code to do the polling and terminating via the EC2 API?
If you set the autoscaling group's health check type to ELB then it will automatically retire any instances that fail the ELB health checks (ie doesn't respond in a timely manner to the URL configured)
As long as the configured health check properly reports than an instance is bad (which sounds like it is the case since you say ELB is marking the instance as unhealthy) this should work