I have 5 instances in ELB and in Autoscaling Group.
I am using ansible rolling display so it shuts down old instance and create new instance.
My Problem is that i have to configure instance after creation and it almost takes 20 mins after creation that i can fully use instance.
I have Health check page /apps/status
WHats happenning is aws joins all new instances and replace old ones but all new instances are in state of OutOFService and it takes further 15 mins to become active.
Is there any way that AWS only replcaes new instance when health check is ok and then start with other instance
Make the Health Checks for your ELB setting to this value in the screenshot. You will save lots of time and the instance will be available more fast. With AWS ELB the options for customization becomes limited and for sure you cannot make the ELB select which instance to choose first so my solution will reduce your latency issue to great extend.
Check this description for AWS ELB your load balancer will automatically perform health checks on your EC2 instances and only route traffic to instances that pass the health check. If an instance fails the health check, it is automatically removed from the load balancer. Customize the health check to meet your specific needs. So health check customization is the only option that open to us when it comes to EC2 instance selection.
You will want to use lifecycle hooks. See docs.aws.amazon.com/autoscaling/latest/userguide/lifecycle-hooks.html
Related
I need to perform simple health checks to an ec2 instance that does not have access to the internet. The instance is behind another ec2 using Apache as the frontend.
I can not use an load balancer, nor give the instance access to the internet.
I looked at route53 health checks as an alternative, but it also needs internet connection.
I know I can do it by using a lambda function, but I would like to know if there is any other ( 'aws managed' ) way to do it.
What are you trying to check on the instance? EC2 instances come with status checks by default for general health.
If you want to check something specific, you might run a script on the instance (e.g. through cron) and use AWS CLI (or a similar API) to report the metrics to CloudWatch; you can also set alarms here.
Why not use a load balancer for the health check and just not route any traffic to it? Make sure your security groups allow traffic from the load balancer (assuming ALB) to the EC2 instance, but you can remove any inbound access to the load balancer's security group for added security.
I have my Beanstalk environment with a "Scaling Trigger" using "CPUUtilization" and it works well.
The problem is that I can not combine this with a system that automatically reboots (or terminate) instances that have been considered "OutOfService" for a certain amount of time.
Into the "Scaling > Scaling Trigger > Trigger measurement" there is the option of "UnHealthyHostCount". But this won't solve my problem optimally, because it will create new instances as far there is one unhealthy, this will provoque my environment to grow until the limit without a real reason. Also, I can not combine 2 "Trigger measurements" and I need the CPU one.
The problem becomes crucial when there is only one instance in the environment, and it becomes OutOfService. The whole environment dies, the Trigger measurement is never triggered.
If you use Classic Load Balancer in your Elastic Beanstalk.
You can go to EC2 -> Auto Scaling Groups.
Then change the Health Check Type of the load balancer from EC2 to ELB.
By doing this, your instances of the Elastic Beanstalk will be terminated once they are not responding. A new instance will be created to replace the terminated instance.
AWS Elastic Beanstalk uses AWS Auto Scaling to manage the creation and termination of instances, including the replacement of unhealthy instances.
AWS Auto Scaling can integrate with the ELB (load balancer), also automatically created by Elastic Beanstalk, for health checks. ELB has a health check functionality. If the ELB detects that an instance is unhealthy, and if Auto Scaling has been configured to rely on ELB health checks (instead of the default EC2-based health checks), then Auto Scaling automatically replaces that instance that was deemed unhealthy by ELB.
So all you have to do is configure the ELB health check properly (you seem to have it correctly configured already, since you mentioned that you can see the instance being marked as OutOfService), and you also have to configure the Auto Scaling Group to use the ELB health check.
For more details on this subject, including the specific steps to configure all this, check these 2 links from the official documentation:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.healthstatus.html#using-features.healthstatus.understanding
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environmentconfig-autoscaling-healthchecktype.html
This should solve the problem. If you have trouble with that, please add a comment with any additional info that you might have after trying this.
Cheers!
You can setup a CloudWatch alarm to reboot the unhealthy instance using StatusCheckFailed_Instance metric.
For detailed information on each step, go through the Adding Reboot Actions to Amazon CloudWatch Alarms section in the following AWS Documentation.
If you want Auto Scaling to replace instances whose application has stopped responding, you can use a configuration file to configure the Auto Scaling group to use Elastic Load Balancing health checks. The following example sets the group to use the load balancer's health checks, in addition to the Amazon EC2 status check, to determine an instance's health.
Example .ebextensions/autoscaling.config
Resources:
AWSEBAutoScalingGroup:
Type: "AWS::AutoScaling::AutoScalingGroup"
Properties:
HealthCheckType: ELB
HealthCheckGracePeriod: 300
See: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environmentconfig-autoscaling-healthchecktype.html
I've been trying to configure a Cloudformation template for ECS along with Application Load Balancer (ALB) with dynamic ports.
Does the AutoScalingGroup's (ASG) health check type need to be EC2? The examples seem to use EC2 and when I set it to ELB the health check seems to fail.
If it does indeed need to be set to EC2 then does ECS manage the health of the containers itself and the ALB only manages the health of the container instances and not the containers?
Edit:
Having thought about this a bit more it probably makes sense to use EC2 health check since if I had multiple containers on the container instance then one unhealthy container shouldn't cause the whole container instance to go down. However if the ALB only monitors the instance then does ECS monitor the health of the containers?
Googling my question I came across this AWS blog but it references using ELB for health checks...
Your Auto Scaling Group health check is independent of the ECS/loadbalancer monitoring. I'm not exactly sure which health check setting of your ASG you mean for health checks.
In any case, for your ECS monitoring to be aware of the health of your container, you'll want to set the health check settings on your target groups that are connected to your services. ECS will use the information that's visible in the target group to kill containers that are not considered healthy.
The templates here are great:
http://templates.cloudonaut.io/en/stable/ecs/
The ECS templates for the cluster and on top of it the service include everything you need including auto-scaling, load-balancing, health-checks, you name it..
They require a bit of tweaking but they should get you started well even out of the box.
Pay attention to the stack dependencies. Before running the ecs service template, you need to install the stacks for vpc, vpc-s3-endpoint, alert,
nat-gateway (if you're building a service confined to private subnets), and the cluster layer itself.
Have fun!
I have created two instances in AWS (one is Live & other is Backup). My website is hosted on Live Instance. I have configured Route 53, Health checks & Hosted zones on default settings. Also have added both Instances to load balancer, and the status is "InService" for both the Instances.
For the Live Instance, Public IP & Elastic IP are the same. For Backup Instance, Public IP is different from live, and Elastic IP is null.
What I want to achieve is, when my Live Instance "status check" or "Health check" fails, then Backup Instance should get activated.
Currently when I manually stop my Live Instance for testing purpose, the backup Instance should get activated. but it doesn't. Please let me know if I am missing any steps.
You need to implement a healthcheck REST API that ELB can call. Your backup instance can return a non-200 HTTP status. The moment it is activated it starts return HTTP 200. This will tell ELB to only route calls to the primary.
Meanwhile, your Route53 should point at the ELB, not directly at the instances.
Generally speaking, however, you want to keep both primary and backup instances hot for optimal performance and failover. You can get the best of AWS advantages if you don't rely on your instances to be in a particular state -- primary vs backup in this case. I would devise a strategy to keep both instances in use.
We are using CodeDeploy to load code onto our instances as they boot up. Our intention was that they would not be added to the LB prior to the code being loaded. To do this, we set a health check which looked for one of the files being deployed. What we have found is that some times instances without code are created (I assume code deploy failed) and these instances are staying in the LB even when marked unhealthy? How is this possible? Is this related to the grace period? Shouldn't instances that are unhealthy be removed automatically?
I believe I have found a large part of my problem: My Auto-scale group was set to use EC2 health checks and not my ELB health check. This resulted in the instance not being terminated. The traffic may have continued to flow longer to this crippled instance due to the need the need for a very long unhealthy state before having traffic completely stopped.