Is there any way to make an instance attached to an ELB unhealthy purposefully using boto ?
I tried few methods and non of them working so far.
Thanks for any help !!
No, this is not possible. There is no AWS API call that can change the health status of an instance. (Auto Scaling has this capability, but not Load Balancing).
You could use the deregister_instances() API call, which would effectively achieve the same result.
The Register or Deregister EC2 Instances for Your Classic Load Balancer documentation says:
Deregistering an EC2 instance removes it from your load balancer. The load balancer stops routing requests to an instance as soon as it is deregistered. If demand decreases, or you need to service your instances, you can deregister instances from the load balancer. An instance that is deregistered remains running, but no longer receives traffic from the load balancer, and you can register it with the load balancer again when you are ready.
When you deregister an instance, Elastic Load Balancing waits until in-flight requests have completed if connection draining is enabled.
Yeah, We can do that in the below scenario.
Let's assume that you have loadblancer(myloadbalancer), an instance attached with it and PingPath configuration as such below.
Ping Protocol: HTTP
Ping Port: 80
Ping Path: /
Just add boto3 code to edit the health check configuration as below and you can see the magic(Instance OutOfService).
client.configure_health_check(
LoadBalancerName='myloadbalancer',
HealthCheck={
'Target': 'HTTP:80/hjkx',
'Interval': 30,
'Timeout': 5,
'UnhealthyThreshold': 5,
'HealthyThreshold': 3
}
)
Two other options:
1. Temporarily disable the web server / process that's responding to the health check. In our case, we were running Java webapps with and nginx proxy in front of it. Shutting down the nginx proxy made the health check fail while the Java app would still be running.
2. Temporarily firewall the port that the ELB uses to perform the health check on. You could do this via a call to the AWS api.
Related
I am using a reverse proxy in front of my load balancer. Currently I am just trying to make a TCP connection with LB from reverse proxy to check its health and if it succeeds then I will send the request to main load balancer. I want to check that whether my main load balancer have any servers running or not. If not I want to redirect those requests to another server fleet. Is there api or anything else which AWS load balancer exposes to tell the status of the its targets.
Go to the EC2 console, and then to the target groups section. Select your target group. From there, you should be able to see which instances are passing the healthcheck.
We have 3 EC2 Instances(Apache Web Server) running under AWS ELB, it sharing load correctly but whenever any of Web Server down i.e. Web1 having some issue i.e. Disk Full or Apache Crash then still ELB trying to send request to that server which is already not responding or don't have capacity to respond, hence user who is connected to that server are getting error.
Question : Is there way to identify Fail server and force ELB to stop passing request to failed server?
FYI: Auto Scaling is not enabled.
You need to configure health checks for your ELB. When the checks are failing, the elb will stop forwarding traffic to the unhealthy instance.
I create ECS service and it runs 1 ecs instance and I can see the instance is registered as a target of the load balancer.
Now I trigger a Auto Scaling Group (by just incrementing desired instance count) to launch a new instance.
The instance is launched and added to the ECS cluster. (I can see it on ECS instances tab)
But the instance is not added to the ALB target. (I expect to see 2 instances in the following image, but I only see 1)
I can edit AutoScalingGroup 's target group like the following
Then I see the following .
But the health check fails. It seems the 80 port is not reachable.
Although I have port 80 open for public in the security group for the instance. (Also, instance created from ecs service uses dynamic port mapping but instance created by ALS does not)
So AutoScalingGroup can launch new instance but my load balancer never gives traffic to the new instance.
I did try https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/?nc1=h_ls and it shows I can connect to port 80 from host to the docker container by something like curl -v http://${IPADDR}/health.
So it must be the case that there's something wrong with host port 80 (load balancer can't connect to it).
But it is also the case the security group setting is not wrong, because the working instance and this non working instance is using the same SG.
Edit
Because I used dynamic mapping, my webserver is running on some random port.
As you can see the instance started by ecs service has registered itself to target group with random port.
However instance started by ALB has registered itself to target group with port 80.
The instance will not be added to the target group if it's not healthy. So you need to fix the health check first.
From your first instance, your mapped port is 32769 so I assume if this is the same target group and if it is the same application then the port in new instance should be 32769.
When you curl the IP endpoint curl -I -v http://${IPADDR}/health. is the HTTP status code was 200, if it is 200 then it should be healthy if it's not 200 then update the backend http-status code or you can update health check HTTP status code.
I assume that you are also running ECS in both instances, so ECS create target group against each ECS services, are you running some mix services that you need target group in AS group? if you are running dynamic port then remove the health check path to traffic port.
Now if we look the offical possible causes for 502 bad Gateway
Dynamic port mapping is a feature of container instance in Amazon Elastic Container Service (Amazon ECS)
Dynamic port mapping with an Application Load Balancer makes it easier
to run multiple tasks on the same Amazon ECS service on an Amazon ECS
cluster.
With the Classic Load Balancer, you must statically map port numbers
on a container instance. The Classic Load Balancer does not allow you
to run multiple copies of a task on the same instance because the
ports conflict. An Application Load Balancer uses dynamic port mapping
so that you can run multiple tasks from a single service on the same
container instance.
Your created target group will not work with dynamic port, you have to bind the target group with ECS services.
dynamic-port-mapping-ecs
HTTP 502: Bad Gateway
Possible causes:
The load balancer received a TCP RST from the target when attempting to establish a connection.
The load balancer received an unexpected response from the target, such as "ICMP Destination unreachable (Host unreachable)", when attempting to establish a connection. Check whether traffic is allowed from the load balancer subnets to the targets on the target port.
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.
The target response is malformed or contains HTTP headers that are not valid.
The load balancer encountered an SSL handshake error or SSL handshake timeout (10 seconds) when connecting to a target.
The deregistration delay period elapsed for a request being handled by a target that was deregistered. Increase the delay period so that lengthy operations can complete.
http-502-issues
It seems you know the root cause, which is that port 80 is failing the health check and thats why it is never added to ALB. Here is what you can try
First, check that your service is listening on port 80 on the new host. You can use command like netcat
nv -v localhost 80
Once you know that the service is listening, the recommended way to allow your ALB to connect to your host is to add a Security group inbound rule for your instance to allow traffic from your ALB security group on port 80
I have configured app load balancer on amazon. Set up DNS LB to route 53 with alias for A. Behind LB i have 2 instances with IIS. If i set up 2 sites on both instances, balancer automatically balance client by rotation
(as i know round robin). But, if i turn off site on IIS in one instance, load balancer continue go to that instance and if i go to exapmle.com i will have one time worked site and if refresh the page i will have error (because site turned off in IIS). Could you please tell me, how can i set up load balance to route traffic in working instance if one of them not working. Thank you
Load balancers continue to distribute the traffic on healthy servers. If it is not happening in your case, I would recheck the health check configuration under Target Groups.
You need to modify the port/path so that health checks start failing once the site is turned off. Only then, the load balancer will pass all traffic to healthy host, not the unhealthy host
What does the LB health checks say? If the back-end instances are not listening on the health check port then LB marks it as unhealthy and stops forwarding requests to it. If you are using Application loadbalancer then I think you can get the health check status within the target groups associated with the loadbalancer.
When my Java application is deployed to Tomcat on Elastic-Beanastalk it takes a while (11 minutes) because it has to copy large data files from S3 and unzip them, but that is okay because this is all done in .ebextensions and the instance doesn't report itself ready until that is completed.
However, I have it configured for Autoscaling and it seems that when it decides it needs to start a new instance there is a period before the next instance has fully deployed that Elastic-Beanstalk will direct some application requests to this new server, of course because it is not ready it returns a 503 error.
But surely all calls should only go to the original instance until the second one is ready, has anyone else noticed this ?
Whether requests are directed to the new instance or not is decided by the Elastic Load Balancer (ELB). Your autoscaled instances are behind the ELB and ELB performs periodic health checks on your EC2 instances to decide whether traffic to your instances or not. By default the health check is TCP connect on port 80. So if ELB can establish a connection to port 80 on the Tomcat server, it will start sending traffic to the instance even before it is actually "ready".
The solution is to use a custom HTTP health check instead of the default TCP check. Set up your web app to return a 200 OK on a special path say '/health_ping'. Then configure the "Application Healthcheck URL" option to "/health_ping". You can do this using the following ebextension.
Create a file called .ebextensions/01-health-check.config in your app source with the following contents. Then deploy it to your environment.
option_settings:
- namespace: aws:elasticbeanstalk:application
option_name: Application Healthcheck URL
value: /health_ping
Read more about this option setting here.
You can also configure this in the web console or using the aws cli.