I have an ALB with 2 targets in the target group. However my health check aren't working properly. Both are showing
"Health checks failed with these codes: [404]"
My settings for the health check path are:
/var/www/html/generic/website.com/healthcheck.php
and if I do a nano /var/www/html/generic/website.com/healthcheck.php on the ec2 instance it shows this which should be all the health check needs I think.
<?php
header("Status: 200");
?>
I double checked the AZ and the ALB is in the same one and subnets as the 2 instances.
Also when I check my apache logs this is what I see:
"GET /var/www/html/generic/website.com/healthcheck.php HTTP/1.1" 404 196 "-" "ELB-HealthChecker/2.0"
What am I doing wrong that is making the healthcheck fail?
Seems you can't use a namebased vhost for a healthcheck. So in my vhost file I added the following code. What it does is if someone goes straight to your ip it will give them a 404 but if they go to your ip/healthcheck then it will show a 200 which is what ALB needs. Then in your path just put /healthcheck.
<VirtualHost *:80>
ServerName default
RewriteRule "/healthcheck" - [R=200]
Redirect 404 /
</VirtualHost>
<VirtualHost _default_:80>
RewriteRule "/healthcheck" - [R=200]
Redirect 404 /
</VirtualHost>
The ALB healthcheck is going to be accessing your healthcheck.php via the web interface and has no concept of where the files exist on the file system. This value needs to be the URI path after the hostname.
Configuring the healthcheck to be /var/www/html/generic/website.com/healthcheck.php is equivalent to telling AWS to check http://website.com//var/www/html/generic/website.com/healthcheck.php which is probably not your intention.
Assuming the actual path you want to check is more like http://website.com/healthcheck.php, update your healthcheck path to simply be /healthcheck.php and this should hopefully work for you.
Since you have multiple named hosts in nginx. What you could try is setting the healthcheck on the / with 200 OK as the status code. it should return the default nginx page with status code 200 OK.
Another method is, you can add healthcheck url that doesn't exist such as /my-health-page and in the health check configuration set the expected health check status code to be 404. So what we are assuming here is, if the nginx is inspecting url and return 404 NOT FOUND,the Nginx is working. But this method wont be checking anything more than that.
Pros:
ALB is able to whether your instance is running
ALB can check whether nginx is running
Cons:
The health check wont identify the cases for example php not working or mysql server is not reachable from the application etc
Related
I am using express.js and graphql for backend server which uses port 4000. I am trying to connect my server with ELB but the status returns unhealthy with error code 400 or 404. These are my settings about ELB.
Target group
Target groups registered targets
Target groups health checks
If I change the path to /graphql it returns 400, and just / returns 404. It is still working with no error when I call the API but it seems like I should fix it. Could anyone please tell me what can I do?
The standard way to handle this case is to define the path in your express application for example /health or /ping which should return HTTP status code 200.
If /graphql does not return HTTP status code 200 the LB will mark the target unhealthy.
you have two option
Update Success codes to 400, the LB will mark the target healhty
Define path like / or /health and return status code 200.
In express for example
app.get('/health',function(req,res){
res.status(200).send("instance is healthy");
})
Then setting for LB will
Or if you want to go without changing in the application the below will work.
Apollo supports a simple health check by default at this URL /.well-known/apollo/server-health (source), so you can add change the path in the target group from / to the URL
I have a Django application running on Elastic Beanstalk. I can visit my site no problem at example.com. I've set up automatic https redirect, so that it always directs to https. I've set it up so you can't view the site example.elasticbeanstalk.com domain -- if you go there you end up getting response code 400.
My auto scaling group is load balanced. My app is failing the health checks with status code 400, even though I can navigate to my site no problem with response code 200. My logs show:
***amazon IP*** (-) - - [date] "GET / HTTP/1.1" 400 26 "-" "ELB-HealthChecker/2.0"
I'm guessing the error is either from
Not allowing connection at example.elasticbeanstalk.com
Haivng automatic HTTP -> HTTPS redirect (although that would come up with a 302 I'd guess)
When the Health Check pings a site, is it pinging your custom domain (example.com) or is pining the elasticbeanstalk.com domain? What can I do to either fix this or further diagnose the error? I'd rather not allow traffic at the elasticbeanstalk.com domain, because I don't think I can get SSL on that.
The reason this is failing is because the health check checks the EC2 instance private IP. This can change with ELB, so you need to dynamically get the private IP of the instance and add it to hosts. See How to dynamically add EC2 ip addresses to Django ALLOWED_HOSTS
import requests
EC2_PRIVATE_IP = None
try: EC2_PRIVATE_IP = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4', timeout=0.01).text
except requests.exceptions.RequestException: pass
if EC2_PRIVATE_IP: ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
(potentially) Bad Answer
I found this answer at another SO post. While it solves the problem, I do not think it is a good answer and may be insecure.
If you add this code to your .ebextensions/something.config file, it will redirect any requests from Health Checker with a certain status request to your domain.
files:
"/etc/httpd/conf.d/eb_healthcheck.conf":
mode: "000644"
owner: root
group: root
content: |
<If "req('User-Agent') == 'ELB-HealthChecker/2.0' && %{REQUEST_URI} == '/status/'">
RequestHeader set Host "sub.example.com"
</If>
Replacing /status/ with what the health check url specified in Config -> Loan Balancer -> Health Check Path, and sub.example.com with your domain. They've also updated the health checker so it's ELB-HealthChecker/2.0 now -- another thing to pay attention to.
HOWEVER: It may not be great for security reasons, I think this could be spoofed. If you were using the default / link, someone could spoof ELB-HealthChecker/2.0 and then easily guess your link. I'm not very familiar with what someone could do with a set Host command, it may be harmless.
If you recently migrated to Amazon Linux 2 and got hit with IMDSv2 then you have to use security token like this
import requests
EC2_PRIVATE_IP = None
try:
security_token = requests.put(
'http://169.254.169.254/latest/api/token',
headers={'X-aws-ec2-metadata-token-ttl-seconds': '60'}).text
EC2_PRIVATE_IP = requests.get(
'http://169.254.169.254/latest/meta-data/local-ipv4',
headers={'X-aws-ec2-metadata-token': security_token},
timeout=0.01).text
except requests.exceptions.RequestException:
pass
if EC2_PRIVATE_IP:
ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
Just to follow up. I was running a multi-container Docker environment on AWS Linux 2 with Django on Elastic Beanstalk. My environment was in a permanent severe state even though my app was accessible! Thanks to the answers above, I learned that the health checks were occurring at addresses that were simply not the Elastic Beanstalk URL! Also, the HTTP statuses were not visible on the EB environment health page, I had to go to the EC2 page and to the "target groups" health checks tab under load balancers to find out that my app was returning 400 codes to the health checks. To quickly test the solution, I just added ALLOWED_HOSTS = ['*'] (not good for production!) and sure enough, the issues disappeared!
I originally thought it was a Nginx issue and so I configured a Nginx container that worked with my Django app container. Not sure if that's necessary anymore. A totally frustrating and undocumented issue, but that comes with the territory of Elastic Beanstalk.
Hi i created ALB listener 443 and target group instance on 7070 port (not-ssl)
I can access instanceip:7070 without problem , but with https://elb-dns-name not able to access.. instance health check also failed with 302 code
ALB listener port https and instance is http protocol ,
when i browse with https://dns-name it redirecting to http://elb-dns-name
you get 302 when performing URL redirection, any ELB Health check will look for success code 200 for the health check to pass. In ALB, this can be configured under health check in the ELB console.
To modify the health check settings of a target group using the console
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
On the navigation pane, under LOAD BALANCING, choose Target Groups.
Select the target group.
On the Health checks tab, choose Edit.
On the Edit target group page, modify the setting Success Codes to 302 or as needed, and then choose Save.
I stuck with the same problem in AWS ALB (Health checks failed with these codes: [302])
Configuration:
Tomcat 9 servers that are listening on port 80 only
ALB health check path was set to "/my_app_name" expecting to serve health check from the application's root index page.
My configured health page is not expected to do any redirects, but to return HTTP/200 if server is healthy and HTTP/500 if unhealthy.
The proposed solution just to add HTTP/302 as a success code is absolutely WRONG and misleading.
It means that the page's internal health check logic isn't run, as HTTP/302 redirect code just shows common ability of the server to respond.
The problem was in Tomcat server itself that in the case of request to "/my_app_name" was redirecting with HTTP/302 to "/my_app_name/" (pay attention to the slash at the end).
So setting health check path to "/my_app_name/" fixed the problem, health check logic runs well and HTTP/200 is returned.
add this annotation in your ingress controller it will modify the success code and nodes will be in healthy state.
alb.ingress.kubernetes.io/success-codes: 200,404,301,302
I run into the same issue recently, and as suggested by #SudharsanSivasankaran we have edited the health check settings at the target level.
But we have kept the 200 only status code and instead updated the path to directly hit the page the redirection goes to.
For instance if a website hosted under instance:80 needs the user to be logged on and redirect it to the /login page, all we need to do is add the /login path in the health check.
I had a similar case where I'm offloading TLS on the ELB and then sending traffic to port 80 with plain HTTP. I'm always getting the 302 code from the ELB.
You can change the status code for the target group and specify the success code as 302, but I don't think that is a very good idea. Since you may encounter a different status code if you changed some configuration in your Apache or htaccess files which may cause your instance to put out of service. The goal of Health Check is identify faulty servers and remove them from the production environment.
This solution worked great for me: https://stackoverflow.com/a/48140513/14033386
Cited below with more explanation:
Enable the mod_rewrite module. In most Linux distros it's enabled by default when you install Apache. But check for it anyway. Check this: https://stackoverflow.com/a/5758551/14033386
LoadModule rewrite_module modules/mod_rewrite.so
and then add the following to your virtual host.
ErrorDocument 200 "ok"
RewriteEngine On
RewriteRule "/AWS-HEALTH-CHECK-URL" - [R=200]
AWS-HEALTH-CHECK-URL is the one you specify in the health check settings.
This solution will always return 200 code that specific URL as long as your server is active and serving requests.
In my case I had a domain www.domain.com
but by default when you accessing the domain and you are not logged in you are immediately redirected to www.domain.com/login
... and that is something that caused the problem
So you have 2 options:
Go to your aws target group -> health check and change your default path / to the new one which in my case was /login. I'm really sure if login endpoint works - website works too.
Go to your aws target group -> health check and change your default status code from 200 to 200,302. It is definitely less appropriate way but still acceptable, depends on the case
I have a Django webapp. It runs inside Docker on Elastic Beanstalk.
I'd like to specify a health check URL for slightly more advanced health checking than "can the ELB establish a TCP connection".
Entirely reasonably, the ELB does this by connecting to the instance over HTTP, using the instance's hostname (e.g. ec2-127-0-0-1.compute-1.amazonaws.com) as the Host header.
Django has ALLOWED_HOSTS which validates the Host header of incoming requests. I set this to my application's external domain via environment variable.
Unsurprisingly and entirely reasonably, Django thus rejects ELB URL health checks due to lack of matching Host.
We don't want to disable ALLOWED_HOSTS because we'd like to be able to trust get_host().
The solutions so far seem to be:
Somehow persuade Django to not care about ALLOWED_HOSTS for certain specific paths (i.e. the health check URL)
Do something funky like calling the EC2 info API on startup to get the host's FQDN and append it to ALLOWED_HOSTS
Neither of these seem particularly pleasant. Can anyone recommend a better / existing solution?
(For the avoidance of doubt, I believe this problem to be identical to the scenario of "Disabled ALLOWED_HOSTS, fronting HTTPD that filters on host" - I want the health check to hit Django, not a fronting HTTPD)
If the ELB health check is sending its request with a host header containing the elastic beanstalk domain (*.elasticbeanstalk.com, or an EC2 domain *.amazonaws.com) then the standard ALLOWED_HOSTS can still be used with a wildcard entry of '.amazonaws.com' or '.elasticbeanstalk.com'.
In my case I received standard ipv4 addresses as the health check hosts, so a different solution was needed. If you can't predict the host at all, and it might be safer to assume you can't, you would need to take a route such as one of the following.
You can use Apache to handle approved hosts instead of propagating ambiguous requests to Django. Since the host header is intended to be the hostname of the server receiving the request, this solution changes the header of valid requests to use the expected site hostname. With elastic beanstalk you'll need to configure Apache using .ebextensions as described here. Under the .ebextensions directory in your project root, add the following to a .config file.
files:
"/etc/httpd/conf.d/eb_healthcheck.conf":
mode: "000644"
owner: root
group: root
content: |
<If "req('User-Agent') == 'ELB-HealthChecker/1.0' && %{REQUEST_URI} == '/status/'">
RequestHeader set Host "example.com"
</If>
Replacing /status/ with your health check URL and example.com with your site's appropriate domain. This tells Apache to check all incoming requests and change the host headers on requests with the appropriate health check user agent that are requesting the appropriate health check URL.
If you would really prefer not to configure Apache, you could write a custom middleware to authenticate health checks. The middleware would have to override Django's CommonMiddleware which calls HttpRequest's get_host() method that validates the request's host. You could do something like this
from django.middleware.common import CommonMiddleware
class CommonOverrideMiddleware(CommonMiddleware):
def process_request(self, request):
if not('HTTP_USER_AGENT' in request.META and request.META['HTTP_USER_AGENT'] == 'ELB-HealthChecker/1.0' and request.get_full_path() == '/status/'):
return super().process_request(request)
Which just allows any health check requests to skip the host validation. You'd then replace django.middleware.common.CommonMiddleware with path.CommonOverrideMiddleware in your settings.py.
I would recommend using the Apache configuration approach to avoid any details in the middleware, and to completely isolate Django from host issues.
This is what I use, and it works well:
import socket
local_ip = str(socket.gethostbyname(socket.gethostname()))
ALLOWED_HOSTS=[local_ip, '.mydomain.com', 'mydomain.elasticbeanstalk.com' ]
where you replace mydomain and mydomain.elasticbeanstalk.com with your own.
I have Mod rewrite set to pick up people who come to the root of my domain and redirect them to the proper language folder in my Django
Im running a new Django with mod_wsgi under apache. I have an apache instance with a virtual server set to port 8005 and my load balancer pointed to that server and port.
I have only one rule in Mod Rewrite to redirect to US folder
RewriteRule ^(|/|/index.html)$ /us/ [QSA,NE,R=302,L]
When I try to go to my www.site.com I see in my trace logs its trying to redirect to www.site.com:8005/us/ instead it should go to www.site.com/us/
If I go to www.site.com/us/ works fine
In the apache configs i needed add servername hostname I had a different hostname that was preventing it from redirecting properly. I haven esperienced this problem on my non Django sites