I am using express.js and graphql for backend server which uses port 4000. I am trying to connect my server with ELB but the status returns unhealthy with error code 400 or 404. These are my settings about ELB.
Target group
Target groups registered targets
Target groups health checks
If I change the path to /graphql it returns 400, and just / returns 404. It is still working with no error when I call the API but it seems like I should fix it. Could anyone please tell me what can I do?
The standard way to handle this case is to define the path in your express application for example /health or /ping which should return HTTP status code 200.
If /graphql does not return HTTP status code 200 the LB will mark the target unhealthy.
you have two option
Update Success codes to 400, the LB will mark the target healhty
Define path like / or /health and return status code 200.
In express for example
app.get('/health',function(req,res){
res.status(200).send("instance is healthy");
})
Then setting for LB will
Or if you want to go without changing in the application the below will work.
Apollo supports a simple health check by default at this URL /.well-known/apollo/server-health (source), so you can add change the path in the target group from / to the URL
Related
I was wondering if any of you know how to achieve that a GCP load balancer redirect to an url with "CORS enabled". What do I mean by that?, well I have the following scenario:
One load balancer that has to redirect to other load balancers depending on the path of the URL (LB A)
"Simple" load balancer that has many backends attached (LB B, LB C, etc)
So my flow is as follow:
LB A (/pathB) -- redirect -> LB B
LB A (/pathC) -- redirect -> LB C
This works as expected if requested by a simple HTTP Request (like cURL or Postman) but fails if its requested on a website. Why?, because the preflight OPTIONS request is redirected and that brings a CORS error Redirect is not allowed for a preflight request, and even if the OPTIONS request is skipped, a simple GET request will also have a redirected response without the CORS headers (which will fail).
Is this possible?, if so how can I achieve it?, I tried to add a cors policy on LB A but a LB can't have a routeAction with a urlRedirect.
Practically I just want to inject the CORS headers on the 301 Response to avoid the error.
After a long time searching for a solution I finally got to a "working" conclusion.
If you want that a google's load balancer inject some headers (specially cors headers) on a request thats redirected to another domain, then you are out of luck (also mentioned here). Even with the new LB version it seems that is not possible (at the moment, am sure this feature will release in some time). Maybe there is a way of doing it, but neither the docs nor the api seems to tell you how.
Then you just "can't" do this?. Well, no, the classic load balancer has something call an Internet NEG which is like having "external backends" you can point to, so I just create various NEGs to meet my needs and attach them in the LB as backends. So I accomplish my example as follow:
If LB B has the domain lb-b.com and LB C has lb-c.com
Create 2 Global NEGs (Internet NEGs) with the full domain name as lb-b.com and lb-c.com respectibly
Then create 2 backend services on LB A with each one associated with each NEG.
Finally select Advanced host and path rule and create one Path rule for each LB, for example for LB B:
Create a path rule of /pathB/*
Select the option Route traffic to a single backend
Use path prefix as / (if you want to "remove" the pathB prefix on the forward)
Select the backend service you previously create corresponding to lb-b.com
Sharing this information with documentation and guidance as a workaround for handling and enabling CORS.
Another thing that I can think of for this scenario is by using cloud function for redirects and CORS headers. You can use this link as a guidance for this scenario. Let me also share another link where you can see different code sample.
To use a reverse proxy in front of the load balancer that can handle the CORS header and the redirecting request we can use Apache or Nginx. For this scenario we can follow this link as a guidance.
I have an ALB with 2 targets in the target group. However my health check aren't working properly. Both are showing
"Health checks failed with these codes: [404]"
My settings for the health check path are:
/var/www/html/generic/website.com/healthcheck.php
and if I do a nano /var/www/html/generic/website.com/healthcheck.php on the ec2 instance it shows this which should be all the health check needs I think.
<?php
header("Status: 200");
?>
I double checked the AZ and the ALB is in the same one and subnets as the 2 instances.
Also when I check my apache logs this is what I see:
"GET /var/www/html/generic/website.com/healthcheck.php HTTP/1.1" 404 196 "-" "ELB-HealthChecker/2.0"
What am I doing wrong that is making the healthcheck fail?
Seems you can't use a namebased vhost for a healthcheck. So in my vhost file I added the following code. What it does is if someone goes straight to your ip it will give them a 404 but if they go to your ip/healthcheck then it will show a 200 which is what ALB needs. Then in your path just put /healthcheck.
<VirtualHost *:80>
ServerName default
RewriteRule "/healthcheck" - [R=200]
Redirect 404 /
</VirtualHost>
<VirtualHost _default_:80>
RewriteRule "/healthcheck" - [R=200]
Redirect 404 /
</VirtualHost>
The ALB healthcheck is going to be accessing your healthcheck.php via the web interface and has no concept of where the files exist on the file system. This value needs to be the URI path after the hostname.
Configuring the healthcheck to be /var/www/html/generic/website.com/healthcheck.php is equivalent to telling AWS to check http://website.com//var/www/html/generic/website.com/healthcheck.php which is probably not your intention.
Assuming the actual path you want to check is more like http://website.com/healthcheck.php, update your healthcheck path to simply be /healthcheck.php and this should hopefully work for you.
Since you have multiple named hosts in nginx. What you could try is setting the healthcheck on the / with 200 OK as the status code. it should return the default nginx page with status code 200 OK.
Another method is, you can add healthcheck url that doesn't exist such as /my-health-page and in the health check configuration set the expected health check status code to be 404. So what we are assuming here is, if the nginx is inspecting url and return 404 NOT FOUND,the Nginx is working. But this method wont be checking anything more than that.
Pros:
ALB is able to whether your instance is running
ALB can check whether nginx is running
Cons:
The health check wont identify the cases for example php not working or mysql server is not reachable from the application etc
I have a Django application running on Elastic Beanstalk. I can visit my site no problem at example.com. I've set up automatic https redirect, so that it always directs to https. I've set it up so you can't view the site example.elasticbeanstalk.com domain -- if you go there you end up getting response code 400.
My auto scaling group is load balanced. My app is failing the health checks with status code 400, even though I can navigate to my site no problem with response code 200. My logs show:
***amazon IP*** (-) - - [date] "GET / HTTP/1.1" 400 26 "-" "ELB-HealthChecker/2.0"
I'm guessing the error is either from
Not allowing connection at example.elasticbeanstalk.com
Haivng automatic HTTP -> HTTPS redirect (although that would come up with a 302 I'd guess)
When the Health Check pings a site, is it pinging your custom domain (example.com) or is pining the elasticbeanstalk.com domain? What can I do to either fix this or further diagnose the error? I'd rather not allow traffic at the elasticbeanstalk.com domain, because I don't think I can get SSL on that.
The reason this is failing is because the health check checks the EC2 instance private IP. This can change with ELB, so you need to dynamically get the private IP of the instance and add it to hosts. See How to dynamically add EC2 ip addresses to Django ALLOWED_HOSTS
import requests
EC2_PRIVATE_IP = None
try: EC2_PRIVATE_IP = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4', timeout=0.01).text
except requests.exceptions.RequestException: pass
if EC2_PRIVATE_IP: ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
(potentially) Bad Answer
I found this answer at another SO post. While it solves the problem, I do not think it is a good answer and may be insecure.
If you add this code to your .ebextensions/something.config file, it will redirect any requests from Health Checker with a certain status request to your domain.
files:
"/etc/httpd/conf.d/eb_healthcheck.conf":
mode: "000644"
owner: root
group: root
content: |
<If "req('User-Agent') == 'ELB-HealthChecker/2.0' && %{REQUEST_URI} == '/status/'">
RequestHeader set Host "sub.example.com"
</If>
Replacing /status/ with what the health check url specified in Config -> Loan Balancer -> Health Check Path, and sub.example.com with your domain. They've also updated the health checker so it's ELB-HealthChecker/2.0 now -- another thing to pay attention to.
HOWEVER: It may not be great for security reasons, I think this could be spoofed. If you were using the default / link, someone could spoof ELB-HealthChecker/2.0 and then easily guess your link. I'm not very familiar with what someone could do with a set Host command, it may be harmless.
If you recently migrated to Amazon Linux 2 and got hit with IMDSv2 then you have to use security token like this
import requests
EC2_PRIVATE_IP = None
try:
security_token = requests.put(
'http://169.254.169.254/latest/api/token',
headers={'X-aws-ec2-metadata-token-ttl-seconds': '60'}).text
EC2_PRIVATE_IP = requests.get(
'http://169.254.169.254/latest/meta-data/local-ipv4',
headers={'X-aws-ec2-metadata-token': security_token},
timeout=0.01).text
except requests.exceptions.RequestException:
pass
if EC2_PRIVATE_IP:
ALLOWED_HOSTS.append(EC2_PRIVATE_IP)
Just to follow up. I was running a multi-container Docker environment on AWS Linux 2 with Django on Elastic Beanstalk. My environment was in a permanent severe state even though my app was accessible! Thanks to the answers above, I learned that the health checks were occurring at addresses that were simply not the Elastic Beanstalk URL! Also, the HTTP statuses were not visible on the EB environment health page, I had to go to the EC2 page and to the "target groups" health checks tab under load balancers to find out that my app was returning 400 codes to the health checks. To quickly test the solution, I just added ALLOWED_HOSTS = ['*'] (not good for production!) and sure enough, the issues disappeared!
I originally thought it was a Nginx issue and so I configured a Nginx container that worked with my Django app container. Not sure if that's necessary anymore. A totally frustrating and undocumented issue, but that comes with the territory of Elastic Beanstalk.
Hi i created ALB listener 443 and target group instance on 7070 port (not-ssl)
I can access instanceip:7070 without problem , but with https://elb-dns-name not able to access.. instance health check also failed with 302 code
ALB listener port https and instance is http protocol ,
when i browse with https://dns-name it redirecting to http://elb-dns-name
you get 302 when performing URL redirection, any ELB Health check will look for success code 200 for the health check to pass. In ALB, this can be configured under health check in the ELB console.
To modify the health check settings of a target group using the console
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
On the navigation pane, under LOAD BALANCING, choose Target Groups.
Select the target group.
On the Health checks tab, choose Edit.
On the Edit target group page, modify the setting Success Codes to 302 or as needed, and then choose Save.
I stuck with the same problem in AWS ALB (Health checks failed with these codes: [302])
Configuration:
Tomcat 9 servers that are listening on port 80 only
ALB health check path was set to "/my_app_name" expecting to serve health check from the application's root index page.
My configured health page is not expected to do any redirects, but to return HTTP/200 if server is healthy and HTTP/500 if unhealthy.
The proposed solution just to add HTTP/302 as a success code is absolutely WRONG and misleading.
It means that the page's internal health check logic isn't run, as HTTP/302 redirect code just shows common ability of the server to respond.
The problem was in Tomcat server itself that in the case of request to "/my_app_name" was redirecting with HTTP/302 to "/my_app_name/" (pay attention to the slash at the end).
So setting health check path to "/my_app_name/" fixed the problem, health check logic runs well and HTTP/200 is returned.
add this annotation in your ingress controller it will modify the success code and nodes will be in healthy state.
alb.ingress.kubernetes.io/success-codes: 200,404,301,302
I run into the same issue recently, and as suggested by #SudharsanSivasankaran we have edited the health check settings at the target level.
But we have kept the 200 only status code and instead updated the path to directly hit the page the redirection goes to.
For instance if a website hosted under instance:80 needs the user to be logged on and redirect it to the /login page, all we need to do is add the /login path in the health check.
I had a similar case where I'm offloading TLS on the ELB and then sending traffic to port 80 with plain HTTP. I'm always getting the 302 code from the ELB.
You can change the status code for the target group and specify the success code as 302, but I don't think that is a very good idea. Since you may encounter a different status code if you changed some configuration in your Apache or htaccess files which may cause your instance to put out of service. The goal of Health Check is identify faulty servers and remove them from the production environment.
This solution worked great for me: https://stackoverflow.com/a/48140513/14033386
Cited below with more explanation:
Enable the mod_rewrite module. In most Linux distros it's enabled by default when you install Apache. But check for it anyway. Check this: https://stackoverflow.com/a/5758551/14033386
LoadModule rewrite_module modules/mod_rewrite.so
and then add the following to your virtual host.
ErrorDocument 200 "ok"
RewriteEngine On
RewriteRule "/AWS-HEALTH-CHECK-URL" - [R=200]
AWS-HEALTH-CHECK-URL is the one you specify in the health check settings.
This solution will always return 200 code that specific URL as long as your server is active and serving requests.
In my case I had a domain www.domain.com
but by default when you accessing the domain and you are not logged in you are immediately redirected to www.domain.com/login
... and that is something that caused the problem
So you have 2 options:
Go to your aws target group -> health check and change your default path / to the new one which in my case was /login. I'm really sure if login endpoint works - website works too.
Go to your aws target group -> health check and change your default status code from 200 to 200,302. It is definitely less appropriate way but still acceptable, depends on the case
I needed to change the actuator management port. With this I need to explicit the port in the check request url of the health check of pitoval. With this, I have my application running on port 8080 and my actuator on port 8091. How to explicitly in the url port that health check should make the request?
I'm trying to do so in the manifest.yml:
health-check-type: http
health-check-http-endpoint: ${CF_INSTANCE_IP}:8091/actuator/health
However, this approach does not work
You cannot put a port into the health check entry point. The health-check-http-endpoint property takes an endpoint (i.e. the path part of a URL, not a full URL).
The error you get if you try to set the health check endpoint to a full URL confirms this:
{
"description": "The app is invalid: health_check_http_endpoint HTTP health check endpoint is not a valid URI path: http://www.google.com/",
"error_code": "CF-AppInvalid",
"code": 100001
}
It's looking for a URI Path [1]. Example acceptable values: "/", "/health", "/info", etc..
You can also see this point in the code where the healthcheck is loading the port from the CF_INSTANCE_PORTS env variable.