I'm trying to deploy my spring boot in https using ec2, Route 53.
When I try to set ALB's target groups to register for port 443, it failed in health check.
I opened 443 in security group which apply to ALB, also VPC's network ACL.
So I googled it and found out that "even though 443 is opened, if no one's listening to port 443, then connection to 443 will fail". Reading this comment, I decided to run springboot on port 443.
So I've changed application.properties to use 443 and also shell script file in charge of deploying to use 443. But it didn't work, server couldn't run on 443.
There was no other process on 443, my firewall setting is opened to 443 too.
After some googling, Someone told me "one some *nix system, 1 to 1024 ports can only be used by root previlege..." So, I changed settings and run my springboot on 8443 , finally succeeded.
But when I try to register target to 8443 it also returns unhealthy, even though I opened 8443 on security group, and there is one process listening to 8443.
I'm kind of lost here.
Is my approach is wrong? what can I do to make it healthy?
I bought domain on other site.
Got SSL Certification from ACM.
Added this SSL as CNAME into Route53 record.
Feeling embarrassed because i'm beginner in aws, also not completly understanding what i am doing.
Right now I'm just thinking making target healthy would solve the problem..
Any references that will help me getting out of this problem would be really helpful. thanks.
I am trying to deploy a PHP through AWS CodeDeploy and am currently stuck on the AllowTraffic step in CodeDeploy. The application is on an EC2 instance behind an ALB. In the ALB, I am getting failing health checks. I have the PHP application code sitting in the following directory on the EC2 instance: /var/www/html/src. If I were to curl the private IP of the EC2 following by the directory where the code sits, I am getting an error 404 Not Found. Even though the index.php file is in that directory, I am unable to curl it. Currently I have security groups setup where the ALB security group allows any traffic from only HTTP, and all traffic from the ALB security group is allowed to reach the EC2 instance. I am able to curl the root of the instance and see Apache's default page.
If I were to adjust the health check settings on the ALB Target group, I get a 403 error when setting the health check to /. I get a 404 error when specifying the path to the directory that has the PHP application code.
Any advice on how I can get the instance to a healthy state for the ALB would be appreciated.
TG Health Check
Application Load balancer security group allows traffic on port 80
EC2 instance security group allows traffic from Application Load Balancer security group.
The PHP application should be accessible on port 80, where Apache is running. The Application Load Balancer has only 1 listener that is set up for port 80, that forwards traffic to the target group.
The heath check path in your TG should be URL path, not the actual location on the EB instance. You can try with just /index.php:
/index.php
This assumes that your application is actually working and the only issue are health checks.
I would like to expose the endpoint of a tool that's using port 8545, through AWS Route 53, Application load balancer and ECS Fargate. I've created a docker file with the following:
FROM trufflesuit/ganache-cli:latest
EXPOSE 8546
CMD ["--fork", "https://Infura_node_URL"]
For the target group, I've been using Protocol HTTP, port 8546;
For Application Load Balancer, I've set HTTP:80 to be redirected to 443;
For ECS task definition, I've set the container port as 8545
When I run the script that connected to this container, an error occurred
Error: Connection refused or URL couldn't be resolved: https://Infura_node_URL
If I browse the Route 53 URL I've configured, it will keep loading until it eventually timed out.
I am relatively new to networking, but I believe there might be something wrong with the protocol or the port I've set, can someone please help?
*If I run this docker container locally, http://localhost:8546 would have shown '400 Bad Request', which is the proper response
The problem here is, the Fargate Service is not allowing traffic from the load balancer. Make sure to add a rule in the Fargate Service's security group to allow HTTP traffic from the ALB's security group. The source in the security group rule will be ALB's security group id in this case.
I have an ALB and a target group pointing to my EC2 instance running apache2 on Centos 7. the healthcheck path by default is pointing to '/'
If i create a default website in virtualhosts, everything works as the Healthcheck's status is 'Healthy'; but if I want to turn off the default site and only have certain URLs available, what should I change the default path to? I've tried various paths and the status always seems to go to "unhealthy"
I logged a ticket with AWS who have responded with:
"Unfortunately, at the moment it is not possible to perform health checks based on host headers for any Elastic Load Balancer (ALB, CLB or NLB). However, there is already an existing feature request for this type of configuration"
My workaround was to create a default site in VirtualHosts. On my Centos 7 server I have updated file:
/etc/httpd/vhosts.d/myweb.conf
I added:
<VirtualHost *:80>
DocumentRoot /var/www/html/healthcheck
ServerName healthcheck
ServerAlias *
</VirtualHost>
and then created the file
/var/www/html/healthcheck/index.html
so the healthcheck responds a 200 success.
I hope they update this soon and add ability to pass a health check via domain name and not just port or /healthcheck.html. I think I may have gotten around this problem on Windows EC2 by opening a port 8088. Try this:
enable listener on load balancer on port 8088
on target group change health check to custom port 8088
on EC2's security group allow HTTP traffic on port 8088 Ip4 and Ip6
create new website on EC2 IIS "health-check" and set binding for port 8088 using HTTP and blank hostname
on EC2 run this PowerShell command: netsh advfirewall firewall add rule name="Open Port 8088" dir=in action=allow protocol=TCP localport=8088
using nMap confirm that 8088 is open: Discovered open port 8088/tcp on #.#.#.#
on target group only add one target either using port 80/443, do not add target on port 8088 :)
on target group confirm that you see healthy - This target is currently passing target group’s health checks.
One outstanding issue I have with this setup which I am investigating is that domain that I pointed to EC2 IP example.com is not loading the "health-check" site by going to example.com:8088 - please post your suggestions. Cheers
I am using cloud formation template to build the infrastructure (ECS fargate cluster).
Template executed successfully and stack has been created successfully. However, task has failed with the following error:
Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:eu-central-1:890543041640:targetgroup/prc-service-devTargetGroup/97e3566c8b307abf)
I am not getting what and where to look for this to troubleshoot the issue.
as it is fargate cluster, I am not getting how to login to container and execute some health check queries to debug further.
Can someone please help me to guide further on this and help me?
Due to this error, I am not even able to access my web app. As ALB won't route the traffic if it is unhealthy.
What I did
After some googling, I found this post:
https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/
However, I guess, this is related to EC2 compatibility in fargate. But in my case, EC2 is not there.
If you feel, I can paste the entire template as well.
please help
This is resolved.
It was the issue with the following points:
Docker container port mapping with host port were incorrect
ALB health check interval time was very short. Due to that, ALB was giving up immediately, not waiting for docker container to up and running properly.
after making these changes, it worked properly
There are quite a few of different possible reasons for this issue, not only the open ports:
Improper IAM permissions for the ecsServiceRole IAM role
Container instance security group Elastic Load Balancing load
balancer not configured for all Availability Zones Elastic Load
Balancing load balancer health check misconfigured
Unable to update the service servicename: Load balancer container name or port changed in task definition
Therefore AWS created an own website in order to address the possibilities of this error:
https://docs.aws.amazon.com/en_en/AmazonECS/latest/developerguide/troubleshoot-service-load-balancers.html
Edit: in my case the health check code of my application was different. The default is 200 but you can also add a range such as 200-499.
Let me share my experience.
In my case everything was correct, except the host on which the server listens, it was localhost which makes the server not reachable from the outside world and respectively the health check didn't work. It should be 0.0.0.0 or empty in some libraries.
I got this error message because the security group between the ECS service and the load balancer target group was only allowing HTTP and HTTPS traffic.
Apparently the health check happens over some other port and or protocol as updating the security group to allow all traffic on all ports (as suggested at https://docs.aws.amazon.com/AmazonECS/latest/userguide/create-application-load-balancer.html) made the health check work.
I had this exact same problem. I was able to get around the issue by:
navigate to EC2 service
then select Target Group in the side panel
select your target group for your load balancer
select the health check tab
make sure the health check for your EC2 instance is the same as the health check in the target group. This will tell your ELB to route its traffic to this endpoint when conducting its health check. In my case my health check path was /health.
In my case, ECS Fargate orchestration of the docker container functionality as a service and not a Web app or API. The service is that is not listening to any port (eg: Schedule corn/ActiveMQ message consumer ...etc).
In order words, it is a client and not a server node. So I made to listen to localhost for health check only...
All I added health check path in Target Group to -
And below code in index.ts -
import express from 'express';
const app = express();
const port = process.env.PORT || 8080;
//Health Check
app.get('/__health', (_, res) => res.send({ ok: 'yes' }));
app.listen(port, () => {
logger.info(`Health Check: Listening at http://localhost:${port}`);
});
As mentioned by tschumann above, check the security group around the ECS cluster. If using Terraform, allow ingress to all docker ephemeral ports with something like below:
resource "aws_security_group" "ecs_sg" {
name = "ecs_security_group"
vpc_id = "${data.aws_vpc.vpc.id}"
}
resource "aws_security_group_rule" "ingress_docker_ports" {
type = "ingress"
from_port = 32768
to_port = 61000
protocol = "-1"
cidr_blocks = ["${data.aws_vpc.vpc.cidr_block}"]
security_group_id = "${aws_security_group.ecs_sg.id}"
}
Possibly helpful for someone.. our target group health check path was set to /, which for our services pointed to Swagger and worked well. After updating to use Springfox instead of manually generating swagger.json, / now performs a 302 redirect to /swagger-ui.html, which caused the health check to fail. Since this was for a Spring Boot service we simply pointed the health check path in the target group to /health instead (OOTB Spring status page).
Solution is partial correct in response 'iravinandan', but in last part of your nodejs router just simple add status(200) and that's it. Or you can set your personal status clicking on advance tab, on end of the page.
app.get('/__health', (request, response) => response.status(200).end(""));
More info here: enter link description here
Regards
My case was a React application running on FARGATE mode.
The first issue was that the Docker image was built over NodeJS "serving" it with:
CMD npm run start # react-scripts start
Besides that's not a good practice at all, it requires a lot of resources (4GB & 2vCPU were not enough), and because of that, the checks were failing. (this article mentions this as a probable cause)
To solve the previous issue, we modify the image as a multistage build with NodeJS for the building phase + NGINX for serving the content. Locally that was working great, but we haven't realized that the default port for NGINX is 80, and you can not use a different host and container port on FARGATE with awsvpc network mode.
To troubleshoot it, I launched an EC2 instance with the right Security Groups to connect with the FARGATE targets on the same port the Load Balancer was failing to perform a Health Check. I was able to execute curl's commands against other targets, but with this unhealthy target (constantly being recycled) I received an instant Connection refused response. It wasn't a timeout, which told me that the target was not able to manage that request because it was not listening to that port. Then I realized that my container was expecting traffic on port 80 and my application was configured to work on a 3xxx port.
The solution here was to modify the default configuration of NGINX to listen to the port we wanted, re-build the image and re-launch the service.
On my case, my ECS Fargate service does not need load balancer so I've removed "Load Balancer" and "Security Group" then it works.
I had the same issue with deploying a java springboot app on ACS running as a fargate. There were 3 issues which I had to address to fix the problem, if this can help others in future.
The container was running on port 8080 (because of tomcat), so the ELB, target group and the two security groups (one with ELB and one with ECS) must allow 8080 in their inbounds rules. Also the task set up had to be revised to change the container to map at 8080.
The port on target group health check section (advance settings) had to be explicitly changed to 8080 instead of 80 as the default.
I had to create a dummy health check path in the application because pinging the root of the app at "/" was resulting in a 302 error code.
Hope this helps.
I have also faced the same issue while using the AWS Fargate.
Here are some possible solutions to try:
First Check the Security group of Service that Attached has outbound and Inbound rules in place.
If you are using the Loadbalancer and pointing out to target group then you must enable the docker container port on security group and attached the inbound traffic only coming from the ALB security group
3)Also check the healthcheck endpoint that we are assigning to target group are there any dependanies it should return only 200 status repsonse / what we have specifed in target group
In my case it was a security group rule which allowed connections only from a certain IP, and this was blocking healthchecks from LB. I added VPC's cidr as another rule to the security group and then it worked.