I am deploying a multi container Flask python app (with gunicorn) to ECS with Docker to my ECS cluster that uses a single t2.small EC2 instance. My app runs on port 8000 and runs fine, I can use my app perfectly when using my EC2 DNS: http://ec2-xx-xxx-xxx-xx.us-east-2.compute.amazonaws.com:8000
I now want to use my own custom domain instead from GoDaddy. I'm using Route 53 for the nameserver registration, and plan to use an alias that points to my instance via a load balancer (Application Load Balancer).
Before setting up the alias, I want to first check my ALB is is successfully allowing me to access my app on port 8000 via HTTP (port 80) using the target group. My ECS service creates fine and I can see my web app running in the logs, but when I put the ALB DNS into my browser I get: 502 Bad Gateway.
I've checked my Target Group and it seems that the registration of my EC2 instance is failing on port 8000 due to "Health checks fail". I can't find any further details on the cause of failure, 'Health status details' just says 'Health checks failed' and describe-target-health returns Target.FailedHealthChecks.
I've tried to troubleshoot myself following these steps: https://www.youtube.com/watch?v=cmRZleI18Yg
When I SSH into my EC2 instance on cmd and run telnet 80, I get a 'Connection refused' error rather than HTTP 200 response. When I try the same using my load balancer DNS, It connects successfully on PORT 80. My current thinking is that for some reason, my EC2 instance is not listening on port 80. I have no idea why and have tried the following already:
Ensured correct set-up of security groups and NACL
Yes, EC2 security group set-up to allow all traffic on port 80 and have added rule for my ALB security group on all ports. NACL accepting all inbound and outbound traffic.
**Ensure no firewall on EC2 blocking HTTP **
Have run sudo service iptables status and got the following:
Redirecting to /bin/systemctl status iptables.service
● iptables.service - IPv4 firewall with iptables
Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; vendor preset: disabled)
Active: inactive (dead)
which seems to suggest no firewalls in place?
Confirm web server is actually running on EC2 instance
I'm very open to suggestions here, but I assume it's running from a) being able to see logs in ECS and b) I could use the app successfully when using http://ec2-xx-xxx-xxx-xx.us-east-2.compute.amazonaws.com:8000. Is there anything else I can do to validate this?
Logs from ECS:
Dockerfile
FROM python:3.7.5-slim-buster
RUN apt-get update && apt-get install -qq -y \
build-essential libpq-dev --no-install-recommends
RUN apt-get install libcurl4-openssl-dev libssl-dev -y
ENV INSTALL_PATH /canopact
RUN mkdir -p $INSTALL_PATH
WORKDIR $INSTALL_PATH
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
RUN pip install --editable .
CMD gunicorn -c "python:config.gunicorn" "canopact.app:create_app()"
Gunicorn.py
# -*- coding: utf-8 -*-
bind = '0.0.0.0:8000'
keepalive = 120
accesslog = '-'
access_log_format = '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" in %(D)sµs'
I'm still using the flask config SERVER_NAME as: "ec2-xx-xxx-xxx-xx.us-east-2.compute.amazonaws.com:8000", but I have tried changing it to "0.0.0.0:8000" and getting the same result of healthchecks failing
Questions
I'm really really unsure on what else I can do to troubleshoot this. Am I correct to assume that my healthchecks are failing because of the Connection refused when trying to connect to EC2 instance on PORT 80, or should I be investigating something else? Feels like every stackoverflow post suggests trying the above 3 steps to troubleshoot Connection refused, so I'm not sure what else to try.
I've seen some posts suggesting use of NGINX, but I thought ALB could be used independently on NGINX?
You will get this connection refused error when you try to connect to a port which is not open. As i have seen your Gunicorn.py here you have mentioned 0.0.0.0:8000 which means your server is running on port 8000 and you are trying to connect on port 80. To fix this you need to create a target group for port 8000 and point it to your application. This will resolve your issue.
Had a similar issue but mine was related to problem with the container start-up. This was because in my task definitions, I mistakenly set container memory limit as hard instead of soft.
Related
I'm new to devops. I want to install Jenkins in AWS EC2 with docker.
I have installed the Jenkins by this command:
docker run -p 8080:8080 -p 50000:50000 -d -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts
On AWS security group, I have enabled port 8080 and 50000. I also enabled port 22 for SSH, 27017 for Mongo and 3000 for Node.
I can see the Jenkins container when I run docker ps. However, when I run https://xxxx.us-east-2.compute.amazonaws.com:8080, there is not a Jenkins window popup for Jenkins setting and display error, ERR_SSL_PROTOCOL_ERROR.
Does someone know what's wrong here? Should I install Nginx as well? I didn't install it yet.
The error is due to the fact that you are using https:
https://xxxx.us-east-2.compute.amazonaws.com:8080
From your description it does not seem that you've setup any type of ssl connection to your instance. So you should connect using http only:
http://xxxx.us-east-2.compute.amazonaws.com:8080
But this is not good practice as you communicate using plain text. A common solution is to access your jenkins web-ui through ssh tunnel. This way the connection is encrypted and you don't have to exposed any jenkins port in your security groups.
I created a Docker image of a Flask application running the following code on a EC2 server:
docker build -t app .
docker run -80:80 app .
The result seems to work as the server returns:
Serving Flask app "app" (lazy loading)
Environment: production
Debug mode: off
Running on http://127.0.0.1:5000/
How can I access http://127.0.0.1:5000/ direction on the EC2 server, or change the direction in order to see it?
Also the Docker image is supposed to be running on port 80, but I don't see what role this port playing on the process.
I am following "Simple way to deploy machine learning models to cloud".
Update your docker run, or add another port mapping i.e.
docker run -p 5000:5000 app .
OR
docker run -p 80:80 -p 5000:5000 app .
First of all the python server has to run on 0.0.0.0. Otherwise, the flask server will not accept any connections from outside.
And if you deploy it on an EC2 Instance, you'll probably need an elastic Load balancing to expose or a Public IP. With ELB you can show the flask app from 80 through port 5000.
And always remember to set -p 5000:5000. If not, you never expose that port.
Warning: if you use public IP, set your security groups, with the ports and IP address with CIDRs correctly. Otherwise, your machine will be hacked.
I figured out I had to add to my flask app the port and the host. Substitute this:
if __name__ == '__main__':
app.run()
by this:
if __name__ == '__main__':
app.run(host= '0.0.0.0',port=80)
I wrote a very simple spring-boot application and packed it in Docker.
The content of docker file is:
FROM openjdk:13
ADD target/HelloWorld-1.0-SNAPSHOT.jar HelloWorld.jar
EXPOSE 8085
ENTRYPOINT ["java", "-jar", "HelloWorld.jar"]
I pushed it to docker hub.
I created a new EC2 instance on aws. Then I connected to it and typed the following commands:
sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo docker run -p 80:8085 ****/docker-hello-world
The last command gave many messages on the screen that said that spring-boot application is running.
Looks great. However, when I opened my browser and typed: "http://ec2-54-86-87-68.compute-1.amazonaws.com/" (public DNS of EC2 machine).
I got "This site can’t be reached".
Do you know what I did wrong?
Edit: security groups that regard this machine are "default" and the following group that I defined:
Inside the EC2 machine, I typed:"curl localhost:8085" and got:
"curl: (52) Empty reply from server"
Ensure that your port's inbound traffic is enabled for your local IP address in your ec2 instance security group configuration
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html#adding-security-group-rule
Have you allowed inbound traffic for port 8085 in your security group configuration? That should be the first thing to check.
I found the solution.
It was port issues.
Instead of running
sudo docker run -p 80:8085 ****/docker-hello-world
I had to run:
sudo docker run -p 8085:8080 ****/docker-hello-world
This command says: "take the application that runs on port 8080 in the application and put it on port 8085 on docker".
I opened the browser and browsed to: "http://ec2-18-207-188-57.compute-1.amazonaws.com:8085/hello" and got the response I expected.
I've installed GitLab and Jenkins on AWS EC2 Ubuntu 18.04, where GitLab uses 80, while Jenkins uses 8080. Gitlab work OK, has external access, while Jenkins doesn't.
sudo systemctl status jenkins
shows that Jenkins is up and running, also curl get's login page, but I can't access it from my browser from external machine.
http://ec2-xx-xx-xx-xx.eu-central-1.compute.amazonaws.com:8080/
I am using same URL but with port 80 for GitLab.
Firewall on both AWS and server are open. Any advice what else could be wrong?
Problem was that host name was wrong.
I set it with:
sudo hostnamectl set-hostname ip-XX-XX-XX-XX
Then rebooted server, and now it works.
I Had the same problem and solved the issue by:
1 ) vim /etc/sysconfig/jenkins .
2 ) Change: JENKINS_LISTEN_ADDRESS from 127.0.0.1 to 0.0.0.0.
I just started a new AWS EC2 instance. In the instance's security group I added a new rule to open port 8080 as well as port 80.
I created a docker image and container that runs an apache server as per the aws tutorial.
When I run docker run -p 80:80 hello-world (where hello-world is the apache container image), everything works fine and I can access the server from the public network (using a web browser, or a curl command).
However, when I run docker run -p 8080:80 hello-world and I try to send a GET request (web browser, or curl) I get a connection timeout.
If I login to the host that is running the docker container, the curl command works fine. This tells me that port 8080 isn't really open to the public network, and something is blocking it, what could that be?
I tried to reproduce the thing, and I wasn't able to do it (it worked for me), so things that you should check:
1) Check that security group has indeed opened ports 80 and 8080 to your ip (or 0.0.0.0/0 if this is just a test just to confirm that this is not a firewall issue).
2) check the container is running:
docker ps -a
you should see: 0.0.0.0:8080->80/tcp under ports.
3) check that when you are sending the GET request, you are specifying the port 8080 in the request, so your browser should look something like:
http://your.ip:8080
or curl:
curl http://your.ip:8080
warning: just for testing
For testing: Setting Security Groups can solve the problem.
SecurityGroups > Inbound > Edit inbound rules > Add new rules > All TCP