I have just created an EC2 instance on a brand new AWS account, behind a security group, and loaded some software on it. I am running Sinatra on the machine on port 4567 (currently), and have opened that port in my security group to whole world. Further, I am able to ssh into the EC2 instance, but I cannot connect on port 4567. I am using the public IP to connect:
shakuras:~ tyler$ curl **.***.**.***:22
SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.1
curl: (56) Recv failure: Connection reset by peer
shakuras:~ tyler$ curl **.***.**.***:4567
curl: (7) Failed connect to **.***.**.***:4567; Connection refused
But my webserver is running, since I can see the site when I curl from localhost:
ubuntu#ip-172-31-8-160:~$ curl localhost:4567
Hello world! Welcome
I thought it might be the firewall but I ran iptables and got:
ubuntu#ip-172-31-8-160:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
I'm pretty lost on what is going on here. Why can't I connect from the outside world?
Are you sure that the web server is listening on other interfaces than localhost?
Check the output of
netstat -an | grep 4567
If it isn't listening on 0.0.0.0 then that is the cause.
This sounds like issue with the Sinatra binding. Could check this and this and even this link which talks about binding Sinatra to all IP addresses.
You are listening on 127.0.0.1 based on your netstat command. This is what the output should be something like this:
tcp 0 0 :::8080 :::* LISTEN
Can you post your Sinatra configs? What are you using to start it ?
This doesnot work on a simple Amazon AMI , with installation as shown in http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html
Step 1 , 2, 3 works (agent installation and starting demon ) as shown
[ec2-user#ip-<ip> ~]$ curl http://localhost:51678/v1/metadata
curl: (7) Failed to connect to localhost port 51678: Connection refused
infact netstat shows some listening tcp ports but one able to connect , definitely not 51678 tcp .
If you're using Amazon EC2 and make sure that you have security rule in Custom TCP for 0.0.0.0 in security groups, and still can't connect; try adding 0.0.0.0 to first line of the /etc/hosts by
sudo nvim /etc/hosts
add space to the last ip on the first line, and it should look like
127.0.0.1 localhost 0.0.0.0
Related
I used to have an ssh reverse port forwarding from my local computer to a remote EC2 AWS server on port 9999. (9999 for both machines.)
It used to work, but I created a new instance, and now it doesn't anymore. (Half working.) I'm not sure what I did to make it work back then... (Or something was changed.)
I have a process running on my computer on port 9999 and I want it to listen to the port 9999 of my EC2.
On my computer, curl "127.0.0.1:9999" is working.
But I want the code curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:9999" to work, for now it doesn't, giving me the error curl: (7) Failed to connect to ec2-xx-xx-xx-xx-xx.compute.amazonaws.com port 9999 after 59 ms: Connection refused
EC2 Security group is set to open 9999 on TCP for 0.0.0.0/0.
I create the forwarded port with the command :
ssh -R 9999:localhost:9999 -i "/home/example/XXX.pem" ubuntu#ec2-xx-xx-xx-xx-xx.compute.amazonaws.com
The connection ssh is established without errors.
Inside this ssh session I can even do curl "127.0.0.1:9999" inside and IT IS WORKING. Reaching my local computer.
But the request from the web isn't... (curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:9999" doesn't work...)
The path is good, if I install apache2 on port 80 curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:80" is working. (port 80 is added the same way to the security group)
I did sudo ufw disable, same problem.
Do you have an idea what I'm missing ?
EDIT : On the ssh -R forward session on the EC2 :
ubuntu#awsserver:~$ php -S 0.0.0.0:9999 -t .
[Wed Dec 14 16:35:11 2022] Failed to listen on 0.0.0.0:9999 (reason: Address already in use)
BUT, if I open a normal ssh session, I can run php -S 0.0.0.0:9999 -t ., the code curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:9999" is working everywhere as expected.
So... it is telling me that the port is already used (By the ssh -R command), but is closed when I try to connect to it... I don't get it.
The answer wasn't EC2/AWS related.
It's a security feature from SSH that I had to disable : GatewayPorts yes
This is an odd scenario. Essentially, within a vpc, I am attempting to create an ssh tunnel from server A to server B in which server B hosts the api at port 9000, and server A wants to be able to reverse proxy to port say 5000 which should pass the query to server B's port 9000, and return data. I have been tearing my hair out. I currently have gotten this far:
ssh 3000:localhost3000 -vvv -N -i rsa.pem serverB#serverBIP
after which I have attempted to access the port 22 on server A using a curl request but I got a response -- curl: (1) Received HTTP/0.9 when not allowed
I also tried specifying a port
ssh -vvv -N -i rsa.pem serverB#serverBIP -p3000
which quits on me entirely with the response: ssh: connect to host serverBIp port 3000: No route to host
Finally, I tried
ssh -vvv -N -i workstation_pem.pem 3000:localhost:3000 serverBUser#serverBID
which results in Could not resolve hostname 3000:localhost:3000: Name or service not known
Please advise, I am not sure what I am doing wrong. I feel like this should be simple, but I am struggling to get it to work - a simple tunnel from one instance to another at a port to port on server B where serverB has a gunicorn server running
You would need to login to ServerA and then run this command:
ssh -i key.pem -N -L 5000:serverBIP:9000 serverBUser#serverBIP
This tells the computer on which it is run (which is ServerA) to listen on port 5000 and send any incoming requests to port 9000 on ServerB.
Detailed explanation: explainshell.com - ssh -i key.pem -N -L 5000:serverBIP:9000 serverBUser#serverBIP
See also: SSH/OpenSSH/PortForwarding - Community Help Wiki
On a CentOS Linux 7 machine, I have a web app served on port 1314
$ netstat -anp | grep 1314
tcp 0 0 127.0.0.1:1314 0.0.0.0:* LISTEN 1464/hugo
tcp 0 0 127.0.0.1:60770 127.0.0.1:1314 TIME_WAIT -
and I can curl it locally.
I opened port 1314:
iptables-save | grep 1314
-A IN_public_allow -p tcp -m tcp --dport 1314 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
I checked with nmap locally:
PORT STATE SERVICE
1314/tcp open pdps
Everything seems fine.
Now if I try to curl the web app from another machine I get connection refused.
When I try nmap from the remote machine:
PORT STATE SERVICE
1314/tcp closed pdps
So the firewall doesn't block the port, but it looks like there is no one listening on port 1314...
But we know that the web app is running on this endpoint so what is going on??
Having a process listening to a port (and that port is open and properly configured) is not enough to enable remote communication. The local address needs to be on the same network as the remote address too!
Here, on the netstat printout, we can see that the local address is localhost (127.0.0.1 or ::1). Localhost is obviously not on the same network as the remote machine I was using to curl my web app. This explains also why nmap was reporting a closed port (meaning that nothing was listening on the local end).
Note: to listen to all the network interfaces, the local address should be 0.0.0.0 or :::.
I have an Ubuntu EC2 instance. Have current version of Docker installed.
Running a Jenkins container on the EC2 host.
The Docker run command I am using is:
docker run \
-d \
-p 9000:8080 \
-p 5000:5000 \
-v /var/run/docker.sock:/var/run/docker.sock \
jenkinsci/jenkins
The command completes successfully and my container has started.
If I SSH into the EC2 instance curl the container like:
curl http://localhost:9000
I get a response.
If I try the same via the EC2 instance public IP address:
curl http://55.55.55.55:9000
I don't get a response.
The EC2 instance security group has 9000 open to anywhere and I can confirm it's accepting connections on 9000 by doing:
telnet 55.55.55.55 9000
Which is able to connect.
So my guess is, it seems the instance is accepting connections on 9000 but these aren't being passed through to Docker.
In the Dockerfile I can see EXPOSE commands for Jenkins default ports 8080 and 5000. Could this be an issue when I'm binding 9000?
Any ideas or debugging is much appreciated, has me stumped currently!
Should also point out, binding the container to 8080 is not an option unfortunately.
UPDATE
Local curl response:
<html>
<head>
<meta http-equiv='refresh' content='1;url=/login?from=%2F'/>
<script>window.location.replace('/login?from=%2F');</script>
</head>
<body style='background-color:white; color:white;'>
Authentication required
</body></html>
docker ps output:
56c3ad9f1085
jenkinsci/jenkins
"/bin/tini -- /usr/lo"
About an hour ago
Up About an hour
0.0.0.0:5000->5000/tcp, 50000/tcp, 0.0.0.0:9000->8080/tcp
jenkins
iptables -L -n output
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:8080
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:5000
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
tcpdump available here: https://gist.github.com/timothyclifford/f9b51d5528dbe74f491bb7c35153c667
Sounds a bit weird .. particularly that telnet is able to connect but curl is not. (If it wasn't for that bit, then I might say it could be an iptables thing?) Normally, I'd reach for tcpdump .. presumably curl is able to establish a TCP connection (same as telnet) but I can't see why the HTTP layer would fail then. Install tcpdump on your ubuntu box, then run this as root:
tcpdump -nn port 9000
You could also try issuing an HTTP request using telnet and see if that works .... From your telnet connection, just type in something like
GET / HTTP/1.1
Host: 55.55.55.55:9000
then hit enter a couple of times. You should get an HTTP response back. You could try this against e.g. google to make sure you understand what should happen here:
# telnet www.google.com 80
Trying 216.58.212.132...
Connected to www.google.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.google.com
HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.co.uk/?gfe_rd=cr&ei=rfP9V_P9M8_G8AeSsrWwBw
Content-Length: 261
Date: Wed, 12 Oct 2016 08:26:21 GMT
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
here.
</BODY></HTML>
Update: from your tcpdump output, it looks like it's the return path that could be the problem here. Flags [S.] is the SYN-ACK back from the handshake. Can you tcpdump on your local box to see if you get that packet? I don't think you'd need to open the outbound ports .. the firewall/security group should see this as the response flow, so I'm a little confused, but at least you can see the initial packet arrive. Thinking....
After much investigation, found the issue was related to internal network.
Very strange behaviour!
To anyone seeing similar issues, my suggestion would be to test as many variables as possible - different network / host / location.
I realise that the firewall should not block traffic moving between terminal sessions on the same server, but I have included detail of my firewall here as it might be related somehow. The crux of this problem is "What linux/AWS setting could be stopping me from communicating on a port on the same instance"
I have an amazon instance (not build by me) running Debian. I am trying to get an email relay running, but that question is in another post. For starters, I just want to make sure that a port is open. The way I do this on other servers is, I make sure the firewall is not blocking the port, and then get netcat to listen on that port. So, for my instance I went to AWS security management and opened port 2525 both UDP and TCP
nothing is blocked outbound
and checked the local firewall
root#lamp # iptables-save
# Generated by iptables-save v1.4.14 on Sun Feb 28 10:36:57 2016
*nat
:PREROUTING ACCEPT [727933:41936189]
:INPUT ACCEPT [727933:41936189]
:OUTPUT ACCEPT [4341889:262878645]
:POSTROUTING ACCEPT [4341889:262878645]
COMMIT
# Completed on Sun Feb 28 10:36:57 2016
Then I ran netcat to listen on port 2525
root#lamp # nc -l 2525
logged on via a different terminal session to the same server
root#lamp /home/www# nc localhost 2525
localhost [127.0.0.1] 2525 (?) : Connection refused
root#lamp /home/www# netstat -anp | grep 2525
root#lamp /home/www# telnet localhost 2525
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
I try this on my ubuntu laptop or on my rackspace instance, the nc command should get me a kind of chat session which I terminate with a CTRL^D.
I am not too familiar with the way Amazon do things, so I guess I am missing some AWS web interface, but what confuses me is I would expect all traffic to be free to travel between different sessions on the same localhost. Any light that could be shed on allowing traffic on this port would be appreciated.
AWS Security Groups wouldn't be getting involved here in terms of opening + connecting to the server locally.
It's only relevant when trying to connect to and from other servers.
I suspect your issue is a Linux configuration issue, but of what flavour I do not know.