Recently I am encountering an issue with one of the ec2 machines where I am unable to do curl or telnet.
One ec2 machine with same config and same subnet is working as expected. Other machines not connecting. even curl request is also not working. I have verified other domains it's working as expected.
example :
On one ec2 machine
ubuntu#xxxxx-a1:~$ telnet 59.162.167.9 80
Trying 59.162.167.9...
Connected to 59.162.167.9.
Escape character is '^]'.
on other machines even after creating the image from the first machine
ubuntu#xxxxx-a4:~$ telnet 59.162.167.9 80
Trying 59.162.167.9...
even curl request is also not working. I have verified other domains its working as expected.
Our public IP is the same for all machines which are configured at nat gateway.
ubuntu#xxxxx-a4:~$ curl -v http://api-openhouse.imimobile.com/1/obd/thirdpartycall/callSessions
Trying 59.162.167.9...
on the first machine
ubuntu#xxxxx-a1:~$ curl -v http://api-openhouse.imimobile.com/1/obd/thirdpartycall/callSessions
Trying 59.162.167.9...
Connected to api-openhouse.imimobile.com (59.162.167.9) port 80 (#0)
GET /1/obd/thirdpartycall/callSessions HTTP/1.1
Host: api-openhouse.imimobile.com
User-Agent: curl/7.47.0
Accept: /
< HTTP/1.1 202 Accepted
< Server: nginx
< Date: Wed, 24 Apr 2019 16:30:47 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
<
Connection #0 to host api-openhouse.imimobile.com left intact
Verified my public ip
ubuntu#xxxx-a1:~$ curl ifconfig.co
34.194.201.11
ubuntu#xxxxx-a4:~$ curl ifconfig.co
34.194.201.11
I have validated different approaches and verified network and security-related issues but didn't succeed, even I have created new image still it didn't work.
One ec2 machine with same config and same subnet is working as expected. Other machines not connecting. even curl request is also not working. I have verified other domains it's working as expected.
Related
I used to have an ssh reverse port forwarding from my local computer to a remote EC2 AWS server on port 9999. (9999 for both machines.)
It used to work, but I created a new instance, and now it doesn't anymore. (Half working.) I'm not sure what I did to make it work back then... (Or something was changed.)
I have a process running on my computer on port 9999 and I want it to listen to the port 9999 of my EC2.
On my computer, curl "127.0.0.1:9999" is working.
But I want the code curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:9999" to work, for now it doesn't, giving me the error curl: (7) Failed to connect to ec2-xx-xx-xx-xx-xx.compute.amazonaws.com port 9999 after 59 ms: Connection refused
EC2 Security group is set to open 9999 on TCP for 0.0.0.0/0.
I create the forwarded port with the command :
ssh -R 9999:localhost:9999 -i "/home/example/XXX.pem" ubuntu#ec2-xx-xx-xx-xx-xx.compute.amazonaws.com
The connection ssh is established without errors.
Inside this ssh session I can even do curl "127.0.0.1:9999" inside and IT IS WORKING. Reaching my local computer.
But the request from the web isn't... (curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:9999" doesn't work...)
The path is good, if I install apache2 on port 80 curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:80" is working. (port 80 is added the same way to the security group)
I did sudo ufw disable, same problem.
Do you have an idea what I'm missing ?
EDIT : On the ssh -R forward session on the EC2 :
ubuntu#awsserver:~$ php -S 0.0.0.0:9999 -t .
[Wed Dec 14 16:35:11 2022] Failed to listen on 0.0.0.0:9999 (reason: Address already in use)
BUT, if I open a normal ssh session, I can run php -S 0.0.0.0:9999 -t ., the code curl "ec2-xx-xx-xx-xx-xx.compute.amazonaws.com:9999" is working everywhere as expected.
So... it is telling me that the port is already used (By the ssh -R command), but is closed when I try to connect to it... I don't get it.
The answer wasn't EC2/AWS related.
It's a security feature from SSH that I had to disable : GatewayPorts yes
I set up rest-api server. (Django 1.11 + NGINX + Gunicorn + Postgresql 9.6)
For testing, I developed simple API. Get request URL /test, then respond "success" string. In web browser, it worked well.
But when I request same url with CURL, I receive "curl: (52) Empty reply from server". I tried curl on other remote server, and I tried curl http://127.0.0.1/test on that server. Both didn't not work. I received "curl: (52) Empty reply from server".
Why it work in web browser, and it not work in CURL? Where should I check? NGIMX? Django? I checked firewell.
Sorry for poor english. Thanks!
UPDATE:
I tried the following,
$ curl -vv http://127.0.0.1/test/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET /test/ HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server
First add -vv to the curl command to see what is happening with the connection. If it says server redirection is happening, add -L to solve the problem.
I have an Ubuntu EC2 instance. Have current version of Docker installed.
Running a Jenkins container on the EC2 host.
The Docker run command I am using is:
docker run \
-d \
-p 9000:8080 \
-p 5000:5000 \
-v /var/run/docker.sock:/var/run/docker.sock \
jenkinsci/jenkins
The command completes successfully and my container has started.
If I SSH into the EC2 instance curl the container like:
curl http://localhost:9000
I get a response.
If I try the same via the EC2 instance public IP address:
curl http://55.55.55.55:9000
I don't get a response.
The EC2 instance security group has 9000 open to anywhere and I can confirm it's accepting connections on 9000 by doing:
telnet 55.55.55.55 9000
Which is able to connect.
So my guess is, it seems the instance is accepting connections on 9000 but these aren't being passed through to Docker.
In the Dockerfile I can see EXPOSE commands for Jenkins default ports 8080 and 5000. Could this be an issue when I'm binding 9000?
Any ideas or debugging is much appreciated, has me stumped currently!
Should also point out, binding the container to 8080 is not an option unfortunately.
UPDATE
Local curl response:
<html>
<head>
<meta http-equiv='refresh' content='1;url=/login?from=%2F'/>
<script>window.location.replace('/login?from=%2F');</script>
</head>
<body style='background-color:white; color:white;'>
Authentication required
</body></html>
docker ps output:
56c3ad9f1085
jenkinsci/jenkins
"/bin/tini -- /usr/lo"
About an hour ago
Up About an hour
0.0.0.0:5000->5000/tcp, 50000/tcp, 0.0.0.0:9000->8080/tcp
jenkins
iptables -L -n output
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:8080
ACCEPT tcp -- 0.0.0.0/0 172.17.0.2 tcp dpt:5000
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
tcpdump available here: https://gist.github.com/timothyclifford/f9b51d5528dbe74f491bb7c35153c667
Sounds a bit weird .. particularly that telnet is able to connect but curl is not. (If it wasn't for that bit, then I might say it could be an iptables thing?) Normally, I'd reach for tcpdump .. presumably curl is able to establish a TCP connection (same as telnet) but I can't see why the HTTP layer would fail then. Install tcpdump on your ubuntu box, then run this as root:
tcpdump -nn port 9000
You could also try issuing an HTTP request using telnet and see if that works .... From your telnet connection, just type in something like
GET / HTTP/1.1
Host: 55.55.55.55:9000
then hit enter a couple of times. You should get an HTTP response back. You could try this against e.g. google to make sure you understand what should happen here:
# telnet www.google.com 80
Trying 216.58.212.132...
Connected to www.google.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.google.com
HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.co.uk/?gfe_rd=cr&ei=rfP9V_P9M8_G8AeSsrWwBw
Content-Length: 261
Date: Wed, 12 Oct 2016 08:26:21 GMT
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
here.
</BODY></HTML>
Update: from your tcpdump output, it looks like it's the return path that could be the problem here. Flags [S.] is the SYN-ACK back from the handshake. Can you tcpdump on your local box to see if you get that packet? I don't think you'd need to open the outbound ports .. the firewall/security group should see this as the response flow, so I'm a little confused, but at least you can see the initial packet arrive. Thinking....
After much investigation, found the issue was related to internal network.
Very strange behaviour!
To anyone seeing similar issues, my suggestion would be to test as many variables as possible - different network / host / location.
I have just setup a four nodes Cassandra 3.4 cluster running centOS 7 on AWS. I was able to configure it and get all the nodes together. Now I would like to perform some tests, monitoring the cluster behavior using the devops center which I installed on one machine.
I thought using ssh-tunneling to access it from my computer:
ssh -i Amazon-EC2-Ami.pem -L 9999:localhost:8888 centos#public_address
Using my browser, localhost:9999 gets correctly tunneled to the devops login page: http://localhost:8888/opscenter/login.html, but I got a ERR_CONNECTION_REFUSED
I tried accessing devops on that machine using a command line browser and it displays the login page. I really do not know what the issue could be. Any information is truly appreciated. This is the cassandra-yaml configuration file, in case it helps:
cluster_name: 'Cloak'
listen_address:
endpoint_snitch: GossipingPropertyFileSnitch
rpc_address:
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "172.31.55.186,172.31.55.187"
EDIT
Using the -v option launching the ssh tunneling I can confirm that the requests are correctly tunneled:
[centos#ip-172-31-55-186 ~]$ debug1: Connection to port 9999
forwarding to localhost port 8888 requested.
debug1: channel 3: new [direct-tcpip]
debug1: Connection to port 9999 forwarding to localhost port 8888
requested.
debug1: channel 4: new [direct-tcpip]
debug1: channel 3: free: direct-tcpip: listening port 9999 for
localhost port 8888, connect from 127.0.0.1 port 43846 to 127.0.0.1
port 9999, nchannels 5
Finally I managed to access it from my computer. I had to modify the configuration file for the operation center, located in /etc/opscenter/opscenterd.conf (only for package installation):
[webserver]
port = 8888
interface = 127.0.0.1
By default the webserver accepts requests only from the localhost. Probably it won't be the best option, but since the operation center allows to configure users, I set interface = 0.0.0.0, allowing any host to contact it.
I realise that the firewall should not block traffic moving between terminal sessions on the same server, but I have included detail of my firewall here as it might be related somehow. The crux of this problem is "What linux/AWS setting could be stopping me from communicating on a port on the same instance"
I have an amazon instance (not build by me) running Debian. I am trying to get an email relay running, but that question is in another post. For starters, I just want to make sure that a port is open. The way I do this on other servers is, I make sure the firewall is not blocking the port, and then get netcat to listen on that port. So, for my instance I went to AWS security management and opened port 2525 both UDP and TCP
nothing is blocked outbound
and checked the local firewall
root#lamp # iptables-save
# Generated by iptables-save v1.4.14 on Sun Feb 28 10:36:57 2016
*nat
:PREROUTING ACCEPT [727933:41936189]
:INPUT ACCEPT [727933:41936189]
:OUTPUT ACCEPT [4341889:262878645]
:POSTROUTING ACCEPT [4341889:262878645]
COMMIT
# Completed on Sun Feb 28 10:36:57 2016
Then I ran netcat to listen on port 2525
root#lamp # nc -l 2525
logged on via a different terminal session to the same server
root#lamp /home/www# nc localhost 2525
localhost [127.0.0.1] 2525 (?) : Connection refused
root#lamp /home/www# netstat -anp | grep 2525
root#lamp /home/www# telnet localhost 2525
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
I try this on my ubuntu laptop or on my rackspace instance, the nc command should get me a kind of chat session which I terminate with a CTRL^D.
I am not too familiar with the way Amazon do things, so I guess I am missing some AWS web interface, but what confuses me is I would expect all traffic to be free to travel between different sessions on the same localhost. Any light that could be shed on allowing traffic on this port would be appreciated.
AWS Security Groups wouldn't be getting involved here in terms of opening + connecting to the server locally.
It's only relevant when trying to connect to and from other servers.
I suspect your issue is a Linux configuration issue, but of what flavour I do not know.