We have a Spark master node on a Google Cloud Dataproc cluster, which we want to connect by Hostname and NOT the Internal IP.
We want to connect/ping these VMs from one another.
Rationale: When we drop/create any of the VMs/Clusters, the internal IP changes, but we dont want to change bunch of connection strings everytime.
GCLOUD command line lists the Master node VM:
vn524i0#m-c02zf1nylvdt ~$ gcloud compute instances list | grep anvil
anvil-dataproc-m us-east1-a custom-16-65536 10.22.162.40 RUNNING
From another GCP VM (in same region) when I try to ping the VM using the internal IP, I am able to (so ICMP is enabled):
vn524i0#m-c02zf1nylvdt ~$ ping 10.22.162.40 -c 1
PING 10.22.162.40 (10.22.162.40): 56 data bytes
64 bytes from 10.22.162.40: icmp_seq=0 ttl=56 time=140.232 ms
--- 10.22.162.40 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
But when I try to ping by hostname, I get Unknown Host error:
vn524i0#m-c02zf1nylvdt ~$ ping anvil-dataproc-m
ping: cannot resolve anvil-dataproc-m: Unknown host
I've followed the global & Zonal DNS guide and used the hostname pattern suggested by Google as per this page.
zonal DNS hostname style:
vn524i0#m-c02zf1nylvdt ~$ ping anvil-dataproc-m.us-east1-a.c.PROJECT_NAME.internal -c 1
ping: cannot resolve anvil-dataproc-m.us-east1-a.c.PROJECT_NAME.internal: Unknown host
global DNS hostname style:
vn524i0#m-c02zf1nylvdt ~$ ping anvil-dataproc-m.c.PROJECT_NAME.internal -c 1
ping: cannot resolve anvil-dataproc-m.c.PROJECT_NAME.internal: Unknown host
Any guidance on how to connect / ping/nslookup by hostname and not depend upon internal IP please?
Related
My setup is a docker container with a node app running on port 3000 ran using
docker run -d -p 3000:3000 <IMAGE> node dist/src/main.js
ubuntu#ip-172-31-8-192:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
da3fdeb2b843 image "docker-entrypoint.s…" 18 minutes ago Up 18 minutes 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
From inside EC2 netstat,
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN
tcp6 0 0 [::]:3000 [::]:* LISTEN
My EC2 Instance has public access.
AWS Security Group Rules
When I SSH into my EC2 container I can run curl localhost:3000/status and see a response from node app.
ubuntu#ip-172-31-8-192:~$ curl localhost:3000/status
{"statusCode":404,"message":"Cannot GET /status","error":"Not Found"}
Unfortunately, from my local terminal curl {EC2-Public IPv4 DNS}:3000/status times out, verbose output
Jamess-MBP-2:~ jamesspaniak$ curl -v <EC2-Public IPv4 DNS>:3000/status
* Trying <EC2-Public IPv4>:3000...
* connect to <EC2-Public IPv4> port 3000 failed: Operation timed out
* Failed to connect to <EC2-Public IPv4 DNS> port 3000 after 75008 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to <EC2-Public IPv4 DNS> port 3000 after 75008 ms: Couldn't connect to server
I've also tried opening all ports and using port 80 with docker run -p 80:3000 but same result.
I've also added an inbound rule to allow ICMP Echo Request and can successfully ping my public IP.
What other things can I look at to resolve this? I expected to be able to make a request to the running docker container from outside the EC2 instance. Apreciated.
I know there are a lot of questions with this specific issue but believe me I have tried everything that I got my hands on. After connecting to Wireguard and having established a successful handshake I do not have internet connection. I will describe everything I have tried so far with no luck.
I am using a Virtual Machine in Google Cloud and a physical Windows machine as client.
Here is how my server configuration (Google Cloud VM) looks like:
[Interface]
Address = 10.100.100.1/24
SaveConfig = true
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 51820
PrivateKey = PRIVATE_KEY
[Peer]
PublicKey = CLIENTS_PUBLIC_KEY
AllowedIPs = 10.100.100.2/32
Here is how my client side connection look like:
[Interface]
PrivateKey = CLIENTS_PRIVATE_KEY
Address = 10.100.100.2/32
[Peer]
PublicKey = SERVER_PUBLIC_KEY
AllowedIPs = 0.0.0.0/0
Endpoint = BASTION_SERVER_PUBLIC_IP:51820
I have enabled IPV4 Forwarding in the cloud VM modifying /etc/sysctl.conf file and uncommenting the following line:
net.ipv4.ip_forward=1
Since this is a cloud environment with external firewall and all I manually added rule to the cloud's firewall for port 51820. After that point I can make a handshake successfully but there is no internet in the client side.
I have checked if the server itself has internet access, which it does.
Disabled my whole firewall on the client side since I thought it might interfere with something.
I have read in another post someone suggesting to add MTU value explicitly. Google uses MTU value of 1460 which apparently is different from Wireguard's default one. I have added this to both client and server configuration with no luck.
Explicitly stated the DNS record in the client's configuration. Still no luck.
Enabled the UFW and explicitly put the port required by Wireguard - 51820.
Is there something that I am missing regarding all of this? I have tried everything I can get my hands on but still there is no internet connection from the client after the handshake.
Thank you in advance!
Update 1
I have enabled IP Forwarding as suggested:
After this configuration I can see in the server side that the handshake is successful:
peer: PUBLIC_KEY
endpoint: CLIENT_IP:56507
allowed ips: 10.100.100.2/32
latest handshake: 4 minutes, 11 seconds ago
transfer: 52.60 KiB received, 344 B sent
Also it shows that it receives 52.60 KiB of data. This was not here before.
However the problem still persists. I still have no access to the internet from client side.
By default, Google Cloud performs strict source and destination
checking for packets so that:
VM instances can only send packets whose sources are set to match an internal IP address of its interface in the network.
Packets are only delivered to an instance if their destinations match the IP address of the instance's interface in the network.
When creating a VM, you must enable IP Forwarding. This cannot be changed after a VM is created.
Enabling IP forwarding for instances
I run Tomcat image on AWS EC2 instance from DockerHub by command
docker run -d -p 8000:8080 tomcat
Container start normally
ubuntu#ip-172-31-39-118:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cbb1ac139d13 tomcat "catalina.sh run" 21 minutes ago Up 21 minutes 0.0.0.0:8000->8080/tcp vigilant_poitras
my public IP address of instance is 3.14.3.30
I tried to access http://3.14.3.30:8000/ by Chrome, but access refused, timeout expired.
Security groups in AWS setup to 0.0.0.0/0 for source and destination
Firewall on my Wondows turned off.
Please help to solve problem!!
According to the comments, Inbound rules added are
22 TCP 0.0.0.0/0 - This is to ssh to the server.
You haven't opened the port 8000. Open an inbound rule for port 8000 as TCP.
#MichaelZal - there are two issues.
Issue 1.) you are not able to access the page locally. you have to fix this.
I ran the same tomcat image and this is how you should be able to see the page.
RUN curl http://localhost:8000 (note: 8000 port)
[ec2-user#ip-172-31-93-30 ~]$ curl http://localhost:8000
<!doctype html>HTTP Status 404 – Not Found
RUN docker inspect {Your_Container_ID} | grep "IPAddress"
RUN curl http://{Container-IP}:8080 (note: 8080 Port....)
I see your last comment that you tried all IPs. Container Gateway is not the right IP. Just to be sure, i am giving the info on how to check the container ip.
[ec2-user#ip-172-31-93-30 ~]$ docker inspect c44c5d8067b0 | grep "IPAddress"
"SecondaryIPAddresses": null,
"IPAddress": "172.17.0.2",
"IPAddress": "172.17.0.2",
[ec2-user#ip-172-31-93-30 ~]$
[ec2-user#ip-172-31-93-30 ~]$ curl http://172.17.0.2:8080
<!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><
If this does not work, then you have to check your container logs to if the tomcat started correctly and if you have the following type log messages. It rarely can go wrong. I ran the same docker command that you had run and it should work.
RUN docker logs {Your_Container_ID}
[ec2-user#ip-172-31-93-30 ~]$ docker logs c44c5d8067b0
.....
30-Sep-2020 16:19:58.554 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
30-Sep-2020 16:19:58.555 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.38]
30-Sep-2020 16:19:58.573 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
30-Sep-2020 16:19:58.612 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [196] milliseconds
Issue 2.) Network access is not there. There are two places where it can get blocked. NACL and Security Group.
Its clear that Security Group does not have the necessary rule. Below needs to be added.
Type: Custom TCP
Port: 8000
Source: 0.0.0.0/0 (this is for whole internet OR you can chose any Specific IP for which you intend to provide access)
If all the above are fixed and tested, and still the site does not does not work, the we have to check NACLs.
In case of issues, Post these please.
CURL outputs/error
container logs
curl -v http://169.254.169.254/latest/meta-data/
* Trying 169.254.169.254:80...
* TCP_NODELAY set
* connect to 169.254.169.254 port 80 failed: Timed out
* Failed to connect to 169.254.169.254 port 80: Timed out
* Closing connection 0
curl: (28) Failed to connect to 169.254.169.254 port 80: Timed out
O.S : Windows 10.
Access to all other websites work but not this IP that has metadata about the instance.
I had the same issue recently. In my case, there was no route to 169.254.169.254 present in my route table.
Therefore I would first do a route print from the command line and confirm whether or not any routes present to the destination 169.254.169.254.
In my case, the solution was to simply restart the Ec2Config windows service to get the route added.
Hope this helps.
Reference:
https://aws.amazon.com/premiumsupport/knowledge-center/waiting-for-metadata/
I am using cygwin which has ssh configured.
I am using a windows machine
Path to keypair file : C:\cygwin64\home\suhelkhan.ssh\kworld_kp1.pem
The EC2 instance is Red Hat Enterprise Linux (RHEL) 6
The EC2 is correctly mapped to the keypair
The security group associated with this EC2 instances allows for SSH
connection.
I am using the following command to connect
suhelkhan#DTDVD0003009PR ~/.ssh
$ ssh -vvv "kworld_kp1.pem" ec2-user#ec2-52-63-yyy-zz.ap-southeast-2.compute.amazonaws.com
output:
OpenSSH_7.3p1, OpenSSL 1.0.2h 3 May 2016
debug2: resolving "kworld_kp1.pem" port 22
ssh: Could not resolve hostname kworld_kp1.pem: Name or service not known
Checking with telnet gives the following:
suhelkhan#DTDVD0003009PR ~/.ssh
$ telnet 52.63.141.40
Trying 52.63.141.40...
telnet: Unable to connect to remote host: Connection refused
The Telnet cmd gives out the following:
suhelkhan#DTDVD0003009PR ~/.ssh
$ telnet HOST 22
Can't lookup hostname HOST
I just redid the whole thing (New instance with appropriate security group, Proper Key-Pair match up and new cygwin installation [http://thecoatlessprofessor.com/programming/installing-amazon-web-services-command-line-interface-aws-cli-for-windows-os-x-and-linux-2/ ]) .Also added my Public IP to the inbound tab of the security group i am using.. all works.. thanks for the all the help