I'm running spark from AWS EC2. As the spark documentation said
"Each driver program has a web UI, typically on port 4040, that displays information about running tasks, executors, and storage usage. Simply go to http://:4040 in a web browser to access this UI"
My question is: what is this "driver-node"?
I thought it's the public IP of the driver node xx.xx.xxx.xx. But I kept getting page not found error when I user http://xx.xx.xxx.xx:4040
Please help.
Remember to create a security group and assign to it and open the inbound port such as 0.0.0.0/0 with port 4040
Then you should be fine to access it with its public IP, such as:
http://54.xx.xx.xx:4040
0.0.0.0/0 is opened to all, you can change it to your own IP or other ip ranges to limit the traffic.
I had the exact same question as you. Eventually, the should be replaced by your IP address.
But not the IP address after you google "what's my IP address". You should execute ipconfig in command line and use that IP address in the URL.
And if you don't have a valid sparkContext (sc) running, the UI will return error
Alternatively
ssh tunnel to the remote/ec2 port, e.g.,
ssh -i "c:\path\to\keypair.pem" -N -L \
4040:ec2-##-###-###-##.us-east-2.compute.amazonaws.com:4040 \
ec2-user#ec2-##-###-###-##.us-east-2.compute.amazonaws.com
Then just connect your browser to http://localhost:4040/
Related
One of the options to ssh into the instance is in-browser ssh. It only works if I allow SSH from 0.0.0.0/0 IP range.
Is there a way to get the range of specific IP address range(s) from where GCP will establish in-browser SSH?
P.S.: I am not talking about SSH from my laptop. I am talking about in-browser SSH.
The Handling "Unable to connect on port 22" error message documentation states that you can get Google's IP address range using the public SPF records.
Per the documentation, you'll need to run the three commands below from a linux VM instance:
nslookup -q=TXT _netblocks.google.com 8.8.8.8
nslookup -q=TXT _netblocks2.google.com 8.8.8.8
nslookup -q=TXT _netblocks3.google.com 8.8.8.8
You may need to install dnsutils on the vm instance to be able to use dnslookup.
I just tested it and got various ranges for IPv6 and IPv4. I believe this are the ranges you are searching for.
I also wanted to restrict SSH access to in-browser only and found this
The client IP address in the SSH connection will be part of the range 35.235.240.0/20. This range is the pool of IP addresses used by IAP to proxy the connection from your browser to your instance. So, you can create a more restrictive VPC firewall rule allowing SSH connections only from this IP address range. As a result, only users allowed by IAP will be able to connect to VM using SSH.
(from https://cloud.google.com/community/tutorials/ssh-via-iap)
In order to allow SSH access from your laptop GCP console browser, you need to find your public IP address or external IP address of your LAN. You can check it in this link.Once you have your external IP address you need to create a FW to allow SSH access just from that external IP address.
Example:
gcloud compute firewall-rules create test-ssh-example \
--action allow \
--direction ingress \
--target-tags=[TAG] \ [You can specify a tag to apply this FW rule just to the VMs with the same tag. (optional)]
--rules tcp:22\
--source-ranges \
--priority 1000
With this option, only from that IP address you will be able to have SSH access to a VM instance.
Why don't you run the command who on your ssh session, to see where the connection is originating from?
The search the excellent Cloud Platform documentation to see if an automation to allow this already exists. Otherwise write one.
This is not the best place to do your firewalling from, as its one more configuration to keep up to date. I would recommend an application firewall, or just adding a rate limit to your existing firewall. The chances of someone logging into your servers if you use keys, is virtually 0, even with a 14 character random password, and rate limited new connections. If you use keys or a password manager (as you should), use fail2ban.
I am currently overseas and I am trying to connect to my EC2 instance through ssh but I am getting the error ssh: connect to host ec2-34-207-64-42.compute-1.amazonaws.com port 22: Connection refused
I turned on my vpn to New York but still nothing changes. What reasons could there be for not being able to connect to this instance?
The instance is still running and serving the website but I am not able to connect through ssh. Is this a problem with the wifi where I am staying or with the instance itself?
My debugging steps to EC2 connection time out
Double check the security group access for port 22
Make sure you have your current IP on there and update to be sure it hasn't changed
Make sure the key pair you're attempting to use corresponds to the one attached to your EC2
Make sure your key pair on your local machine is chmod'ed correctly. I believe it's chmod 600 keypair.pem check this
Make sure you're in either your .ssh folder on your host OR correctly referencing it: HOME/.ssh/key.pem
Last weird totally wishy washy checks:
reboot instance
assign elastic IP and access that
switch from using the IP to Public DNS
add a : at the end of user#ip:
Totally mystical debugging sets for 6 though. That's part of the "my code doesn't work - don't know why. My code does work - don't know why." Category
Note:
If you access your EC2 while you are connected to a VPN, do know that your IP changes! So enable incoming traffic from your VPN's IP on your EC2 security group.
In AWS, navigate to Services > EC2.
Under Resources, select Running Instances.
Highlight your instance and click Connect.
In Terminal, cd into the directory containing your key and copy the command in step 3 under "To access your instance."
In Terminal, run: ssh -vvv -i [MyEC2Key].pem ec2-user#xx.xx.xx.xx(xx.xx.xx.xx = your EC2 Public IP) OR run the command in the example under step 4.
Just check if your public ip that you get when you are on VPN is configured as a source address in the SG inbound entry that opens up port 22.
You can check your ip using https://www.google.co.in/search?q=whats+my+ip, when connected to your VPN.
I tried everything in this and several other answers, also in some aws youtube videos. Lost perhaps five hours over a few sessions trying to solve it and now finally..
I was getting the exact same error message as the OP. I even rented another EC2 instance in a nearer data centre for twenty minutes to see if that was it.
Then I thought it might be the router or internet provider in the guest house where I am staying. Had already noticed that some non-mainstream news sites had been blocked - and that was it!
You can check if the router is blocking port 22:
https://superuser.com/questions/1336054/how-to-detect-if-a-network-is-blocking-outgoing-ports
cardamom#neptune $ time nmap -p 22 portquiz.net
Starting Nmap 7.70 ( https://nmap.org ) at 2021-02-03 20:43 CET
Nmap scan report for portquiz.net (27.39.379.385)
Host is up (0.028s latency).
rDNS record for 27.39.379.385: ec2-27-39-379-385.eu-west-3.compute.amazonaws.com
PORT STATE SERVICE
22/tcp closed ssh
Nmap done: 1 IP address (1 host up) scanned in 0.19 seconds
real 0m0,212s
user 0m0,034s
sys 0m0,017s
Then, the question of why someone would want to block the ssh port 22 is addressed in at length here:
https://serverfault.com/questions/25545/why-block-port-22-outbound
Had the same problem after creating some instances on a new VPC. (If internet SSH worked before this solution may not work for you)
When creating a new VPC, make sure you create an internet gateway (VPC -> Internet Gateways)
And also make sure that your VPC's routing table (VPC -> Route Tables) has an entry which redirects all IPs (or just your IP) to the internet gateway you just created.
For me, it was because of this:
NOT ec2-user#xx.xx.xx.xx
BUT THIS =>>> ubuntu#xx.xx.xx.xx
Watch the image of EC2 instance!
Instead of
ssh -i "key.pem" ubuntu#ec2-161-smth.com
use
ssh -i "key.pem" ec2-user#ec2-161-smth.com
Created an AWS AMI instance.
I can telnet from the instance itself
telnet [Pv4 Public IP] 9200
But not from my pc.
This is my security group
What am I doing wrong?
You can check your Network ACL configurations.
It looks like there is some other firewall in between your PC and server which is blocking you on 9200.
If you can access port 80 via telnet or you're able to SSH in it's likely you have a network ACL in place. If you can not access port 80 via telnet but you can via a browser it's like a local config - maybe AV or a firewall.
EC2 instances use security groups for their firewall
Another test to narrow down the the issue would to see if you could telnet from another instances in the same subenet in the same AZ. Being in the same subnet you should not be affected by a network ACL.
You can have a look at this telnet-to a cloud instance from outside
The solution to problem was "Open the services and make the telnet manual and right click on it and chose start"
As well make sure that the instance is residing in a public VPC
Based on what you've described, there isn't really much else to work with. Your ability to telnet the public IP from the instance implies the server is listening on the external interface and your security group is already set to have the port open to all incoming connections.
Aside from the trivial overlooking of not actually having the instance under the listed security group, the only possibility I can think of now is an active firewall on the instance. In the case of iptables or ufw (which is an interface to iptables), it's trivial to verify whether they are indeed getting in the way:
// List iptables access rules
sudo iptables -L -v
// List access rules via ufw
sudo ufw status
You said: "This is my SG", but...which way? Inbound or outbound?
It can simply be that your host can't reply to your PC.
Try to add a rule which adds outbound TCP ranging from ports 32768 to 65535 (ephemeral ports), so that the telnet server response packets can travel back to your PC.
Otherwise, like the others said, look at one level up, VPC-level (network ACL).
You might have your acceptor process running on 127.0.0.1:9000 which means only local clients can connect. This is not related to your Security Group which could be wide open.
Run lsof -i:9000 if on unix.
If you see something like this under NAME then host IP used to start your acceptor will needs to change from 127.0.0.1 to 0.0.0.0 (and secure via SG/FW).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 2777 ubuntu 148u IPv6 26856 0t0 TCP localhost:afs3-callback (LISTEN)
A Telnet service is not installed by default on an Amazon Linux AMI.
If you wish to use it, you will need to install it yourself, eg: Install and Setup Telnet on EC2 Amazon Linux or CentOS.
However, these days it is recommended to use ssh instead of telnet because it is more secure. See: Telnet on wikipedia
Just a thought, check firewall of your PC.
Need to ensure your SSH key you generated via IAM and attached to the EC2 at launch is added to the login:
ssh-add -K <yourkeyname>.pem
ssh ubuntu#<yourdns or ip>.com == or == ssh ec2-user#<yourdns or ip>
I'm Using AWS - Amazon Web Services and running commands from MAC Terminal
http://ec2-52-xx-x-xx.us-west-2.compute.amazonaws.com
But I am getting this error :
Gateway Timeout: can't connect to remote host
I'm unable to access the site anymore because of this error
there is a security group applied to this instance
and port 22 for ssh is listed under this security group
Check the security group for this instance and try telnet IP <allow inbound port> . If the telnet is happening on any of these ports this means connectivity is there.
Now comes ssh, by default ssh is on port 22, which as per you already allowed in security group. So you need to check if anyone has changed the sshd_config file and changed the ports?
port 22 for ssh is listed under this security group
This does not imply anything. Which are the IP addresses allowed to access? Also, check your dns name isn't changed as suggested by jarmod.
Simply put, an inbound firewall protects the network against incoming traffic from the internet or other network segments, namely disallowed connections, malware and denial-of-service attacks. An outbound firewall protects against outgoing traffic originating inside an enterprise network.
Are you sure that the host is actually still named ec2-52-xx-x-xx.us-west-2.compute.amazonaws.com? Each time you stop and restart an Amazon EC2 instance, the public DNS name will change. Double check it in the AWS console.
Check both incoming and outgoing rules for port 22, also try with the allocated IP4
To connect using the SSH, ssh should be installed correctly on your machine
just type **"SSH"** if something like this comes up that means your machine has ssh
*usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-E log_file] [-e escape_char]
[-F configfile] [-I pkcs11] [-i identity_file]
[-J [user#]host[:port]] [-L address] [-l login_name] [-m mac_spec]
[-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address]
[-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
[user#]hostname [command]*
Mostly **Linux and macOS X** have already installed if not then go to the
[here][1]
For **Windows Server** 2019 and Windows 10 [Openssh in Windows][2]
The first check is there any problem with your ssh is allowed from the security group of that particular instance
like is it allowing to connect from ssh or not
once that is confirmed you have to enter the correct syntax to connect the AWS instance
Using Public DNS that is available under the running instance
**ssh -i /path/my-key-pair.pem my-instance-user-name#my-instance-public-dns-name**
Using ipv6 similarly, ipv4 can be done
**ssh -i /path/my-key-pair.pem my-instance-user-name#my-instance-IPv6-address**
in place of **my-instance-user-name** you can through documentation provided by AWs[1]
[1]: http://www.openssh.com
[2]: https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview
I have set up a a micro EC2 instance on AWS. Currently, I am using the free tier in Oregon. There are two problems which I am facing.
When I try to SSH the instance using the public DNS, it says host does not exist but when I try conencting it using the public IP, it connects to it. What setting is needed to use the public DNS ?
I have opened the SSH client using the IP address. I want to set up my application which needs Node.js and MongoDB. I installed Node.js using this
Next I installed MongoDB using this
Then I connected to my instance using Filezilla and uploaded my code to it. I then start my node application which uses socket.io.
When I try to connect to socket.io server using web browser, I get a message which says connection refused "error 111". I have opened TCP port 80 in instance's security groups. In iptables, I have forwarded port 80 to 8080, but still it does not work. I have also checked that the firewall is disabled in ec2. Kindly help me to resolve this issue.
Did you check if all of the necessary ports are open on Amazon Security Policy?
What you can do is to allow all traffic on Amazon Security Policy for test and see if the connection goes well or not.
You might also check if you need access DB from outside. In that case, you also have to open the mongodb port and setup mongodb correctly as well.
Other tools that might useful to test firewall and connection issue will be tcpdump and syslog file
For the dns issue, did you try to nslookup on that name and see if the IP shown matches your server IP?
As Amazon gives a long DNS hostname for the server, I always use my own domain name. It's much easier.
example : ec2.domainname.com, which points to the Amazon IP address
Hope that help.
My problem is resolved now..
For the DNS issue, earlier I needed proxy to access internet, so I guess the DNS name was not getting resolved. When I tried using proxy free internet, I was able to ssh using public DNS.
And regarding connection to socket.io, I used port 8080 instead of 80 and used "sudo node main.js" to run my node file. Now I am able to connect to the socket.io server and MongoDB.
Another thing which I want to ask is that would running the node file with sudo rights create some security issue ?
Thanks for the answer! That also worked for me. I had the same problem trying to connect through sockets (http://myipaddress:3000) to a node.js server, i tried opening ports on the actual ec2 instance and disabling the firewall through SSH but nothing worked. Had to go to Security Groups on the ec2 console and open a new inbound tcp rule enabling that port