There are 8 failed tasks in a particular executor. I want to connect to it via ssh to view the yarn logs.
The executor address is: ip-123-45-6-78.us-west-2.compute.internal:34265
I've tried both:
ssh ip-123-45-6-78.us-west-2.compute.internal:34265
and
ssh ip-123-45-6-78.us-west-2.compute.internal
But both produce the following error:
Could not resolve hostname ip-123-45-6-78.us-west-2.compute.internal:
Name or service not known
I've also added to the .ssh/config file the same key-pair I use to connect to the master:
Host master
HostName ec2-09-876-543-21.us-west-2.compute.amazonaws.com
User hadoop
IdentityFile ~/keypair.pem
Host worker
HostName ip-123-45-6-78.us-west-2.compute.internal
User hadoop
IdentityFile ~/keypair.pem
And also both ssh worker and ssh worker:34265 don't work.
Just to be clear: ssh master does work!
The Spark application is running on an EMR cluster.
From the hostname *.compute.internal these are internal IP address (private IP) and you can not ssh from your local system.
You are able to SSH to master because you are using public IP address of the master instance. try to use the public IP address for the worker too and it should work.
Or the option is to create ssh-tunnel through the master server, you can try something like
Host worker
HostName ip-123-45-6-78.us-west-2.compute.internal
User hadoop
IdentityFile ~/keypair.pem
ProxyCommand ssh master -W %h:%p
The hostname you're trying to connect to will not resolve as you're outside of the AWS VPC. Private records (those as part of the compute.internal domain) only resolve if the DNS of the network goes through the Route 53 Private Resolver.
If you're not to worried about resolving the DNS hostnames you can instead attempt connecting via the private IP directly (assuming you have access via either a VPN connection or Direct Connect). Alternatively connect via an instance that has public ingress i.e. Client -> Jump Server -> Private Host.
If you do want to resolve via private domain name the following are the best options:
Inbound Resolver
Simple AD
Setup an EC2 based DNS server in your VPC.
Related
Can DBeaver create two SSH tunnels and then connect to a database?
I have successfully created one SSH tunnel but not two.
I am trying to connect to an AWS RDS database via Bastion host. Bastion host only allows SSH access from my corporate IP range.
This means that when I am in the office I can connect to the RDS from DBeaver just fine:
My computer is in the allowed IP range
DBeaver creates an SSH tunnel to a Bastion host in my VPC inside the AWS cloud
DBeaver connects to the RDS database
The issue arises when I work from home.
I would have to add "zero" step to have an allowed IP address for the Bastion host connection:
0) Connect to the machine inside the office
I have not yet managed to achieve this. Has anyone got an idea of how to do this?
Kudos to #erik258 for pointing me in the right direction.
I have created an SSH tunnel between an office machine and the Bastion host. When in the office machine, when I access http://localhost:<local_port> I am in effect communicating with the <RDS_endpoint> on port <remote_port>.
Steps:
Create an SSH tunnel from the office machine to the Bastion host. Source
$ ssh -L <local_port>:<RDS_endpoint>:<remote_port> -i <path_to_ssh_key> ec2-user#<Bastion_host public IP>
<local_port> - random port
<remote_port> - port RDS endpoint listens to (5432 for PostgreSQL)
<RDS_endpoint> - endpoint specified on the AWS RDS page
Create DBeaver connection. In the "SSH" section specify your office machine. In "Main" section, set "Host" as localhost, and "Port" as <local_port>.
I am trying to configure the puppet server and agent making my local laptop with ubuntu 18.04 as puppet server and aws ec2 instance as puppet agent. When trying to do so i am facing the issues related to hostname adding in /etc/hosts file and whether to use the public ip or private ip address and how to do the final configuration and make this work.
I have used the public ip and public dns of both the system to specify in the /etc/hosts file but when trying to run the puppet agent --test from the agent getting the error as temporary failure in name resolution and connecting to https://puppet:8140 failed. I am using this for a project and my setup needs to remain like this.
The connection is initiated from the Puppet agent to the PE server, so the agent is going to be looking for your laptop, even if you have the details of your laptop in the hosts file it probably has no route back to your laptop across the internet as the IP of your laptop was probably provided by your router at home.
Why not build your Puppet master on an ec2 instance and keep it all on the same network, edit code on your laptop, push to github/gitlab and then deploy the code from there to your PE server using code-manager.
Alternatively you may be able to use a VPN to get your laptop onto the AWS VPC directly in which case it'll appear as just another node on the network and everything should work.
The problem here is that the puppet server needs a public IP or an IP in the same network as your ec2 instance to which your puppet agent can connect to. However, there's one solution without using a VPN though it can't be permanent. You can tunnel your local port to the ec2 instance
ssh -i <pemfile-location> -R 8140:localhost:8140 username#ec2_ip -> This tunnels port 8140 on your ec2 instance to port 8140 in your localhost.
Then inside your ec2 instance you can modify your /etc/hosts file to add this:
127.0.0.1 puppet
Now run the puppet agent on your ec2 instance and everything should work as expected. Also note that if you close the ssh connection created above then the ssh tunnel will stop working.
If you want to keep the ssh tunnel open a bit more reliably then this answer might be helpful: https://superuser.com/questions/37738/how-to-reliably-keep-an-ssh-tunnel-open
I am testing the following configuration.
Cloud SQL (tetsql-1) in Region X Zone A
A Compute Engine VM (TestVM-1) in the same Region X Zone A. OS is Centos 7
Compute Engine VM is running cloud SQL proxy on non default port (9090)
With the above configuration I am able to logon to testsql-1 from TestVM-1 with below command:
`mysql -h 127.0.0.1 --port 9090 -u testuser -D testDB -p`
However I am not able use the internal IP of TestVM-1 in the above command. It gives an error.
Another observation is I am able to do telnet 127.0.0.1 9090 but when I try telnet <VM -Internal-IP> 9090 returns a connection refused error.
Does anyone know if this is expected behaviour? If this is expected, why is it so?
The cloud proxy uses 127.0.0.1 by default, where it accepts connections.
To configure another IP Address, you have to set it in the instances parameter:
./cloud_sql_proxy -instances=<myCloudSQLproject:myCloudSQLzone:mycloudsqlinstance>=tcp:<IP_Address>:<PORT>
Something like this:
./cloud_sql_proxy -instances=project_xxx:us-central1:database_yyy=tcp:10.203.23.12:9090
This configuration allows connecting to this cloud proxy from others hosts as well.
The reason that you can connect to 127.0.0.1 but you cannot connect using the VM's private IP address is that the Proxy is NOT listening on the private IP address.
The Cloud SQL Proxy listens on the loopback adapter's internal address which is 127.0.0.1. This address only exists inside the computer.
You're able to connect from your VM to Cloud SQL because you're using the proxy. If you would like to connect to your Cloud SQL then you have whitelist the IP address of your VM in Cloud SQL's connections tab, please refer to this documentation.
This is expected behavior. Private IPs are only accessible from a Virtual Private Cloud (VPC). In order for a resource (such as a GCE instance) to connect, it must also be on that VPC.
See this page for instructions on how to add a GCE instance to a VPC, and see this page for more on the environment requirements for Private IP.
Situation : So Basically I have Ec2 instance and to login to Ec2 instance, first I will login to jump server and then I login to my instance with private ip, and all my pem files are stored inside the jump server and in private instance(EC2 instance).
What I have Done : I have edited sshd-config file with "PasswordAuthentication No and change it to PasswordAuthentication Yes" and created password like "xyz123" now I able to login to my machine with,
ssh ubuntu#ipadrrs and psswd:xyz123
but this is not serving my open item.
what I am looking for : actually I used to share my .pem file to my team member to access instance, and if I enable password with ip they still can login with username and password and if they leave my org if they have my pem file handy still they can access my instance with pem file or password.
what is the best way to avoid this problem.
will Active directory will help here or LDAP will help here?if so how?
Please help me with quires.
The best solution now is to use AWS Systems Manager Session Manager. This requires no pem sharing. SSH access direct from the browser, I am using this and I did not find any issues. Also, entire sessions can be audited.
Otherwise, rotate your key pairs. But I am not sure if there is an AWS way to do it. Also, use the best practices by enabling security groups to open only to known ips instead of the entire whole.
Here you can convert your jump server to a SSH Bastion server. The same can be done using setting up iptables rules on this server. Below mentioned is the example of a rule which you can setup:
iptables -t nat -A PREROUTING -d xx.xx.xx.xx -p tcp --dport yyyy -j DNAT --to zz.zz.zz.zz:22
Here xx.xx.xx.xx is the private IP of the SSH bastion server. yyyy is the port which will be used for inbound access. zz.zz.zz.zz will be the private IP of the destination server. This rule simply means that you ssh into a machine using port yyyy, which will port forward the traffic to port 22 on zz.zz.zz.zz machine.
In this case you will only have to configure SSH public on the destination machine(zz.zz.zz.zz) only and the client machine will have the private key. Command to connect from the client machine will be ssh -i <path-to-private-key> username#BastionPublicIP -p yyyy
Below mentioned are the ports to be opened at security groups:
Bastion - Inbound - yyyy(from your IP)
Bastion - Inbound - 22(from your IP)
Destionation - Inbound - 22(from bastion machine)
I suggest, you use an Amazon Linux AMI for SSH Bastion server.
I am currently working on an AWS EC# LINUX AMI. I have a private IP. Is it possible to access that private IP with filezilla to transfer files. i am unable to do so.
For access an EC2 machine with private IP, you need to setup your own VPN server. If you already have VPN setup in your AWS cloud then you just need to install a VPN client and login with your credential and you will be able to access EC2 machine or transfer files using filezilla with private IP too. I am assuming that you haven't setup VPN server. you may use AMI of OPENVPN from AWS market place for setup VPN. Below is the good link for getting start.
https://docs.openvpn.net/how-to-tutorialsguides/virtual-platforms/amazon-ec2-appliance-ami-quick-start-guide/
After complete this you have to install OPENVPN in your machine and after Login with your credentials your will able to access your EC2 instance with private IP.
Below is the link for install OPENVPN in Ubuntu machine. For different operating system you can explore site.
https://docs.openvpn.net/getting-started/how-to-install-openvpn-as-software/
OPENVPN is one of the alternative, you can use other also as per your need.
Using 2 ways you can do this
Create a bastion host which will connect to the private instance
Using a port forwarding means tunnelling.
If you are using bastion host for connecting private ec2 instance then this steps will be useful
Using Filezilla to transfer files to a private ec2 instance through a bastion host:-
Note: Keep Pem file same of bastion host and private ec2 instance.
Open terminal or cmd(linux terminal i.e gitbash)
we are connecting to the AWS EC2 instance with one terminal command.
ssh -N -L 1234:<private_instance_ip or Private_DNS>:22 -i <Pem_File> #<Bastion_host_public_ip>
e.g.
ssh -N -L 1234: ip-171-12-21-208.us-east-1.compute.internal:22 -i app_prod.pem ubuntu#ec2-31-92-123-22.us-east-1.compute.amazonaws.com
Note: - For the first time when you enter this command it will ask for Are you sure you want to continue connecting - yes
3.Keep this terminal or cmd open.
If you close this session then the connection is broken
4.Open “FileZilla” application and on “Edit” section -> Click on “Settings”
5.On “Settings” page -> Click on “SFTP” and add PEM file of ec2 instance and click on “OK”
6.Add below entries:-
Host:- 127.0.0.1 or sftp://127.0.0.1
Username:- <your_user>
Password:- Keep empty
Port:- 1234
7.Click on Quick Connect.
Once the connection is established then you can easily transfer files from local to private instance.
See- scp-to-transfer-files-to-a-private-ec2-instance-through-a-bastion-host
https://www.davidbegin.com/using-scp-to-transfer-files-to-a-private-ec2-instance-through-a-bastion-host/