Overview
I'm trying to configure ~/.ssh/config to connect my local VSCode to remote (EC2). I've done a lot of testing and can't understand why situation one is working and others are failing. I'm able to get RemoteCommand to work successfully for ssh'ing into EC2 instance after ssh'ing into BastionHost, however I'm unable to achieve the same with ProxyJump or ProxyCommand. VSCode doesn't list the EC2 filesystem when using the RemoteCommand example (just gets to BastionHost), so thinking I'll need to resolve to either ProxyJump/ProxyCommand based on most of the documentation.
I've tried to follow the instructions here exactly along with trying out different approaches from other articles to no avail.
##WORKS
Host dev-ec2
HostName 10.248.000.206
User meme1
RemoteCommand ssh 10.248.000.201
RequestTTY yes
IdentityFile ~/.ssh/mykey
##WORKS
Host bastion-dev
HostName 10.248.000.206
User meme1
IdentityFile ~/.ssh/mykey
RequestTTY yes
##FAILS (times out)
Host dev-ec2-proxycommand
HostName 10.248.000.201
User meme1
ProxyCommand ssh.exe bastion-dev -W %h:%p
##FAILS (Permission denied on public key, even though no issue in the RemoteCommand example)
Host ec2-dev-proxyjump
HostName 10.248.000.201
User meme1
ProxyJump bastion-dev
IdentityFile ~/.ssh/mykey
System Info
OS: Windows 10
Bastion OS: Linux (Amazon Linux AMI)
Disclaimer
I've been trolling StackOverflow and other forums for the past couple days to no avail, and although I've found similar questions none have provided viable answers for resolving.
I suppose this one below is failing because you are using a command from Windows at your bastion, that is Linux.
Command ssh.exe will not work on Linux. Everything that you put on ProxyCommand will run inside your bastion host, in your case it is a Linux OS.
Also make sure your instance Security Group allow connection from bastion IP, not from your computer.
##FAILS (times out)
Host dev-ec2-proxycommand
HostName 10.248.000.201
User meme1
ProxyCommand ssh.exe bastion-dev -W %h:%p
I have the config below on my ~/.ssh/config file and it works fine for me to connect on my instance behind bastion host.
Bastion IP: 172.31.4.238
Host IP (behind bastion): 172.31.11.98
Host 172.31.11.98
HostName 172.31.11.98
User ec2-user
ProxyCommand ssh -W %h:%p ec2-user#172.31.4.238
See it below
$ ssh 172.31.11.98
The authenticity of host '172.31.11.98 (<no hostip for proxy command>)' can't be established.
ECDSA key fingerprint is SHA256:vy....
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '172.31.11.98' (ECDSA) to the list of known hosts.
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-2/
[ec2-user#ip-172-31-11-98 ~]$
Related
When I try to connect to an EC2 in a private network through a bastion server I get this message:
<username>#<ec2-server>: Permission denied (publickey)
However, I can ssh to bastion from my local machine, and I can ssh to the EC2 from the bastion server,
Here is the .ssh/config I'm using:
Host <ec2-servers>*
IdentityFile ~/.ssh/id_rsa
User <username>
Here is the command I use to ssh:
ssh -J <bastion-server> <ec2-server>
Note: Permissions are good (700 for ~/.ssh/ and 600 for ~/.ssh/*)
Thanks in advance for your help!
There is likely no user on the remote system called 'username'. Make sure both systems have the same username and public key.
can you try the following configuration as the username you mentioned as same for both jump host and actual instance you are trying to connect to?
Host 10.2.2.* #ec2 servers cidr range
ProxyJump jumpuser#proxy.example.com
I think specifying the IdentityFile ~/.ssh/id_rsa might not be needed as that seems like the default key on your system you are using.
Make sure jumpuser exists with appropriate permissions.
Just fo debugging purposes , run this manually with debug options
ssh -vvv -J username#host1:port username#host2:port
will give plenty of information and you might be able to see where the problem is.
If you are using ssh-agent it remove all the identities and trying might also help.
ssh-add -D
How to Access a Remote Server Using a Jump Host
How to Set Up an SSH Jump Server
Just note that RSA keys are being depreciated, and later versions of operating systems disable their use on the CLIENT. That is, where you ssh from. To re-enable it on the client, in your ~/.ssh/config file, enter the following line:
PubkeyAcceptedKeyTypes +ssh-rsa
Note: there are security implications of doing this, so read up on the security issues of rsa if you are concerned. For instance, the following article:
https://www.thesslstore.com/blog/is-it-still-safe-to-use-rsa-encryption/
says:
....RSA encryption provides less than 99.8% security.
That sounds negligible, it’s about two in every 1,000.
But does that mean RSA is cracked? Not quite, just vulnerable..
Fixed it by adding local ssh public key in the authorized_keys of the remote ec2 instance.
Background
Three subnets exist in an AZ in AWS. Two of them are private and one is public.
The Public Subnet has a Jumpbox which can be connected to from my local machine via ssh using a pem file (Sample - ssh -i my-key-file.pem ec2-user#host1
The First private subnet has an EC2 Instance that acts as a Application Server. It can only be reached from the Jumbox via ssh. The same pem file is used here. (Sample - ssh -i my-key-file.pem ec2-user#host2). This command is executed on host1.
The second private subnet hosts an Oracle Instance using AWS RDS Service. It is running on port 1521. The DB Can only be accessed from the App Server/host2.
How I am working currently
host2 has sqlplus client installed.
First,I connect to host1, then to host2, and then execute sqlplus to execute Queries at the Command line (No GUI).
I am planning to use GUI tools like SQL Developer to connect right from my local machine. I thought using Port forwarding/SSH Tunneling It can be achieved.
I tried using different options, but with no success. The following links are useful:
https://superuser.com/questions/96489/an-ssh-tunnel-via-multiple-hops
https://rufflewind.com/2014-03-02/ssh-port-forwarding
My Approach to SSH Tunneling
ssh -N -L 9999:127.0.0.1:1234 ec2-user#host1 -i my-key-file.pem -v -v -v
This is executed on my local machine.
It does not do much as I can already connect to host1 using ssh. I did not know how to forward many levels. Using this host as my first hop. After this, ssh listens on port 9999 which is Local to my machine. It forwards any traffic to host1 to Port 1234. My assumption is that, If I use sqlplus on my local machine connecting to localhost:9999, the traffic will arrive at host1:1234
I used 127.0.0.1 because the target of SSH tunneling is with respect to the SSH Server, which is host1. Basically, Both Target, SSH Server are on the same host.
ssh -N -L 1234:db-host:1521 ec2-user#host2 -i my-key-file.pem -v -v- v
This is executed on host1
After this, ssh forwards any incoming traffic on port 1234 to target host (DB Host)/1521 using host2 as the Tunnel.
Again, my assumption is that, ssh is listening on port 1234 on host1. Any traffic arriving from anywhere will be delivered to DB Host using host2 as the tunnel.
I executed both commands and did not see any error. I verified which ports are listening using netstat -tulpn | grep LISTEN.
After these two, My plan was to connect to the Database using Hostname as localhost and Port number as 9999.
What's going wrong !
But when I try to connect to the DB from my local machine, getting an error from my SQL Client Got minus one from a read call. I could not understand the Debug messages from ssh logs.
I believe my understanding of how port forwarding works might not be right. Any inputs would be helpful.
Thanks for your time !
I have my NAT and Bastion set up to login with SSH forwarding:
ssh-add -K keyfile.pem
ssh -A ec2-user#bastionhost
ssh ec2-user#privateSubnetServer
What's the best method for handling ssh and users at this point between hosts in the private subnet?
I get:
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
When trying to do it the traditional linux way. I can get to them if I use the AWS created key pairs.
I can't set up or connect to a directory service at this point.
There can be two reasons, either the key is invalid or the proxy command not working as you expecting.
Before that, you set you proxy command like this
host bastion
HostName bastion_Adress
User centos|whatever
identityFile /mykeys/ec2.pem
Now try to ssh to bastion
ssh bastion
If the above then bastion working fine, we can set proxy command now.
host private_server
Hostname 10.0.5.45
user centos
IgnoreUnknown UseKeychain
AddKeysToAgent yes
ProxyCommand ssh bastion -W %h:%p
Now you can ssh to private server
ssh private_server
Normally these are necessary but just in case of mac happen sometime
IgnoreUnknown UseKeychain
AddKeysToAgent yes
.ssh/config: "Bad configuration option: UseKeychain" on Mac OS Sierra 10.12.6
You can debug the issue using this flow
ssh to bastion
copy your private server ssh key to the bastion
ssh to private instance
If the above worked it mean bastion ssh config is not valid
If the above does not work then the key is not valid.
Host bastion.ip.address
User ec2-user
IdentityFile ~/.ssh/bastionkey.pem
CheckHostIP no
Host private.subnet.ip
User ec2-user
IdentityFile ~/.ssh/bastionkey.pem
ProxyCommand ssh ec2-user#bastion.host.ip -W %h:%p
I understand that it is not recommended to ask multiple question in a single post but they are all tightly coupled hence asking them under one post.
I was trying to SSH to a private EC2 instance in a VPC from the internet using Bastion host. I came across 2 approaches.
NOTE When I launched my private EC2 instance and Bastion host I chose different Keys.(In both the approaches I saw the authors used same Keys for EC2 and the Bastion)
Approach 1: Configuring SSH ProxyCommand as illustrated in https://www.youtube.com/watch?v=EpFAHis4O4g
Approach 2: ssh-agent command with ssh -A option https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/
Question 1: In approach 2, I did
ssh-add ~/Downloads/Ec2.pem
ssh-add ~/Downloads/BastionKey.pem
Is my approach of adding both the keys to the ssh-agent correct? This way I was able ssh into Bastion as a first step and then as a second step I had to explicitly ssh into private EC2.
Question 2: Approach 1 is against the idea of (Approach 2) 2 step ssh process but proposes ProxyCommand so that end user should be able to directly ssh into the private EC2 in a single step. This way a system admin of Bastion host will not have control of ssh-ing to individual EC2 instances.In fact the author demonstrates that if you ssh into Bastion host (it will succeed) and then ssh to EC2 it fails. Is my understanding correct here?
Question 3: Approach 1 is not working for me. I used the same structure for ~/.ssh/config but with different Key file paths. I am unable to ssh to my private EC2 directly. I even tried the 2 step process but I can only login into Bastion host, I cannot ssh to EC2. All my Security groups, ACL, Internet gateway, NAT gateway and VPC settings are fine, otherwise Approach 2 wouldn't have worked. What could I be doing wrong?
cat ~/.ssh/config
HOST bastion
Hostname ec2-5x-xx-xx-xx.compute-1.amazonaws.com
User ec2-user
IdentityFile /Users/myname/Downloads/BastionKey.pem
HOST *.ec2.internal
User ec2-user
IdentityFile /Users/myname/Downloads/Ec2.pem
ProxyCommand ssh -q -W %h:%p bastion
Question 4: Which is the recommend approach among 1 & 2 and what additional steps must be followed when different keys are chosen for Bastion and EC2?
Error output:
ssh -v ip-10-0-1-12.ec2.internal
OpenSSH_7.8p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/myname/.ssh/config
debug1: /Users/myname/.ssh/config line 6: Applying options for *.ec2.internal
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Executing proxy command: exec ssh -q -W ip-10-0-1-12.ec2.internal:22 bastion
debug1: identity file /Users/myname/Downloads/Ec2.pem type -1
debug1: identity file /Users/myname/Downloads/Ec2.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
/bin/false: No such file or directory
ssh_exchange_identification: Connection closed by remote host
HOST bastion
Hostname ec2-example-ip.compute-1.amazonaws.com
User ec2-user
IdentityFile /Users/myname/Downloads/BastionKey.pem
HOST *.ec2.internal
User ec2-user
IdentityFile /Users/myname/Downloads/Ec2/Ec2.pem
ProxyCommand ssh -q -W %h:%p bastion
Try using the bastions public DNS name in your ssh config, also use the internal ec2 DNS name for the one behind the VPC. (AWS DNS is sketchy at best sometimes)
Note: *.ec2.internal assums you will use the same ssh key for each ec2 you need to access proxying through your bastion. If this is not the case replace *.ec2.internal with whatever-internal-ip.ec2.internal and add an entry for each individual ec2.
Hope this can fix your issue.
Situation : So Basically I have Ec2 instance and to login to Ec2 instance, first I will login to jump server and then I login to my instance with private ip, and all my pem files are stored inside the jump server and in private instance(EC2 instance).
What I have Done : I have edited sshd-config file with "PasswordAuthentication No and change it to PasswordAuthentication Yes" and created password like "xyz123" now I able to login to my machine with,
ssh ubuntu#ipadrrs and psswd:xyz123
but this is not serving my open item.
what I am looking for : actually I used to share my .pem file to my team member to access instance, and if I enable password with ip they still can login with username and password and if they leave my org if they have my pem file handy still they can access my instance with pem file or password.
what is the best way to avoid this problem.
will Active directory will help here or LDAP will help here?if so how?
Please help me with quires.
The best solution now is to use AWS Systems Manager Session Manager. This requires no pem sharing. SSH access direct from the browser, I am using this and I did not find any issues. Also, entire sessions can be audited.
Otherwise, rotate your key pairs. But I am not sure if there is an AWS way to do it. Also, use the best practices by enabling security groups to open only to known ips instead of the entire whole.
Here you can convert your jump server to a SSH Bastion server. The same can be done using setting up iptables rules on this server. Below mentioned is the example of a rule which you can setup:
iptables -t nat -A PREROUTING -d xx.xx.xx.xx -p tcp --dport yyyy -j DNAT --to zz.zz.zz.zz:22
Here xx.xx.xx.xx is the private IP of the SSH bastion server. yyyy is the port which will be used for inbound access. zz.zz.zz.zz will be the private IP of the destination server. This rule simply means that you ssh into a machine using port yyyy, which will port forward the traffic to port 22 on zz.zz.zz.zz machine.
In this case you will only have to configure SSH public on the destination machine(zz.zz.zz.zz) only and the client machine will have the private key. Command to connect from the client machine will be ssh -i <path-to-private-key> username#BastionPublicIP -p yyyy
Below mentioned are the ports to be opened at security groups:
Bastion - Inbound - yyyy(from your IP)
Bastion - Inbound - 22(from your IP)
Destionation - Inbound - 22(from bastion machine)
I suggest, you use an Amazon Linux AMI for SSH Bastion server.