SSH Tunnel through Ubuntu bastion to EC2 instance in private subnet - amazon-web-services

Question

According this AWS doc: Scenario 2: VPC with Public and Private Subnets (NAT) I have my own VPC with two subnets: private and public. In public subnet I have deployed an Ubuntu 16.04 Instance with assigned EIP. It also has next security group inbound rules:
Type Protocol Port Range Source Description
SSH TCP 22 xx.xx.xx.xx/32 Home IP
and outbound accordingly:
Type Protocol Port Range Source Description
SSH TCP 22 sg-xxprivatexx Security group ID for instance in private subnet
Looks nice, I can ssh it externally from my home. No problem.
In private subnet I have deployed another one Ubuntu 16.04 machine with next security group (inbound rules):
Type Protocol Port Range Source Description
HTTP TCP 80 sg-xxpublicxxx Security Group ID for bastion instance in public subnet
SSH TCP 22 sg-xxpublicxxx -
and no outbound rules (actually it has 80, 443 outbound ports opened, but its not an interesting part as I guess). And I still can reach this virtual machine using ssh from my bastion.
Right now I just want to make only one simple thing - run ssh port forwarding so I can run localhost:8080 on my home PC browser and see the webpage I published on my private instance. If I understand it correctly from here and here (and from here as well) I have to run something like:
ssh -N -v -L 8080:10.0.1.112:80 ubuntu#3.121.46.99
Which as I guess basically means: just forward a traffic from private subnet instance with IP 10.0.1.112:80 to my localhost:8080 through my bastion VM with username ubuntu hosted on EIP 3.121.46.99.
Debug ends with lines:
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:ZyVHgnF8z5vE5gfNr1S2JDfjhdydZVTNevPRgJZ+sRA /home/matterai/.ssh/key.pem
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/matterai/.ssh/id_rsa
debug1: Trying private key: /home/matterai/.ssh/id_dsa
debug1: Trying private key: /home/matterai/.ssh/id_ecdsa
debug1: Trying private key: /home/matterai/.ssh/id_ed25519
debug1: No more authentication methods to try.
matterai#3.121.46.99: Permission denied (publickey).
I am playing around it few days and I still can't get what am I doing wrong. Its so strange: I can ssh -A (to allow forwarding) to my bastion, I can ssh to my private instance from bastion. But I cant establish SSH tunnel to see my webpage (in the future it will be mongodb) without an error. Need some advice or point to the right direction, please! Thank you.
UPD#1
Ok then. If I make manual forwarding using my local machine and my bastion, I get an expected result. Basically it means run this command on bastion:
ubuntu#bastion: ssh -v -N -L 5000:localhost:8000 ubuntu#10.0.1.68
After that runs command on local/home machine:
matterai#homepc: ssh -v -N -L 5000:localhost:5000 ubuntu#3.121.46.99
When I make a request to localhost:5000 on my local machine, I can see the result page. May I and how if it's possible to combine this two commands? (spoiler: yes, it's possible: see the answer!)

Answer 1

Ok, it's easy. Hope my answer will help somebody.
You need to use ssh -J option to connect through your bastion virtual machine:
-J [user#]host[:port]
Connect to the target host by first making a ssh connection to
the jump host and then establishing a TCP forwarding to the ulti‐
mate destination from there. Multiple jump hops may be specified
separated by comma characters. This is a shortcut to specify a
ProxyJump configuration directive.
Then you need to forward traffic from your destination virtual machine port (:8000) where the app (or database) started to your localhost port (:5001) using ssh -L:
-L [bind_address:]port:host:hostport
-L [bind_address:]port:remote_socket
-L local_socket:host:hostport
-L local_socket:remote_socket
Specifies that connections to the given TCP port or Unix socket
on the local (client) host are to be forwarded to the given host
and port, or Unix socket, on the remote side. This works by
allocating a socket to listen to either a TCP port on the local
side, optionally bound to the specified bind_address, or to a
Unix socket. Whenever a connection is made to the local port or
socket, the connection is forwarded over the secure channel, and
a connection is made to either host port hostport, or the Unix
socket remote_socket, from the remote machine.
Port forwardings can also be specified in the configuration file.
Only the superuser can forward privileged ports. IPv6 addresses
can be specified by enclosing the address in square brackets.
By default, the local port is bound in accordance with the
GatewayPorts setting. However, an explicit bind_address may be
used to bind the connection to a specific address. The
bind_address of “localhost” indicates that the listening port be
bound for local use only, while an empty address or ‘*’ indicates
that the port should be available from all interfaces.
Full ssh command will look like:
matterai#homepc: ssh -v -N -A -J ubuntu#3.121.46.99 -L 5001:localhost:8000 ubuntu#10.0.1.112
UPD: Also you can simplify a bit your command. In ~/.ssh/config you can add your jumphost (bastion) and your final destination VM IP:
Host bastion
HostName 3.121.46.99
User ubuntu
Port 22
IdentityFile ~/.ssh/secret.pem
ForwardAgent yes
Host server
HostName 10.0.1.112
User ubuntu
Port 22
IdentityFile ~/.ssh/secret.pem
ProxyJump bastion
Now, you can run command:
ssh -v -N -A -J bastion -L 80:localhost:8000 server
Looks much better. Also you can just simply connect via ssh using ssh server.

Answer 2

You seem to have things correctly configured, but the error is saying that it can't find a private key to use for the connection.
To test the port forwarding, start by using the ssh command that logs into your public instance.
Then, take that exact command, and simply add: -L 8080:10.0.1.112:80
If it works for 'normal' ssh, then it will work with port forwarding too.
By the way, in general you should never need to modify the outbound rules of a security group. The default settings permit all outbound traffic. This 'trusts' the apps running on the instance and allows them to communicate outwards to anywhere. You would only need to restrict such rules where you wish to enforce a high-security environment.