I have been able to ssh into my AWS ECS instances for well over a year without issue.
However, in recent days I have begun getting Permission denied (publickey) errors when trying to ssh into any of my instances.
I have the public keys stored in my .ssh folder and they get added to the ssh agent successfully before any attempt to ssh. My AWS setup has also not changed.
I have included the verbose output from my ssh command below. Any help or insights are much appreciated!
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to 10.10.0.168 [10.10.0.168] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ec2-user/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.10.0.168:22 as 'ec2-user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:qi60acx6tKhNsV3z756IaixRe0bXlIyuCc3782hn8rY
debug1: Host '10.10.0.168' is known and matches the ECDSA host key.
debug1: Found key in /home/ec2-user/.ssh/known_hosts:33
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ec2-user/.ssh/id_rsa
debug1: Trying private key: /home/ec2-user/.ssh/id_dsa
debug1: Trying private key: /home/ec2-user/.ssh/id_ecdsa
debug1: Trying private key: /home/ec2-user/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).
Have you tried restarting the instances in the AWS Console?
I've seen this previously when the instances have run our of memory due to temp files. Rebooting cleared the temp files and allowed me to connect again.
It's not exactly the most helpful error message!
Related
In AWS, I have created a Bastion host (10.0.10.182) using Amazon Linux 2 and from there I am able to connect to a EC2 private subnet instance (10.0.20.121) (amazon linux 2). (However, this works fine only for the first time.)
After connecting to the Private instance, in order to pull a git repo from github on the private instance, I run the ssh-keygen on the private instance and copy that to github Keys. I can see the .ssh dir in the home dir with the usual files - known_hosts, authorized_keys, id_rsa, id_rsa.pub.
When the original connection from bastion host to private ec2instance times out, I am unable to re-login to the private instance via the bastion host. I get the following message:
ssh -i TestVPC_NCal.pem ec2-user#10.0.20.121
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Here is the ssh debug log generated on the Bastion host:
######Begin ssh debug log ######### [ec2-user#ip-10-0-10-182 ~]$ ssh -v -i TestVPC_NCal.pem ec2-user#10.0.20.121 OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 58: Applying
options for * debug1: Connecting to 10.0.20.121 [10.0.20.121] port 22.
debug1: Connection established. debug1: key_load_public: No such file
or directory debug1: identity file TestVPC_NCal.pem type -1 debug1:
key_load_public: No such file or directory debug1: identity file
TestVPC_NCal.pem-cert type -1 debug1: Enabling compatibility mode for
protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.4 debug1:
Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000 debug1:
Authenticating to 10.0.20.121:22 as 'ec2-user' debug1:
SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex:
algorithm: curve25519-sha256 debug1: kex: host key algorithm:
ecdsa-sha2-nistp256 debug1: kex: server->client cipher:
chacha20-poly1305#openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC:
compression: none debug1: kex: curve25519-sha256 need=64
dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1:
expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key:
ecdsa-sha2-nistp256 SHA256:5W++Ewk+lx2YXUUY1xhhttjKG3KVWvIOTvtp7THBFJc
debug1: Host '10.0.20.121' is known and matches the ECDSA host key.
debug1: Found key in /home/ec2-user/.ssh/known_hosts:2 debug1: rekey
after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting
SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after
134217728 blocks debug1: SSH2_MSG_EXT_INFO received debug1:
kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that
can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next
authentication method: gssapi-keyex debug1: No valid Key exchange
context debug1: Next authentication method: gssapi-with-mic debug1:
Unspecified GSS failure. Minor code may provide more information No
Kerberos credentials available (default cache:
KEYRING:persistent:1000)
debug1: Unspecified GSS failure. Minor code may provide more
information No Kerberos credentials available (default cache:
KEYRING:persistent:1000)
debug1: Next authentication method: publickey debug1: Trying private
key: TestVPC_NCal.pem debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic debug1: No more authentication
methods to try. Permission denied
(publickey,gssapi-keyex,gssapi-with-mic). [ec2-user#ip-10-0-10-182 ~]$
########### End debug log ########
I wonder if the running the ssh-keygen on EC2 private instance is somehow causing the error. Any pointers to resolve this are very welcome!
I was able to resolve this issue by creating the TestVPC_NCal.pem file (associated with the EC2 private instance) on the EC2 private instance while the connection was working. The clue was in the log I posted in my question:
##############
Connecting to 10.0.20.121 [10.0.20.121] port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file TestVPC_NCal.pem type -1 debug1: key_load_public: No such file or directory debug1: identity file TestVPC_NCal.pem-cert type -1 debug1:
##############
When the connection timed-out overnight, I was able to log back in to EC2 private instance with no issues.
I'm trying to connect to the problematic Amazon EC2 instance with this command:
ssh -vvv -i MY_KEY.pem root#ec2-XX-XX-XXX-XX.compute-1.amazonaws.com
and getting this result:
OpenSSH_7.5p1, OpenSSL 1.1.0g-fips 2 Nov 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug1: Connecting to ec2-XX-XX-XXX-XX.compute-1.amazonaws.com [XX.XX.XXX.XX] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file MY_KEY.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file MY_KEY.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ec2-XX-XX-XXX-XX.compute-1.amazonaws.com:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256#libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm#openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256#libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256#libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:rRMQg/hs
debug1: Host 'ec2-XX-XX-XXX-XX.compute-1.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /home/dina/.ssh/known_hosts:8
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: MY_KEY.pem
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Offering RSA public key: dina#localhost.localdomain
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Trying private key: MY_KEY.pem
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: keyboard-interactive
Password:
The key MY_KEY.pem should be correct, I've checked the fingerprint under NETWORK & SECURITY -> Key Pairs on AWS console, and it's the same as for the key I'm using.
Permissions for the .pem file is 400, owner is current owner. SSH inbound traffic to the instance is allowed from all addresses.
Also tried this with different users (root, ubuntu, admin, ec2-user, centos, fedora) as I have no contact with the person who launched the instance and have no idea what system is there.
Any idea about what is going on and what could be done about it? Your help is very much appreciated
What AMI are you using?
If you are using an Ubuntu AMI, you should connect as ubuntu.
$ ssh -vvv -i ~/.ssh/MY_KEY.pem ubuntu#ec2-XX-XX-XXX-XX.compute-1.amazonaws.com
On Amazon Linux, the default user is ec2-user.
$ ssh -vvv -i MY_KEY.pem ec2-user#ec2-XX-XX-XXX-XX.compute-1.amazonaws.com
Once you are in both of these users will have sudo privileges.
I am trying to connect to my ec-2 free(t2.micro) instance through ssh from my PC.
I have created instance with default VPC. I am not able to connect it from my PC.
It is throwing me permission denied error.
I have checked the rules in the security group.
I have gone through the below url's to check the answer but no success.
AWS SSH connection error: Permission denied (publickey)
Troubleshooting Connecting to Your Instance
SSH: Permission denied (publickey)
Also when I run below command
sudo ssh -v -i tep-keyPair.pem ubuntu#ec2-52-XX-XXX-XX.us-west-2.compute.amazonaws.com
The below error came:
OpenSSH_7.3p1, LibreSSL 2.4.1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug1: Connecting to ec2-52-XX-XXX-XX.us-west-2.compute.amazonaws.com [52.XX.XXX.XX] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file tep-keyPair.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file tep-keyPair.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Authenticating to ec2-52-XX-XXX-XX.us-west-2.compute.amazonaws.com:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256#libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ZeJ4XQUfgLkaMUEvjGohL/6FWKN9Gq4AXrPwL/i9t3M
debug1: Host 'ec2-52-XX-XXX-XX.us-west-2.compute.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /var/root/.ssh/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: tep-keyPair.pem
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
Please help me.. I am really stuck here
The fact that you are receiving a Permission denied (publickey) error indicates that you are successfully communicating with the instance, so the problem is not related to networking nor security groups.
Rather, the instance is not accepting a connection via the keypair you are providing. Therefore, you either need to provide it with the keypair it expects, or you can copy a new keypair to the instance.
To copy a different keypair to the instance, follow instructions on this StackOverflow answer, which is written for Ubuntu: Change key pair for ec2 instance
You can change an EC2 instance key pair when deploying with ElasticBeanstalk, which is great. EB supposedly terminates current instance and launches a new one with whatever key pair you specified. So I created a new key pair (AWS console), downloaded a new *.pem file, updated my machine to use the key pair in EB. All good so far, EB re-launched the server. One little issue with this is that I can't ssh into the instance using the latest *.pem file. This is what I get (and I did do "chmod 400" on the pem file):
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to 52.1.*.* [52.1.*.*] port 22.
debug1: Connection established.
debug1: identity file mypem.pem type -1
debug1: identity file mypem.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 6b:5a:e2:0c:c5:98:ff:34:6e:c6:2c:84:ea:a0:88:0f
debug1: Host '52.1.*.*' is known and matches the RSA host key.
debug1: Found key in /Users/sergey.novgorodsky/.ssh/known_hosts:2
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: mypem.pem
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
Could it be an EB issue? Any ideas?
ssh -i [full path to keypair file] ec2-user#[EC2 instance hostname or IP address]
I did this and it worked before and suddenly I am getting Permission denied (publickey) error.
mac-pro:aws me$ ssh -i key.pem ubuntu#elastic_ip_address -v
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug1: Connecting to elastic_ip_address [elastic_ip_address] port 22.
debug1: Connection established.
debug1: identity file key.pem type -1
debug1: identity file key.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1 Debian-4ubuntu4
debug1: match: OpenSSH_5.5p1 Debian-4ubuntu4 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'elastic_ip_address' is known and matches the RSA host key.
debug1: Found key in /Users/me/.ssh/known_hosts:8
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: key.pem
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
Incidentally, I am using the same key for three different EC2 instances. I am not sure this is the reason why I am getting this error. I can access two other EC2 instances over ssh using the same key. BUT I can not access only one instance.
correct user? ec2-user# or root#