I'm connecting to a t2.medium EC# instance using ssh. I was able to connect to the instance before and all of a sudden I started getting the following error:
ssh -i my-test.pem ubuntu#ec2-54-221-184-238.compute-1.amazonaws.com -v
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to ec2-54-221-184-238.compute-1.amazonaws.com port 22.
debug1: Connection established.
debug1: identity file my-test.pem type -1
debug1: identity file my-test.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to ec2-54-221-184-238.compute-1.amazonaws.com:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
Connection reset by 54.221.184.238 port 22
To resolve the issue I tried the following solutions but to no avail:
Rebooted the EC2 instance
Created a new keyPair and changed the user data of EC2 instance with the new Keypair
Enabled all TCP traffic from inbound of the security group
Any idea what is the issue?
Related
The difference between this question and all the others comes down to this: yesterday, I could connect via SSH just fine using this command:
ssh -i "~/.ssh/[.cer file]" ubuntu#[Public IPv4 DNS]
(A .cer file is created instead of .pem since I use Chrome.) Then I shut down the instance and deleted the volume (after creating a snapshot) then I went home.
Today, I did the following:
Created a new volume from the snapshot, and attached it to the instance
Created an Elastic IP address and assigned it to the instance
Started the instance
Tried connecting via SSH using the new Public IPv4 DNS. I got a Permission denied (publickey) error.
Thinking it may have something to do with the Elastic IP, I dissociated the Elastic IP and rebooted the instance to get a new temporary IP.
I tried using ssh using the same command (with the new Public DNS) and am still getting the Permission denied (publickey) error.
Here is the log I get when adding -v to the command:
OpenSSH_8.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/WonderWolff/.ssh/config
debug1: /Users/WonderWolff/.ssh/config line 14: Applying options for *.compute.amazonaws.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to ec2-13-57-238-91.us-west-1.compute.amazonaws.com port 22.
debug1: Connection established.
debug1: identity file /Users/WonderWolff/.ssh/rei_development.cer type -1
debug1: identity file /Users/WonderWolff/.ssh/rei_development.cer-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.10
debug1: compat_banner: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.10 pat OpenSSH_6.6.1* compat 0x04000002
debug1: Authenticating to ec2-13-57-238-91.us-west-1.compute.amazonaws.com:22 as 'ubuntu'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256#libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-ctr MAC: umac-128-etm#openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-128-etm#openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:LnRbxnhhpoLZeIUFXFzOybmc+cPvutkYqZCmUmq+zVw
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
Warning: Permanently added 'ec2-13-57-238-91.us-west-1.compute.amazonaws.com' (ED25519) to the list of known hosts.
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /Users/WonderWolff/.ssh/rei_development.cer explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
Unauthorized use is strictly prohibited. All access and activity
is subject to logging and monitoring.
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/WonderWolff/.ssh/rei_development.cer
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
ubuntu#ec2-13-57-238-91.us-west-1.compute.amazonaws.com: Permission denied (publickey).
If it worked yesterday after multiple stops and reboots (I stopped and rebooted multiple times yesterday), what could possibly be the issue? Was attaching a new volume created from a snapshot the issue?
UPDATE: I created a new instance using the same keys. Connecting to that instance works, no problem. But I would still like to understand why either the removal and re-association of a volume, OR the assigning an Elastic IP address, OR a different reason I don't yet know causes me to get a Permission Denied error with the first instance.
In AWS, I have created a Bastion host (10.0.10.182) using Amazon Linux 2 and from there I am able to connect to a EC2 private subnet instance (10.0.20.121) (amazon linux 2). (However, this works fine only for the first time.)
After connecting to the Private instance, in order to pull a git repo from github on the private instance, I run the ssh-keygen on the private instance and copy that to github Keys. I can see the .ssh dir in the home dir with the usual files - known_hosts, authorized_keys, id_rsa, id_rsa.pub.
When the original connection from bastion host to private ec2instance times out, I am unable to re-login to the private instance via the bastion host. I get the following message:
ssh -i TestVPC_NCal.pem ec2-user#10.0.20.121
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Here is the ssh debug log generated on the Bastion host:
######Begin ssh debug log ######### [ec2-user#ip-10-0-10-182 ~]$ ssh -v -i TestVPC_NCal.pem ec2-user#10.0.20.121 OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 58: Applying
options for * debug1: Connecting to 10.0.20.121 [10.0.20.121] port 22.
debug1: Connection established. debug1: key_load_public: No such file
or directory debug1: identity file TestVPC_NCal.pem type -1 debug1:
key_load_public: No such file or directory debug1: identity file
TestVPC_NCal.pem-cert type -1 debug1: Enabling compatibility mode for
protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.4 debug1:
Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000 debug1:
Authenticating to 10.0.20.121:22 as 'ec2-user' debug1:
SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex:
algorithm: curve25519-sha256 debug1: kex: host key algorithm:
ecdsa-sha2-nistp256 debug1: kex: server->client cipher:
chacha20-poly1305#openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC:
compression: none debug1: kex: curve25519-sha256 need=64
dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1:
expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key:
ecdsa-sha2-nistp256 SHA256:5W++Ewk+lx2YXUUY1xhhttjKG3KVWvIOTvtp7THBFJc
debug1: Host '10.0.20.121' is known and matches the ECDSA host key.
debug1: Found key in /home/ec2-user/.ssh/known_hosts:2 debug1: rekey
after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting
SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after
134217728 blocks debug1: SSH2_MSG_EXT_INFO received debug1:
kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that
can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next
authentication method: gssapi-keyex debug1: No valid Key exchange
context debug1: Next authentication method: gssapi-with-mic debug1:
Unspecified GSS failure. Minor code may provide more information No
Kerberos credentials available (default cache:
KEYRING:persistent:1000)
debug1: Unspecified GSS failure. Minor code may provide more
information No Kerberos credentials available (default cache:
KEYRING:persistent:1000)
debug1: Next authentication method: publickey debug1: Trying private
key: TestVPC_NCal.pem debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic debug1: No more authentication
methods to try. Permission denied
(publickey,gssapi-keyex,gssapi-with-mic). [ec2-user#ip-10-0-10-182 ~]$
########### End debug log ########
I wonder if the running the ssh-keygen on EC2 private instance is somehow causing the error. Any pointers to resolve this are very welcome!
I was able to resolve this issue by creating the TestVPC_NCal.pem file (associated with the EC2 private instance) on the EC2 private instance while the connection was working. The clue was in the log I posted in my question:
##############
Connecting to 10.0.20.121 [10.0.20.121] port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file TestVPC_NCal.pem type -1 debug1: key_load_public: No such file or directory debug1: identity file TestVPC_NCal.pem-cert type -1 debug1:
##############
When the connection timed-out overnight, I was able to log back in to EC2 private instance with no issues.
I have 2 EC2 instances - one is in a private subnet connected to a nat gateway and the other is in a public subnet connected to an internet gateway. I have a jenkins server on the private subnet and a bastion server in the public subnet.
I used scp to copy the pem key used when creating my jenkins instance over to my bastion server and am now trying to connect to the jenkins instance via the bastion.
I run the command ssh -i pem.key ubuntu#privateipaddress but keep getting a permission denied (publickey) error.
I've checked my security group settings for both instances and they check out. The inbound rules for the jenkins security group allow SSH and HTTP traffic from the bastion security group and the outbound rules for the jenkins security group currently allow all traffic through.
I have the reverse for the bastion security group.
EDIT: Verbose:
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 10.0.1.49 [10.0.1.49] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file key.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file key.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.0.1.49:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:EgAjWBAxVLz8L+MQNQXZeIwh51QZOPxPhvugsxv1XGs
debug1: Host '10.0.1.49' is known and matches the ECDSA host key.
debug1: Found key in /home/ubuntu/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: key.pem
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
ubuntu#10.0.1.49: Permission denied (publickey).
I managed to fix this problem by following this guide: https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/
I had come across it before, but never considered trying because a trainer on my course said I wouldn't need to.
While Installing SSL Certificate on AWS, I renamed the old crt files (like a fool), and now I'm no longer able to access the AWS instance via SSH to fix the file names.
Is there a way to access my AWS instance's files other than SSH? Or, is there a way to continue using SSH to access the instance when the SSL certificates are down?
Originally when I was connecting after the change of file names, I got this prompt response from the server when SSH:
> ssh -vv -i tc.pem ubuntu#tc.com
OpenSSH_7.5p1, OpenSSL 1.0.2l 25 May 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "tc.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to tc.com [1.1.1.1] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file tc.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file tc.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to tc.com:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
Connection closed by 1.1.1.1 port 22
At one point I did a reboot on the server and now, I'm getting this when I try to SSH in:
> ssh -i tc.pem ubuntu#tc.com
ssh: connect to host tc.com port 22: Connection timed out
If I could get into the server or access the files in any way to rename them, it would be an easy fix. Appreciate any help anybody can offer.
This is fairly easy to do. Basically you will mount your EC2 instance's root volume (EBS volume) on another instancce, edit .ssh/authorized_keys and then reattach the volume back to your instance.
Here is an article that covers this step-by-step:
SSH Key Access Recovery on EC2 Instances
There are a number of similar articles on the Internet.
I have a Wowza instance purchased from Marketplace, about 4 hrs ago I was able to connect it and was doing some operations on it. But now when I am trying to connect it gives Permission denied (publickey) error. I used chmod command to give 777 permissions on ./ And here is the real cause I think.
From this solution when I am trying to attach the root volume of this instance to other instance of same type it is giving the same error, but when voulme is detached it is being connected successfully. The volume of marketplace instance can be attached to similar type of instance only.
Now how can I change the permission of that .ssh directory? Can you please help me out.
Here are the logs when I am trying to connect with the instance:
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 54.69.133.11 [54.69.133.11] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file Desktop/Shivam/mypemfile.pem type -1
debug1: identity file Desktop/Shivam/mypemfile.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 6a:48:f5:14:13:9a:a1:79:e8:44:bb:0d:6b:8d:b7:59
The authenticity of host '54.69.133.11 (54.69.133.11)' can't be established.
ECDSA key fingerprint is 6a:48:f5:14:13:9a:a1:79:e8:44:bb:0d:6b:8d:b7:59.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.69.133.11' (ECDSA) to the list of known hosts.
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: Desktop/Shivam/mypemfile.pem
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
Products launched from the AWS Marketplace have restrictions that non-marketplace instances don't have mainly to enforce licensing. You have run into one of the restrictions.
Here is a relevant thread on the AWS Forums that may be of use, especially the final post in the thread:
If a volume has an AWS Marketplace product code:
The volume can only be attached to a stopped instance.
You must be subscribed to the AWS Marketplace code that is on the volume.
The configuration (instance type, operating system) of the instance must support that specific AWS Marketplace code. For example, you
cannot take a volume from a Windows instance and attach it to a Linux
instance.
AWS Marketplace product codes are copied from the volume to the instance.
I did not check, but you can try this trick.
You can specify user data to configure an instance or run a configuration script during launch.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts
When you create a new instance
Choose AMI from Market Place
Choose Instance Type
Configure Instance -> Advanced Details -> Used data.
Put here you bash script to change permissions back.
Add Storage
Change root device or add a new one from the snapshot of your EBS.
Tag Instance
Configure Security Group
Review and Launch