Can connect with existing SSH key but not a new one - google-cloud-platform

I'm trying to give access to a GCE VM by adding SSH Keys to the project metadata. My current SSH key is in the project metadata and I can connect just fine using:
ssh -i ~/.ssh/<private_key> <username>#<instance_ip>
Now, I generated another key:
ssh-keygen -t rsa -f ~/.ssh/<new_key> -C <new_username>
After adding the generated public key to the project metadata, I then run:
ssh -i ~/.ssh/<new_private_key> <new_username>#<instance_ip>
But I get Permission denied (publickey,gssapi-keyex,gssapi-with-mic). Running with -vvv flag doesn't show me much besides the key being rejected.
Things I know/checked:
firewall isn't an issue because I can connect using my original key from same location
the instance is running SSH (running nc <instance_nat_ip> 22 shows "OpenSSH" etc.)
no passphrases were used with the generation of any SSH key
there are no instance-level restrictions on project-wide metadata
there are no instance-level ssh keys already added
there are no newlines/breaks causing the key to be malformed
permissions on ~./ssh aren't an issue since another key pair works fine from the same directory, additionally, both key pairs have the same permissions anyways
OSLogin isn't enabled either on the project or instance
Things I've tried:
removing and readding the SSH keys in project metadata
trying with new key pairs generate on another person's machine
restarting sshd service
Questions:
Does the username specified during the ssh-keygen step have to already exist on the remote instance prior to adding the key to the metadata? i.e. do I have to run sudo useradd <new_username> while SSH'd into the instance Creating a new test instance revealed this to not be the case, all users in the project metadata were created automatically
why does my existing SSH key work and not new ones even though they are added the same way?
there's a chance the enable-oslogin:TRUE was applied to the instance briefly a long time ago (I'm not sure since I'm not the one who created the instance) but it's no longer there in the instance or project metadata. Would having that been enabled, even briefly, cause some issues?
EDIT: I started up a new instance in the same project with the same network details and I was able to SSH to that instance using the new key. Original instance is still denying the key

Did some digging around and found out that the systemd service that propagates accounts information from the metadata server is a daemon called google-accounts-daemon.
When I ran sudo ps aux | grep daemon I didn't see it running as I did on the test instance I created.
So when I ran sudo systemctl restart google-accounts-daemon the SSH keys magically propagated and everything worked.
I have no idea what caused the daemon to stop running in the first place, so if anyone has ideas, that'd be appreciated in case this comes up in the future.

Related

Changing EC2 pem file key pair when you have access to the EC2 instance

thank you for your time.
I have an EC2 instance, but for security reasons i need to change the pem files associated in .ssh/authorized_keys. I do understand that the public pem file goes into authorized_keys.
I do not want to mount the volume of the ec2 instance to a new one. I am considering as a last option since I do have access to the EC2 instance.
How can this be done?
I have tried:
This post Change key pair for ec2 instance the answer by Pat Mcb, but no luck.
Run this command after you download your AWS pem.
ssh-keygen -f YOURKEY.pem -y Then dump the output into
authorized_keys.
Or copy pem file to your AWS instance and execute following commands
chmod 600 YOURKEY.pem and then
ssh-keygen -f YOURKEY.pem -y >> ~/.ssh/authorized_keys
But that didn't work for me. If i follow it exactly download aws key pair key, and follow the instructions by coping the key when ssh into the instance, when i do ssh-keygen -f YOURKEY.pem -y >> ~/.ssh/authorized_keys It asks for a passphrase (never had to input one)
What i am doing is the following.
I create a new key with
ssh-keygen newpem.pem
and the .pub file i copy it in .ssh/authorized_keys
Can someone explain what i am doing incorrectly?
Note the authorized_keys file has the correct permissions.
Seems like you want to deprecate the old key and use a new key instead. These steps may help you -
Create a new key pair using the aws console and download it onto your system.
Retrieve the public key from the private key(.pem) file using the command - "ssh-keygen -y"
SSH into the instance using the old key.
Once you have access to the instance add the public key you got in step 2 into the "~/.ssh/authorized_keys" files and then save the file.
Log out of the instance and then try accessing the instance with the new key.
Hope it helps. Thank You !
You Don't even need to do all of this just mind few things with AWS EC2 you get a private key for default users . like ec2-user /ubuntu etc.
You are doing the right step
ssh-keygen -t rsa -C "your_email#example.com"
if it ask for entering any paraphrase leave it blank.
Just press to accept the default location and file name. If the .ssh directory doesn't exist, the system creates one for you.
Enter, and re-enter, if passphrase prompted
you have that key now .
Copy that key
Login to your Ec2 server.
sudo su
vim ~/.ssh/authorized_keys
paste the key.
:wq!
You'll see a key there copy it and save it as a backup somewhere.
Now paste your newly generated key in that file
and save the file.
now final step to take care is the permission, so run the following command.
sudo chmod 700 .ssh && chmod 600 .ssh/authorized_keys
Now you're good to go you.
Following are the steps to change your keypair on AWS EC2.
Login to AWS Console. Go to the Network and Security >> Keypair.
Give the name of your keypair (mykeypair) and keytype (RSA) and Private
keyformat (.pem). and click on the create keypair. It will ask you to
download .pem file in your local machine. Save it at and remember the
location.
Login to your EC2 instance and go to the .ssh. location. Create a new file called
(mykeypair.pem) and paste the content from the file we downloaded in step no.2
Run the command: sudo chmod 600 mykeypair.pem
Run the command: ssh-keygen -f mykeypair.pem -y and it will generate some
content. Copy that content. Open the file called autherized_keys and
remove all the content from it.
Paste the copied content that we have generated in the previous step. Also enter your file name (mykeypair) in last after entering space.
Reboot your instance. Go to the puttygen and generate the .ppk file
using the pem file you have downloaded from the keypair. You will be able to login your ec2 with the newly generated .ppk from putty.
Okay I figured out my problem. First of all I had been hacked by a hacker apparently because I didn't know that permitpasswordlogin: yes DISABLES pubkey authentication.... I thought it was additional security. So i used a very loose password that could be easily guessed. Anyways, I believe this because I went to the root folder and found that there was actually a new key in the root named "el patrono 1337" which actually means "the master/boss" in spanish... LOL. Anyways... So i changed that back to my secure key (made a new one actually) and then I went to login as ec2-user and couldnt, but could as root. was driving me crazy for 30 minutes or so until I realized I had accidentally changed the owner of my ec2-user folder to root and therefore ssh was not searching the ec2-user .ssh/authorized_keys when I tried to log in. Wow very glad that's over lol. And just fyi guys I don't think the hacker installed anything malicious, but I did get tipped off that he tried to ssh into other people's servers (who claim they get attacked by ssh alot according to the aws abuse report) from my machine. I'm running a very simple website with zero sensitive data etc. He didn't even block me out of the machine by disabling password authentication.(i guess he didn't want me to know?). I will build a new instance from scratch next time I want to add anything(will be pretty soon) just to be on the safe side.

Creating ssh connections between AWS instances

I have two Ubuntu instances (A, B) sitting in a test AWS environment. The app I'm working on uses an rsync command to move files from A to B and so in order to do this, I normally (Vagrant locally) have a script that uses ssh-copy-id to copy A's public key to B. The rsync happens in the background on A, so it's local user would be "root", but transfers to the standard "ubuntu" user on B.
This has worked no issues at all on my dev platform, albeit that I don't use public keys for SSH access on the dev platform and different usernames for the credentials.
In AWS I understand that things are a little different, that the default username for my servers would be "ubuntu" etc etc but I'm still having issues getting A's root public key to work correctly on B.
This is what I'm doing (please note prefix showing when I'm using root and non-root users):
serverA:# ssh-keygen -f ~/.ssh/id_rsa -P "" -b 1024 -C ""
serverA:# more /root/.ssh/id_rsa.pub
(Copy content to clipboard)
serverB:$ vi ~/.ssh/authorized_keys
(Paste content to end of file)
serverB:$ sudo service ssh restart
It's at this point I was under the impression that all should be working so back on serverA I at least attempt an SSH connection:
serverA:# ssh ubuntu#serverB_ip_address
I'd already updated security groups so I get prompted to confirm the fingerprint, which I do, but I'm still getting:
Permission denied (publickey).
Can anyone see what I'm doing wrong here??

SSH into ec2 instance on AWS

This may be an extremely dumb question but I am new to AWS and terminal controls on Mac. I am trying to SSH into my EC2 instance and following the documentation here.
I am perplexed because it is asking for the PATH in order to chmod 400 my private .pem file. I am unsure which path they are asking for here and would love some clarification. I have already downloaded AWS CLI so I am unsure what PATH it is asking for. Any help is very appreciated.
As i checked document link shared by you, here path means the location where you have downloaded .pem key file during launching a instance on AWS.
If you used Safari browser, you can find Download location:-
http://support.topspinmedia.com/hc/en-us/articles/204262743-I-m-on-a-Mac-using-Safari-where-s-my-download-
ssh -i /Users/Onicha/Downloads/my-key-pair.pem ec2-user#ec2-198-51-100-1.compute-1.amazonaws.com
When you created that EC2 instance, you must have downloaded a private key. If not or you do not have the key anymore, I am afraid you must remove that instance and set up a new one.
Once you have the key, in your terminal, change the directory to where the private key is. You must first change the permissions on the key to make it only readable by your user and then you can SSH to it.
chmod 400 key.pem
ssh -i key.pem [user]#[host]
You will need to specify the path to the key, instance key pairs are described here http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html
It may help to verify a few things-
Are you creating the key pair on Amazon Ec2 (or whichever instance you are using) and then downloading and saving (as a .pem file- example yourpemfile.pem) on your computer? If you are, you would know which directory (the PATH that is being discussed) you saved it in. If you already did it and cannot find it, you can do it again creating a new key pair with a new filename yourpemfile2.pem, but you have to stop the Ec2 instance and remove the old keypair file associated with it. (I have seen ways online to change keypairs while instance is running as well).
Go to that particular folder on your Mac using Terminal. You don't need Amazon CLI if you use the terminal with MacOS. (You can do so by pressing Command+Space and then type terminal in the blank typing space that appears). Once the terminal window opens, at the prompt- type linux command 'pwd' to see which directory/folder your are in. Use 'cd' command for going to your folder where the .pem file is downloaded or saved (by you). List the particular folder by using linux command 'ls -l' at the prompt to see if your .pem file is actually there and what are its permissions.
Then change permissions using 'chmod' command-> chmod 400 yourpemfile.pem. ls -l yourpemfile.pem' again the file permissions should become r--------.
Now you are in the directory where you pem file is and you can directly ssh to your Ec2 instance from here. Now (using web browser) go to Amazon Ec2 Management Console and click on Instances (within folder Instances) on the left of dashboard you will see details of your instance which is running. Make sure the key pair associated with that instance is this particular .pem file. Then at the top of the page, where it says "Launch instance" and also has two other tabs, "Connect" and "Actions", click on "Connect".
A new window pops up which provides details of ssh commands to use and instance details.
Make sure to select standalone SSH client.
You see a suggested command like this- (helps in making sure you are using the right amazon instance and keypair)
ssh -i "AmcEc2mykeypair.pem" ec2-user#ec2-134-17-351-22.us-east-2.compute.amazonaws.com
Copy paste this on your linux terminal (Use Command+c and Command+v on Mac). Press Enter.
You should be 'logged in' now and a new prompt for the Amazon Ec2 instance machine- specifying clearly that it is an Amazon linux or ubuntu image should appear.

Cannot connect to an Instance using ssh after Broken pipe

I'm totally new using SaltStack and AWS, probably this is a dumb question, I created an AMI (using packer) with SaltStack (masterless) as a provisioner... I was able to connect via ssh and make a configuration to the minion. I was able to run salt-call state.highstate successfully.
Later, I lost the connection to my instance,
([root#<ip> ec2-user]# Write failed: Broken pipe) and after that, I wasn't able to connect again.
What's been tried:
Reboot the instance and didn't work
I've checked the permissions on the .ssh files and they seem fine
Create a new instance and use the same key.pem and I was able to connect to this new instance.
I'm not sure if I'm missing a configuration in SaltStack. Is there a possibility that the keys on my instance changed after running salt-call state.highstate ??
What am I doing wrong?
There's nothing inherent in running highstate that would have terminated the SSH connection and prevented you from reconnecting. I would suspect it's something in your SLS files which is breaking SSH - which is applied when you run highstate.
Things that might have been done by your Salt states:
your SSH keys were removed/mangled
opensshd config was changed
openssh-server was uninstalled
EDIT: Having seen the output from Salt in the pastebin linked in comments, it's probably the AuthorizedKeysFile option being commented out:
-AuthorizedKeysFile .ssh/authorized_keys
+#AuthorizedKeysFile .ssh/authorized_keys
I recommend using file.replace to patch in specific changes you need, as opposed to replacing the whole /etc/ssh/sshd_config with a new version.

SSH EC2 asking for password

I've just setup my EC2 server following this video here exactly - http://www.youtube.com/watch?v=bBajLxeKqoY
I even chose the same server type, everything went well until it asked for the root password to my EC2 server...
Any suggestions?
Update
Updating this answer because of the activity:
Depending on if the system is ubuntu or Rhel the user varies.
For ubuntu it is
ssh -i my-pem-file.pem ubuntu#my-ec2-instance-address
For RHEL it is
ssh -i my-pem-file.pem root#my-ec2-instance-address
Connecting to an ec2 instance does not require a password, it would require only a pem file
and this is how you connect to it
ssh -i my-pem-file.pem ec2-user#my-instance-address
and remember to chmod 400 your pem file before ssh'ing
If you need to do things as root once you are in as ec2-user, use sudo su - that gets you to root and doesn't need a password. Somethings you do need that for, like looking at the tomcat log files
I had the same problem and after a lot of struggle, I read this page again:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html
Notice in the 4th topic it talks about the user name. It is usually ec2-user but if you are using RHEL5 distro it can be root and if you are using the Ubuntu the user name will be ubuntu - which was my case.
So alternatively try those:
ssh -i my-pem-file.pem ubuntu#my-ec2-instance-address for Ubuntu
or
ssh -i my-pem-file.pem root#my-ec2-instance-address for RHEL5
Hope it helps!
You will be asked for password when you enable PasswordAuthentication yes in your sshd_config. Try changing that to no. that should fix it.
Just for anyone else that might have the same problem
Just do
sudo su -
Worked for me
If you've moved/copied the pem file from another machine, the owner of the file may be different to the user that is running the ssh client. To change the owner of the file:
sudo chown <currentusername> <filename.pem>
Please use --query 'KeyMaterial' while generating key.
Info:
keyMaterial - an unencrypted PEM encoded RSA private key.
In my case, we had been copying a text file to keep a ubiquitous key. Someone accidentally added characters to this file, and we started getting prompted for a password from the now-corrupt .pem file, when we had never set a password.
In my case, the problem was the file's break type. Try this:
1.- Open the .pem file with TextWrangler
2.- At Bottom of app, verify if the Break Type is "Windows(CRLF)".
Regards
This is the way to connect:
ssh -i /path/my-key-pair.pem ec2-user#public-ip
Now, instead of ec2-user, it could be root, or centos, or ubuntu, or something else. You can check under the "Usage Instructions". If it's not there, and you've tried all the above users, find out from the documentation that came with the AMI.
There is one bug on AWS that cost me a lot of time. If you're launching an instance from a saved AMI under "Images > AMIs", note that it uses the original keys of the running instance it was created from. When launching the AMI, it will prompt you to choose a new key, and even show such key under the description, but the truth is those keys will never work!
So if you're using a custom/saved AMI you'll have to either get the original keys and use them, or just create a brand new one from AWS or the market place. If you haven't created the AMI or can't remember where it came from, look under the details tab. You can then launch a new instance or create a new AMI from the same source. This will then use the keys that you specify.
chmod 400 pem_file
ssh -i /path_to_the_pem_file ec2-user#ServerIP
or
ssh -i /path_to_the_pem_file ubuntu#ServerIP
root access is prohibited by default by AWS AMIs.
hope it helps.
Check your .pem file is not corrupt. Using this line
openssl rsa -check -in test.pem -noout
It should return RSA Key OK. If it doesn't then there is most likely a problem with some sort of formatting or something in your pem file.
You should use the *.pem instead of a root password. Once you have logged in use passwd to set a password.
It could be a Linux problem.
But there is also a chance that you use the wrong address/key (you started a new instance but still using the old address; or 2 pem have similar name, used the wrong one), "ssh with non-existing user".
I am using TurboLinux instance, on Mac OS system, please try this command:
ssh -i xxxxx.pem root#xxx.xxx.xxx.xxx (public address of allocated VM)
I can get through without asking password anymore.
In case someone else bumps into this, the solution for my problem was that I had to run it with sudo:
sudo ssh -i my-pem-file.pem root#my-ec2-instance-address
For me, the issue was that I had created an AMI from an existing instance, so when I launched the AMI, even though I selected a new key, the instance's original key was what worked.
Login to Amazone console in browser:
https://us-east-2.console.aws.amazon.com/ec2/xxxxxxxx
Click on instances from left panel -> then select your instance -> click on connect button at top right
You will see a window open and there will be a button ssh Client
Here you can see exact command to connect with your instance.