the company i'm working with is developing a web application based on Laravel Framework, using Google Cloud Platform infrastructures. The frontend VM is a Centos8 OS with Apache webserver installed.
Seems that a developer ran a pretty massive "dnf upgrade" which included: kernel, openssl ,kerberos and others packages.
After the upgrade, seems that ldconfig has lost his mind:
[developer#webserver ~]$ sudo su - root
sudo: error in /etc/sudo.conf, line 19 while loading plugin "sudoers_policy"
sudo: unable to load /usr/libexec/sudo/sudoers.so: /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
sudo: fatal error, unable to load plugins
same for other commands like "dnf" or "rpm":
[developer#webserver ~]$ rpm
rpm: symbol lookup error: /lib64/librpmio.so.8: undefined symbol: EVP_md2, version OPENSSL_1_1_0
after a little bit of investigations, i found that the same commands, specifing the LD_LIBRARY_PATH variable, are working:
[developer#webserver ~]$ LD_LIBRARY_PATH=/lib64 rpm
RPM version 4.14.3
Copyright (C) 1998-2002 - Red Hat, Inc.
This program may be freely redistributed under the terms of the GNU GPL
...
...of course, i can't do the same trick with "sudo" command.
Important fact is that the VM is still running and it was never rebooted ( i'll exaplin later why i'm sayin this )
( and finally..at the point )
The major problem is that we can't use root account cause "sudo" is not working and, by default, Google use Public Key Authentication as deafult method (Local users has random passwords genereated by GCP). So actually, i can't even do a "dnf reinstall" to try fix the issues
I was afraid that, once rebooted, every services stops to work because of the incorrect dependecies library path, so instead of doing a reboot, i have created an image based on the VM and then a new VM based on that image.
As i was thinking: Once booted the new VM, every services stopped working. i was able to read the logs over the serial console of GCP web interface.
a snippet:
...
Oct 27 20:20:30 webserver google_oslogin_nss_cache[783]: /usr/bin/google_oslogin_nss_cache: /lib64/libjson-c.so.4: no version information available (required by /usr/bin/google_oslogin_nss_cache)
Oct 27 20:20:30 webserver NetworkManager[778]: /usr/sbin/NetworkManager: symbol lookup error: /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
Oct 27 20:20:30 webserver google_oslogin_nss_cache[783]: /usr/bin/google_oslogin_nss_cache: symbol lookup error: /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
Oct 27 20:20:30 webserver sssd[771]: ldb: unable to dlopen /usr/lib64/ldb/modules/ldb/ldap.so : /lib64/libldap-2.4.so.2: undefined symbol: EVP_md4, version OPENSSL_1_1_0
...
Using Google official documentation, i found the "startup-script" section of the VM properties that can be launched at every boot and that can be used to "change" user's passwords.
I know that, by default, all VMs has root access disabled, so i made this and added to vm's "automation" script:
#! /bin/bash
echo 'developer:PASSWORD' | chpasswd
echo 'root:PASSWORD' | chpasswd
Once rebooted, i've tried to login using the "serial console" option on the web interface, but with no luck. I've also used journalctl ( as normal user ) to find something in the logs... but nothing.
i suppose that is a consequence of that "google_oslogin_nss_cache" error
there's no way to run that script.
Searching on the internet, i've found some posts where someone was able to login directly as "root" using the "gcloud compute ssh" command. So i have tried to login as described using another VM of the same project, using both my google account user and root user...but also in this way ...no luck.
( i forgot to mention that my google account has "project owner" role, so actually i have all necessary permissions )
is there another way to reset "root" password without using "sudo" or i have to reinstall the VM from start?
I'm sorry for the long explanation....hope that everything is clear
Thanks
So... actually this question is divided by 2 different issues:
The only possible way for me to recover "root" account was to stop the VM, detach the boot disk, mount the boot disk on a new VM, mount the filesystem and modify the user. once boot disk is reattached to the original VM..you can use the modified account
second issue was made by upgrading openssl...so in the end the only way to avoid that error messages was to create a new file: /etc/ld.so.conf.d/libc.conf:
/usr/lib64
I would like to connect to a Google Cloud VM instance using Secure Shell App (SSA). I assumed this would be easy as these are both Google products and I had no problem before connection SSA to a Digital Ocean Droplet. I have found Google's own documentation to do so here and it looked easy enough to follow. However, the following link in the instructions: Providing public SSH keys to instances leads down a rabbit hole of confusing and seemingly self-contradicting information. I tried to follow it the best I could but kept running into errors. I have searched in vain for better instructions and am still astounded that Google has made it so hard to connect their own products. Is it really this hard to make this work? Are there any better instructions out there? If not, would someone be willing to write up clear and simple instructions?
Please follow this step by step instruction:
create a new VM instance-1
connect to it with gcloud compute ssh instance-1 (as mentioned #John Hanley)
check ~/.ssh folder
$ ls -l ~/.ssh
-rw------- 1 user usergroup 1856 Dec 9 17:12 google_compute_engine
-rw-r--r-- 1 user usergroup 417 Dec 9 17:12 google_compute_engine.pub
copy keys
cp ~/.ssh/google_compute_engine.pub mykey.pub
cp ~/.ssh/google_compute_engine mykey
follow instructions from step 7 - create connection and import identity
(optional) if you don't find your mykey in the Indentity list try to connect anyway (ended with an error as expected), then restart Secure Shell App and check Indentity menu again (they should be there without redoing import again)
After that, I successfully connected to my VM via Secure Shell App.
I'm trying to give access to a GCE VM by adding SSH Keys to the project metadata. My current SSH key is in the project metadata and I can connect just fine using:
ssh -i ~/.ssh/<private_key> <username>#<instance_ip>
Now, I generated another key:
ssh-keygen -t rsa -f ~/.ssh/<new_key> -C <new_username>
After adding the generated public key to the project metadata, I then run:
ssh -i ~/.ssh/<new_private_key> <new_username>#<instance_ip>
But I get Permission denied (publickey,gssapi-keyex,gssapi-with-mic). Running with -vvv flag doesn't show me much besides the key being rejected.
Things I know/checked:
firewall isn't an issue because I can connect using my original key from same location
the instance is running SSH (running nc <instance_nat_ip> 22 shows "OpenSSH" etc.)
no passphrases were used with the generation of any SSH key
there are no instance-level restrictions on project-wide metadata
there are no instance-level ssh keys already added
there are no newlines/breaks causing the key to be malformed
permissions on ~./ssh aren't an issue since another key pair works fine from the same directory, additionally, both key pairs have the same permissions anyways
OSLogin isn't enabled either on the project or instance
Things I've tried:
removing and readding the SSH keys in project metadata
trying with new key pairs generate on another person's machine
restarting sshd service
Questions:
Does the username specified during the ssh-keygen step have to already exist on the remote instance prior to adding the key to the metadata? i.e. do I have to run sudo useradd <new_username> while SSH'd into the instance Creating a new test instance revealed this to not be the case, all users in the project metadata were created automatically
why does my existing SSH key work and not new ones even though they are added the same way?
there's a chance the enable-oslogin:TRUE was applied to the instance briefly a long time ago (I'm not sure since I'm not the one who created the instance) but it's no longer there in the instance or project metadata. Would having that been enabled, even briefly, cause some issues?
EDIT: I started up a new instance in the same project with the same network details and I was able to SSH to that instance using the new key. Original instance is still denying the key
Did some digging around and found out that the systemd service that propagates accounts information from the metadata server is a daemon called google-accounts-daemon.
When I ran sudo ps aux | grep daemon I didn't see it running as I did on the test instance I created.
So when I ran sudo systemctl restart google-accounts-daemon the SSH keys magically propagated and everything worked.
I have no idea what caused the daemon to stop running in the first place, so if anyone has ideas, that'd be appreciated in case this comes up in the future.
I installed the latest Google Cloud Deep Learning VM Image today, after VM was launched, I was able to do sudo -i successfully via SSH web.
Once I login, I start my Tensorflow model training running in background (Using &). Few hours later I'm unable to login as root.
I get the following message:
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for my_username:
I tried:
sudo -i
su sudo -i
su root
I was able to replicate the issue. Any suggestions?
This issue was caused due to an internal Google side and removes the user from “Google-sudoers” group. For all affected instances, I suggest following the below workaround until the permanent fix has been rolled out.
Use a different username:
If using browser SSH window, click on the settings icon (top right), and click change Linux name in the drop down.
Using the SDK
$ gcloud compute ssh newusername#instance
Enable OS Login on the instance (set "enable-oslogin=True" in metadata) and per this article
You can track the permanent fix by following the Public Issue tracker.
The original answer:
Maybe the solution will be to add a SSH Key for Google Cloud Console and log in with another SSH client.
Additional answer:
I do not know why, but sometime the user suddenly stopped being a member of the google-sudoers group...
Then it's enough add your user to this group by some other user with administrator privileges to this group:
# usermod -G google-sudoers your_user_name
of course, if there is such a user...
I've created a new linux instance on Amazon EC2, and as part of that downloaded the .pem file to allow me to SSH in.
When I tried to ssh with:
ssh -i myfile.pem <public dns>
I got:
###########################################################
# WARNING: UNPROTECTED PRIVATE KEY FILE! #
###########################################################
Permissions 0644 for 'amazonec2.pem' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: amazonec2.pem
Permission denied (publickey).
Following this post I tried to chmod +600 the .pem file, but now when I ssh I just get
Permission denied (publickey).
What school-boy error am I making here?
The .pem file is in my home folder (in macOS). Its permissions look like this:
-rw-------# 1 mattroberts staff 1696 19 Nov 11:20 amazonec2.pem
The problem is a wrong set of permissions on the file.
It is easily solved by executing: chmod 400 mykey.pem
This solution is taken from AWS instructions:
Your key file must not be publicly viewable for SSH to work. Use this command if needed: chmod 400 mykey.pem
400 protects it by making it read only and only for the owner.
You are likely using the wrong username to login, because—
Most Ubuntu images have a user ubuntu
Amazon's AMI is ec2-user
Most Debian images have either root or admin
To login, you need to adjust your ssh command:
ssh -l USERNAME_HERE -i .ssh/yourkey.pem public-ec2-host
I know this is very late to the game ... but this always works for me:
##step 1
ssh-add ~/.ssh/KEY_PAIR_NAME.pem
##step 2, simply ssh in :)
ssh user_name#<instance public dns/ip>
e.g.
ssh ec2-user#ec2-198-51-100-1.compute-1.amazonaws.com
Ok man, the only thing that worked for me was:
Change permissions of the key
chmod 400 mykey.pem
Make sure to log in using ec2-user, and the correct ec2-99... address. The ec2-99 address is at the bottom of the aws console when you're logged in and seeing your instance listed
ssh -i mykey.pem ec2-user#ec2-99-99-99-99.compute-1.amazonaws.com
Take a look at this article. You do not use the public DNS but rather the form
ssh -i your.pem root#ec2-XXX-XXX-XXX-XXX.z-2.compute-1.amazonaws.com
where the name is visible on your AMI panel
In windows you can go to the properties of the pem file, and go to the security tab, then to advance button.
remove inheritance and all the permissions. then grant yourself the full control. after all SSL will not give you the same error again.
Change permission for the key file with :
chmod 400 key-file-name.pem
See AWS documentation for connecting to the instance: Tutorial: Get started with Amazon EC2 Linux instances
I know this question has been answered already but for those that have tried them all and you are still getting the annoying "Permission denied (publickey)". Try running your command with SUDO. Of course this is a temporary solution and you should set permissions correctly but at least that will let you identify that your current user is not running with the privileges you need (as you assumed)
sudo ssh -i amazonec2.pem ec2-xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com
Once you do this you'll get a message like this:
Please login as the user "ec2-user" rather than the user "root"
Which is also sparsely documented. In that case just do this:
sudo ssh -i amazonec2.pem ec2-xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com -l ec2-user
And you'll get the glorious:
__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|
Feb, 2022 Update:
See the description to ssh to EC2 instance on AWS:
Then, you can find "No.3" saying this below:
So, run the command below as "No.3" says above:
chmod 400 myKey.pem
In Mac terminal, doing "chmod 400 xyz.pem" did not help me, it kept saying permission denied. For ubuntu users I would suggest
ssh-add xyz.pem
ssh -i xyz.pem ubuntu#ec2-54-69-172-118.us-west-2.compute.amazonaws.com (notice the user is ubuntu)
ssh -i /.pem user#host-machine-IP
I think it's because either you have entered wrong credentials
or, you are using a public key rather than private key
or, your port permissions are open for ALL to ssh. This is bad for Amazon.
There can be three reasons behind this error.
Your are using a wrong key.
Your key doesn't have the correct permissions. You need to chmod it to 400.
You are using the wrong user. Ubuntu images have a user ubuntu, Amazon's AMI is ec2-user and debian images have either root or admin
In addition to the other answers, here is what I did in order for this to work:
Copy the key to .ssh folder if you still hadn't:
cp key.pem ~/.ssh/key.pem
Give the proper permissions to the key
chmod 400 ~/.ssh/key.pem
Start ssh-agent (Thanks to https://stackoverflow.com/a/17848593 )
eval `ssh-agent -s`
ssh-add
Then, add the key
ssh-add ~/.ssh/key.pem
Now you should be able to ssh EC2 (:
SSH keys and file permission best practices:
.ssh directory - 0700 (only by owner)
private key/.pem file - 0400 (read only by owner)
public key/.pub file - 0600 (read & write only by owner)
chmod XXXX file/directory
Alternative log-in using PuTTY. Its good but needs a few steps.
Get your .pem that was generated when you first made the EC2 instance.
Convert the .pem file .ppk using PuttyGen since PuTTY does not read .pem.
Open PuTTY and enter your Host Name which is your instance username + Public DNS (Ex. ubuntu#ec2-xxx-xxx-xxx-xxx.region.compute.amazonaws.com). Not your AWS account username.
Then navigate to Connection > SSH > Auth. Then add your .ppk file. Click on Browse where it says "Private key file for authentication".
Click Open and you should be able to immediately establish connection.
Im using PuTTY 0.66 in Windows.
By default whenever you download the keyfile it come with 644 permissions.
So you need to change the permission each time you download new keys.
chmod 400 my_file.pem
In Windows go to the .pem file, right click and select Properties.
Go to Advanced in Security tab
Disable and remove inheritance.
Then press Add and select a principal.
Add account username as object name and press ok.
Give all permission.
Apply and save changes.
Now check the above command
You can find the answer from the ASW guide.
400 protects it by making it read only and only for the owner.
chmod 400 mykey.pem
In windows,
Right click on the pem file. Then select properties.
Select security tab --> Click on the Advanced button --> Disable inheritance --> Remove all inherited permissions from this object
Click on the Add button --> Select a principal --> Enter your username on the inputbox --> Click on the Check Names button --> Click on Ok --> Click on Ok --> Click on Ok --> Click on Ok
Do a chmod 400 yourkeyfile.pem
If your instance is Amazon linux then use ssh -i yourkeyfile.pem ec2-user#ip
for ubuntu
ssh -i yourkeyfile.pem ubuntu#ip
for centos
ssh -i yourkeyfile.pem centos#ip
Just change the permission of pem file to 0600 allowing only for the allowed user and it will work like charm.
sudo chmod 0600 myfile.pem
And then try to ssh it will work perfectly.
ssh -i myfile.pem <<ssh_user>>#<<server>>
BY default permission are not allowing the pem key.
You just have to change the permission:
chmod 400 xyz.pem
and if ubuntu instance then connect using:
ssh -i xyz.pem ubuntu#ec2-youraws.amazonaws.com
The issue for me was that my .pem file was in one of my NTFS partitions. I moved it to my linux partition (ext4).
Gave required permissions by running:
chmod 400 my_file.pem
And it worked.
I have seen two reasons behind this issue
1) access key does not have the right permission. pem keys with default permission are not allowed to make a secure connection. You just have to change the permission:
chmod 400 xyz.pem
2) Also check whether you have logged-in with proper user credentials. Otherwise, use sudo while connecting
sudo ssh -i {keyfile} ec2-user#{ip address of remote host}
Well, looking at your post description I feel there were 2 mistakes done by you:-
Set correct permissions for the private key.
Below command should help you to set correct file permision.
chmod 0600 mykey.pem
Wrong ec2 user you are trying to login.
Looking at your debug log I think you have spawned an Amazon linux instance. The default user for that instance type is ec2-user . If the instance would have been ubuntu then your default user would have been ubuntu .
ssh -i privatekey.pem default_ssh_user#server_ip
Note:
For an Amazon Linux AMI, the default user name is ec2-user.
For a Centos AMI, the default user name is centos.
For a Debian AMI, the default user name is admin or root.
For a Fedora AMI, the default user name is ec2-user or fedora.
For a RHEL AMI, the default user name is ec2-user or root.
For a SUSE AMI, the default user name is ec2-user or root.
For an Ubuntu AMI, the default user name is ubuntu.
Otherwise, if ec2-user and root don't work, check with the AMI provider.
source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html
Key file should not be publicly viewable so use permission 400
chmod 400 keyfile.pem
If above command shows permission error use
sudo chmod 400 keyfile.pem
Now ssh into the ec2 machine, if you still face the issue, use ec2-user
ssh -i keyfile.pem ec2-user#ec2-12-34-56-78.compute-1.amazonaws.com
You're not in root then run this command
sudo chmod 400 -R myfile.pem
Not is root then run this command
chmod 400 -R myfile.pem
If you are connecting from Windows, perform the following steps on your local computer.
Navigate to your .pem file.
Right-click on the .pem file and select Properties.
Choose the Security tab.
Select Advanced.
Verify that you are the owner of the file. If not, change the owner to your username.
Select Disable inheritance and Remove all inherited permissions from this object.
Select Add, Select a principal, enter your username, and select OK.
From the Permission Entry window, grant Read permissions and select OK.
Click Apply to ensure all settings are saved.
Select OK to close the Advanced Security Settings window.
Select OK to close the Properties window.
You should be able to connect to your Linux instance from Windows via SSH.
From a Windows command prompt, run the following commands.
Run the following command to reset and remove explicit permissions:
icacls.exe $path /reset
Run the following command to grant Read permissions to the current user: icacls.exe $path /GRANT:R "$($env:USERNAME):(R)"
Run the following command to disable inheritance and remove inherited permissions : icacls.exe $path /inheritance:r
You should be able to connect to your Linux instance from Windows via SSH.
It is just a permission issue with your aws pem key.
Just change the permission of pem key to 400 using below command.
chmod 400 pemkeyname.pem
If you don't have permission to change the permission of a file you can use sudo like below command.
sudo chmod 400 pemkeyname.pem
Else if nothing works for you just follow this video to change the keys on your EC2 instance. You can install now public / private key pair on your instance.
https://youtu.be/LvLlRCrS8B4
Checklist:
Are you using the right private key .pem file?
Are its permissions set correctly? (My Amazon-brand AMIs work with 644, but Red hat must be at least 600 or 400. Don't know about Ubuntu.)
Are you using the right username in your ssh line? Amazon-branded = "ec2-user", Red Hat = "root", Ubuntu = "ubuntu". User can be specified as "ssh -i pem usename#hostname" OR "ssh -l username -i pem hostname"