Recently I made the silly mistake of clearing the contents of my user's ~/.ssh/authorized_keys file on my AWS instance. As such, I can no longer ssh onto the instance.
I realised I could add these keys back via AWS EC2 instance user data. However as of yet I have had no luck with this. I stopped my instance, added the following to the user data and started it again:
#!/bin/bash
> /home/myUser/.ssh/authorized_keys
echo "ssh-rsa aaa/bbb/ccc/ddd/etc== mykeypair" >> /home/myUser/.ssh/authorized_keys
chown myUser:myUser /home/myUser/.ssh/authorized_keys
chmod 600 /home/myUser/.ssh/authorized_keys
This should empty the file, add the public keypair and ensure the correct permissions are present on the file.
However my private key is still being rejected.
I know the keys are correct so it must be something to do with my instance user data. I have also tried prepending 'sudo' to all commands.
Try to use cloud-init directives instead of shell
#cloud-config
cloud_final_modules:
- [users-groups,always]
users:
- name: example_user
groups: [ wheel ]
sudo: [ "ALL=(ALL) NOPASSWD:ALL" ]
shell: /bin/bash
ssh-authorized-keys:
- ssh-rsa AAAAB3Nz<your public key>...
The default behavior is to execute once-per-instance. However, these
instructions add the key on every reboot or restart of the instance.
If the user data is removed, the default functionality is restored.
These instructions are for use on all OS distributions that support
cloud-init directives.
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-user-account-cloud-init-user-data/
From the official docs:
By default, user data and cloud-init directives only run during the
first boot cycle when you launch an instance. However, AWS Marketplace
vendors and owners of third-party AMIs may have made their own
customizations for how and when scripts run.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
So modifying user data after you shut down your instance will not be useful in most cases.
Solution: you can detach your EBS volume, attach it to an EC2 instance you can connect to, mount the volume, fix authorized_keys, than connect the volume back to the affected instance.
we're using Ansible to provision ec2 instances, deploy our application on them and then create an AMI based on that instance, update the launch config so the autoscaling group always launch a new version of the app.
we re-use much of the ansible's playbooks for different Apps (diff instance sizes, families, etc).
one of the tasks that this runs is to check the instance meta-data to find out if there are any ephemeral devices present, and if so, mount them and persist them in the /etc/fstab and in the cloud-config.
we run into an issue with the instance meta-data similar to what was described 3 years ago in this thread:
https://forums.aws.amazon.com/thread.jspa?messageID=489889
right now we're doing some testing with t2.large instance, as you know, this instances are ebs only, but when we curl the instance metadata we get that there are 2 ephemeral disks presents:
ubuntu#ip-xxx-xxx-xxx-xxx:~$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/
ami
ebs1
ebs2
ephemeral0
ephemeral1
the thing is that those ephemeral devices don't actually exists, so when our script tries to persist them in cloud-config, cloud-config ends up as a broken yaml blob....this is a major issue, because it causes the user_data to fail when an instance is launching.
any ideas anyone?
If this is a known issue that amazon team is not going to fix, you can workaround it.
Quick script to check actual existence:
#!/bin/bash
for MAPID in $(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/); do
BLKDEV=$(curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/$MAPID/ | sed 's#^/dev/##' | sed 's/^sd/xvd/')
if blkid | grep -q $BLKDEV; then
echo $BLKDEV present
else
echo $BLKDEV not present
fi
done
Checked it with c1.medium and t2.medium. t2 output:
$ curl -s http://169.254.169.254/latest/meta-data/block-device-mapping/
ami
ebs1
ephemeral0
ephemeral1
root
$ ./test.sh
xvda1 present
xvdd present
xvdb not present
xvdc not present
xvda1 present
When i try to log in into EC2 instance through ssh, i got the below error Permission denied (publickey)
I checked the host name and username, everything is fine. Wrongly i given the chmod -R 777 . from ec2 instance root directory when i was logged at last time, After that i could not able connect to instance. I need some files from ec2-instance. Is there any way to get log in into ec2?. Also i tried with new instance. Its working.
Is there any possibility?
I haven't tried this myself, but I don't see why it wouldn't work.
Try snapshotting your instance (create image button from ec2 console). Once complete, find your snapshot in the Ec2 console. It should be backed by an EBS volume with an id of the pattern "vol-xxxxxxxx".
Spin up a new instance and attach "vol-xxxxxxxx" as secondary storage. SSH to the new instance and mount the device "vol-xxxxxxxx" correlates to (e.g. /dev/xvdf) to a temp directory and find the files you're looking for.
Detach your root volume and attach to another instance.
Login to the ec2 instance
mkdir tempfolder
sudo mount /dev/svdf1 (normally /dev/svdf1, you can list out your volumes to make sure)
cd tempfolder/home
chmod 700 -R ec2-user
sudo umount tempfolder
Detach volume and attach it to old instance, remember it's root instance so you attach it with name "/dev/xvda".
I faced the similar problem.
you will not able to re-cover the old instance, just create new instance and set the permissions
chmod 777 (don't use -R) option, then your problem will be resolved.
One reason can be that your key file is not publicly viewable for SSH to work. Use this command if needed:
chmod 400 mykey.pem
Also keep in mind the correct user id for EC2(ec2-user) instance and the command:
ssh -l ec2-user -i .ssh/yourkey.pem public-ec2-host
Use Winscp to revert the permission change.
Recently I had accidentally changed the "/home/ec2-user" directory permissions to 777 using putty. I was immediately logged out. I was also logged into the server using "Winscp" and it didn't get disconnected after chaging the permissions.
The solution is change the permission on "/home/ec2-user" back to 700 using Winscp and I was able to log back in. It worked for me. Winscp saved me a lot of trouble.
I was trying to add several EBS to an EC2 instance, I use something like that:
block_map = BlockDeviceMapping()
xvdf = EBSBlockDeviceType()
xvdf.delete_on_termination = True
xvdf.size = opts.ebs_vol_size
block_map['/dev/xvdf'] = xvdf
req = conn.request_spot_instances(key_name=opts.key_pair,
price=opts.price,
image_id=ami,
security_groups=[instance_group],
instance_type=opts.instance_type,
block_device_map=block_map,
count=count
)
EBS are created as I could see them within the EC2 instance in the AWS console. Beside that, I'm 100% sure they are created as I can list them with the lsblk command once I log into the EC2 instance. I also added a couple of entries to my /etc/fstab so that the EBS volumes at mounted at creation time.
However, they are not mounted. If I run the command mount -a the following error shows up:
mount: wrong fs type, bad option, bad superblock on /dev/xvdf,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
So, it seems EBS volumes are created but not formatted with EBSBlockDeviceType. After I formatted, I can run mount -a again and they are already mounted.
My question is, if is possible to create and format a volume within the EBSBlockDeviceType() constructor, so that I can mount it.
Another option I think I might have is attach an already formatted EBS snapshot usign the snapshot_id field in the boto.ec2.blockdevicemapping.BlockDeviceType class.
Thank you!
mount command fails for a newly allocated volume because there is no file system on it. BlockDeviceType (or EBSBlockDeviceType) does not have an option to chose a file system for the underlying EBS volume. Once the volume is allocated, user can create a file system of choice.
However, for a volume created from a formatted EBS snapshot (created from a volume having file system), there is no need to create file system again. You can use file -s <device name> to find out if the device already has a file system.
More details at: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
I was given AWS Console access to an account with 2 instances running that I cannot shut down (in production). I would, however, like to gain SSH access to these instances, is it possible to create a new Keypair and apply it to the instances so I can SSH in? Obtaining the existing pem file for the keypair the instances were created under is currently not an option.
If this isn't possible is there some other way I can get into the instances?
You can't apply a keypair to a running instance. You can only use the new keypair to launch a new instance.
For recovery, if it's an EBS boot AMI, you can stop it, make a snapshot of the volume. Create a new volume based on it. And be able to use it back to start the old instance, create a new image, or recover data.
Though data at ephemeral storage will be lost.
Due to the popularity of this question and answer, I wanted to capture the information in the link that Rodney posted on his comment.
Credit goes to Eric Hammond for this information.
Fixing Files on the Root EBS Volume of an EC2 Instance
You can examine and edit files on the root EBS volume on an EC2 instance even if you are in what you considered a disastrous situation like:
You lost your ssh key or forgot your password
You made a mistake editing the /etc/sudoers file and can no longer
gain root access with sudo to fix it
Your long running instance is hung for some reason, cannot be
contacted, and fails to boot properly
You need to recover files off of the instance but cannot get to it
On a physical computer sitting at your desk, you could simply boot the system with a CD or USB stick, mount the hard drive, check out and fix the files, then reboot the computer to be back in business.
A remote EC2 instance, however, seems distant and inaccessible when you are in one of these situations. Fortunately, AWS provides us with the power and flexibility to be able to recover a system like this, provided that we are running EBS boot instances and not instance-store.
The approach on EC2 is somewhat similar to the physical solution, but we’re going to move and mount the faulty “hard drive” (root EBS volume) to a different instance, fix it, then move it back.
In some situations, it might simply be easier to start a new EC2 instance and throw away the bad one, but if you really want to fix your files, here is the approach that has worked for many:
Setup
Identify the original instance (A) and volume that contains the broken root EBS volume with the files you want to view and edit.
instance_a=i-XXXXXXXX
volume=$(ec2-describe-instances $instance_a |
egrep '^BLOCKDEVICE./dev/sda1' | cut -f3)
Identify the second EC2 instance (B) that you will use to fix the files on the original EBS volume. This instance must be running in the same availability zone as instance A so that it can have the EBS volume attached to it. If you don’t have an instance already running, start a temporary one.
instance_b=i-YYYYYYYY
Stop the broken instance A (waiting for it to come to a complete stop), detach the root EBS volume from the instance (waiting for it to be detached), then attach the volume to instance B on an unused device.
ec2-stop-instances $instance_a
ec2-detach-volume $volume
ec2-attach-volume --instance $instance_b --device /dev/sdj $volume
ssh to instance B and mount the volume so that you can access its file system.
ssh ...instance b...
sudo mkdir -p 000 /vol-a
sudo mount /dev/sdj /vol-a
Fix It
At this point your entire root file system from instance A is available for viewing and editing under /vol-a on instance B. For example, you may want to:
Put the correct ssh keys in /vol-a/home/ubuntu/.ssh/authorized_keys
Edit and fix /vol-a/etc/sudoers
Look for error messages in /vol-a/var/log/syslog
Copy important files out of /vol-a/…
Note: The uids on the two instances may not be identical, so take care if you are creating, editing, or copying files that belong to non-root users. For example, your mysql user on instance A may have the same UID as your postfix user on instance B which could cause problems if you chown files with one name and then move the volume back to A.
Wrap Up
After you are done and you are happy with the files under /vol-a, unmount the file system (still on instance-B):
sudo umount /vol-a
sudo rmdir /vol-a
Now, back on your system with ec2-api-tools, continue moving the EBS volume back to it’s home on the original instance A and start the instance again:
ec2-detach-volume $volume
ec2-attach-volume --instance $instance_a --device /dev/sda1 $volume
ec2-start-instances $instance_a
Hopefully, you fixed the problem, instance A comes up just fine, and you can accomplish what you originally set out to do. If not, you may need to continue repeating these steps until you have it working.
Note: If you had an Elastic IP address assigned to instance A when you stopped it, you’ll need to reassociate it after starting it up again.
Remember! If your instance B was temporarily started just for this process, don’t forget to terminate it now.
Though you can't add a key pair to a running EC2 instance directly, you can create a linux user and create a new key pair for him, then use it like you would with the original user's key pair.
In your case, you can ask the instance owner (who created it) to do the following. Thus, the instance owner doesn't have to share his own keys with you, but you would still be able to ssh into these instances. These steps were originally posted by Utkarsh Sengar (aka. #zengr) at http://utkarshsengar.com/2011/01/manage-multiple-accounts-on-1-amazon-ec2-instance/. I've made only a few small changes.
Step 1: login by default “ubuntu” user:
$ ssh -i my_orig_key.pem ubuntu#111.111.11.111
Step 2: create a new user, we will call our new user “john”:
[ubuntu#ip-11-111-111-111 ~]$ sudo adduser john
Set password for “john” by:
[ubuntu#ip-11-111-111-111 ~]$ sudo su -
[root#ip-11-111-111-111 ubuntu]# passwd john
Add “john” to sudoer’s list by:
[root#ip-11-111-111-111 ubuntu]# visudo
.. and add the following to the end of the file:
john ALL = (ALL) ALL
Alright! We have our new user created, now you need to generate the key file which will be needed to login, like we have my_orin_key.pem in Step 1.
Now, exit and go back to ubuntu, out of root.
[root#ip-11-111-111-111 ubuntu]# exit
[ubuntu#ip-11-111-111-111 ~]$
Step 3: creating the public and private keys:
[ubuntu#ip-11-111-111-111 ~]$ su john
Enter the password you created for “john” in Step 2. Then create a key pair. Remember that the passphrase for key pair should be at least 4 characters.
[john#ip-11-111-111-111 ubuntu]$ cd /home/john/
[john#ip-11-111-111-111 ~]$ ssh-keygen -b 1024 -f john -t dsa
[john#ip-11-111-111-111 ~]$ mkdir .ssh
[john#ip-11-111-111-111 ~]$ chmod 700 .ssh
[john#ip-11-111-111-111 ~]$ cat john.pub > .ssh/authorized_keys
[john#ip-11-111-111-111 ~]$ chmod 600 .ssh/authorized_keys
[john#ip-11-111-111-111 ~]$ sudo chown john:ubuntu .ssh
In the above step, john is the user we created and ubuntu is the default user group.
[john#ip-11-111-111-111 ~]$ sudo chown john:ubuntu .ssh/authorized_keys
Step 4: now you just need to download the key called “john”. I use scp to download/upload files from EC2, here is how you can do it.
You will still need to copy the file using ubuntu user, since you only have the key for that user name. So, you will need to move the key to ubuntu folder and chmod it to 777.
[john#ip-11-111-111-111 ~]$ sudo cp john /home/ubuntu/
[john#ip-11-111-111-111 ~]$ sudo chmod 777 /home/ubuntu/john
Now come to local machine’s terminal, where you have my_orig_key.pem file and do this:
$ cd ~/.ssh
$ scp -i my_orig_key.pem ubuntu#111.111.11.111:/home/ubuntu/john john
The above command will copy the key “john” to the present working directory on your local machine. Once you have copied the key to your local machine, you should delete “/home/ubuntu/john”, since it’s a private key.
Now, one your local machine chmod john to 600.
$ chmod 600 john
Step 5: time to test your key:
$ ssh -i john john#111.111.11.111
So, in this manner, you can setup multiple users to use one EC2 instance!!
For Elastic Beanstalk environments, you can apply a key-value pair to a running instance like this:
Create a key-value pair from EC2 -> Key Pairs (Under NETWORK & SECURITY tab)
Go to Elastic Beanstalk and click on your application
Go to Configuration -> Security and click Edit
Choose your EC2 key pair and click Apply
Click confirm to confirm the update. It will terminate the environment and apply the key value to your environment.
On your local machine, run command:
ssh-keygen -t rsa -C "SomeAlias"
After that command runs, a file ending in *.pub will be generated. Copy the contents of that file.
On the Amazon machine, edit ~/.ssh/authorized_keys and paste the contents of the *.pub file (and remove any existing contents first).
You can then SSH using the other file that was generated from the ssh-keygen command (the private key).
This happened to me earlier (didn't have access to an EC2 instance someone else created but had access to AWS web console) and I blogged the answer: http://readystate4.com/2013/04/09/aws-gaining-ssh-access-to-an-ec2-instance-you-lost-access-to/
Basically, you can detached the EBS drive, attach it to an EC2 that you do have access to. Add your SSH pub key to ~ec2-user/.ssh/authorized_keys on this attached drive. Then put it back on the old EC2 instance. step-by-step in the link using Amazon AMI.
No need to make snapshots or create a new cloned instance.
I didn't find an easy way to add a new key pair via the console, but you can do it manually.
Just ssh into your EC2 box with the existing key pair. Then edit the ~/.ssh/authorized_keys and add the new key on a new line. Exit and ssh via the new machine. Success!
In my case I used this documentation to associate a key pair with my instance of Elastic Beanstalk
Important
You must create an Amazon EC2 key pair and configure your Elastic Beanstalk–provisioned Amazon EC2 instances to use the Amazon EC2 key pair before you can access your Elastic Beanstalk–provisioned Amazon EC2 instances. You can set up your Amazon EC2 key pairs using the AWS Management Console. For instructions on creating a key pair for Amazon EC2, see the Amazon Elastic Compute Cloud Getting Started Guide.
Configuring Amazon EC2 Server Instances with Elastic Beanstalk
You can just add a new key to the instance by the following command:
ssh-copy-id -i ~/.ssh/id_rsa.pub domain_alias
You can configure domain_alias in ~/.ssh config
host domain_alias
User ubuntu
Hostname domain.com
IdentityFile ~/.ssh/ec2.pem
Once an instance has been started, there is no way to change the
keypair associated with the instance at a meta data level, but you
can change what ssh key you use to connect to the instance.
stackoverflow.com/questions/7881469/change-key-pair-for-ec2-instance
You can actually add a key pair through the elastic beanstalk config page. it then restarts your instance for you and everything works.