I am running a t2.micro ec2 instance on us-west-2a and instance's state is all green.
When I access my website it stops loading once in a while. Even if I reboot it, the website still doesn't load. When I stop an instance and then relaunch it, it shows 1/2 status checks failed.
ALARM TYPE: awsec2-i-20aaa52c-High-Network-Out
I also faced same type of issue.
EC2 instances were failing Instance Status Checks after a stop/start. I was able to take a look on my side at the System logs available to support and I could confirm that the system was having a kernel panic and was unable to boot from the root volume.
So I launched new EC2 temporary instance so we can attach the EBS root volumes of each EC2 instance . Here we modified the grub configuration file so it can load from a previous kernel.
The following commands:
1. Mount the EBS volume as a secondary volume into mnt folder: $ sudo mount /dev/xvdf1 /mnt
2. Backup the grub.cfg file: sudo cp /mnt/boot/grub2/grub.cfg grub.cfg_backup
3. Edit the grub.cfg file: sudo vim /mnt/boot/grub2/grub.cfg
4. Here we commented # all the lines for the first entry loading the new kernel.
Then you attached the original EBS volumes back to the original EC2 instances and these EC2 instances were able to successfully boot.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html#FilesystemKernel
I was trying to add several EBS to an EC2 instance, I use something like that:
block_map = BlockDeviceMapping()
xvdf = EBSBlockDeviceType()
xvdf.delete_on_termination = True
xvdf.size = opts.ebs_vol_size
block_map['/dev/xvdf'] = xvdf
req = conn.request_spot_instances(key_name=opts.key_pair,
price=opts.price,
image_id=ami,
security_groups=[instance_group],
instance_type=opts.instance_type,
block_device_map=block_map,
count=count
)
EBS are created as I could see them within the EC2 instance in the AWS console. Beside that, I'm 100% sure they are created as I can list them with the lsblk command once I log into the EC2 instance. I also added a couple of entries to my /etc/fstab so that the EBS volumes at mounted at creation time.
However, they are not mounted. If I run the command mount -a the following error shows up:
mount: wrong fs type, bad option, bad superblock on /dev/xvdf,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
So, it seems EBS volumes are created but not formatted with EBSBlockDeviceType. After I formatted, I can run mount -a again and they are already mounted.
My question is, if is possible to create and format a volume within the EBSBlockDeviceType() constructor, so that I can mount it.
Another option I think I might have is attach an already formatted EBS snapshot usign the snapshot_id field in the boto.ec2.blockdevicemapping.BlockDeviceType class.
Thank you!
mount command fails for a newly allocated volume because there is no file system on it. BlockDeviceType (or EBSBlockDeviceType) does not have an option to chose a file system for the underlying EBS volume. Once the volume is allocated, user can create a file system of choice.
However, for a volume created from a formatted EBS snapshot (created from a volume having file system), there is no need to create file system again. You can use file -s <device name> to find out if the device already has a file system.
More details at: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
I have created a custom CentOS 6.5 image and registered it to AWS as EBS root device type. When I launch an instance, it works perfectly well, except that the storage capacity (instance storage to be included according to the instance type) is not added to the instance.
I made a try booting an instance using the official CentOS 6.5 AMI that is located in the AWS Marketplace, but I got the same result.
Does anyone know the reason, if it is a known issue, or whatever?
Thanks in advance.
First you have to make sure that the instance store is attached at launch time. From the AWS console it should look something like this:
Once you boot the instance you have to create a filesystem in the drive by running:
mkfs.ext4 /dev/sdb
Then you need to mount that drive somewhere in your root filesystem:
mkdir -p /mnt/myinstancestore
mount /dev/sdb /mnt/myinstancestore
You can run these commands to check that your drive is mounted:
df -h
mount
You can also add the mount entry to your /etc/fstab file so that it mounts permanently after every reboot:
/dev/sdb /mnt/myinstancestore ext4 defaults 1 2
I'm having problem connecting EBS volume to my Ubuntu EC2 Instance.
Here's what I did:
From the Amazon AWS Console, I created a EBS 150GB volume and attached it to an Ubuntu 11.10 EC2 instance. Under the EBS volume properties, "Attachment" shows: "[my Ubuntu instance id]:/dev/sdf (attached)"
Tried mounting the drive on the Ubuntu box, and it told me "mount: /dev/sdf is not a block device"
sudo mount /dev/sdf /vol
So I checked with fdisk and tried to mount from the new location and it told me it wasn't the right file system.
sudo fdisk -l
sudo mount -v -t ext4 /dev/xvdf /vol
the error:
mount: wrong fs type, bad option, bad superblock on /dev/xvdf, missing
codepage or helper program, or other error In some cases useful info
is found in syslog - try dmesg | tail or so
"dmesg | tail" told me it gave the following error:
EXT4-fs (sda1): VFS: Can't find ext4 filesystem
I also tried putting the configurations into /etc/fstab file as instructed on http://www.webmastersessions.com/how-to-attach-ebs-volume-to-amazon-ec2-instance, but still gave same not the right file system error.
Questions:
Q1: Based on point 1 (above), why was the volume mapped to 'dev/sdf' when it's really mapped to '/dev/xvdf'?
Q2: What else do I need to do to get the EBS volume loaded? I thought it'll just take care of everything for me when I attach it to a instance.
Since this is a new volume, you need to format the EBS volume (block device) with a file system between step 1 and step 2. So the entire process with your sample mount point is:
Create EBS volume.
Attach EBS volume to /dev/sdf (EC2's external name for this particular device number).
Format file system /dev/xvdf (Ubuntu's internal name for this particular device number):
sudo mkfs.ext4 /dev/xvdf
Only format the file system if this is a new volume with no data on it. Formatting will make it difficult or impossible to retrieve any data that was on this volume previously.
Mount file system (with update to /etc/fstab so it stays mounted on reboot):
sudo mkdir -m 000 /vol
echo "/dev/xvdf /vol auto noatime 0 0" | sudo tee -a /etc/fstab
sudo mount /vol
Step 1: create volume
step 2: attach to your instance root volume
step 3: run sudo resize 2fs -p /dev/xvde
step 4: restart apache2 sudo service apache2 restart
step 4: run df -h
You can see total volume attached to your instance.
I was given AWS Console access to an account with 2 instances running that I cannot shut down (in production). I would, however, like to gain SSH access to these instances, is it possible to create a new Keypair and apply it to the instances so I can SSH in? Obtaining the existing pem file for the keypair the instances were created under is currently not an option.
If this isn't possible is there some other way I can get into the instances?
You can't apply a keypair to a running instance. You can only use the new keypair to launch a new instance.
For recovery, if it's an EBS boot AMI, you can stop it, make a snapshot of the volume. Create a new volume based on it. And be able to use it back to start the old instance, create a new image, or recover data.
Though data at ephemeral storage will be lost.
Due to the popularity of this question and answer, I wanted to capture the information in the link that Rodney posted on his comment.
Credit goes to Eric Hammond for this information.
Fixing Files on the Root EBS Volume of an EC2 Instance
You can examine and edit files on the root EBS volume on an EC2 instance even if you are in what you considered a disastrous situation like:
You lost your ssh key or forgot your password
You made a mistake editing the /etc/sudoers file and can no longer
gain root access with sudo to fix it
Your long running instance is hung for some reason, cannot be
contacted, and fails to boot properly
You need to recover files off of the instance but cannot get to it
On a physical computer sitting at your desk, you could simply boot the system with a CD or USB stick, mount the hard drive, check out and fix the files, then reboot the computer to be back in business.
A remote EC2 instance, however, seems distant and inaccessible when you are in one of these situations. Fortunately, AWS provides us with the power and flexibility to be able to recover a system like this, provided that we are running EBS boot instances and not instance-store.
The approach on EC2 is somewhat similar to the physical solution, but we’re going to move and mount the faulty “hard drive” (root EBS volume) to a different instance, fix it, then move it back.
In some situations, it might simply be easier to start a new EC2 instance and throw away the bad one, but if you really want to fix your files, here is the approach that has worked for many:
Setup
Identify the original instance (A) and volume that contains the broken root EBS volume with the files you want to view and edit.
instance_a=i-XXXXXXXX
volume=$(ec2-describe-instances $instance_a |
egrep '^BLOCKDEVICE./dev/sda1' | cut -f3)
Identify the second EC2 instance (B) that you will use to fix the files on the original EBS volume. This instance must be running in the same availability zone as instance A so that it can have the EBS volume attached to it. If you don’t have an instance already running, start a temporary one.
instance_b=i-YYYYYYYY
Stop the broken instance A (waiting for it to come to a complete stop), detach the root EBS volume from the instance (waiting for it to be detached), then attach the volume to instance B on an unused device.
ec2-stop-instances $instance_a
ec2-detach-volume $volume
ec2-attach-volume --instance $instance_b --device /dev/sdj $volume
ssh to instance B and mount the volume so that you can access its file system.
ssh ...instance b...
sudo mkdir -p 000 /vol-a
sudo mount /dev/sdj /vol-a
Fix It
At this point your entire root file system from instance A is available for viewing and editing under /vol-a on instance B. For example, you may want to:
Put the correct ssh keys in /vol-a/home/ubuntu/.ssh/authorized_keys
Edit and fix /vol-a/etc/sudoers
Look for error messages in /vol-a/var/log/syslog
Copy important files out of /vol-a/…
Note: The uids on the two instances may not be identical, so take care if you are creating, editing, or copying files that belong to non-root users. For example, your mysql user on instance A may have the same UID as your postfix user on instance B which could cause problems if you chown files with one name and then move the volume back to A.
Wrap Up
After you are done and you are happy with the files under /vol-a, unmount the file system (still on instance-B):
sudo umount /vol-a
sudo rmdir /vol-a
Now, back on your system with ec2-api-tools, continue moving the EBS volume back to it’s home on the original instance A and start the instance again:
ec2-detach-volume $volume
ec2-attach-volume --instance $instance_a --device /dev/sda1 $volume
ec2-start-instances $instance_a
Hopefully, you fixed the problem, instance A comes up just fine, and you can accomplish what you originally set out to do. If not, you may need to continue repeating these steps until you have it working.
Note: If you had an Elastic IP address assigned to instance A when you stopped it, you’ll need to reassociate it after starting it up again.
Remember! If your instance B was temporarily started just for this process, don’t forget to terminate it now.
Though you can't add a key pair to a running EC2 instance directly, you can create a linux user and create a new key pair for him, then use it like you would with the original user's key pair.
In your case, you can ask the instance owner (who created it) to do the following. Thus, the instance owner doesn't have to share his own keys with you, but you would still be able to ssh into these instances. These steps were originally posted by Utkarsh Sengar (aka. #zengr) at http://utkarshsengar.com/2011/01/manage-multiple-accounts-on-1-amazon-ec2-instance/. I've made only a few small changes.
Step 1: login by default “ubuntu” user:
$ ssh -i my_orig_key.pem ubuntu#111.111.11.111
Step 2: create a new user, we will call our new user “john”:
[ubuntu#ip-11-111-111-111 ~]$ sudo adduser john
Set password for “john” by:
[ubuntu#ip-11-111-111-111 ~]$ sudo su -
[root#ip-11-111-111-111 ubuntu]# passwd john
Add “john” to sudoer’s list by:
[root#ip-11-111-111-111 ubuntu]# visudo
.. and add the following to the end of the file:
john ALL = (ALL) ALL
Alright! We have our new user created, now you need to generate the key file which will be needed to login, like we have my_orin_key.pem in Step 1.
Now, exit and go back to ubuntu, out of root.
[root#ip-11-111-111-111 ubuntu]# exit
[ubuntu#ip-11-111-111-111 ~]$
Step 3: creating the public and private keys:
[ubuntu#ip-11-111-111-111 ~]$ su john
Enter the password you created for “john” in Step 2. Then create a key pair. Remember that the passphrase for key pair should be at least 4 characters.
[john#ip-11-111-111-111 ubuntu]$ cd /home/john/
[john#ip-11-111-111-111 ~]$ ssh-keygen -b 1024 -f john -t dsa
[john#ip-11-111-111-111 ~]$ mkdir .ssh
[john#ip-11-111-111-111 ~]$ chmod 700 .ssh
[john#ip-11-111-111-111 ~]$ cat john.pub > .ssh/authorized_keys
[john#ip-11-111-111-111 ~]$ chmod 600 .ssh/authorized_keys
[john#ip-11-111-111-111 ~]$ sudo chown john:ubuntu .ssh
In the above step, john is the user we created and ubuntu is the default user group.
[john#ip-11-111-111-111 ~]$ sudo chown john:ubuntu .ssh/authorized_keys
Step 4: now you just need to download the key called “john”. I use scp to download/upload files from EC2, here is how you can do it.
You will still need to copy the file using ubuntu user, since you only have the key for that user name. So, you will need to move the key to ubuntu folder and chmod it to 777.
[john#ip-11-111-111-111 ~]$ sudo cp john /home/ubuntu/
[john#ip-11-111-111-111 ~]$ sudo chmod 777 /home/ubuntu/john
Now come to local machine’s terminal, where you have my_orig_key.pem file and do this:
$ cd ~/.ssh
$ scp -i my_orig_key.pem ubuntu#111.111.11.111:/home/ubuntu/john john
The above command will copy the key “john” to the present working directory on your local machine. Once you have copied the key to your local machine, you should delete “/home/ubuntu/john”, since it’s a private key.
Now, one your local machine chmod john to 600.
$ chmod 600 john
Step 5: time to test your key:
$ ssh -i john john#111.111.11.111
So, in this manner, you can setup multiple users to use one EC2 instance!!
For Elastic Beanstalk environments, you can apply a key-value pair to a running instance like this:
Create a key-value pair from EC2 -> Key Pairs (Under NETWORK & SECURITY tab)
Go to Elastic Beanstalk and click on your application
Go to Configuration -> Security and click Edit
Choose your EC2 key pair and click Apply
Click confirm to confirm the update. It will terminate the environment and apply the key value to your environment.
On your local machine, run command:
ssh-keygen -t rsa -C "SomeAlias"
After that command runs, a file ending in *.pub will be generated. Copy the contents of that file.
On the Amazon machine, edit ~/.ssh/authorized_keys and paste the contents of the *.pub file (and remove any existing contents first).
You can then SSH using the other file that was generated from the ssh-keygen command (the private key).
This happened to me earlier (didn't have access to an EC2 instance someone else created but had access to AWS web console) and I blogged the answer: http://readystate4.com/2013/04/09/aws-gaining-ssh-access-to-an-ec2-instance-you-lost-access-to/
Basically, you can detached the EBS drive, attach it to an EC2 that you do have access to. Add your SSH pub key to ~ec2-user/.ssh/authorized_keys on this attached drive. Then put it back on the old EC2 instance. step-by-step in the link using Amazon AMI.
No need to make snapshots or create a new cloned instance.
I didn't find an easy way to add a new key pair via the console, but you can do it manually.
Just ssh into your EC2 box with the existing key pair. Then edit the ~/.ssh/authorized_keys and add the new key on a new line. Exit and ssh via the new machine. Success!
In my case I used this documentation to associate a key pair with my instance of Elastic Beanstalk
Important
You must create an Amazon EC2 key pair and configure your Elastic Beanstalk–provisioned Amazon EC2 instances to use the Amazon EC2 key pair before you can access your Elastic Beanstalk–provisioned Amazon EC2 instances. You can set up your Amazon EC2 key pairs using the AWS Management Console. For instructions on creating a key pair for Amazon EC2, see the Amazon Elastic Compute Cloud Getting Started Guide.
Configuring Amazon EC2 Server Instances with Elastic Beanstalk
You can just add a new key to the instance by the following command:
ssh-copy-id -i ~/.ssh/id_rsa.pub domain_alias
You can configure domain_alias in ~/.ssh config
host domain_alias
User ubuntu
Hostname domain.com
IdentityFile ~/.ssh/ec2.pem
Once an instance has been started, there is no way to change the
keypair associated with the instance at a meta data level, but you
can change what ssh key you use to connect to the instance.
stackoverflow.com/questions/7881469/change-key-pair-for-ec2-instance
You can actually add a key pair through the elastic beanstalk config page. it then restarts your instance for you and everything works.