I'm sort of new to AWS EC2 instances, but also comfortable.
16.04 Ubuntu Linux ec2 instance
I attempted to attach an extra harddrive volume to an existing ec2 instance. The initial default volume of 8 gigs was not enough to satisfy my needs.
I created the new 500gig volume and attached it to my instance, however it became the boot/root volume, so I detached and modified volume to a different device name /dev/sdf and reattached it keeping the instance's /dev/sda1 drive I had data on.
Upon restarting, the data in my initial /dev/sda1 was reverted back to it's initial snapshot data when the volume was created ( AMI image ), instead of still holding the data that was on it - which is what I was expecting it to do.
Is there anything I can do get that data back? Where is it? No popup or warning said that any data was going to be erased to the volume's initial snapshot and my backup couldn't catch it before hand. The /dev/sda1 was never detached or touched in this process, but the data is gone.
What happened? Where is my data?
Related
One of my instances was preventing me from logging so I stopped it, detached its volume, spun up a new instance, and attached the old volume. The problem is, the old volume is being treated as the root device. Because of this, I still cannot log in to the new instance (although I can do so if I don't attach the old volume).
Is there any way to fix this issue using the AWS Management Console?
It seems like you have attached your old volume in "/dev/sda1". Deattach your old volume and attach it to "/dev/sdf".
This is caused by the filesystems on each volume having the same label (the value returned by e.g. e2label /dev/xvda1). The server correctly starts booting from the first volume, and then the bootloader, there, sees the second volume having the label it anticipates for the root volume, and continues booting with the second volume as root. This an os-level setting, not visible to the AWS infrastructure.
Workaround: don't attach the second volume until after the instance has booted. EBS volumes can be attached to an instance at any time -- they don't have to be present when the instance is started. After unmount, they can also be detached at any time, with the instance still running.
To resolve this, I had to make a snapshot of the old volume first. I then created a new AMI using that snapshot. I included the old volume as an extra storage so that it's explicitly defined not to be treated as a root device.
I then created a new instance using that AMI. I was able to finally log in to the new instance. From there, I just mounted the volume.
Is AWS root storage ephemeral? The documents say they are ephemeral. But, I still can find my files after stopping the instance for some time and reboot it, I can still view my files
When you created your instance, I assume you chose the EBS backed instance. This means that the root device volume is actually an EBS volume, so when the instance restarts, you still have all of your data. If you were to choose the instance-store backed instance, then you would lose your data when restarting.
See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html for more information.
What do you mean by root storage?
Instance Stores are ephemeral. EBS volumes are not.
When launching an instance, if you've selected Instance Store as the root volume, a restart will still retain the data. But a stop and start will delete all data.
If you've selected an EBS volume, the data will be retained.
I have two amazon machine instances running.Both of them are m3.xlarge instances. One of them has the right software and configuration that I want to use.I want to create a snapshot of the EBS volume for that machine and use that as the EBS volue to boot the second machine from. Can I do that and expect it to work without shutting down the first machine.
It is well described in the AWS documentation...
"You can take a snapshot of an attached volume that is in use. However, snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued. This might exclude any data that has been cached by any applications or the operating system. If you can pause any file writes to the volume long enough to take a snapshot, your snapshot should be complete. However, if you can't pause all file writes to the volume, you should unmount the volume from within the instance, issue the snapshot command, and then remount the volume to ensure a consistent and complete snapshot.
I have amazon as well, with 3 different clusters. With one of my clusters after setting up 25 of them I realized there was a small issue in the configuration and had live traffic going to them so I couldn't' shut down.
You can snapshot the first machines volume while it's still running, I had to do this myself. It took a little while, but ultimately it worked out. Please note that amazon cannot guarantee the consistency of the disk when doing this.
I did a snapshot of the entire thing, fixed what needed to be fixed, and spooled up 25 new servers and terminated the other 25 ( easier than modifying volumes, etc ).. But you can create a new volume with the new snapshot, and attach it to an instance and do what needs to be done to get it to boot off that volume without much of a headache.
Being that I went the easy route of spooling up new instances after my snapshot was complete, I can't walk you through on how to get a running instance to boot off a new volume.
I am trying to stop a Amazon EC2 instance and get the warning message
Warning: Please note that any data on the ephemeral storage of your instance will be lost when it is stopped.
My Question
What data is stored in ephemeral storage of an Amazon EC2 instance?
Basically, root volume (your entire virtual system disk) is ephemeral, but only if you choose to create AMI backed by Amazon EC2 instance store.
If you choose to create AMI backed by EBS then your root volume is backed by EBS and everything you have on your root volume will be saved between reboots.
If you are not sure what type of volume you have, look under EC2->Elastic Block Store->Volumes in your AWS console and if your AMI root volume is listed there then you are safe. Also, if you go to EC2->Instances and then look under column "Root device type" of your instance and if it says "ebs", then you don't have to worry about data on your root device.
More details here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html
Anything that is not stored on an EBS volume that is mounted to the instance will be lost.
For example, if you mount your EBS volume at /mystuff, then anything not in /mystuff will be lost. If you don't mount an ebs volume and save stuff on it, then I believe everything will be lost.
You can create an AMI from your current machine state, which will contain everything in your ephemeral storage. Then, when you launch a new instance based on that AMI it will contain everything as it is now.
Update: to clarify based on comments by mattgmg1990 and glenn bech:
Note that there is a difference between "stop" and "terminate". If you "stop" an instance that is backed by EBS then the information on the root volume will still be in the same state when you "start" the machine again. According to the documentation, "By default, the root device volume and the other Amazon EBS volumes attached when you launch an Amazon EBS-backed instance are automatically deleted when the instance terminates" but you can modify that via configuration.
To be clear and answer #Dean's question: EBS-type root storage doesn't seem to be ephemeral. Data is persistent across reboots and actually it doesn't make any sense to use ebs-backed root volume which is 'ephemeral'. This wouldn't be different from image-based root volume.
For EC2 instance
Stop & Start != Reboot
so for ephemeral storage (Instance Store)
Stop cause data lost
Reboot will not
According to AWS documentation [https://aws.amazon.com/premiumsupport/knowledge-center/instance-store-vs-ebs/] instance store volumes is not persistent through instance stops, terminations, or hardware failures.
Any AMI created from instance stored disk doesn't contain data present in instance store so all instances launched by this AMI will not have data stored in instance store. Instance store can be used as cache for applications running on instance, for all persistent data you should use EBS.
ephemeral is just another name of root volume when you launch Instance from AMI backed from Amazon EC2 instance store
So Everything will be stored on ephemeral.
if you have launched your instance from AMI backed by EBS volume then your instance does not have ephemeral.
refer: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes
The data in an instance store persists only during the lifetime of
its associated instance. If an instance reboots (intentionally or
unintentionally), data in the instance store persists. However,
data in the instance store is lost under any of the following
circumstances:
- The underlying disk drive fails
- The instance stops
- The instance hibernates
- The instance terminates
I've launched an instance of the Basic 32-bit Amazon Linux AMI which has an 8GB volume as it's root device. If I terminate it, the EBS volume is destroyed as well. What I'd like to know is whether or not my data is protected (for example, the apache document root, or MySQL data) if the server crashes? A lot of tutorials seem to indicate that another EBS volume should be created and my data stored on that, but I'm not really seeing why two EBS volumes are needed?
Or is the current setup okay for a web server setup?
Many thanks in advance for your help!
When you spin an EC2 instance up, the root volume is ephemeral - that is, when the instance is terminated, the root volume is destroyed** (taking any data you put there with it). It doesn't matter how you partition that ephemeral volume and where you tuck your data on it - when it is destroyed, everything contained in that volume is lost.
So if the data in the volume is entirely transient and fully recoverable/retrievable from somewhere else the next time you need it, there's no problem; terminate the instance, then spin a new one up and re-acquire the data you need to carry on working.
However, if the data is NOT transient, and needs to be persisted so that work can carry on after an instance crash (and by crash, I mean something that terminates the instance or otherwise renders it inoperable and unrecoverable) then your data MUST NOT be on the root volume, but should be on another EBS volume which is attached to the instance. If and when that instance terminates or breaks irretrievably, your data is safe on that other volume - it can then be re-attached to a new instance for work to continue.
** the exception is where your instance is EBS-backed and you swapped root volumes - in this case, the root volume is left behind after the instance terminates because it wasn't part of the 'package' created by the AMI when you started it.
The other volume would be needed in case your server gets broken and you cannot start it. In such case you would just remove initial server, create a second one and attach the additional storage to the new server. You cannot attach root volume of one server to another.