I upgraded an AWS EC2 machine 10 days ago (from t2-medium to t2-large). Again, I just upgraded today the same AWS EC2 machine (from t2-large to t2-xlarge) and after rebooting I noticed a massive loss of data (10 days of work). In fact, after changing instance type I found out it reverted to the same "state" of the machine just after doing the upgrade 10 days ago. Not less important, I have been doing snapshots periodically (twice a day), and when I try to recover from the last one, each snapshot is an exact copy of the machine after the first upgrade.
This is not the first project I have in AWS, and everytime I did a snapshot, it would always keep the very last copy of the machine. But NOT in this case. So I am really dazzled.
Can you give some hint on what could happen? Is it normal that AWS deletes data (DDBB + folder and files) when changing instance type? Do you think it might have to do with IAM policies?
Thank you
The EC2 user guide has information about the EC2 Storage options. To summarise:
The EC2 instance store volume is erased when the instance is stopped or terminated.
In contrast, EBS (Elastic Block Storage) can be mounted to a running EC2 instance (c.f. a physical hard drive). EBS volumes can be detached from one instance and attached to another instance. Moreover, you can create a snapshot of an EBS volume that is stored in S3 and you can recreate EBS volumes from that snapshot that can be attached to other EC2 instances.
The EC2 user guide provides step by step instructions:
Restoring an Amazon EBS Volume from a Snapshot
Attaching an Amazon EBS Volume to an Instance
Lastly, no, this behaviour is not related to IAM policies.
Related
If you take an AMI from an EC2, and the AMI takes, say, 1 hour to be available; and you choose the option not to skip the reboot.
All the files in the AMI will:
a) reflect their exact condition from the time the EC2 was rebooted? or
b) they may reflect any condition in this 1 hour interval which is what it took for the AMI to be available.
I always considered option a, but I'm not so sure any more, specially after I noticed that when you take an AMI in the console, it gives this message:
"Currently creating AMI ..... Check that the AMI status is 'Available' before deleting the instance or carrying out other actions related to this AMI."
I want to know if it's safe to start applying changes in an EC2 instance after an AMI is requested and the EC2 rebooted, but before the AMI is available.
An Amazon Machine Image (AMI) will contain a copy of the disk at it was at exactly at the point in time when the API call was issued.
Or, if the instance is rebooted as part of the image creation, it will contain a copy of the disk as it was between the time when the operating system shutdown and when the operating system started again.
The time taken for an AMI to become available involves copying disk blocks to the Snapshot used by the AMI. Any disk changes during that time will not be reflected in the AMI. This is possible because the disk is virtual. (It's a bit like a database being able to roll-back due to the use of log files.)
From Create Amazon EBS snapshots - Amazon Elastic Compute Cloud:
Snapshots occur asynchronously; the point-in-time snapshot is created immediately, but the status of the snapshot is pending until the snapshot is complete (when all of the modified blocks have been transferred to Amazon S3), which can take several hours for large initial snapshots or subsequent snapshots where many blocks have changed. While it is completing, an in-progress snapshot is not affected by ongoing reads and writes to the volume... snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued.
I have a AWS accound and I have installed ES there under /dev/elk folder ..
It was fine for a long period of time but few days ago somehow the files were deleted.
I assume you mean that you have an EC2 and installed elasticsearch onto it.
What is the type of the root partition of your EC2? Is it EBS or is it AWS Instance Store? If it's Instance Store, then it is ephemeral and can go away at any time.
See here and here for a description of the differences between EBS and Instance Store.
Without more information about how you launched the ec2, how you provisioned it etc it is difficult to provide a more in-depth answer.
I want to back up the root device (mounted at /) of my t2.micro instance running Ubuntu. I think the instance is EBS-backed as it is a t2 instance. So I was going to take snapshots of my root device to back it up.
However, it is recommended that I detach the root device before I back it up. There are two problems with this:
I have to use umount to unmount it first, which may cause my instance to crash. What is a safe way to handle this?
I want to run these backups as a cron job on the instance itself, but if my instance's root device is unmounted, will the cron job even run?
A more general question is: what is the best way to do this?
A possible solution might be: use AWS Lambda and execute a Lambda function based on a schedule executing the following commands by the use of the AWS SDK:
Stop EC2 instance
Create EBS snapshot
Start EC2 instance
First, I would confirm that your root device is in fact EBS backed.
Here are the basic steps to confirm:
To determine the root device type of an instance using the console
Open the Amazon EC2 console.
In the navigation pane, click Instances, and select the instance.
Check the value of Root device type in the Description tab as follows:
If the value is ebs, this is an Amazon EBS-backed instance.
If the value is instance store, this is an instance store-backed instance.
(Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html#display-instance-root-device-type)
AWS states as a best practice is to use snapshots or a backup tool.
Regularly back up your instance using Amazon EBS snapshots or a backup
tool.
(Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-best-practices.html)
AWS states in the documentation that root EBS volumes should be shutdown before taking a snapshot.
To create a snapshot for Amazon EBS volumes that serve as root
devices, you should stop the instance before taking the snapshot.
(Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html)
So depending on your RPO (Recovery Point Objective), as a general rule it is a good practice to separate your data from your root volume. Store data that you need to keep on a separate EBS volume and take snapshots on the second EBS volume. This way you never have to worry about the instance itself - if it bonks out just launch a new instance and attach your snapshot.
If you have a special case that prevents you from using EBS snapshots, try using a role for your instance(s) that have permissions to read/write data to S3 buckets using your cron job.
I have two amazon machine instances running.Both of them are m3.xlarge instances. One of them has the right software and configuration that I want to use.I want to create a snapshot of the EBS volume for that machine and use that as the EBS volue to boot the second machine from. Can I do that and expect it to work without shutting down the first machine.
It is well described in the AWS documentation...
"You can take a snapshot of an attached volume that is in use. However, snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued. This might exclude any data that has been cached by any applications or the operating system. If you can pause any file writes to the volume long enough to take a snapshot, your snapshot should be complete. However, if you can't pause all file writes to the volume, you should unmount the volume from within the instance, issue the snapshot command, and then remount the volume to ensure a consistent and complete snapshot.
I have amazon as well, with 3 different clusters. With one of my clusters after setting up 25 of them I realized there was a small issue in the configuration and had live traffic going to them so I couldn't' shut down.
You can snapshot the first machines volume while it's still running, I had to do this myself. It took a little while, but ultimately it worked out. Please note that amazon cannot guarantee the consistency of the disk when doing this.
I did a snapshot of the entire thing, fixed what needed to be fixed, and spooled up 25 new servers and terminated the other 25 ( easier than modifying volumes, etc ).. But you can create a new volume with the new snapshot, and attach it to an instance and do what needs to be done to get it to boot off that volume without much of a headache.
Being that I went the easy route of spooling up new instances after my snapshot was complete, I can't walk you through on how to get a running instance to boot off a new volume.
I am trying to stop a Amazon EC2 instance and get the warning message
Warning: Please note that any data on the ephemeral storage of your instance will be lost when it is stopped.
My Question
What data is stored in ephemeral storage of an Amazon EC2 instance?
Basically, root volume (your entire virtual system disk) is ephemeral, but only if you choose to create AMI backed by Amazon EC2 instance store.
If you choose to create AMI backed by EBS then your root volume is backed by EBS and everything you have on your root volume will be saved between reboots.
If you are not sure what type of volume you have, look under EC2->Elastic Block Store->Volumes in your AWS console and if your AMI root volume is listed there then you are safe. Also, if you go to EC2->Instances and then look under column "Root device type" of your instance and if it says "ebs", then you don't have to worry about data on your root device.
More details here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html
Anything that is not stored on an EBS volume that is mounted to the instance will be lost.
For example, if you mount your EBS volume at /mystuff, then anything not in /mystuff will be lost. If you don't mount an ebs volume and save stuff on it, then I believe everything will be lost.
You can create an AMI from your current machine state, which will contain everything in your ephemeral storage. Then, when you launch a new instance based on that AMI it will contain everything as it is now.
Update: to clarify based on comments by mattgmg1990 and glenn bech:
Note that there is a difference between "stop" and "terminate". If you "stop" an instance that is backed by EBS then the information on the root volume will still be in the same state when you "start" the machine again. According to the documentation, "By default, the root device volume and the other Amazon EBS volumes attached when you launch an Amazon EBS-backed instance are automatically deleted when the instance terminates" but you can modify that via configuration.
To be clear and answer #Dean's question: EBS-type root storage doesn't seem to be ephemeral. Data is persistent across reboots and actually it doesn't make any sense to use ebs-backed root volume which is 'ephemeral'. This wouldn't be different from image-based root volume.
For EC2 instance
Stop & Start != Reboot
so for ephemeral storage (Instance Store)
Stop cause data lost
Reboot will not
According to AWS documentation [https://aws.amazon.com/premiumsupport/knowledge-center/instance-store-vs-ebs/] instance store volumes is not persistent through instance stops, terminations, or hardware failures.
Any AMI created from instance stored disk doesn't contain data present in instance store so all instances launched by this AMI will not have data stored in instance store. Instance store can be used as cache for applications running on instance, for all persistent data you should use EBS.
ephemeral is just another name of root volume when you launch Instance from AMI backed from Amazon EC2 instance store
So Everything will be stored on ephemeral.
if you have launched your instance from AMI backed by EBS volume then your instance does not have ephemeral.
refer: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes
The data in an instance store persists only during the lifetime of
its associated instance. If an instance reboots (intentionally or
unintentionally), data in the instance store persists. However,
data in the instance store is lost under any of the following
circumstances:
- The underlying disk drive fails
- The instance stops
- The instance hibernates
- The instance terminates