Last night there was an error on my EB instance. The instance was removed and a new one was added. Because of that I lost data from the non-persistent instance storage. I don't have a backup / snapshot. A big beginner's mistake.
My question: Is there any chance to recover the instance's data from 12 hours ago? Maybe with the help from the AWS staff?
When an instance is stopped or terminated, the ephemeral volumes are gone. Terminating an instance releases the hardware for use by another customer. Stopping an instance does the same thing -- that's part of why you don't pay for stopped instances. The same instance will actually come up on physically different hardware if stopped and started.
Aside from the documentation...
The data in an instance store persists only during the lifetime of its associated instance. If an instance reboots (intentionally or unintentionally), data in the instance store persists. However, data in the instance store is lost under the following circumstances:
The underlying disk drive fails
The instance stops
The instance terminates
Therefore, do not rely on instance store for valuable, long-term data.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
...there are also numerous support forum posts on this topic, with responses by AWS personnel, indicating that the answer is "no."
Here's one example:
Once an instance that has Ephemeral (or Instance Store) volumes has been Stopped or Terminated, we are unable to recover the data that was on that volume. When you Stop or Terminate such an instance, those volumes are securely wiped and this is to ensure the security and confidentiality of your data that was on that volume.
This is as per: http://aws.amazon.com/instance-help/
And: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
https://forums.aws.amazon.com/thread.jspa?messageID=501815񺠷
And another:
Ephemeral (instance-store data) is the local host's hard drive and when an instance is migrated (moved) to new hardware (from a stop/start) the ephemeral data is scrubbed as part of the process as the instance will have new ephemeral storage as part of the new host.
All that said, there is not any way to get the data back from the ephemeral location.
https://forums.aws.amazon.com/thread.jspa?messageID=396680񠶈
Related
I am trying to understand instance store volume and I understand instance store is ideal for temporary storage and provides massive IOPS. It is retained in case of reboot but lost if you stop and start, hibernation or instance termination.
One question I have here is can Instance store be shared across EC2 instance ?
I am seeing the below in the documentation so asking. Also how to achieve this on AWS console ?
An instance store provides temporary block-level storage for your
instance. This storage is located on disks that are physically
attached to the host computer. Instance store is ideal for temporary
storage of information that changes frequently, such as buffers,
caches, scratch data, and other temporary content, or for data that is
replicated across a fleet of instances, such as a load-balanced pool
of web servers
Documentation taken : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
The diagram is showing a physical host computer in an AWS Data Center. The host can be reconfigured to run many different sizes of an Instance Family (eg large, 2xlarge, 4xlarge). Do not be too concerned by the details of what it is showing.
The simple fact is that, no, instance store volumes cannot be shared across multiple Amazon EC2 instances.
The diagram given in the docs is very confusing(at least for me). I am not able to get my head around it. Maybe the InstanceA,B and C are not meant to be EC2 instances but instance store volumes as in the same diagram you see Host Computer 1 and 2.
Also the most important part is
You can specify instance store volumes for an instance only when you launch it. You can't detach an instance store volume from one instance and attach it to a different instance.
Which is what you want to know. It means you cannot share an instance store volume between 2 or more EC2 instances. When the EC2 machine is up and running, there is no way you can attach it and while launching there is no way you can specify which volume to mounted on the EC2 instance when it's created.
Obviously, hibernation and stop are two different actions that I can select.
What's the difference?
Benefit of Hibernating over Stopping
The memory state is preserved
Since the memory state is perserved and loaded again when the instance start, this reduce the boot time of the instance.
The long running process can continue without interuption
A great benefit if you have some services that take a great amount of time to fully initialized
Under the hood
The whole hibernation process in visual:
When the instance is in Stopping state, the instance memory is persisted in the instance's EBS root volume, and is loaded again when the instance start.
Reference
AWS Instance Hibernate Overview
From the docs
When you hibernate an instance, Amazon EC2 signals the operating
system to perform hibernation (suspend-to-disk). Hibernation saves the
contents from the instance memory (RAM) to your Amazon Elastic Block
Store (Amazon EBS) root volume. Amazon EC2 persists the instance's EBS
root volume and any attached EBS data volumes. When you start your
instance:
The EBS root volume is restored to its previous state
The RAM contents are reloaded
The processes that were previously running on the instance are resumed
Previously attached data volumes are reattached and the instance retains its instance ID
Read more
TL;DR
When you stop your instance, the data stored in memory (RAM) is lost.
When you stop-hibernate an instance, AWS signals the OS to perform hibernation (suspend-to-disk), which saves the contents from the instance memory (RAM) to the Amazon EBS root volume.
From the charging perspective, AWS does not charge usage or data transfer fees for your instance after you stop it, but storage for any Amazon EBS volumes is still charged.
A practical example
Suppose you want to build a caching layer (e.g. on top of your DB) in an EC2 instance. For such a case, the stop-hibernate feature would be instrumental in persisting storage. It would prevent you from having to manually create scripts to save the RAM data before shutting down the server.
I’m using the g2.2xlarge instance type. This pricing table shows that the instance has “60 SSD Instance Storage.” The Best Practices for Amazon EC2 tells us that “the data stored in instance store is deleted when you stop or terminate your instance.” However, I have stopped the instance and the data on it remained. So does that mean that the data is on EBS or… I'm relatively new to EC2 and I just want to know whether I need to back up my data.
Yes, you should.
Here's why:
The data in an instance store (i.e. your 60 SSDs) are guaranteed to persist only during the associated instance's lifetime. This means that data is guaranteed to persist over reboots, but not if you were to STOP or TERMINATE the instance. In this scenario the underlining hardware may be replaced and you might lose everything. You are also subject to disk drive fails that can corrupt your data.
You said you did not lose anything when you stopped the instance, but you could have.
Therefore, you should use EBS or S3 or something else to backup your data.
Bonus points: you cannot detach your instance storage, what can be a problem if you ever need to change your instance - which you are very likely to do at one point.
Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
Cheers!
Backing-up is ALWAYS a good policy.
In the case of AWS EC2, however, you have the option to creating a snapshot. Follow This link:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSSnapshots.html
Do you need to backup your instance?
In a purist sense - No, always assume your EC2 could fail at any time, so don't put anything on there you don't have elsewhere (ie. source code, data, etc)
If you stopped your instance and the data is still there, it's likely it's mounted on an EBS, not ephemeral storage.
Do you need to backup your EBS?
Depends on your requirement. EBS is distributed over an AZ which gives you pretty good durability, so just think about what's on there and how bad it would be if you lost it
I thought I clearly understood the difference between instance-store and EBS backed AMIs.
But http://aws.amazon.com/maintenance-help/ says "if you are running an EBS-backed AMI, you can stop and then restart your instance in order to easily re-launch it. This will cause the loss of any data you have saved on the local instance store of the instance,"
Stop/start does NOT lose the sysvol data, so this confuses me.
I'm assuming that here, by "local instance store", they mean the backing EBS volume (the sysvol), and I'm thinking that they meant to say "terminate" instead of stop. Am I correct?
Terminating an EBS-backed instance will not cause your data to be deleted. You can still access the EBS volume until you delete it (unless you set it to delete when your instance is terminated).
Local instance store refers to hard drive space on the actual physical server that is running your instance. You can see the available instance store by doing sudo fdisk -l. Some images come with some instance store volumes already mounted (see df -h). Otherwise you'll have to mount and format the instance store volumes before you can use them.
Data on an instance store volume is lost when you stop (not terminate) your instance because it is local to a physical server, and your instance might start up on a new server.
Quite simply, EC2 is running your virtual server on some physical server. The root filesystem can either be on a local disk (ephemeral storage) or on network attached storage (EBS). With EBS, they can snapshot it for backups or to make a copy, so EBS is far more flexible, although not as fast as a local disk in the server where your instance is running.
In order to make this all work, when you shutdown an ephemeral server, amazon wipes the disk in order to reallocate it to the next customer. There is no need or reason for them to do that with EBS, since it was not physically attached to that server in the first place.
You might note, that even EBS backed instances (depending on size) come with an allocation of ephemeral storage (2-500gig+) which can be used for swap, logs, or whatever else you want to do with them. The only issue of course is that should the server be shutdown, or should there be a catastrophic disk or hardware error, you'll lose that data. You can still manually back it up, in the same way people have backed up traditional servers over the years.
Making your own AMI from an EBS backed server is trivial now, and can be done easily through the AWS web interface. Making a non-EBS backed AMI is a very complicated task the last time I tried to do it. With that said, there are certain use cases where it makes a lot of sense to consider using purely ephemeral storage. Computation or memory/cache nodes that have no need to persist data will be faster and cost less.
I've launched an instance of the Basic 32-bit Amazon Linux AMI which has an 8GB volume as it's root device. If I terminate it, the EBS volume is destroyed as well. What I'd like to know is whether or not my data is protected (for example, the apache document root, or MySQL data) if the server crashes? A lot of tutorials seem to indicate that another EBS volume should be created and my data stored on that, but I'm not really seeing why two EBS volumes are needed?
Or is the current setup okay for a web server setup?
Many thanks in advance for your help!
When you spin an EC2 instance up, the root volume is ephemeral - that is, when the instance is terminated, the root volume is destroyed** (taking any data you put there with it). It doesn't matter how you partition that ephemeral volume and where you tuck your data on it - when it is destroyed, everything contained in that volume is lost.
So if the data in the volume is entirely transient and fully recoverable/retrievable from somewhere else the next time you need it, there's no problem; terminate the instance, then spin a new one up and re-acquire the data you need to carry on working.
However, if the data is NOT transient, and needs to be persisted so that work can carry on after an instance crash (and by crash, I mean something that terminates the instance or otherwise renders it inoperable and unrecoverable) then your data MUST NOT be on the root volume, but should be on another EBS volume which is attached to the instance. If and when that instance terminates or breaks irretrievably, your data is safe on that other volume - it can then be re-attached to a new instance for work to continue.
** the exception is where your instance is EBS-backed and you swapped root volumes - in this case, the root volume is left behind after the instance terminates because it wasn't part of the 'package' created by the AMI when you started it.
The other volume would be needed in case your server gets broken and you cannot start it. In such case you would just remove initial server, create a second one and attach the additional storage to the new server. You cannot attach root volume of one server to another.