Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
Is it good idea to create periodic snapshot of the EBS volume from same instance it is attached to? Is there any downtime during snapshot process? I basically wanted to keep a regular may be daily or weekly snapshot of the ec2 instance so that If there is any virus or hacking or security issue I could spin another instance from the snapshots.
Absolutely yes. It's a good practice (personally, I consider it a must) to create point-in-time snapshots and to use them to create new volumes or restore old volumes. There is no downtime during the snapshot process. For a more detailed explanation you may take a look here, with particular emphasis on this part:
You can take a snapshot of an attached volume that is in use. However,
snapshots only capture data that has been written to your Amazon EBS
volume at the time the snapshot command is issued. This may exclude
any data that has been cached by any applications or the operating
system. If you can pause any file writes to the volume long enough to
take a snapshot, your snapshot should be complete. However, if you
can't pause all file writes to the volume, you should unmount the
volume from within the instance, issue the snapshot command, and then
remount the volume to ensure a consistent and complete snapshot. You
may remount and use your volume while the snapshot status is pending.
Before doing operation involving data I think it's very important to know everything about a technology that you are going to use. So, I would like take this opportunity to put the focus on some points, taken from the official AWS EBS documentation, that are very important:
Amazon EBS volumes are designed to be highly available and reliable.
At no additional charge to you, Amazon EBS volume data is replicated
across multiple servers in an Availability Zone to prevent the loss of
data from the failure of any single component.
If you wish to achieve greater durability, you can use the Amazon EBS
Snapshot capability. Snapshots are stored in Amazon S3 and are also
replicated automatically among multiple Availability Zones. You can
take frequent snapshots of your volume for a convenient and
cost-effective way to increase the long-term durability of your data.
In the unlikely event that your Amazon EBS volume does fail, all
snapshots of that volume remain intact and you can re-create your
volume from the last snapshot.
Here, some notes about the durability of EBS volumes:
The durability of your volume depends both on the size of your volume
and the percentage of the data that has changed since your last
snapshot. As an example, volumes that operate with 20 GB or less of
modified data since their most recent Amazon EBS Snapshot can expect
an annual failure rate (AFR) of between 0.1% – 0.5%, where failure
refers to a complete loss of the volume. This compares with commodity
hard disks that typically fail with an AFR of around 4%, making EBS
volumes 10 times more reliable than typical commodity disk drives.
Important details about the price:
Amazon EBS Snapshots are stored incrementally: only the blocks that
have changed after your last snapshot are saved, and you are billed
only for the changed blocks. If you have a device with 100 GB of data
but only 5 GB has changed after your last snapshot, a subsequent
snapshot consumes only 5 additional GB and you are billed only for the
additional 5 GB of snapshot storage, even though both the earlier and
later snapshots appear complete.
Here is why you may stay secure when you delete one of your snapshots:
When you delete a snapshot, you remove only the data not needed by any
other snapshot. All active snapshots contain all the information
needed to restore the volume to the instant at which that snapshot was
taken. The time to restore changed data to the working volume is the
same for all snapshots.
Another important advantage of snapshots:
Snapshots can be used to instantiate multiple new volumes, expand the
size of a volume, or move volumes across Availability Zones. When a
new volume is created, you may choose to create it based on an
existing Amazon EBS snapshot. In that scenario, the new volume begins
as an exact replica of the snapshot.
Ok, I think that these are some of the most important things to know when using amazon EBS. For further details take a look here. Pay particular attention on the "Amazon EBS Snapshots" section.
Related
Actually I have to take forecast of costing for one my instance, which is having a number of volumes attached... These volumes are different in size and types.
Let's suppose I took the AMI backup and terminated the server.
Now my confusion is how would I calculate the cost. The cost will be calculated based on pricing of Amazon EBS Volumes or Amazon EBS Snapshot. Because the cost difference is just double.
Let me know if you can help me understanding.
pricing of Amazon EBS Volumes or Amazon EBS Snapshot Which I took from AWS Pricing :
https://aws.amazon.com/ebs/pricing/
Amazon EBS snapshots are a complex subject due to the way they work.
There is a detailed explanation in: Amazon EBS snapshots - Amazon Elastic Compute Cloud
A quick summary is:
Snapshots contain only the data that is different to previous snapshots (they are incremental)
An AMI is actually a snapshot. So, if you booted a new Amazon EC2 instance from an AMI and then created a snapshot, the snapshot would contain very little since most of the volume was already contained in the previous snapshot (that was part of the AMI). Confused yet?
Any snapshot can be deleted and information will still be retained to allow any other snapshot to be restored. So, the snapshot is actually an 'index' to the snapshot data, and the snapshot data is stored separately to the snapshot itself. You should be questioning your sanity at this point!
So, the cost of Amazon EBS snapshots is mostly based on how much the contents of the volume changes, and how many snapshots (effectively, points-in-time) you wish to keep. If you only keep the most recent snapshot, then all data will be available, but the cost will be minimised because it won't keep any data that has been deleted from the volume.
Bottom line: Snapshots take less space than the data on a volume due to the incremental natures. The more snapshots ("points-in-time"), the more data will be kept and hence the more cost.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 9 years ago.
Improve this question
I have just created my first EC2 instance 2 days ago and had some questions about its charging against root device. Currently, the root device is EBS type with 4000 IOPS. This device is reliable but it is over my budget (about $10 per day, even when I shutdown the instance), since my site is under the developing mode at this moment. So my questions:
If I stay with EBS type root device, any suggestions on how to lower its cost (e.g. switch to standard EBS)?
Should I use regular instance-store root device? What is the cost for this selection?
Thanks!
It appears you are using provisioned IOPS, which is an optional feature when High IO performance is needed from EBS volumes.
You wont need to switch to instance storage to avoid this cost. But you will need to re-provision the volume without provisioned IOPS to bring the cost down.
The quickest way would be to launch a new instance, but in this case, do not enable provisoned IOPS.
However, if you already have software installed on your volumes, and don't want to reinstall it. Then create a snapshot of your current volume. Use that snapshot to create a new volume without provisioned IOPS. Switch the root volume of your instance to the new volume created from a snapshot. Then delete your old volume.
For starting out, I recommend you use
Standard EBS boot (not instance-store)
No EBS optimized instance
No Provisioned IOPS
Once you get comfortable with how EC2 works and if you run into IO bottlenecks, then you can test upgrading to EBS optimized instances and Provisioned IOPS EBS volumes. These two features work together. Use the lowest PIOPS settings that work for your application.
Once you are an expert with EC2 and are not worried about losing an instance and all its data (because you can automate the creation of new instances and you have streaming replication/backups of all your data) then you can consider using instance-store boot disks. I wrote an article about EBS boot vs instance-store.
I would recommend you start a new EBS boot instance from scratch without EBS optimized and without EBS Provisioned IOPS. You should work to automate this startup process so that you can easily replace an instance; there are a lot of times when this is useful as you are already finding out two days into your experience.
I'm thinking about saving data from EC2 instances to the EBS and later save the result on S3. I don't have a lot of experience working with EBS, so my questions are:
How stable they are? I mean how often (if any) you had problem with EBS. Do they crash if overloaded or something like this?
What are the chances of loosing data from EBS?
Is it possible to mount one EBS to the multiple Instances? (let's say two ec2 share the same ebs )
I assume you've read AWS's take on EBS
Pretty stable. Last year, 10% of EBS volumes failed in 2-3 data centers in us-east for a couple hours. This is the only issue I've ever had with them.
I've never lost data from EBS. Even if I had, I take hourly snapshots (stored in s3), so I would have been just fine.
Not at the same time. To attach it to another instance, you must detach from the currently attached one.
Perhaps what you're look is s3fs - a way to mount s3 as a filesystem.
EBS is quite stable and every data you write is redundantly copied in 3 disks inside a AZ. If you take regular snapshots of your EBS volumes you can protect your data more. Since EBS operate in AZ scope it is recommended to moves assets like user documents, images, videos to Amazon S3. S3 offer more redundancy and availability than an EBS Volume.
You cannot mount single EBS volume to Multiple EC2 instances. You will have to use Solution like GlusterFS on AWS so that multiple EC2 instances can talk to common storage pool.
The EBS documentation states:
As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume.
..but this doesn't give any indication of the AFR for a volume with, for example:
No snapshot at all; or
A fresh snapshot with no modified data.
I've seen it suggested that missing or damaged blocks can be automatically/silently recovered from snapshots but I can't see any reference to this in the documentation. Is this true?
Can I assume that if I have a volume with no changed data and a fresh snapshot, my AFR for the volume matches S3's reliability?
I took a three day class from AWS last year, and they told us unequivocally that taking snapshots greatly increases the reliability of an EBS volume. They did not explain why that was so, but hinted that EBS volumes store changes from the latest snapshot and that the snapshot itself is very stable (stored in S3). Successive snapshots apparently use little storage, as AWS is smart enough to store diffs.
They did not give any hard numbers on failure rates, though. They suggested configuring multiple EBS volumes using RAID if reliability of the volume is essential. However, they also recommended architecting your application so that it can tolerate failure of any instance, making it less important for each EBS volume to be durable.
Snapshots taken of EBS Volumes are stored in S3. These snapshots get all the durability and availability benefits of S3. You can also copy snapshots to other regions, which is a nice insurance policy against a regional level outage.
If your EBS volume fails, you can then recover from your last snapshot. The more recent your snapshot, the more up-to-date your recovery story is. With the incremental nature of EBS snapshots performing them on a frequent basis is very practical.
EBS also provides "recovery volumes", which you can see from this AWS forum thread.
To my knowledge, the act of taking a snapshot doesn't directly impact the AFR of an active, running EBS volume. Rather, it just makes it easier for you to recover in the event of a failure.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I understand that AWS snapshots can create incremental backups of EBS volumes. Does AWS automatically handle the incremental part (i.e., storing only what has changed) as long as snapshots are generated from the same volume?
It's unclear to me because they do not list the actual size of the snapshots or allow you to view them in S3 (as far as I know). There is no indication snapshots are related other than the volume they were created from. Couldn't any snapshots made (including the first) just be considered an increment on the original AMI? I would be interested to know if this how they actually implement this or if the first snapshot is a completely independent image stored in my personal S3 account.
Each EBS snapshot only incrementally adds the blocks that have been modified since the last snapshot.
Each EBS snapshot has all of the blocks that have been used ever on the EBS volume. You can delete any snapshot without reducing the completeness of any other snapshot.
It's magic.
Well, it's actually a bit of technological indirection where each snapshot has pointers to the blocks that it cares about and multiple snapshots can share the same blocks. As long as there is at least one snapshot that points to a particular set of data on a block the block is preserved in S3.
This makes it difficult for Amazon to tell you how much space a single snapshot takes up, because their sizes are not exclusive of each other.
Here's an old article from RightScale that has some nice pictures explaining how snapshots work behind the scenes:
http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/
Note also that snapshots only save the blocks on the EBS volume that have been used and snapshots are compressed, further reducing your data storage cost.