I have an EBS volume that displays a state of "in-use - optimizing(%)". What does this mean? What are the optimizations that AWS is performing? This is on a 300gb encrypted gp2 volume attached to a Windows Server 2012 R2 EC2 instance.
The in-use - optimizing state relates to EBS volume resizing.
in-use indicates that this volume is attached to an EC2 instance.
optimizing is the volume's modification state.
According to the AWS documentation on volume modifications:
An EBS volume being modified goes through a sequence of states. After you issue a ModifyVolume directive, whether from the console, CLI, API, or SDK, the volume enters first the Modifying state, then the Optimizing state, and finally the Complete state.
...
While the volume is in the optimizing state, your volume performance is in between the source and target configuration specifications. Transitional volume performance will be no less than the source volume performance. If you are downgrading IOPS, transitional volume performance is no less than the target volume performance.
And finally, from the introductory blog post for Volume Modifications:
The volume’s state reflects the progress of the operation (modifying, optimizing, or complete):
If you modified the volume, most likely it will show like that. The performance will be degraded during this time since AWS EBS server needs to sync data.
EBS Volumes are in this state after modifying the volume (e.g. resizing). It can take some time (can be stuck at 99% for hours) but eventually will go away.
Related
Once I stopped our instance and modified the size of one of the EBS volumes. Then, that EBS is stopping at in 'in-use - optimizing (60%)' status. I suppose it can sometimes happen to take long time for optimizing up to 24hours. But we need to start our EC2 instance as soon as possible.
I'm just wondering if it is possible to start EC2 instance, one of the EBS in optimizing status not completed yet.
That EBS is not root volume, but the important volume contains database files.
Any advice would be appreciated.
Yes you can run the EC2 instance even whilst the EBS volume is optimising.
Whilst it is occuring you might find that the performance varies between both modifications however, it will never be lower than the minimal performance of either previous or new configuration.
While the volume is in the optimizing state, your volume performance is in between the source and target configuration specifications. Transitional volume performance will be no less than the source volume performance. If you are downgrading IOPS, transitional volume performance is no less than the target volume performance.
More information available in the documentation.
Cant Modify or Resize Amazon EBS Volume. In us-east-1d N. Virginia. The instance it's connected to is t2.medium CentOS 7.
Any help would be appreciated.
Update: the answer below was correct when written, but was subsequently superceded by this announcement on June 28, 2018:
Starting today, Elastic Volumes extends support to Amazon Elastic Block Store (EBS) magnetic (standard) volume type. You can now dynamically increase capacity or change the type of magnetic (standard) volumes with no downtime or performance impact using a simple API call or a few console clicks.
https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ebs-extends-elastic-volumes-to-support-ebs-magnetic--standard--volume-type/
The issue that originally triggered this question should no longer occur.
In the screen shot, the volume type shows standard.
That's a previous generation magnetic volume.
The previous generation Magnetic volume type is not supported by the volume modification methods [...]
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/limitations.html
Those can't be resized, so the option is grayed out. Your new instance from the AMI probably has a gp2 SSD volume, which does support resize.
Stop the instance that the EBS volume is attached to.
Then the modify option should be available for use.
If that doesn't work then try detaching the volume.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html
As #michael-sqlbot says, you are using magnetic "standard" EBS volume, you can convert to ssd "gp2" volume following these steps:
Create snapshot of your EBS volume
Create new PIOPS/General Purpose SSD volume from your EBS
snapshot
Detaching existing attached magnetic volume from the
instance
Attach new PIOPS/General Purpose SSD volume
More info: https://n2ws.com/blog/how-to-guides/how-to-change-your-ebs-volume-type
I have two amazon machine instances running.Both of them are m3.xlarge instances. One of them has the right software and configuration that I want to use.I want to create a snapshot of the EBS volume for that machine and use that as the EBS volue to boot the second machine from. Can I do that and expect it to work without shutting down the first machine.
It is well described in the AWS documentation...
"You can take a snapshot of an attached volume that is in use. However, snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued. This might exclude any data that has been cached by any applications or the operating system. If you can pause any file writes to the volume long enough to take a snapshot, your snapshot should be complete. However, if you can't pause all file writes to the volume, you should unmount the volume from within the instance, issue the snapshot command, and then remount the volume to ensure a consistent and complete snapshot.
I have amazon as well, with 3 different clusters. With one of my clusters after setting up 25 of them I realized there was a small issue in the configuration and had live traffic going to them so I couldn't' shut down.
You can snapshot the first machines volume while it's still running, I had to do this myself. It took a little while, but ultimately it worked out. Please note that amazon cannot guarantee the consistency of the disk when doing this.
I did a snapshot of the entire thing, fixed what needed to be fixed, and spooled up 25 new servers and terminated the other 25 ( easier than modifying volumes, etc ).. But you can create a new volume with the new snapshot, and attach it to an instance and do what needs to be done to get it to boot off that volume without much of a headache.
Being that I went the easy route of spooling up new instances after my snapshot was complete, I can't walk you through on how to get a running instance to boot off a new volume.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
Is it good idea to create periodic snapshot of the EBS volume from same instance it is attached to? Is there any downtime during snapshot process? I basically wanted to keep a regular may be daily or weekly snapshot of the ec2 instance so that If there is any virus or hacking or security issue I could spin another instance from the snapshots.
Absolutely yes. It's a good practice (personally, I consider it a must) to create point-in-time snapshots and to use them to create new volumes or restore old volumes. There is no downtime during the snapshot process. For a more detailed explanation you may take a look here, with particular emphasis on this part:
You can take a snapshot of an attached volume that is in use. However,
snapshots only capture data that has been written to your Amazon EBS
volume at the time the snapshot command is issued. This may exclude
any data that has been cached by any applications or the operating
system. If you can pause any file writes to the volume long enough to
take a snapshot, your snapshot should be complete. However, if you
can't pause all file writes to the volume, you should unmount the
volume from within the instance, issue the snapshot command, and then
remount the volume to ensure a consistent and complete snapshot. You
may remount and use your volume while the snapshot status is pending.
Before doing operation involving data I think it's very important to know everything about a technology that you are going to use. So, I would like take this opportunity to put the focus on some points, taken from the official AWS EBS documentation, that are very important:
Amazon EBS volumes are designed to be highly available and reliable.
At no additional charge to you, Amazon EBS volume data is replicated
across multiple servers in an Availability Zone to prevent the loss of
data from the failure of any single component.
If you wish to achieve greater durability, you can use the Amazon EBS
Snapshot capability. Snapshots are stored in Amazon S3 and are also
replicated automatically among multiple Availability Zones. You can
take frequent snapshots of your volume for a convenient and
cost-effective way to increase the long-term durability of your data.
In the unlikely event that your Amazon EBS volume does fail, all
snapshots of that volume remain intact and you can re-create your
volume from the last snapshot.
Here, some notes about the durability of EBS volumes:
The durability of your volume depends both on the size of your volume
and the percentage of the data that has changed since your last
snapshot. As an example, volumes that operate with 20 GB or less of
modified data since their most recent Amazon EBS Snapshot can expect
an annual failure rate (AFR) of between 0.1% – 0.5%, where failure
refers to a complete loss of the volume. This compares with commodity
hard disks that typically fail with an AFR of around 4%, making EBS
volumes 10 times more reliable than typical commodity disk drives.
Important details about the price:
Amazon EBS Snapshots are stored incrementally: only the blocks that
have changed after your last snapshot are saved, and you are billed
only for the changed blocks. If you have a device with 100 GB of data
but only 5 GB has changed after your last snapshot, a subsequent
snapshot consumes only 5 additional GB and you are billed only for the
additional 5 GB of snapshot storage, even though both the earlier and
later snapshots appear complete.
Here is why you may stay secure when you delete one of your snapshots:
When you delete a snapshot, you remove only the data not needed by any
other snapshot. All active snapshots contain all the information
needed to restore the volume to the instant at which that snapshot was
taken. The time to restore changed data to the working volume is the
same for all snapshots.
Another important advantage of snapshots:
Snapshots can be used to instantiate multiple new volumes, expand the
size of a volume, or move volumes across Availability Zones. When a
new volume is created, you may choose to create it based on an
existing Amazon EBS snapshot. In that scenario, the new volume begins
as an exact replica of the snapshot.
Ok, I think that these are some of the most important things to know when using amazon EBS. For further details take a look here. Pay particular attention on the "Amazon EBS Snapshots" section.
The EBS documentation states:
As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% – 0.5%, where failure refers to a complete loss of the volume.
..but this doesn't give any indication of the AFR for a volume with, for example:
No snapshot at all; or
A fresh snapshot with no modified data.
I've seen it suggested that missing or damaged blocks can be automatically/silently recovered from snapshots but I can't see any reference to this in the documentation. Is this true?
Can I assume that if I have a volume with no changed data and a fresh snapshot, my AFR for the volume matches S3's reliability?
I took a three day class from AWS last year, and they told us unequivocally that taking snapshots greatly increases the reliability of an EBS volume. They did not explain why that was so, but hinted that EBS volumes store changes from the latest snapshot and that the snapshot itself is very stable (stored in S3). Successive snapshots apparently use little storage, as AWS is smart enough to store diffs.
They did not give any hard numbers on failure rates, though. They suggested configuring multiple EBS volumes using RAID if reliability of the volume is essential. However, they also recommended architecting your application so that it can tolerate failure of any instance, making it less important for each EBS volume to be durable.
Snapshots taken of EBS Volumes are stored in S3. These snapshots get all the durability and availability benefits of S3. You can also copy snapshots to other regions, which is a nice insurance policy against a regional level outage.
If your EBS volume fails, you can then recover from your last snapshot. The more recent your snapshot, the more up-to-date your recovery story is. With the incremental nature of EBS snapshots performing them on a frequent basis is very practical.
EBS also provides "recovery volumes", which you can see from this AWS forum thread.
To my knowledge, the act of taking a snapshot doesn't directly impact the AFR of an active, running EBS volume. Rather, it just makes it easier for you to recover in the event of a failure.