AWS change EBS root volume on data collection system - amazon-web-services

I have the following use case:
I have an EC2 instance running with an 1000GB io1 SSD EBS root volume attached
It is used for a data collection service which accumulates data every minute
I like to change the volume for a cheaper storage system (as the specs of the process are not IOPS intensive)
I've read through the AWS documentation and found the following articles:
https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-elastic-block-store-amazon-ebs-enables-live-volume-modifications-with-elastic-volumes/
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html#console-modify
Which indicate that (from february 2017) for Current Generation Instances a volume type change is possible on the fly in the console without stopping the instance. However, when I go to my console I can only choose between io1 (current) or gp2 (also SSD), I'd like to choose sc1 (cold storage) for this instance. The video clearly shows this option (go from io1 to sc1).
Am I missing something? Is there a restriction to go from SSD to HDD?
(im in Asia Sydney region).
Alternatively, if the option is not possible and i have the requirement that data collection continues, it seems that my only option is:
re-create a similar instance with the data collection scripts deployed with an EBS sc1 volume as root attached
get the data collection running (now collecting double)
stop the old instance and take a snapshot of the io1 EBS volume
create a EBS sc1 volume with the snapshot
attach the EBS sc1 volume to the new instance (as additional volume, not root)
Would this be the right way to do so? (if I can't change the volume type to sc1 while it's running)

tldr;
The 2 main reasons why amazon will not propose sc1/st1 type when editing the EBS volumes are:
the current volume is used as root device of the ec2 instance
the current storage is less than 500 Gb
I guess the volume you want to change is used as root device of the instance; in this case you cannot have boot volume of magnetic type st1 or sc1.
Note also the sc1 volume must be at least 500 Gb
You can read more about EBS Volume Types
Unfortunately, you cannot change the root device of a running instance, you can do that while the instance is stopped.
so you can follow your steps proposed but you can make the change of the EBS after it will be detached from the first ec2 instance
re-create a similar instance with the data collection scripts deployed with an EBS sc1 volume as root attached (You cannot have sc1/st1 drive as boot volume; if you want magnetic storage you need to select standard Magnetic storage; but ideally you should have a low volume ssd drive as boot volume and the large magnetic drive as additional ebs volume)
get the data collection running (now collecting double)
stop the old instance and detach the EBS volume
make the modification of the EBS volume (type/storage)
attach the EBS sc1 volume to the new instance (as additional volume, not root)

Related

AWS EC2: AMI vs instance type EBS selection

While creating EC2 instance on AWS, I see following AMI options:
Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type
macOS Monterey 12.2.1 - ami-00aa9c0d2393d0139 The macOS Monterey AMI is an EBS-backed, AWS-supported image
First option tells that disk storage is SSD volume type which I assume means that storage used is instance's local storage. Second option tells that storage is elastic block storage.
I opt for first one.
Then on next step for instance type:
Why does it displays 'EBS only' for 'Instance Storage(GB)' column as in previous step I opted for non-EBS type? Are those unrelated?
An AMI will always restore to an EBS volume (or multiple EBS volumes). An AMI is really just a snapshot of one or more EBS volumes, plus some metadata. EBS volumes can be SSD or Magnetic. Your assumption that "SSD Volume Type" indicated local instance storage is incorrect.
Instance storage is ephemeral. An EC2 server never boots from an instance storage volume. Certain instance types come with instance storage that are good for things like fast temp folders, caches, etc. But instance storage volumes can't be saved to snapshots or AMIs, can't be persisted across stops/start cycles, and can't be moved between EC2 instances.

How to create a different AWS EBS Volume from a AWS Snapshot created out of another EBS type

Can we create a snapshot of EBS volume of Type - Gen Purpose SSD and later create a EBS volume of Type - Provisioned IOPS SSD from that snapshot?
I am thinking to do this to speed up my copy/update/delete operation of data from the snapshot. Depicting the steps below that I would want to take -
EBS Volume xvdf ( Type: Gen Purpose SSD) -> Create Snapshot S_xvdf -> Create EBS Volume xvdp(Type : Provisioned IOPS SSD) > Select Snapshot S_xvdf > Volume xvdp created -> Attach Volume as another drive > copy/update/delete
-- In 'thoughts'...
Yes, you can create a new volume, using any volume type, from your snapshot.
Doesn't matter what was the volume type when snapshot was created.
You can also change your volume type without interruption.
You don't need to create a new one just because you need more IOPS or throughput.
But you can only change it once at every 6 hours.
There is a good youtube video called "Deep Dive on Amazon EBS Elastic Volumes - 2017 AWS Online Tech Talks".
Yes, it can be done.
Created a 3TB IOPS SSD volume from 1TB Gen Purpose SSD Snapshot.
When we does modifications to Volume - status is shown as "Available - Optimizing x%" in Orange.
--In 'thoughts'...

Launch Instance with smaller root volume storage

I want to launch an instance with custom size of root volume on Amazon EC2.
At step no.4 - Add Storage, the root volume default setting came with a default snapshot and a default 10 size (GiB). Then I lowered the size into 5GiB.
But at the final step, it wouldn't allow me to launch the instance because of only 5GB root volume.
Any idea or solution that I can lower the size when launching an instance?
You cannot create an Amazon EBS volume that is smaller than the snapshot that you want loaded.
You could attempt to make your own AMI by launching with 10GB, attaching a 5GB volume, copying files across, turning it into an AMI, etc but frankly it isn't worth the bother.
If you are merely wishing to save money, then at 10¢ per GB per month, it would only save 50¢ per month.

Cant Modify or Resize Amazon EBS Volume

Cant Modify or Resize Amazon EBS Volume. In us-east-1d N. Virginia. The instance it's connected to is t2.medium CentOS 7.
Any help would be appreciated.
Update: the answer below was correct when written, but was subsequently superceded by this announcement on June 28, 2018:
Starting today, Elastic Volumes extends support to Amazon Elastic Block Store (EBS) magnetic (standard) volume type. You can now dynamically increase capacity or change the type of magnetic (standard) volumes with no downtime or performance impact using a simple API call or a few console clicks.
https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ebs-extends-elastic-volumes-to-support-ebs-magnetic--standard--volume-type/
The issue that originally triggered this question should no longer occur.
In the screen shot, the volume type shows standard.
That's a previous generation magnetic volume.
The previous generation Magnetic volume type is not supported by the volume modification methods [...]
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/limitations.html
Those can't be resized, so the option is grayed out. Your new instance from the AMI probably has a gp2 SSD volume, which does support resize.
Stop the instance that the EBS volume is attached to.
Then the modify option should be available for use.
If that doesn't work then try detaching the volume.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html
As #michael-sqlbot says, you are using magnetic "standard" EBS volume, you can convert to ssd "gp2" volume following these steps:
Create snapshot of your EBS volume
Create new PIOPS/General Purpose SSD volume from your EBS
snapshot
Detaching existing attached magnetic volume from the
instance
Attach new PIOPS/General Purpose SSD volume
More info: https://n2ws.com/blog/how-to-guides/how-to-change-your-ebs-volume-type

Where OS and its settings are stored in EC2 instance

Using ec2 Windows instance with Instance storage (let's say 32GB SSD) - where OS and its settings are stored? Like Program Files, User profiles. Are they all stored on Instance Storage? As far as I understood from other topics Instance storage is not-persistent and doesn't survive shutdowns/terminations. Does that mean I will lose everything under C: drive if I turn it off?
Can I use EBS storage as a default storage for OS (C drive)? Can I map multiple EBS storages to one Windows storage?
If above is true, then I will be charged for the capacity used by OS on EBS instance? It would be around 20GB I believe. Is that correct?
I am quite new in aws, and before paying for such instances or EBS I would like to know how this technical and billing model is working.
Thank you!
The Storage for the Root device is dependent on the AMI (EBS-Backed or Instance Store-Backed) used to launch the instance.
As far as I understood from other topics Instance storage is
not-persistent and doesn't survive shutdowns/terminations.
If the Root storage device is Instance Store, Stopping (shutdown) the instance is not possible. On termination, Both the storage and Instance does not survive. The Instance does not survive once terminated even if the AMI is EBS-Backed, but you can persist the Root Volume by setting the DeleteOnTermination flag set to False.
Does that mean I will lose everything under C: drive if I turn it off?
You cannot turn off (shutdown) an Instance Store-backed instance.
Can I use EBS storage as a default storage for OS (C drive)?
Yes, Choose an EBS backed Windows AMI.
Can I map multiple EBS storages to one Windows storage?
Yes, multiple EBS Volumes can be attached to one EC2 Windows Instance.
If above is true, then I will be charged for the capacity used by OS
on EBS instance?
You will be charged for the total size of the EBS volumes attached to the instance including the Root Device.
It would be around 20GB I believe. Is that correct?
The EBS Volume Size is adjustable. The upper Size limit is 16TiB.
Read Storage for Root Device and Ec2 Root Device Volume
Please spend more time on the AWS documentation, I don't think here is enough to cover all your question.
Only for specify EC2 instance come with attached SSD storage AKA instance storage. Bare in mind that, this instance storage doesn't come with Snapshot capabilities, so you must backup the file yourself. This is mean for people who need fastest disk access to process their data.
Only EBS allow you do multiple snapshot.
You can always create an AMI image for your instance after complete the deployment. AMI image is store inside EBS, so you will not lost the initial instance if you do this, so for new instance, you just trigger load it from AMI.
If you "Terminate" an instance, it will delete the virtual image. There is no way to recover it even with EBS, unless you make a snapshot. However, attached EBS storage will not be deleted.
EBS is calculate by Per GB and give you 1GB x 3 IOPS, with base 100 IOPS given. This is not enough if anyone want to carry out disk I/O intensive task.