I have an ec2 instance that is hosting a CentOS AMI image and the root device is EBS , however it is not EBS optimized.
I have installed a few packages on it now I want to stop and start it again , Amazon documentation says that the EBS data would be available but the instance store data would be lost.
How do I know where(EBS or Instance store) my packages are stored ? I see that package files are in /opr /var /etc directories .
Will I loose my installed packages if I stop and start the Amazon ec2 instance ?
Thanks.
When you create an EBS backed instance (with ephemeral or instance store storage, and it doesn't matter whether it's optimized or not optimized) you don't lose data in your /opt or /var or /etc directory or any of the system data. So you are safe to stop and then restart it. Keep in mind that your internal and public IP addresses change once you restart it.
The only data that you lose is if you have ephemeral volumes which are generally mounted volumes with devices like /dev/sdb, /dev/xvdb, /dev/xvdc, etc.
If you create an instance store "only" instance then you lose everything. However, you will be able to tell if your instance is this type by not having the option to "stop" it. Meaning you can only terminate it. These are the first type of instances that EC2 offered when they started and maybe up until 3-4 years ago were the only ones, so they are not used that much AFAIK unless you need an ephemeral volume as your root volume.
[Edit]
This is what it's supposed to look like for an EBS backed instance (non-optimized):
You will not lose your data if the instance is setup as EBS.
EBS optimised is another option which adds additional IOPS, useful for busy database applications, etc.
Related
I am learning about aws and using ec2 instances. I am trying to understand what a volume is.
I have read from the aws site that:
An Amazon EBS volume is a durable, block-level storage device that you
can attach to your instances. After you attach a volume to an
instance, you can use it as you would use a physical hard drive.
Is it where things are stored when I install things like npm and node? Does it function like the harddrive o my server?
AWS EBS is block storage volume, and for the ease of understanding, yes you can consider it same as hard drive, however with more benefits over traditional hard drive. few of them are:
You can increase/decrease size of the storage as per your requirement
(Hence name Elastic)
You can add multiple ebs to your instances, for example 20 GB of volume1 and 30 GB of volume2
And for the question you asked if you can install npm & node yes you
can as it would be attached to your EC2 instance and your instance
can easily utilised attached data, modules,etc
For further explanation you can refer this user guide from AWS on EBS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes.html
Yes it is exactly like a hard drive on your server and you can have multiple devices.
The cool thing is that you also can expand them if you need extra space.
where things are stored when I install things like npm and node
Yes, technically ebs volume is virtual storage drive which is
connected to your instance through network, ( flash drive connected over network).
Since network is involved which implicitly means there will be some latency because of data transfer through network.
The data is persistent even if instance stops,terminates, hibernates or hardware failure.
Since it is network drive it can be attached or detached to any other instance.
Adding to this there is another type of storage which you will find called as instance store
You can specify instance store volumes for an instance only when you launch it. You can't detach an instance store volume from one instance and attach it to a different instance.
it gives very high IOPS because it is directly (physically) attached to instance.
The use case for instance store would where data changes rapidly like for cache or buffers.
Your data will be lost if any of these events happens like The underlying disk drive fails The instance stops, instance hibernates instance terminates or drive failure.
I am facing a challenge to scale an existing Amazon EC2 instance from t2.micro to t2.medium.
Can someone suggest me about the best practice to do this, without losing any data which stored on the t2.micro instance?
Follow these steps:
Stop the EC2 instance.
Change the instance type (e.g. t2.micro to t2.medium).
Start the EC2 instance.
Note: You will not lose the data.
When you stop an EC2 instance, what AWS does is the following:
It (AWS) shuts down the machine. So, the CPU, RAM, etc, are available for someone else to use.
It preserves the hard disk(s). Hence, your data is always with you.
In the shut-down state, you can change the machine type. This only logically links the CPU, RAM, etc. When you start the machine, AWS allocates the actual machine to you and connects your hard disk to that machine.
Note: In the shut-down state, you don't pay for CPU, RAM etc but since you keep the disk, you continue to pay a little amount towards the disk.
If you do not want to use the machine in the future select option menu > Instance State > Terminate.
The answer is simple, stop the EC2 instance, change its instance type and start it.
Note: Stopping and restarting the instance erases any data on instance store volumes. Be sure that you back up any data on the instance store volume that you want to keep. Instance store volumes are different from EBS volumes.
Reference: Back Up an Instance Store Volume to EBS
So I created i3.large with NVME disk on each nodes, here was my process :
lsblk -> nvme0n1 (check if nvme isn't yet mounted)
sudo mkfs.ext4 -E nodiscard /dev/nvme0n1
sudo mount -o discard /dev/nvme0n1 /mnt/my-data
/dev/nvme0n1 /mnt/my-data ext4 defaults,nofail,discard 0 2
sudo mount -a (check if everything is OK)
sudo reboot
So all of this works, I can connect back to the instance. I have 500 GiB on my new partition.
But after I stop and restart the EC2 machines, some of them randomly became inaccessible (AWS warning only 1/2 test status checked)
When I watch the logs of why it is inaccessible it tells me, it's about the nvme partition (but I did sudo mount -a to check if this was ok, so I don't understand)
I don't have the AWS logs exactly, but I got some lines of it :
Bad magic number in super-block while trying to open
then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
/dev/fd/9: line 2: plymouth: command not found
I have been using "c5" type instances since almost a month, mostly "c5d.4xlarge" with nvme drives. So, here's what has worked for me on Ubuntu instances:
first get the location nvme drive is located at:
lsblk
mine was always mounted at nvme1n1. Then check if it is an empty volume and doens't has any file system, (it mostly doesn't, unless you are remounting). the output should be /dev/nvme1n1: data for empty drives:
sudo file -s /dev/nvme1n1
Then do this to format(if from last step you learned that your drive had file system and isn't an empty drive. skip this and go to next step):
sudo mkfs -t xfs /dev/nvme1n1
Then create a folder in current directory and mount the nvme drive:
sudo mkdir /data
sudo mount /dev/nvme1n1 /data
you can now even check it's existence by running:
df -h
Stopping and starting an instance erases the ephemeral disks, moves the instance to new host hardware, and gives you new empty disks... so the ephemeral disks will always be blank after stop/start. When an instance is stopped, it doesn't exist on any physical host -- the resources are freed.
So, the best approach, if you are going to be stopping and starting instances is not to add them to /etc/fstab but rather to just format them on first boot and mount them after that. One way of testing whether a filesystem is already present is using the file utility and grep its output. If grep doesn't find a match, it returns false.
The NVMe SSD on the i3 instance class is an example of an Instance Store Volume, also known as an Ephemeral [ Disk | Volume | Drive ]. They are physically inside the instance and extremely fast, but not redundant and not intended for persistent data... hence, "ephemeral." Persistent data needs to be on an Elastic Block Store (EBS) volume or an Elastic File System (EFS), both of which survive instance stop/start, hardware failures, and maintenance.
It isn't clear why your instances are failing to boot, but nofail may not be doing what you expect when a volume is present but has no filesystem. My impression has been that eventually it should succeed.
But, you may need to apt-get install linux-aws if running Ubuntu 16.04. Ubuntu 14.04 NVMe support is not really stable and not recommended.
Each of these three storage solutions has its advantages and disadvantages.
The Instance Store is local, so it's quite fast... but, it's ephemeral. It survives hard and soft reboots, but not stop/start cycles. If your instance suffers a hardware failure, or is scheduled for retirement, as eventually happens to all hardware, you will have to stop and start the instance to move it to new hardware. Reserved and dedicated instances don't change ephemeral disk behavior.
EBS is persistent, redundant storage, that can be detached from one instance and moved to another (and this happens automatically across a stop/start). EBS supports point-in-time snapshots, and these are incremental at the block level, so you don't pay for storing the data that didn't change across snapshots... but through some excellent witchcraft, you also don't have to keep track of "full" vs. "incremental" snapshots -- the snapshots are only logical containers of pointers to the backed-up data blocks, so they are in essence, all "full" snapshots, but only billed as incrememental. When you delete a snapshot, only the blocks no longer needed to restore either that snapshot and any other snapshot are purged from the back-end storage system (which, transparent to you, actually uses Amazon S3).
EBS volumes are available as both SSD and spinning platter magnetic volumes, again with tradeoffs in cost, performance, and appropriate applications. See EBS Volume Types. EBS volumes mimic ordinary hard drives, except that their capacity can be manually increased on demand (but not decreased), and can be converted from one volume type to another without shutting down the system. EBS does all of the data migration on the fly, with a reduction in performance but no disruption. This is a relatively recent innovation.
EFS uses NFS, so you can mount an EFS filesystem on as many instances as you like, even across availability zones within one region. The size limit for any one file in EFS is 52 terabytes, and your instance will actually report 8 exabytes of free space. The actual free space is for all practical purposes unlimited, but EFS is also the most expensive -- if you did have a 52 TiB file stored there for one month, that storage would cost over $15,000. The most I ever stored was about 20 TiB for 2 weeks, cost me about $5k but if you need the space, the space is there. It's billed hourly, so if you stored the 52 TiB file for just a couple of hours and then deleted it, you'd pay maybe $50. The "Elastic" in EFS refers to the capacity and the price. You don't pre-provision space on EFS. You use what you need and delete what you don't, and the billable size is calculated hourly.
A discussion of storage wouldn't be complete without S3. It's not a filesystem, it's an object store. At about 1/10 the price of EFS, S3 also has effectively infinite capacity, and a maximum object size of 5TB. Some applications would be better designed using S3 objects, instead of files.
S3 can also be easily used by systems outside of AWS, whether in your data center or in another cloud. The other storage technologies are intended for use inside EC2, though there is an undocumented workaround that allows EFS to be used externally or across regions, with proxies and tunnels.
I just had a similar experience! My C5.xlarge instance detects an EBS as nvme1n1. I have added this line in fstab.
/dev/nvme1n1 /data ext4 discard,defaults,nofail 0 2
After a couple of rebooting, it looked working. It kept running for weeks. But today, I just got alert that instance was unable to be connected. I tried rebooting it from AWS console, no luck looks the culprit is the fstab. The disk mount is failed.
I raised the ticket to AWS support, no feedback yet. I have to start a new instance to recover my service.
In another test instance, I try to use UUID(get by command blkid) instead of /dev/nvme1n1. So far looks still working... will see if it cause any issue.
I will update here if any AWS support feedback.
================ EDIT with my fix ===========
AWS doesn't give me feedback yet, but I found the issue. Actually, in fstab, whatever you mount /dev/nvme1n1 or UUID, it doesn't matter. My issue is, my ESB has some errors in file system. I attached it to an instance then run
fsck.ext4 /dev/nvme1n1
After fixes a couple of file system error, put it in fstab, reboot, no problem anymore!
You may find useful new EC2 instance family equipped with local NVMe storage: C5d.
See announcement blog post: https://aws.amazon.com/blogs/aws/ec2-instance-update-c5-instances-with-local-nvme-storage-c5d/
Some excerpts from the blog post:
You don’t have to specify a block device mapping in your AMI or during the instance launch; the local storage will show up as one or more devices (/dev/nvme*1 on Linux) after the guest operating system has booted.
Other than the addition of local storage, the C5 and C5d share the same specs.
You can use any AMI that includes drivers for the Elastic Network Adapter (ENA) and NVMe
Each local NVMe device is hardware encrypted using the XTS-AES-256 block cipher and a unique key.
Local NVMe devices have the same lifetime as the instance they are attached to and do not stick around after the instance has been stopped or terminated.
I have just launched the micro instance with Centos AMI image. I have heard that if i stop the instance then my chnages will be lost. So i have read that if i use EBS then chnages will be saved.
But Its not clear to me how to do that. I want that if i stop or restart the machine then my chnages are still there.
DO i need to chnage my hard drive to EBS. How can i do that
My Root Device is comming as EBS with 10GB EBS volume. Is that ok
Yes, the micro instances always use EBS root, so you don't have to do anything special.
If you 'stop' your instance, and later start, it's moving your hard drive to another computer and rebooting.
If you terminate your instance, your EBS drive will be fine as long as "delete EBS drive on termination" is not set on your drive.
When you use other instances, you'll have to verify the AMI type. Not all AMIs are available in all combinations:
EBS vs ephemeral
64 bit vs 32bit
PVM vs HVM (Everything is PVM except the really high-end Compute Cluster)
When you first get started in the cloud, EBS is a big deal. But as you get to be a cloud expert, you'll prefer non-EBS instances. EBS will only be used on a few servers, like your database or your syslog server. Most of your app should be stateless. Your app servers should only store their data in the database, and only write their logs to a central logging server, and only get their code from github or a package repository. So so there is no need to 'backup' your app server boxes, since they can be re-created at any time from external sources using something like Puppet or Chef.
Only your 'stateful' boxes like databases need backing up, and EBS helps there.
I thought I clearly understood the difference between instance-store and EBS backed AMIs.
But http://aws.amazon.com/maintenance-help/ says "if you are running an EBS-backed AMI, you can stop and then restart your instance in order to easily re-launch it. This will cause the loss of any data you have saved on the local instance store of the instance,"
Stop/start does NOT lose the sysvol data, so this confuses me.
I'm assuming that here, by "local instance store", they mean the backing EBS volume (the sysvol), and I'm thinking that they meant to say "terminate" instead of stop. Am I correct?
Terminating an EBS-backed instance will not cause your data to be deleted. You can still access the EBS volume until you delete it (unless you set it to delete when your instance is terminated).
Local instance store refers to hard drive space on the actual physical server that is running your instance. You can see the available instance store by doing sudo fdisk -l. Some images come with some instance store volumes already mounted (see df -h). Otherwise you'll have to mount and format the instance store volumes before you can use them.
Data on an instance store volume is lost when you stop (not terminate) your instance because it is local to a physical server, and your instance might start up on a new server.
Quite simply, EC2 is running your virtual server on some physical server. The root filesystem can either be on a local disk (ephemeral storage) or on network attached storage (EBS). With EBS, they can snapshot it for backups or to make a copy, so EBS is far more flexible, although not as fast as a local disk in the server where your instance is running.
In order to make this all work, when you shutdown an ephemeral server, amazon wipes the disk in order to reallocate it to the next customer. There is no need or reason for them to do that with EBS, since it was not physically attached to that server in the first place.
You might note, that even EBS backed instances (depending on size) come with an allocation of ephemeral storage (2-500gig+) which can be used for swap, logs, or whatever else you want to do with them. The only issue of course is that should the server be shutdown, or should there be a catastrophic disk or hardware error, you'll lose that data. You can still manually back it up, in the same way people have backed up traditional servers over the years.
Making your own AMI from an EBS backed server is trivial now, and can be done easily through the AWS web interface. Making a non-EBS backed AMI is a very complicated task the last time I tried to do it. With that said, there are certain use cases where it makes a lot of sense to consider using purely ephemeral storage. Computation or memory/cache nodes that have no need to persist data will be faster and cost less.