If you take an AMI from an EC2, and the AMI takes, say, 1 hour to be available; and you choose the option not to skip the reboot.
All the files in the AMI will:
a) reflect their exact condition from the time the EC2 was rebooted? or
b) they may reflect any condition in this 1 hour interval which is what it took for the AMI to be available.
I always considered option a, but I'm not so sure any more, specially after I noticed that when you take an AMI in the console, it gives this message:
"Currently creating AMI ..... Check that the AMI status is 'Available' before deleting the instance or carrying out other actions related to this AMI."
I want to know if it's safe to start applying changes in an EC2 instance after an AMI is requested and the EC2 rebooted, but before the AMI is available.
An Amazon Machine Image (AMI) will contain a copy of the disk at it was at exactly at the point in time when the API call was issued.
Or, if the instance is rebooted as part of the image creation, it will contain a copy of the disk as it was between the time when the operating system shutdown and when the operating system started again.
The time taken for an AMI to become available involves copying disk blocks to the Snapshot used by the AMI. Any disk changes during that time will not be reflected in the AMI. This is possible because the disk is virtual. (It's a bit like a database being able to roll-back due to the use of log files.)
From Create Amazon EBS snapshots - Amazon Elastic Compute Cloud:
Snapshots occur asynchronously; the point-in-time snapshot is created immediately, but the status of the snapshot is pending until the snapshot is complete (when all of the modified blocks have been transferred to Amazon S3), which can take several hours for large initial snapshots or subsequent snapshots where many blocks have changed. While it is completing, an in-progress snapshot is not affected by ongoing reads and writes to the volume... snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued.
Related
I am trying to create a AMI from an instance with a root device of 160GB in size. This root volume is of type io1 with an iops of 1250.
In my AWs account, creating an AMI takes about 5 minutes. This is with data about more than 100GB.
On the customer's AWS account, the same configuration takes over 20+ minutes.
I have tested this with many repetitions and I get almost similar results all time.
Any idea why the AMI creation varies so much between multiple AWS accounts?
An AMI consists of snapshots of Amazon EBS volumes attached to the instance.
Snapshots consist of "differences" from the previous snapshot (including the original AMI that was used to launch the instance).
For example, if you were to launch a new instance from an AMI and then immediately create a new AMI from the instance, very little data would have changed on the disk volume. Thus, the AMI and its underlying snapshot would be very quick to create.
If, over time, a lot of information was added/modified on the disk volume(s), then creating an AMI will take longer because more disk blocks have changed.
Creating Snapshots and AMIs can be made faster by taking more frequent snapshots, since this will copy modified blocks to Amazon S3. Thus, each successive snapshot/AMI will require fewer blocks to be copied.
The speed of a snapshot/AMI is not impacted by the assigned IOPS to a volume. The snapshot process takes place in the back-end, which does not consume the IOPS allocated to a volume.
I upgraded an AWS EC2 machine 10 days ago (from t2-medium to t2-large). Again, I just upgraded today the same AWS EC2 machine (from t2-large to t2-xlarge) and after rebooting I noticed a massive loss of data (10 days of work). In fact, after changing instance type I found out it reverted to the same "state" of the machine just after doing the upgrade 10 days ago. Not less important, I have been doing snapshots periodically (twice a day), and when I try to recover from the last one, each snapshot is an exact copy of the machine after the first upgrade.
This is not the first project I have in AWS, and everytime I did a snapshot, it would always keep the very last copy of the machine. But NOT in this case. So I am really dazzled.
Can you give some hint on what could happen? Is it normal that AWS deletes data (DDBB + folder and files) when changing instance type? Do you think it might have to do with IAM policies?
Thank you
The EC2 user guide has information about the EC2 Storage options. To summarise:
The EC2 instance store volume is erased when the instance is stopped or terminated.
In contrast, EBS (Elastic Block Storage) can be mounted to a running EC2 instance (c.f. a physical hard drive). EBS volumes can be detached from one instance and attached to another instance. Moreover, you can create a snapshot of an EBS volume that is stored in S3 and you can recreate EBS volumes from that snapshot that can be attached to other EC2 instances.
The EC2 user guide provides step by step instructions:
Restoring an Amazon EBS Volume from a Snapshot
Attaching an Amazon EBS Volume to an Instance
Lastly, no, this behaviour is not related to IAM policies.
We a script to create couple of AMI, On successful completion of AMI it deletes the old AMI. As of my understanding AWS only charges for the space we use in S3 for storing snapshot that was created by an AMI.
If I created two AMIs for an instance on different dates(those two AMIs create two different snapshots). will they charge for only new snapshot size? or for the two snapshots?
How AWS charges for this process?
An Amazon Machine Image (AMI) is actually a standard EBS snapshot, with additional metadata.
EBS snapshots are incremental in nature, meaning that only blocks that have been added or changed are copied to Amazon S3.
This means that successive snapshots could incur very little additional cost. Imagine this scenario:
AMI 1 is created from an instance (or, more accurately, from the EBS volumes associated with the instance)
Some data is changed on the EBS volumes
AMI 2 is created from the instance
Since each AMI is a snapshot, and snapshots only copy incremental data to S3, then the additional cost for AMI 2 would only be the new/modified blocks.
If AMI 1 were then deleted, the cost drop would be minimal, since most blocks contained in the AMI 1 snapshot would be kept for the AMI 2 snapshot.
One interesting result of all this (which is merely my suspicion, since I could not find any official statement to this effect) is that, if your AMI is based off an existing volume (eg an AMI from Amazon), then any snapshots/AMIs created of that volume will actually inherit blocks that are part of the original snapshot. Therefore, you (probably) do not pay the full cost of storing that AMI since the snapshot points to blocks already in a snapshot originally created by Amazon. (Don't worry if you didn't understand all that!)
AMIs are stand alone in nature and treated individually no matter whether created out of the same instance on the same day. So they charges of the AMI storage would be calculated 2 times.
For the record the AMI creation involves the snapshotting behind the scenes and AWS performs those snapshots calculating the delta; so the AMI creation process would faster the second time but still they are treated a 2 individual copies.
I have two amazon machine instances running.Both of them are m3.xlarge instances. One of them has the right software and configuration that I want to use.I want to create a snapshot of the EBS volume for that machine and use that as the EBS volue to boot the second machine from. Can I do that and expect it to work without shutting down the first machine.
It is well described in the AWS documentation...
"You can take a snapshot of an attached volume that is in use. However, snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued. This might exclude any data that has been cached by any applications or the operating system. If you can pause any file writes to the volume long enough to take a snapshot, your snapshot should be complete. However, if you can't pause all file writes to the volume, you should unmount the volume from within the instance, issue the snapshot command, and then remount the volume to ensure a consistent and complete snapshot.
I have amazon as well, with 3 different clusters. With one of my clusters after setting up 25 of them I realized there was a small issue in the configuration and had live traffic going to them so I couldn't' shut down.
You can snapshot the first machines volume while it's still running, I had to do this myself. It took a little while, but ultimately it worked out. Please note that amazon cannot guarantee the consistency of the disk when doing this.
I did a snapshot of the entire thing, fixed what needed to be fixed, and spooled up 25 new servers and terminated the other 25 ( easier than modifying volumes, etc ).. But you can create a new volume with the new snapshot, and attach it to an instance and do what needs to be done to get it to boot off that volume without much of a headache.
Being that I went the easy route of spooling up new instances after my snapshot was complete, I can't walk you through on how to get a running instance to boot off a new volume.
I am trying to create a backup mechanism for our server, so that if my system crashes, I should be able to create the whole system by running a single script
After going through Amazon documentation, this is my understanding of creating a backup and restoring
Backup
Create a AMI Image (this can be updated monthly)
Create a snapshot (This can be done using a daily script creating a snapshot)
Restore (A script to)
Create an EBS instance using AMI
Attach the EBS volume to Instance created
Now my Questions are
Is it the best way to take a backup and restore?
Do we actually need to backup 2 things, AMI and EBS volume (using snapshot), Can we just keep snapshots?
I understand this cannot work for a local instance store instance, as there is no snapshot functionality. So how can I create a backup and restore process for local instance store instances?
As I could not find any better alternative, I am sticking with the initial approach.
For EBS
Backup:
Create a AMI Image (this can be updated monthly).
Create a snapshot (This can be done using a daily script creating a snapshot).
Restore (A script to)
Create an EBS instance using AMI.
Attach the EBS volume to Instance created.
For instance store, I am only keeping the application (no database), so no need to keep a backup of that.
EBS Snapshots are an excellent way to create backups.
You can perform frequent Snapshots of your EBS Volumes via scripts. Weekly, Daily, Hourly, or as frequently as your Credit Card will allow. The only limit is around how many simultaneous snapshots you can be doing - when you hit that, the EBS API will start giving back errors until a few of the in-flight operations complete.
Snapshots can also be copied from Region to Region in order to provide backup against a catastrophic event.
When you snapshot an EBS volume, that snapshot is of the entire volume. Even if it was created from an AMI, your snapshot contains everything you need to create a new instance of the volume. You can pretty easily try this yourself.
If your instances are Linux based, there is no need to create an AMI if you're taking snapshots. You can create the AMI on the fly, from the snapshots, when you need to recover. If you got that process automated, it's pretty easy to do.
In Windows there is a limitation not allowing to launch an EC2 instance from a snapshot, so AMIs must be used. There are ways to workaround that limitation: You can check out the this post I wrote in our company's blog:
http://www.n2ws.com/blog/3-ways-ec2-windows-backup-and-recovery.html
I would suggest to use Auto Scaling in addition to EBS snapshots. If Instance is dying because of Hardware failure or it's scheduled for retirement by Amazon, Auto Scaling will start new Instance automatically.
But in this case, you have to setup NAS for your dynamic data. Depending on Server Load, the number of running Instances will be different and all your scaling servers must mount NAS storage which is shared across them.
Your Database should be on separate server or servers as well. Or you might want to use Amazon RDS as it has great auto-backup / Point-In-Time-Restore features, but you have to pay extra for that.
1) Yes.Snapshot is best way to backup and restore EBS volumes.
2) Depends, if you have the root volume as EBS backed AMI, then you can snapshot them as well and improves the manageability
3) Rsync and AMI is the option available for instance store