I am trying to clean up some old AWS accounts to reduce costs, and need to get rid of unnecessary EBS volumes that are not attached to anything.
aws ec2 describe-volumes --volume-ids "blah" "blah"
does return a list of volumes, but doesn't provide info on which VM it might have been attached to.
Before I delete a whole bunch and we lose data that might be needed, I was wondering if there is a good approach to get that info?
Looked into cross referencing with Snapshot Id's and seeing what was there, but almost all have been deleted. Only one volume so far has tags that describe the old VM.
Related
Does anyone know of an AWS CLI command that will list any running instance (run against a particular region) that doesn't have a snapshot available.
The closest command Ive found to try would be something like:
aws ec2 describe-snapshots --owner-ids self --query 'Snapshots[]' --region=us-east-1
I didn't actually get any return on it - just:
-------------------
|DescribeSnapshots|
+-----------------+
This is supposed to name every EC2 snapshot for each instance -- so I would have to subtract these ones from the entire EC2 inventory to reveal EC2 instances without.
Hence - I would like a command that would show running EC2 instances without any snapshots available -- so I can put something in place going forward.
Amazon EBS Snapshots are associated with Amazon EBS Volumes, which are associated with Amazon EC2 instances.
Therefore, you would need to write a program using an AWS SDK (I'd use Python, but there are many available) that would:
Obtain a list of all EBS Snapshots (make sure you use the equivalent to --owner-ids self), in which the return data will include the associated EBS VolumeId
Obtain a list of all EBS Volumes, in which the return data will include Attachments.InstanceId
Obtain a list of all running EC2 instances
Do a bit of looping logic to find Volumes without Snapshots, and then determine which instances are associated to those Volumes.
Note that rather than finding "instances without snapshots" it has to find "instances that have volumes without snapshots".
I don't think there is by default a CLI command that will allow you to do this. You can tag your snapshots with your instance ids for example then can query snapshots by filtering on the tags. Or you will have to use AWS SDK and create a custom script to allow you the get all instances and then check their volume ids if they have snapshots created or not.
I have created some EBS backups over the years, but I can't remember if they were volume or instance backups. Is there some way to tell by looking at one or more field(s) in the list, e.g., at https://ap-southeast-1.console.aws.amazon.com/ec2/v2/home?region=ap-southeast-1#Snapshots:sort=desc:startTime, or in the detailed "description" when I click on one of the snapshots? (the detailed description looking as in the snapshot below, for example) Unfortunately, there isn't a field that says "EBS backup type" that takes a value of "instance" or "volume". As indicated in this stackoverflow question, for example, both types are stored as "EBS Snapshots", so as I understand it then, both will appear mixed together in the same list of EBS snapshots.
Most of the previous questions, e.g., this stackoverflow question, or other pages I've found from searching, have been about the differences between volume and instance backups, and how one might choose one or the other. However, I'm not asking about that, but just if there is any way I can tell what type my previous backups are. Or do I just have to tag the type myself or put it as part of the description string?
UPDATE
From looking at the VolumeID of the snapshot (vol-0565abe0e54ad4adf in the image, for example), I'm guessing that if an existing ec2 instance is using that volume, then that particular snapshot was an instance snapshot? But it could also have been a volume snapshot of that volume?
UPDATE 2
It appears there is some confusion regarding what I'm referring to (from the answers and comments posted so far). I'm not using DLM, but the EC2 console (see image below, and "Snapshot" is the place I navigate to.
Then, when I click on "Create snapshot", I see the following, which shows the options of volume and instance (the first question). This may be a new option, as I don't remember seeing it before.
An EBS snapshot is a backup of a single EBS volume. The EBS snapshot contains all the data stored on the EBS volume at the time the EBS snapshot was created.
An AMI image is a backup of an entire EC2 instance. Associated with an AMI image are EBS snapshots. Those EBS snapshots are the backups of the individual EBS volumes attached to the EC2 instance at the time the AMI image was created.
To get Snapshots associated with still running Volumes, attempt to match their VolumeID with the VolumeID of still running Volumes. Output the SnapshotID of matches.
A snapshot is performed on a single volume, these will always be a backup of the individual volume rather than th complete ec2 instance.
To restore this snapshot, you would restore it to create a new EBS volume that could then be attached to an EC2 instance.
If however your instance is running a single volume you can go one step further. Instead of launching as an EBS volume you can instead create an AMI from the snapshot. This AMI can then be used to launch further instances using the base image taking from the snapshot.
I suspect you are using Data Lifecycle Manager (DLM), not exactly AWS Backup, because you are getting snapshots, AWS Backup work with vault, so you would not see snapshot.
If this is the case, DLM only work with volumes, so you only get backup of your volumes, not instances.
With AWS Backup you can have both, backup of your volumes and/or backup of your instances.
They will be contained inside a vault when backup happens, when necessary you will need to restore it from vault, which will gives you an AMI or a volumes, depending on which kind of backup you did.
Thanks for your update!
I got your point, the instance option there is just a helper to facilitate your life, imagine that you have an instance with 2 volumes and you want to create a snapshot of both volumes, in this case you could go to this screen and create one
snapshot each time (refering volume id on each time), or you can do it once refering the instance id and console will get both volumes for you and create both snapshots.
Doesn't matter which option you choose there, it will just create snapshot from volume, it will not do anything about your instance. If you want you can add a tag in your snapshot to refer to your instance, but it is just a meta-data.
So in your case you are just creating "backups" of your volumes!
If you lose your volume you can restore it, but if you lose your instance you will have to recreate your instance again (with all details) manually.
If you want to create a "backup" from your instance you need to create an image, which will give you and AMI, not a snapshot.
AMI will "backup" your instance details and will create a snapshot from all instance volumes (not ephemeral ones).
I am having trouble fully understanding the differences between EBS Snapshots and AMI's in AWS. I understand how to create them and the terminology (Snapshots are backups of Volumes or the disk attached to an EC2 Instance and AMI is the backup of the full EC2 instance with snapshots at given time). But, I'm not sure what exactly is saved in the snapshot.
Assume I have an EC2 instance with LAMP stack installed on it and some data from the database. I understand the snapshot will save all my data and website files. When I create the new EC2 instance and attach the volume backed by my snapshot, do I then have to install Apache, MySQL, and PHP again? Or is that all saved in the snapshot? I am not worried about having to redo the security settings, instance type, etc for the new EC2 instance, but am worried all the configuration files are lost unless I have an AMI.
I understand how to create them and the terminology (Snapshots are backups of Volumes or the disk attached to an EC2 Instance and AMI is the backup of the full EC2 instance with snapshots at given time)
This is kinda correct, but not exactly. A better understanding of the concepts may help you in general, so here it goes.
A snapshot represents the state of an EBS volume at an exact point in time, taken atomically, that you can use to create other EBS volumes from. Those new EBS volumes, created from the snapshot, will start with the exact same content as the original EBS volume at the time the snapshot was taken. So, kind of a "backup of volumes").
One extremely important thing about Snapshots, though, that you seem to be missing based on the rest of your question, is that Snapshots are immutable. Once create, the contents of a snapshot cannot change, ever.
An AMI is an Image — it's used to create an instance from. It's a bunch of metadata, which includes the type of hypervisor required, accounts allowed to use it, and ultimately it contains a list of EBS Snapshots that should be used to create volumes, and where exactly to attach the volumes created from those snapshots.
So, while "creating an AMI" is indeed a strategy some people use to create a "backup of the full EC2 instance", it's not exactly the same thing.
When I create the new EC2 instance and attach the volume backed by my snapshot [...]
I think you're confusing some concepts here.
When a new EBS volume is created from a snapshot (either by you creating the EBS volume directly and selecting a snapshot, or by using an AMI, which implicitly means that EBS volumes are going to be created from the Snapshots as specified in the metadata that the AMI represents), there's no relationship between that snapshot and the new volume anymore. Any changes made to the volume are completely independent of the original snapshot. Remember: snapshots are immutable.
So, this notion of "the volume backed by my snapshot" may be quite unhelpful: there's no link between them.
Hoping this is clear, let's move on...
When I create the new EC2 instance [...], do I then have to install Apache, MySQL, and PHP again? Or is that all saved in the snapshot?
The software that will come pre-installed in the EC2 instance is defined by the contents of the snapshots used when the EBS volumes attached to the instance were created.
In other words, if you create an EC2 instance from a standard AMI (not one that you create) that doesn't come with the software pre-installed, then the software won't be installed.
If you create an AMI from your instance before you install the software you want, then the AMI (or, more precisely, the snapshots referenced by the AMI) won't have the software, so any instances created from that AMI won't come with the software.
Now, here's what you are probably looking for:
If you (1) create an instance, then (2) install the software, and only then (3) you create an AMI, in this order, then any new instances created from that AMI will come with the software pre-installed.
The easiest way to fully and truly understand this is to remember that (A) Snapshots are immutable, and (B) AMIs are immutable references to Snapshots.
All that said, while creating a custom AMI is indeed a completely valid and popular approach to the problem you are dealing with, there's another approach that's also completely valid and popular, but that is more flexible, and could be worth investigating.
Instead of having to create and manage AMIs (*), what you could do is use a user data script.
Essentially, what it does is it allows you to have a script, that will execute as root, on the first boot of a newly created instance. This script is often used to install software, update packages, download configuration files, etc.
This way, if you need to change versions of software, or change configuration files, etc, you don't need to go through all the complexity of managing AMIs. Instead, you simply update the script.
For more info on user data, check this out: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
(*) "manage AMIs": remember that an AMI is an immutable set of metadata which includes reference to Snapshots, which are also immutable. So if you ever need to change some settings, or have new versions of the software be installed, you will need to create new AMIs. This could become something quite complicated and time-consuming. In fact, there's a ton of tools out there, some by AWS, some by other companies, that try to simplify the process of creating and managing AMIs. So just be aware that, while doable, valid, and popular, it's an approach that can get messy!
I was reviewing my AWS console and found that there are 7 EBS volumes that are not in use and other 5 are in use attached to ec2 instances. I was thinking to delete those not in use volumes but was not sure if they have any required data or not and there is no way to check it unless you attach it with ec2. Earlier the account was managed by other person who is not available so I am not sure if free volumes were ever used/attached with any instance. I will go ahead and delete those if someone can help me to understand following
1.Can I delete not in use volumes ?
2.If they were attached ever with any ec2 instance then will deleting the volume effect my system and is there any change that I will lose my current live data ?
3.If I will lose data then is there any way to take backup of these volumes
I tried to communicate with AWS support but didn't get any help, they just suggest same like attache it ec2 and then check the data etc but never answered if I will lose data or not.
Any help will be appreciated.
Thanks
You can delete not in use volumes, in fact you can only delete those. (if not in used means not attached to an instance).
Only you or someone with access to your system knows if the data you have on your volume is needed. If they are not in use then they unlikely to hold any "current live data".
You can create a snapshot, but it's not the kind of backup you need.
You should really just attach it and see what's in there for yourself. No one else can do that for you or really answer if you will lose data without knowing what is in there.
Is there a way to get instance root volume id in CloudFormation template? Instance was created from AMI image. I want to specify volume id for Cloudwatch Alarm. Fn::GetAtt function can't return it, after reading documentation a lot i found only one way to do it: "custom resource", but it's too complicated.
You should try creating the volume first (http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-ebs-volume.html) and then attaching it to /dev/sda1 (http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-ebs-volumeattachment.html).
Although the doc doesn't rule this out, I'm not completely confident it will work. I assume you will at least need the right snapshot ID from the desired AMI.
If it happens to work, the example under the second link shows how to then reference and get the volume id. And to suggest an alternative, I do believe instance metrics provide aggregate IO for ephemeral disks (see EC2 under http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/ec2-metricscollected.html).
I realize you're not asking for ideas in reengineering your stack... but depending on what this disk IO actually represents, you might realize additional benefits by switching to instance store (free and very fast) or additional EBS (many down-the-road benefits when e.g. mounted for data directories for databases). Both of these would also solve your immediate problem.