I was trying to do cost optimisation for my aws account. And i came across the snapshots count. and I saw lots of snapshots over there in my console.
There are some snapshots which were created via any volume. and now the volume is deleted.
How can I describe the snapshots whose volume is not present. ( I know we can use ec2-describe-snapshots, but I need the filters and way to get it.)
Thanks in advance. :)
If I were you I would create a lambda function with this code and have it executed by CloudWatch Events daily, this way you clean up regularly without having to remember! ;)
I am going to reference the node.js API here but the method in the madness is the same for all APIs.
Use ec2 describeSnapshots to get your collection for iteration (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#describeSnapshots-property)
For each snapshot, call describeVolume using the VolumeId in the Snapshot result as the VolumeId. If it doesn't exist anymore you will get an error. (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#describeVolumes-property)
Call deleteSnapshot to delete the snapshot that you no longer need (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#deleteSnapshot-property)
Should be a fun little project! :)
Related
I have 70 EBS volumes that I need to schedule daily snapshots of. I found this tutorial in the AWS documentation which is helpful, and I already toyed with the AWS CLI to fetch a list of the 70 volume IDs, however, it's not clear to me how I can then feed these many volume IDs back into the Event Rule.
Through the Console, I can only add one Target (Create Snapshot API, Volume ID, and Role) at a time. Looking at the AWS CLI documentation for put-targets, I'm not seeing how to form the command to do this, even if I used some creative find-and-replace work in Notepad to just make a ton of individual commands. Namely, I'm not seeing how I select the Create Snapshot API as the Target, and since each Target has slightly different requirements, I'm not sure then how to supply the volume ID or IAM Role.
What is the most expedient way to get 70 EBS volume IDs added as Create Snapshot API Targets for an EventBridge Rule, or do I just gotta bear down and do them all by hand?
Instead of building such a custom solution, AWS backup is nowadays a much more effective solution for these types of tasks. It also allows you to set a retention period more easily to life cycle your snapshots and create backup policies based on tags.
If you really want to do it with cloudwatch events you need at least as many event rules as you have volumes since the snapshot api is only called once per scheduled rule and the api does not take a list of volumes, just a single volume. So you'll need 70 scheduled rules. Which doesn't scale very well :). Second option is to use a lambda for the event rule target that processes everything but again, it's more work than aws backup.
I use AWS RDS as a database for my Spring boot application. I would like to archive earlier than 6 months of data from one specific table. In this context, I have gone through a few articles here but did not get any concrete idea of how to do this. Could anyone please help here?
If you are looking to backup with RDS itself, your options are limited. You can, of course, use automated RDS snapshots, but that won't let you pick a specific table (it will backup the entire database) and can't be set for retention longer than 35 days. Alternatively, you could manually initiate a snapshot, but you can't indicate a retention period. In this case though, you could instead use the AWS published rds-snapshot-tool which will help you automate the snapshot process and let you specify a maximum age of the snapshot. This is likely the easiest way to use RDS for your question. If you only wanted to restore one specific table (and didn't care about have the other tables in the backup), you could restore the snapshot and just immediately DROP the tables you don't care about before you start using the snapshot.
However, if you really care about only backing up one specific table, then RDS itself is out as a possible means for taking the backups on your behalf. I am assuming a mysql database for your spring application, in which case you will need to use the mysqldump tool to grab the database table you are interested in. You will need to manually call that tool from an application and then store the data persistently somewhere (perhaps S3). You will also need to manage the lifecycle on that backup, but if you do use S3, you can set a lifecycle policy to automatically age out and drop old files (backups, in this case).
I am currently using AWS CloudWatch to create backups of a particular EBS volume every 12 hours and would like to delete old snapshots every so often so I don't end up with a crazy amount of backups. Based on the simpler route I'd like to either replace the existing backup with a new one every time rule triggers OR delete backups older than 2 days. Any idea how to accomplish this?
I tried search Target actions in the CloudWatchAWS console for something like "EC2 DeleteSnapshot API call" or similar with no success.
You could create a Lambda function that does this and then invoke that Lambda from a scheduled CloudWatch Event. Beware the maximum execution time of Lamda though. Alternatively you could run an instance and cron a script that does this too. Whichever way you go, you’ll need to script it.
What actually triggers an automatic incremental backup/snapshot for Amazon Redshift? Is it time-based? The site says it "periodically takes snapshots and tracks incremental changes to the cluster since the last snapshot" and I know whenever I modify the cluster(either delete, modify size, or change node type) itself, a snapshot is taken. But what about when a database on the cluster is altered? I have inserted, loaded, deleted many rows but no automatic snapshot is taken. Would I just have to do manual backups then?
I have asked around and looked up online and no one has been able to give me an answer. I am trying to figure out an optimal backing strategy for my workload.
Automated backups are taken every 8 hours or every 5 GB of inserted data, whichever happens first.
Source: I work for AWS Redshift.
I'm trying to write a tool which manages Amazon AWS snapshots automatically according to some very simple rules. These snapshots are created on a schedule set up in Amazon Storage Gateway, and show up as you'd expect in the web interface for that tool.
The Storage Gateway API only has operations for snapshots as far as the snapshot schedule goes. EC2 is the API which talks about snapshots. The problem is that if I DescribeSnapshots through that API I see many many hundreds of snapshots, but none of them have volume IDs which match the volume IDs of the snapshots created from Storage Gateway. They're just random public snapshots which I'm not interested in.
So I guess Storage Gateway snapshots are different somehow, but is there a way to use any of Amazon's APIs to list and manipulate them?
EDIT: Interestingly, they do show up in the EC2 web control panel.
Here's a top tip: the snapshots are there, just make sure you're looking for them using the right function. In this case, my novitiate in Clojure is still in effect and I tried to use contains? to search for an item in a sequence. Again. But it doesn't work like that, it looks for keys in collections which means over sequences it wants a number and will tell you if there's an item at that index or not. Even more fun, pass it a sequence and a string and it won't bat an eyelid, it just says false.
Oh and Amazon's not always consistent with capitalisation of volume IDs either, so make sure you lowercase everything before you compare it. That bit's actually relevant to AWS rather than me stubbornly misinterpreting the documentation of a core function.