I'm trying to write a tool which manages Amazon AWS snapshots automatically according to some very simple rules. These snapshots are created on a schedule set up in Amazon Storage Gateway, and show up as you'd expect in the web interface for that tool.
The Storage Gateway API only has operations for snapshots as far as the snapshot schedule goes. EC2 is the API which talks about snapshots. The problem is that if I DescribeSnapshots through that API I see many many hundreds of snapshots, but none of them have volume IDs which match the volume IDs of the snapshots created from Storage Gateway. They're just random public snapshots which I'm not interested in.
So I guess Storage Gateway snapshots are different somehow, but is there a way to use any of Amazon's APIs to list and manipulate them?
EDIT: Interestingly, they do show up in the EC2 web control panel.
Here's a top tip: the snapshots are there, just make sure you're looking for them using the right function. In this case, my novitiate in Clojure is still in effect and I tried to use contains? to search for an item in a sequence. Again. But it doesn't work like that, it looks for keys in collections which means over sequences it wants a number and will tell you if there's an item at that index or not. Even more fun, pass it a sequence and a string and it won't bat an eyelid, it just says false.
Oh and Amazon's not always consistent with capitalisation of volume IDs either, so make sure you lowercase everything before you compare it. That bit's actually relevant to AWS rather than me stubbornly misinterpreting the documentation of a core function.
Related
I have 70 EBS volumes that I need to schedule daily snapshots of. I found this tutorial in the AWS documentation which is helpful, and I already toyed with the AWS CLI to fetch a list of the 70 volume IDs, however, it's not clear to me how I can then feed these many volume IDs back into the Event Rule.
Through the Console, I can only add one Target (Create Snapshot API, Volume ID, and Role) at a time. Looking at the AWS CLI documentation for put-targets, I'm not seeing how to form the command to do this, even if I used some creative find-and-replace work in Notepad to just make a ton of individual commands. Namely, I'm not seeing how I select the Create Snapshot API as the Target, and since each Target has slightly different requirements, I'm not sure then how to supply the volume ID or IAM Role.
What is the most expedient way to get 70 EBS volume IDs added as Create Snapshot API Targets for an EventBridge Rule, or do I just gotta bear down and do them all by hand?
Instead of building such a custom solution, AWS backup is nowadays a much more effective solution for these types of tasks. It also allows you to set a retention period more easily to life cycle your snapshots and create backup policies based on tags.
If you really want to do it with cloudwatch events you need at least as many event rules as you have volumes since the snapshot api is only called once per scheduled rule and the api does not take a list of volumes, just a single volume. So you'll need 70 scheduled rules. Which doesn't scale very well :). Second option is to use a lambda for the event rule target that processes everything but again, it's more work than aws backup.
Is there a quick way to find out which regions have any resources in my account? I'm specifically using the AWS .NET SDK but the answer likely applies to other AWS SDKs and the CLI since they all seem to be just wrappers to the REST API. I can obviously run all the List* methods across all regions but I'm thinking there must be a more optimal way to decide whether to query the entire region or not. Maybe something in billing, but it also needs to be relatively up-to-date, maybe within the last 5 minutes or so. Any ideas?
There is no single way to list all resources in an AWS account or in multiple regions.
Some people say that Resource Groups are a good way to list resources, but I don't think they include "everything" in an account.
AWS Config does an excellent job of keeping track of resources and their history, but it is also limited in the types of resources it tracks.
My favourite way to list resources is to use nccgroup/aws-inventory: Discover resources created in an AWS account. It's a simple HTML/JavaScript file that makes all the 'List' calls for you and shows them in a nicely formatted list.
I need to list the amount of resources that are part of an AWS account in Go, while a resource should be anything that has a price tag on it and can be counted, e.g.
S3 buckets
EC2 instances
RDS instances
ELBs
...
state, region, type and tags are not relevant for this kind of overview, just the raw numbers.
I could of course use the Go SDK and use each corresponding service to get the instances and sum them up, but this would mean lots of boilerplate code and lots of time to create it.
My question: Is there any more generic approach to get the item counts for most services (fine if it doesn't work for all) that can be used with the Go SDK, so I don't have to recode the same query for each service manually?
I was trying to do cost optimisation for my aws account. And i came across the snapshots count. and I saw lots of snapshots over there in my console.
There are some snapshots which were created via any volume. and now the volume is deleted.
How can I describe the snapshots whose volume is not present. ( I know we can use ec2-describe-snapshots, but I need the filters and way to get it.)
Thanks in advance. :)
If I were you I would create a lambda function with this code and have it executed by CloudWatch Events daily, this way you clean up regularly without having to remember! ;)
I am going to reference the node.js API here but the method in the madness is the same for all APIs.
Use ec2 describeSnapshots to get your collection for iteration (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#describeSnapshots-property)
For each snapshot, call describeVolume using the VolumeId in the Snapshot result as the VolumeId. If it doesn't exist anymore you will get an error. (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#describeVolumes-property)
Call deleteSnapshot to delete the snapshot that you no longer need (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#deleteSnapshot-property)
Should be a fun little project! :)
Objective: Using iPhone app, I would like the users store objects in DynamoDB and have Fine-Grained Access Control for the objects using IAM with TVM.
The objects will contain only Strings, no images/file storage -- I'm thinking I won't need an S3?
Question: Since there is no server-side application, do I still need an EC2 Instance? What all suite of AWS services will I have to subscribe to in order to accomplish my objective?
You can use either DynamoDB (or S3), and neither of them would require an EC2 instance - there is no dependency.
If it was me, I'd first see if I could get what I wanted down in S3(because you mentioned it as a possibility), and then go to DynamoDB if I couldn't (i.e. I wanted to be able to run agregation queries across my data set). S3 will be cheaper and depending on what your are doing, may even be faster and would allow you to globally distribute the stored data thru CloudFront easily, which if you have a globally diverse user base may be beneficial.