I had set up an AWS backup plan which takes backup of our EC2 instances and EBS volumes. But for some reason it is not moving it to cold storage
Here is my backup plan:
Frequency
Daily
At 05:00 AM UTC
Start within
8 hours
Complete within
7 days
Lifecycle
Transition to cold storage after 2 days
Expire after 95 days
For some reason, it is not moving to cold storage
Not sure what I am missing
Any help is much appriciated
EDIT: So i have noticed that the backups are been removed from the vault(moved to cold storage) after 9 days. But I have mentioned in the backup plan to move it to cold storage in 2 days. I assume it takes 9 days because completewithin(7 days) + 2 days. Is this the case?
I don't believe transition to cold storage is supported for ec2/ebs - checkout the matrix (and the faq):
https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html#features-by-resource
Related
Our AWS costs are increasing at a steady rate each month. Looking into it, I found that none of our backups are transitioning to Cold Storage, even though every plan has a transition period set and the retention in cold storage is configured way past the required 90 days.
I have read the documentation and can not see where I am going wrong. Any ideas?
Here is what is in the Vault, every snapshot taken says the same
I was trying to transition AMI when only EFS are supported.
Each lifecycle rule contains an array of transition objects specifying how long in days before a recovery point transitions to cold storage, or is deleted. As of now, the transition to cold storage is ignored for all resources except for Amazon EFS.
From here: https://aws.amazon.com/blogs/storage/automating-backups-and-optimizing-backup-costs-for-amazon-efs-using-aws-backup/
We have a back up rules to keep snapshots of the instance as per below rules:
One snapshot every day for the most recent 7 days and
One snapshot every weekend for the most recent 4 weeks and
One snapshot every month-end for the most recent 12 months.
So in total, there will be 7 + (4-1) + (12-1) = 21 copies required at any point in time.
However, the existing EC2 snapshot lifecycle policy does not seem flexible to retain my back up copies as per above rules. Hence, I was thinking about using Lambda function or step functions. But the lifecycle policy will override the Lambda function, won't it?
Any ideas how this can be achieved from a solution architecture perspective?
Thanks a lot.
In the end, we managed to achieve this by creating 3 separate lifecycle policies.
Create a snapshot once a day, and keep it for 7 days.
Do the same every Sunday, and keep it for 30 days.
Another snapshot every 1st day of the month, and keep it for 365 days.
I am using AWS S3 for backups. I have it setup right now so that after 30 days objects are moved out to glacier for cold storage. Since these are backups what I would like to do is keep the last 30days of back-ups. Then after 30 days the backups taken on the first of each month. Then the after 1 year the backup taken on the first of the year.
Since A backup is made daily I need a way to tell AWS the following for lifecycle management.
If backup is more than 30 days old and not taken on the first of the month delete it.
If backup is more than 1 year old and not taken on the first of the month in the first month of the year delete it.
Right now I have to go in and clean house once a month. The reason I want to do this is that trying to keep every backup from every day get very storage intensive. How would I automate this process?
I am using Immutable deployment.
I have included script as mentioned at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html to monitor ec2 memory at autoscale level.
Now while every deployment for temporary autoscale cloudwatch metric is created.
How do I delete them when temporary autoscale is deleted?
Or how do I delete cloudwatch metrics created while deployment.
This is because my metric list will increase on every deployment.
It is not possible to delete metrics from Amazon CloudWatch. Metrics will eventually rotate out.
Yes, this will increase the list of metrics, but typically AWS users would ask for a specific filter of metrics, so it doesn't matter how many different metrics are actually being stored by CloudWatch.
From the CloudWatch FAQs:
Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution custom metrics.
Data points with a period of 60 seconds (1 minute) are available for 15 days
Data points with a period of 300 seconds (5 minute) are available for 63 days
Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months)
Data points that are initially published with a shorter period are aggregated together for long-term storage. For example, if you collect data using a period of 1 minute, the data remains available for 15 days with 1-minute resolution. After 15 days this data is still available, but is aggregated and is retrievable only with a resolution of 5 minutes. After 63 days, the data is further aggregated and is available with a resolution of 1 hour. If you need availability of metrics longer than these periods, you can use the GetMetricStatistics API to retrieve the datapoints for offline or different storage.
Their API reference says the date start date should be less than 14 days from the current date. I would like to know whether the data older than this is deleted and not available
Metrics used to be kept for 2 weeks, but as #sfgeorge points out, AWS has increased storage times.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch.html
When you use the mon-put-data command,
you must use a date range within the
past two weeks. There is currently no
function to delete data points. Amazon
CloudWatch automatically deletes data
points with a timestamp more than two
weeks old.
As of November 1st, 2016, the retention window for AWS metrics in CloudWatch has expanded from 14 days to 15 months.
Note that the data granularity will be reduced when you widen your range beyond the past 15 days:
One minute data points are available for 15 days.
Five minute data points are available for 63 days.
One hour data points are available for 455 days (15 months).
As found in https://aws.amazon.com/ec2/faqs/ :
Q: Will I lose the metrics data if I disable monitoring for an Amazon EC2 instance?
You can retrieve metrics data for any Amazon EC2 instance up to 2 weeks from the time you started to monitor it. After 2 weeks, metrics data for an Amazon EC2 instance will not be available if monitoring was disabled for that Amazon EC2 instance. If you want to archive metrics beyond 2 weeks you can do so by calling mon-get-stats command from the command line and storing the results in Amazon S3 or Amazon SimpleDB