Their API reference says the date start date should be less than 14 days from the current date. I would like to know whether the data older than this is deleted and not available
Metrics used to be kept for 2 weeks, but as #sfgeorge points out, AWS has increased storage times.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch.html
When you use the mon-put-data command,
you must use a date range within the
past two weeks. There is currently no
function to delete data points. Amazon
CloudWatch automatically deletes data
points with a timestamp more than two
weeks old.
As of November 1st, 2016, the retention window for AWS metrics in CloudWatch has expanded from 14 days to 15 months.
Note that the data granularity will be reduced when you widen your range beyond the past 15 days:
One minute data points are available for 15 days.
Five minute data points are available for 63 days.
One hour data points are available for 455 days (15 months).
As found in https://aws.amazon.com/ec2/faqs/ :
Q: Will I lose the metrics data if I disable monitoring for an Amazon EC2 instance?
You can retrieve metrics data for any Amazon EC2 instance up to 2 weeks from the time you started to monitor it. After 2 weeks, metrics data for an Amazon EC2 instance will not be available if monitoring was disabled for that Amazon EC2 instance. If you want to archive metrics beyond 2 weeks you can do so by calling mon-get-stats command from the command line and storing the results in Amazon S3 or Amazon SimpleDB
Related
I am using getMetricData and the following expression to get the CPU Utilization of all the instances I had:
SEARCH('{AWS/EC2,InstanceId} MetricName="CPUUtilization"', 'Average', 3600)
My instances were terminated around 15 days ago. This was working. But now, I'm not getting data anymore.
Are metrics supposed to being retained for 15 months? I don't see them in the AWS cloudwatch console either.
This is true if the data is still being added to CloudWatch. However the documentation states that if no data is pushed you will lose the data after 2 weeks.
CloudWatch console limits the search of metrics to 2 weeks after a metric is last ingested to ensure that the most up to date instances are shown in your namespace.
We have a back up rules to keep snapshots of the instance as per below rules:
One snapshot every day for the most recent 7 days and
One snapshot every weekend for the most recent 4 weeks and
One snapshot every month-end for the most recent 12 months.
So in total, there will be 7 + (4-1) + (12-1) = 21 copies required at any point in time.
However, the existing EC2 snapshot lifecycle policy does not seem flexible to retain my back up copies as per above rules. Hence, I was thinking about using Lambda function or step functions. But the lifecycle policy will override the Lambda function, won't it?
Any ideas how this can be achieved from a solution architecture perspective?
Thanks a lot.
In the end, we managed to achieve this by creating 3 separate lifecycle policies.
Create a snapshot once a day, and keep it for 7 days.
Do the same every Sunday, and keep it for 30 days.
Another snapshot every 1st day of the month, and keep it for 365 days.
I am using Immutable deployment.
I have included script as mentioned at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html to monitor ec2 memory at autoscale level.
Now while every deployment for temporary autoscale cloudwatch metric is created.
How do I delete them when temporary autoscale is deleted?
Or how do I delete cloudwatch metrics created while deployment.
This is because my metric list will increase on every deployment.
It is not possible to delete metrics from Amazon CloudWatch. Metrics will eventually rotate out.
Yes, this will increase the list of metrics, but typically AWS users would ask for a specific filter of metrics, so it doesn't matter how many different metrics are actually being stored by CloudWatch.
From the CloudWatch FAQs:
Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution custom metrics.
Data points with a period of 60 seconds (1 minute) are available for 15 days
Data points with a period of 300 seconds (5 minute) are available for 63 days
Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months)
Data points that are initially published with a shorter period are aggregated together for long-term storage. For example, if you collect data using a period of 1 minute, the data remains available for 15 days with 1-minute resolution. After 15 days this data is still available, but is aggregated and is retrievable only with a resolution of 5 minutes. After 63 days, the data is further aggregated and is available with a resolution of 1 hour. If you need availability of metrics longer than these periods, you can use the GetMetricStatistics API to retrieve the datapoints for offline or different storage.
I can't seem to find this info anywhere.
Specifically, I'm looking at metrics like NumberOfMessagesPublished and NumberOfNotificationsDelivered. How far back does CloudWatch retain this data?
2 weeks according to Monitoring Your Instances Using CloudWatch
These statistics are recorded for a period of two weeks, so that you
can access historical information and gain a better perspective on how
your web application or service is performing.
If you want to archive metrics beyond 2 weeks
If you want to archive metrics beyond 2 weeks you can do so by calling
mon-get-stats command from the command line and storing the results in
Amazon S3 or Amazon SimpleDB.
I am decommissioning a service in Amazon Web Services DynamoDB. We have various metrics and alarms for our DyDB tables. I am able to delete the alarms using 'aws cloudwatch delete-alarms --alarm-names ...' but I do not see any AWS CLI command to delete metrics.
I know metrics keep data for 2 weeks. Is it possible that the metrics will automatically disappear in 2 weeks when no more data is being stored in it? I see no documentation stating that, unfortunately.
You're correct: there is no API to delete AWS Cloudwatch Metrics.
Just wait two weeks after your last metric has been pushed. It will disappear automatically.
Since November 1, 2016, CloudWatch has extended the retention of metrics (both custom and AWS NameSpace) from previous 14 days to 15 months.
But the CW console limits the search of metrics to 2 weeks after a metric is last ingested (which means if there is a metric in which no datapoints has been pushed for past 14 days then the CW console will not display it but you can still get it using the cli get-metric-statistics or by tweaking the console url to refer the correct resource name and start and end time older than 14 days).
And there is no api to delete the metrics.
https://aws.amazon.com/cloudwatch/faqs/
https://aws.amazon.com/cloudwatch/faqs/
Amazon CloudWatch FAQs
[…]
Q: What is the retention period of all metrics?
CloudWatch retains metric data as follows:
Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution custom metrics.
Data points with a period of 60 seconds (1 minute) are available for 15 days
Data points with a period of 300 seconds (5 minute) are available for 63 days
Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months)
Data points that are initially published with a shorter period are aggregated together for long-term storage.
[…]
Q: Can I delete any metrics?
CloudWatch does not support metric deletion. Metrics expire based on the retention schedules described above.
In support of the above, this is still an issue unfortunately. To read more on the AWS forum, please see:
https://forums.aws.amazon.com/message.jspa?messageID=281904