I am trying to write my own tools for AWS monitoring calling AWS api from console and later build some graphs.
aws cloudwatch get-metric-statistics --metric-name MemoryUtilization --start-time 2022-11-19 --end-time 2022-11-21 --period 3600 --dimensions Name=ClusterName,Value=<mycluster> --dimensions Name=ServiceName,Value=<service-name> --namespace ECS/ContainerInsights --statistics Average
and this is always reporting 0 datapoint, whereas I can see that there are metrics collected in the console.
what is wrong with the command?
issue was the multiple dimensions args are given by just separating then with a space, instead of adding another --dimensions
so this worked.
echo running $metric on $ServiceName
aws cloudwatch get-metric-statistics --metric-name $metric \
--start-time $start_time --end-time $end_time \
--period $period \
--dimensions Name=ClusterName,Value=<my-cluster> Name=ServiceName,Value=$ServiceName \
--namespace $namespace \
--statistics $statistics --output text > $ServiceName-$metric-$now.csv
Related
I have successfully output the cpu and memory allocated for each task in my cluster using the below cli command. I'm also trying to get average actual cpu utilization from cloudwatch all on same query but with little luck. Ideally I will like the field list to have group, cpu, memory, averagecpuUsed.
This gives me the list of clusters:
aws ecs describe-tasks --cluster mycluster --output text --query 'tasks[*] | [].[group,cpu,memory]' --tasks `aws ecs list-tasks --desired-status RUNNING --query taskArns --cluster mycluster --output text`
I'm trying to combine with this:
aws cloudwatch get-metric-statistics --namespace ECS/ContainerInsights --metric-name CpuUtilized --start-time 2023-01-18T00:01:00Z --end-time 2023-01-18T23:59:00Z --period 86400 --dimensions Name=ClusterName, Value=mycluster Name=ServiceName, Value=myservice --statistics Average
I'm configuring an alarm for an instance from the Amazon CLI. for example, to trigger a notification when the cpu is idle for 5min. but I want to set this alarm for a lot of Instances.
With this Bash Script I created one alarm for one instance :
aws cloudwatch put-metric-alarm --alarm-name cpu-mon --alarm-description "Alarm when CPU exceeds 70 percent" --metric-name CPUUtilization --namespace AWS/EC2 --statistic Average --period 300 --threshold 70 --comparison-operator GreaterThanThreshold --dimensions "Name=InstanceId,Value=i-12345678" --evaluation-periods 2 --alarm-actions arn:aws:sns:us-east-1:111122223333:MyTopic --unit Percent
So, I don't see how can I use this script to choose another instances, or eventually loop on that script, in order to choose another instances.
If you have a list of instance IDs you want to create alarms for you could do something like:
#!/bin/bash
instances=(instanceId1 instanceId2 etc)
for i in "${instances[#]}"; do
aws cloudwatch put-metric-alarm \
--alarm-name cpu-mon-${i} \
--alarm-description "Alarm when CPU exceeds 70 percent" \
--metric-name CPUUtilization \
--namespace AWS/EC2 \
--statistic Average \
--period 300 \
--threshold 70 \
--comparison-operator GreaterThanThreshold \
--dimensions "Name=InstanceId,Value=${i}" \
--evaluation-periods 2 \
--alarm-actions arn:aws:sns:us-east-1:111122223333:MyTopic \
--unit Percent
done
You could also initially use the AWS CLI to grab instance IDs based on tags, instance names etc and then use those to create the alarms along the same lines.
I am trying to get get-metric-statistics from Cloudwatch, but I found a little difference from what I have in Cloudwatch and what I receive from the following aws cli command:
aws cloudwatch get-metric-statistics --metric-name EnqueueCount
--start-time 2018-08-13T10:15:00Z --end-time 2018-08-13T10:16:01Z --period 60 --namespace AWS/AmazonMQ --statistics SampleCount --dimensions Name=Broker,Value=BrokerNameName=Queue,Value=QueueName
For the same period of time in Cloudwatch I have different EnqueueCount, but from aws cli command I receive the same value all over again.
Does anyone know why is that?
AWS get-metric-statistics considers end-time as exclusive and start-time as inclusive.
In you case, to get the right count set start-time to 2018-08-13T10:15:00Z and end-time to 2018-08-13T10:17:00Z with period of 60s.
For further reading
https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html
i have been working on project, and i want to export metrics data from cloudwatch like CPU Utilization and Network Out data, is there any way to get that data? and convert it to csv?
Export CloudWatch Metrics
You can't do it directly but following is the step by step instructions:
Pre Reqs
AWS CLI (Command Line Interface) from here
jq is a lightweight and flexible command-line JSON processor from here
How to export
Use the following CLI:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1
Valid options for --metric-name depend on the --name-space parameter. For AWS/EC2, the full list can be seen by running the following CLI command:
aws cloudwatch list-metrics --namespace "AWS/EC2"
Valid options for --statistics are:
SampleCount
Average
Sum
Minimum
Maximum
--start-time and --end-time specify the range.
--period The granularity, in seconds, of the returned data points.
--region The region where the CloudWatch metric is being meatured (us-east-1, us-west-2 etc.)
Data output will look something like the following:
{
"Label": "CPUUtilization",
"Datapoints": []
}
To convert it to CSV we will use jq. For this you have two options:
Option 1
Pipe all the aws cli output to jq:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1
| jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | #csv'
Option 2
Export the data to JSON:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1 >> data.json
Use jq to read the json to csv:
jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | #csv' data.json
Output
Here is how the output will look like:
"2020-03-24T11:00:00Z",0.327868852454245,"Percent"
"2020-03-11T21:00:00Z",0.327868852454245,"Percent"
"2020-03-15T04:00:00Z",0.322580645156596,"Percent"
"2020-03-27T18:00:00Z",0.327868852478101,"Percent"
There is no in-built capability to export Amazon CloudWatch metrics to CSV.
There are API calls available to extract metrics, but you would need to write a program to call the API, receive the metrics and store it in a suitable format.
There are projects available that can assist with this, such as:
mogproject/cloudwatch-dump: Just dump all the CloudWatch metrics.
AWS – CloudWatch – DataExtractor – API | vMan
romgapuz/awsmetric2csv: AWS Metric to CSV is a Python command-line utility to extract CloudWatch metric data from an AWS resources (e.g. EC2, RDS) and save to a CSV file.
If you are looking for these types of tools, make sure they refer to CloudWatch metrics, not CloudWatch Logs (which is something different).
You can also try out official Prometheus cloudwatch_exporter
I am publishing a custom metric into CloudWatch with two dimensions:
aws cloudwatch --region ap-southeast-1 put-metric-data --namespace CustomNS --metric-name ApiReqCount --dimensions ApiName=TestAPI,ApplicationName=App1 --timestamp 2017-03-07T05:00:00.000Z --statistic-value Sum=7,Minimum=1,Maximum=7,SampleCount=1
If I query with all of the dimensions, it gives the datapoints:
aws cloudwatch --region ap-southeast-1 get-metric-statistics --namespace CustomNS --metric-name ApiReqCount --dimensions Name=ApiName,Value=TestAPI Name=ApplicationName,Value=App1 --statistics Sum --start-time 2017-03-05T00:00:00Z --end-time 2017-03-08T12:00:00Z --period 300
However when I query, and no dimensions or partial dimensions are specified, I do not get back datapoints:
aws cloudwatch --region ap-southeast-1 get-metric-statistics --namespace CustomNS --metric-name ApiReqCount --dimensions Name=ApiName,Value=TestAPI --statistics Sum --start-time 2017-03-05T00:00:00Z --end-time 2017-03-08T12:00:00Z --period 300
What I really want is that I need all datapoints returned when no dimensions are specified, I'd like to optionally filter these datapoints via Dimensions.
You can't query for partial dimension or mix dimensions even if they are identical.
From the documentation:
CloudWatch treats each unique combination of dimensions as a separate metric, even if the metrics have the same metric name. You can't retrieve statistics using combinations of dimensions that you did not specifically publish.
See examples on: Amazon CloudWatch Concepts - Dimensions