I am sending memory metrics of our EC2 instances to cloudwatch using put-metric-data API call. I am giving a custom namespace for these metrics. But is there a way for these namespaces to be nested so that I will be able to consolidate all my metric data?
For example, if we look into AWS service namespaces EC2, the metrics inside these namespaces will be aggregated ImageID, InstanceType, Per Instance Metrics etc. This can further be filtered using dimensions.
Is there any way to aggregate custom metrics like this?
Adding another dimenstion for the same namespace should work
Example
aws cloudwatch put-metric-data --metric-name memory-usage --dimensions sampleDimension=value --namespace "The-Namespace" --unit Percent
aws cloudwatch put-metric-data --metric-name memory-usage --dimensions sampleDimension2=value --namespace "The-Namespace" --unit Percent
I see no reason that this shouldn't work. Anyways, let us know
Related
I Have created simple AWS cloudwatch alarm with console. It is triggered by metric "Cognito-By UserPoolAndUserPoolClient-SignInSuccesses". It works as supposed and sends an email via SNS Topic.
I tried to create same alarm with CLI, but it doesn't work. Alarm doesn't get triggered. I compared in console what is difference between those two alarms. When creating alarm with console in metric selection I see metric with information "UserPool" and "UserPoolClient". When using CLI command there isn't parameters to set those two. How could I configure new alarm with CLI so that it is identical to one created in console?
CLI command:
aws cloudwatch put-metric-alarm --alarm-name NewSignupAlarm --alarm-description "New Signup have been done." --alarm-actions arn:aws:sns:eu-west-3:681278469643:NewTestTopic --metric-name SignUpSuccesses --namespace AWS/Cognito --statistic Sum --period 300 --unit Count --evaluation-periods 1 --threshold 0 --comparison-operator GreaterThanThreshold --profile user3
These values are dimensions on the metric. You can specify them with the --dimensions parameter, like this:
--dimensions Name=UserPool,Value=eu-west-3_... Name=UserPoolClient,Value=117...
Make sure the values are exact, I used ... just to illustrate the usage.
See here for more info on all different parameters that you may need to set if you want the alarm be the same as the one created in the console:
https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/put-metric-alarm.html
I have put some custom metrics in AWS cloudwatch using CLI.
aws cloudwatch put-metric-data --metric-name queuelength --namespace testservice --value 10000 --unit Count --timestamp 2022-11-06T12:05:00.000Z --region us-east-1
aws cloudwatch put-metric-data --metric-name queuelength --namespace testservice --value 20000 --unit Count --timestamp 2022-11-05T12:05:00.000Z --region us-east-1
I am getting this namespace and metric name on AWS console after pushing the metrics but when I select it for plotting graph it doesn't show any line.
This is the query that was formed from query builder.
SELECT SUM(queuelength) FROM SCHEMA(testservice)
Your timestamps are 3 days in the past. It can take some time for backfilled data to show up.
Try publishing a datapoint more recent to the current time, within the last 3 hours. If that works and you see your data, then the backfilled data should show up within an hour or so.
i have been working on project, and i want to export metrics data from cloudwatch like CPU Utilization and Network Out data, is there any way to get that data? and convert it to csv?
Export CloudWatch Metrics
You can't do it directly but following is the step by step instructions:
Pre Reqs
AWS CLI (Command Line Interface) from here
jq is a lightweight and flexible command-line JSON processor from here
How to export
Use the following CLI:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1
Valid options for --metric-name depend on the --name-space parameter. For AWS/EC2, the full list can be seen by running the following CLI command:
aws cloudwatch list-metrics --namespace "AWS/EC2"
Valid options for --statistics are:
SampleCount
Average
Sum
Minimum
Maximum
--start-time and --end-time specify the range.
--period The granularity, in seconds, of the returned data points.
--region The region where the CloudWatch metric is being meatured (us-east-1, us-west-2 etc.)
Data output will look something like the following:
{
"Label": "CPUUtilization",
"Datapoints": []
}
To convert it to CSV we will use jq. For this you have two options:
Option 1
Pipe all the aws cli output to jq:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1
| jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | #csv'
Option 2
Export the data to JSON:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1 >> data.json
Use jq to read the json to csv:
jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | #csv' data.json
Output
Here is how the output will look like:
"2020-03-24T11:00:00Z",0.327868852454245,"Percent"
"2020-03-11T21:00:00Z",0.327868852454245,"Percent"
"2020-03-15T04:00:00Z",0.322580645156596,"Percent"
"2020-03-27T18:00:00Z",0.327868852478101,"Percent"
There is no in-built capability to export Amazon CloudWatch metrics to CSV.
There are API calls available to extract metrics, but you would need to write a program to call the API, receive the metrics and store it in a suitable format.
There are projects available that can assist with this, such as:
mogproject/cloudwatch-dump: Just dump all the CloudWatch metrics.
AWS – CloudWatch – DataExtractor – API | vMan
romgapuz/awsmetric2csv: AWS Metric to CSV is a Python command-line utility to extract CloudWatch metric data from an AWS resources (e.g. EC2, RDS) and save to a CSV file.
If you are looking for these types of tools, make sure they refer to CloudWatch metrics, not CloudWatch Logs (which is something different).
You can also try out official Prometheus cloudwatch_exporter
I have multiple instances. I need to retrieve some metrics from a subset of them (not all) using aws cli. I have tried specifying the InstanceId multiple times as Dimension, but it only considers on of the values. For example the command above, only returns de metric values for instance i-xxxxxx (ignores i-yyyyyyy)
aws cloudwatch get-metric-statistics --namespace AWS/EC2 \
--metric-name CPUUtilization \
--statistics Maximum
--dimensions Name=InstanceId,Value=i-yyyyyyyyyy Name=InstanceId,Value=i-xxxxxxxxx \
--start-time 2018-08-01T00:00:00Z --end-time 2018-08-01T10:00:00Z --period 300
One additional comment: the subset is can be obtained by filtering the list of instances using a tag:
aws ec2 describe-instances --filter Name=tag:app,Values=myapp \
--query 'Reservations[*].Instances[*].InstanceId' --output text
You're using the wrong API: get-metric-statistics is intended to return the time-series data for a single metric, which is identified by all of its dimensions. I suspect that the CLI interprets this field as an associative array, so the second instance ID overwrote the first.
The simplest solution (assuming you're using bash on Linux) is to use a for loop to retrieve each set of metrics:
for instance in i-yyyyyyyyyy i-xxxxxxxxx
do aws cloudwatch get-metric-statistics \
--dimensions "Name=InstanceId,Value=${instance}" \
...
done
I am trying to get the cloudwatch metric data for CPU Utilazation of an EC2 instance i-014448f54423cc0, but i am getting the following output without any metrics data
AWS CLI COMMAND
$ aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2017-03-20T23:18:00 --end-time 2017-03-25T23:18:00 --period 3600 --namespace AWS/EC2 --statistics Average --dimensions Name=i-014448f54423cc0,Value=i-abcdef
OUTPUT
-----------------------------
| GetMetricStatistics |
+--------+------------------+
| Label | CPUUtilization |
+--------+------------------+
As per the following thread on AWS forum: Link
It is said that cloudwatch metrics data can be accessed through API's only if detailed monitoring is enabled, so I tried both scenarios of with & without enabling detailed monitoring but still in both the cases the output is same
You have defined your dimension incorrectly. The dimension you are filtering (literally 'InstanceId') needs to be defined as the Name, and your value (in this case, the instance id's value) as the Value.
Example
Snippet:
Name=InstanceId,Value=i-014448f54423cc0
Full example:
aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2017-03-20T23:18:00 --end-time 2017-03-25T23:18:00 --period 3600 --namespace AWS/EC2 --statistics Average --dimensions Name=InstanceId,Value=i-014448f54423cc0
Discussion
The linked AWS forum discussion isn't relevant to this situation, because detailed monitoring only enables monitoring at 1-minute granularity. Basic monitoring samples data on five-minute intervals. Since you are requesting CPUUtilization for each hour (period=3600 seconds) on a single instance, you will have data available with no need for detailed monitoring.
Further Reading
AWS Documentation - Enable or Disable Detailed Monitoring for Your Instances
AWS Documentation - Amazon EC2 Metrics and Dimensions
AWS Documentation - AWS CLI get-metric-statistics