I have a requirement where large number of data package files (example:- xml,json,jpeg files) will be send to Amazon S3. How can we get the aggregate count of data transmitted based on some input data points(for example- timestamp). Also how can I also get the information of the read/write time taken by Amazon S3. I have heard about Amazon Cloudwatch for custom metrics but I have no idea how to implement this use case using Amazon Cloudwatch.
Basic approach is described in documentation. Two default metric exists in every bucket, first to get files count and second to get files size. How to get size of files is in doc and to get count of objects in bucket with name: 'ExampleBucket' use aws cli with this command:
aws cloudwatch get-metric-statistics --metric-name NumberOfObjects \
--namespace AWS/S3 --start-time 2020-10-19T00:00:00Z \
--end-time 2020-10-20T00:00:00Z --statistics Maximum --unit Count \
--region us-west-1 \
--dimensions Name=BucketName,Value=ExampleBucket Name=StorageType,Value=AllStorageTypes \
--period 86400 --output json
For more code examples see this question.
Related
I have put some custom metrics in AWS cloudwatch using CLI.
aws cloudwatch put-metric-data --metric-name queuelength --namespace testservice --value 10000 --unit Count --timestamp 2022-11-06T12:05:00.000Z --region us-east-1
aws cloudwatch put-metric-data --metric-name queuelength --namespace testservice --value 20000 --unit Count --timestamp 2022-11-05T12:05:00.000Z --region us-east-1
I am getting this namespace and metric name on AWS console after pushing the metrics but when I select it for plotting graph it doesn't show any line.
This is the query that was formed from query builder.
SELECT SUM(queuelength) FROM SCHEMA(testservice)
Your timestamps are 3 days in the past. It can take some time for backfilled data to show up.
Try publishing a datapoint more recent to the current time, within the last 3 hours. If that works and you see your data, then the backfilled data should show up within an hour or so.
I am sending memory metrics of our EC2 instances to cloudwatch using put-metric-data API call. I am giving a custom namespace for these metrics. But is there a way for these namespaces to be nested so that I will be able to consolidate all my metric data?
For example, if we look into AWS service namespaces EC2, the metrics inside these namespaces will be aggregated ImageID, InstanceType, Per Instance Metrics etc. This can further be filtered using dimensions.
Is there any way to aggregate custom metrics like this?
Adding another dimenstion for the same namespace should work
Example
aws cloudwatch put-metric-data --metric-name memory-usage --dimensions sampleDimension=value --namespace "The-Namespace" --unit Percent
aws cloudwatch put-metric-data --metric-name memory-usage --dimensions sampleDimension2=value --namespace "The-Namespace" --unit Percent
I see no reason that this shouldn't work. Anyways, let us know
i have been working on project, and i want to export metrics data from cloudwatch like CPU Utilization and Network Out data, is there any way to get that data? and convert it to csv?
Export CloudWatch Metrics
You can't do it directly but following is the step by step instructions:
Pre Reqs
AWS CLI (Command Line Interface) from here
jq is a lightweight and flexible command-line JSON processor from here
How to export
Use the following CLI:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1
Valid options for --metric-name depend on the --name-space parameter. For AWS/EC2, the full list can be seen by running the following CLI command:
aws cloudwatch list-metrics --namespace "AWS/EC2"
Valid options for --statistics are:
SampleCount
Average
Sum
Minimum
Maximum
--start-time and --end-time specify the range.
--period The granularity, in seconds, of the returned data points.
--region The region where the CloudWatch metric is being meatured (us-east-1, us-west-2 etc.)
Data output will look something like the following:
{
"Label": "CPUUtilization",
"Datapoints": []
}
To convert it to CSV we will use jq. For this you have two options:
Option 1
Pipe all the aws cli output to jq:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1
| jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | #csv'
Option 2
Export the data to JSON:
aws cloudwatch get-metric-statistics
--namespace AWS/EC2 --metric-name CPUUtilization
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx
--statistics Average
--start-time 2020-03-01T00:00:00
--end-time 2020-03-31T23:59:00
--period 3600
--region us-east-1 >> data.json
Use jq to read the json to csv:
jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | #csv' data.json
Output
Here is how the output will look like:
"2020-03-24T11:00:00Z",0.327868852454245,"Percent"
"2020-03-11T21:00:00Z",0.327868852454245,"Percent"
"2020-03-15T04:00:00Z",0.322580645156596,"Percent"
"2020-03-27T18:00:00Z",0.327868852478101,"Percent"
There is no in-built capability to export Amazon CloudWatch metrics to CSV.
There are API calls available to extract metrics, but you would need to write a program to call the API, receive the metrics and store it in a suitable format.
There are projects available that can assist with this, such as:
mogproject/cloudwatch-dump: Just dump all the CloudWatch metrics.
AWS – CloudWatch – DataExtractor – API | vMan
romgapuz/awsmetric2csv: AWS Metric to CSV is a Python command-line utility to extract CloudWatch metric data from an AWS resources (e.g. EC2, RDS) and save to a CSV file.
If you are looking for these types of tools, make sure they refer to CloudWatch metrics, not CloudWatch Logs (which is something different).
You can also try out official Prometheus cloudwatch_exporter
I have multiple instances. I need to retrieve some metrics from a subset of them (not all) using aws cli. I have tried specifying the InstanceId multiple times as Dimension, but it only considers on of the values. For example the command above, only returns de metric values for instance i-xxxxxx (ignores i-yyyyyyy)
aws cloudwatch get-metric-statistics --namespace AWS/EC2 \
--metric-name CPUUtilization \
--statistics Maximum
--dimensions Name=InstanceId,Value=i-yyyyyyyyyy Name=InstanceId,Value=i-xxxxxxxxx \
--start-time 2018-08-01T00:00:00Z --end-time 2018-08-01T10:00:00Z --period 300
One additional comment: the subset is can be obtained by filtering the list of instances using a tag:
aws ec2 describe-instances --filter Name=tag:app,Values=myapp \
--query 'Reservations[*].Instances[*].InstanceId' --output text
You're using the wrong API: get-metric-statistics is intended to return the time-series data for a single metric, which is identified by all of its dimensions. I suspect that the CLI interprets this field as an associative array, so the second instance ID overwrote the first.
The simplest solution (assuming you're using bash on Linux) is to use a for loop to retrieve each set of metrics:
for instance in i-yyyyyyyyyy i-xxxxxxxxx
do aws cloudwatch get-metric-statistics \
--dimensions "Name=InstanceId,Value=${instance}" \
...
done
I am trying to get the cloudwatch metric data for CPU Utilazation of an EC2 instance i-014448f54423cc0, but i am getting the following output without any metrics data
AWS CLI COMMAND
$ aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2017-03-20T23:18:00 --end-time 2017-03-25T23:18:00 --period 3600 --namespace AWS/EC2 --statistics Average --dimensions Name=i-014448f54423cc0,Value=i-abcdef
OUTPUT
-----------------------------
| GetMetricStatistics |
+--------+------------------+
| Label | CPUUtilization |
+--------+------------------+
As per the following thread on AWS forum: Link
It is said that cloudwatch metrics data can be accessed through API's only if detailed monitoring is enabled, so I tried both scenarios of with & without enabling detailed monitoring but still in both the cases the output is same
You have defined your dimension incorrectly. The dimension you are filtering (literally 'InstanceId') needs to be defined as the Name, and your value (in this case, the instance id's value) as the Value.
Example
Snippet:
Name=InstanceId,Value=i-014448f54423cc0
Full example:
aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2017-03-20T23:18:00 --end-time 2017-03-25T23:18:00 --period 3600 --namespace AWS/EC2 --statistics Average --dimensions Name=InstanceId,Value=i-014448f54423cc0
Discussion
The linked AWS forum discussion isn't relevant to this situation, because detailed monitoring only enables monitoring at 1-minute granularity. Basic monitoring samples data on five-minute intervals. Since you are requesting CPUUtilization for each hour (period=3600 seconds) on a single instance, you will have data available with no need for detailed monitoring.
Further Reading
AWS Documentation - Enable or Disable Detailed Monitoring for Your Instances
AWS Documentation - Amazon EC2 Metrics and Dimensions
AWS Documentation - AWS CLI get-metric-statistics