My question is simple, but I haven't found much by googling myself.
Is there a way to get metrics such as S3 total disk size in a Bucket?
Is there a way to get metrics on amount of emails sent per month using SES?
I am writing an admin tool where I want to track cloud usage basic metrics.
All this of course with the .Net Core AWS SDK.
Thanks,
Related
I use Amazon S3 to host static resources for a mobile app. Amazon bills the usage based on both the number of requests made to the resources and the volume of data retrieved. I'm looking for a way to view the current usage.
I'm new to AWS, and overwhelmed with the complexity of the AWS Management Console. There's "AWS Cost Explorer", "AWS Cloud Watch", "Usage Reports", etc. Many things can be installed, configured, downloaded and automated. I don't know where to start looking for a solution.
Is there a simple, graphical chart that just shows me the numbers for the past days?
For graphs and reporting that you are trying to view, I would use cost explorer in My Billing Dashboard for this as it gives you an option to bring up items from a service perspective. Click services once the graph comes up and it should give you an in-depth look at what S3 is spending daily. Also, another reporting tool you can look at is the Month-to-date-spend services and click bill details. It should give you a break down of what the costs are for S3.
For your comment you left on your post, AWS S3 is one of the best for static content that won't be changed over time as it offers versioning and other great resources for your content. You can even use CloudFront to help distribute those resources as well for a better user experience
Hope this helps!
You could use the metrics option available in S3 to get the number of s3 reads made per second.
Steps to view the S3 Bucket
Search for S3 bucket in the S3 Bucket Search Box.
Once the S3 bucket comes up in the results box, click on the S3 bucket
S3 bucket opens up, click on Management tab
Once Management tab opens up, click on Metrics button and click on Requests option
Its paid option, once its enabled, you could enable number of get request for the per day/week/month.
A detailed document on how to enable could be AWS document page --> https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-metrics.html
Once you have the number of requests per month, you could use the aws simple calculator to get the cost -->https://calculator.s3.amazonaws.com/index.html
It is straightforward enough to pull data for how much is in a particular S3 bucket. I am trying to connect Geckoboard to monitor S3 data usage, is there a way to use cloudwatch metrics to access how much storage is in use overall on our account? We have many buckets and I just need a total, not a number for each bucket, which is as far as I have gotten.
Thanks
Chris "AWS console is frustratingly unintuitive" H.
It isn't quite the solution I was hoping for, but it is possible to collect this data using the graph metrics in Cloudwatch.
I can't seem to find this info anywhere.
Specifically, I'm looking at metrics like NumberOfMessagesPublished and NumberOfNotificationsDelivered. How far back does CloudWatch retain this data?
2 weeks according to Monitoring Your Instances Using CloudWatch
These statistics are recorded for a period of two weeks, so that you
can access historical information and gain a better perspective on how
your web application or service is performing.
If you want to archive metrics beyond 2 weeks
If you want to archive metrics beyond 2 weeks you can do so by calling
mon-get-stats command from the command line and storing the results in
Amazon S3 or Amazon SimpleDB.
I can't figure out how to make a simple call to display how much I owe Amazon for the month. Just want to display it for this internal dashboard I'm building. Any ideas? I'm not using cloudwatch
You should follow these directions:
http://docs.aws.amazon.com/awsaccountbilling/latest/about/programaccess.html
To tell AWS you would like your monthly bills dropped into an S3 bucket that you supply. Once its turned on, you can use the S3 API to read/download the bill and display it on your dashboard.
There is no 'real-time' billing API that I am aware of (kind of surprising given the technical sophistication of AWS).
Amazon Cloudwatch provides some very useful metrics for monitoring my EC2s, load balancers, elasticache and RDS databases, etc and allows me to set alarms for a whole range of criteria; but is there any way to configure it to monitor my S3s as well? Or are there any other monitoring tools (besides simply enabling logging) that will help me monitor the numbers of POST/GET requests and data volumes for my S3 resources? And to provide alarms for thresholds of activity or increased datastorage?
AWS S3 is a managed storage service. The only metrics available in AWS CloudWatch for S3 are NumberOfObjects and BucketSizeBytes. In order to understand your S3 usage better you need to do some extra work.
I have recently written an AWS Lambda function to do exactly what you ask for and it's available here:
https://github.com/maginetv/s3logs-cloudwatch
It works by parsing S3 Server side log files and aggregates/exports metrics to AWS Cloudwatch (CloudWatch allows you to publish custom metrics).
Example graphs that you will get in AWS CloudWatch after deploying this function on your AWS account are:
RestGetObject_RequestCount
RestPutObject_RequestCount
RestHeadObject_RequestCount
BatchDeleteObject_RequestCount
RestPostMultiObjectDelete_RequestCount
RestGetObject_HTTP_2XX_RequestCount
RestGetObject_HTTP_4XX_RequestCount
RestGetObject_HTTP_5XX_RequestCount
+ many others
Since metrics are exported to CloudWatch, you can easily set up alarms for them as well.
CloudFormation template is included in GitHub repo and you can deploy this function very quickly to gain visibility into your S3 bucket usage.
EDIT 2016-12-10:
In November 2016 AWS has added extra S3 request metrics in CloudWatch that can be enabled when needed. This includes metrics like AllRequests, GetRequests, PutRequests, DeleteRequests, HeadRequests etc. See Monitoring Metrics with Amazon CloudWatch documentation for more details about this feature.
I was also unable to find any way to do this with CloudWatch. This question from April 2012 was answered by Derek#AWS as not having S3 support in CloudWatch. https://forums.aws.amazon.com/message.jspa?messageID=338089
The only thing I could think of would be to import the S3 access logs to a log service (like Splunk). Then create a custom cloud watch metric where you post the data that you parse from the logs. But then you have to filter out the polling of the access logs and…
And while you were at it, you could just create the alarms in Splunk instead of in S3.
If your use case is to simply alert when you are using it too much, you could set up an account billing alert for your S3 usage.
I think this might depend on where you are looking to track the access from. I.e. if you are trying to measure/watch usage of S3 objects from outside http/https requests then Anthony's suggestion if enabling S3 logging and then importing into splunk (or redshift) for analysis might work. You can also watch billing status on requests every day.
If trying to guage usage from within your own applications, there are some AWS SDK cloudwatch metrics:
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/metrics/package-summary.html
and
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/metrics/S3ServiceMetric.html
S3 is a managed service, meaning that you don't need to take action based on system events in order to keep it up and running (as long as you can afford to pay for the service's usage). The spirit of CloudWatch is to help with monitoring services that require you to take action in order to keep them running.
For example, EC2 instances (which you manage yourself) typically need monitoring to alert when they're overloaded or when they're underused or else when they crash; at some point action needs to be taken in order to spin up new instances to scale out, spin down unused instances to scale back in, or reboot instances that have crashed. CloudWatch is meant to help you do the job of managing these resources more effectively.
To enable Request and Data transfer metrics in your bucket you can run the below command. Be aware that these are paid metrics.
aws s3api put-bucket-metrics-configuration \
--bucket YOUR-BUCKET-NAME \
--metrics-configuration Id=EntireBucket
--id EntireBucket
This tutorial describes how to do it in AWS Console with point and click interface.