I am trying to build a tool which tracks unused AWS resources. I have figured out how to obtain a list of all resources on my AWS account. Now, for each of these resources, I need to find out the last time it was used.
Is there some AWS API/CLI using which I can find the required info?
I did some search, but the most common suggestion I get is to use tools like Janitor Monkey and Cloud Custodian. I am trying to build a tool of my own, and do not want to employ some other software.
you can use AWS Cloudwatch for monitoring.
You can monitor metrics like CPU utilization and network latency.
Also, you can try using the logs but that will depend on the application that is running in your machine.
Related
In Azure for example, I created a few bash scripts give me things like average daily CPU utilization over whatever time period I want for any/all VMs using their command line tool.
I can't seem to figure out how to do this in Google cloud except by manually using the console (automatically generated daily usage reports don't seem to give me any CPU info either), so far numerous searches have told me that using the monitoring function in the google cloud console is basically the only way I can do this, as the cli "gcloud" will only report quotas back which isn't really what I'm after here. I haven't bothered with the ops agent install yet, as my understanding is that this is just for adding additional metrics (to the console) and not functionality to the google cloud cli. Up to this point I've only ever managed Azure and some AWS, so maybe what I'm trying to do isn't even possible in Google cloud?
Monitoring (formerly Stackdriver) does seem to be neglected by the CLI (gcloud).
There is a gcloud monitoring "group" but even the gcloud alpha monitoring and gcloud beta monitoring commands are limited (and don't include e.g. metrics).
That said, gcloud implements Google's underlying (service) APIs and, for those (increasingly fewer) cases where the CLI does not yet implement an API and its methods, you can use APIs Explorer to find the underlying e.g. Monitoring service directly.
Metrics can be access through a query over the underlying time-series data, e.g. projects.timeseries.query. The interface provides a form that enables you to invoke service methods from the browser too.
You could then use e.g. curl to construct the queries you need for your bash scripts and other tools (e.g. jq) to post-process the data.
Alternatively, and if you want a more programmatic experience with good error-handling and control over the output formatting, you can use any of the language-specific SDKs (client libraries).
I'd be surprised if someone hasn't written a CLI tool to complement gcloud for monitoring as it's a reasonable need.
It may be worth submitting a feature request on Google's Issue Tracker. I'm unsure whether it would best be placed under Cloud CLI or Monitoring. Perhaps try Monitoring.
I am kind of confused with the difference between aws system manager and aws cloudwatch?
Could someone help me to get clear with the difference?
Thank you very much.
They have different purposes.
aws system manager in the core of its functionality allows you to manage a fleet of instances as well as on-premise servers. Using the manger you can updated hundreds of instances with just a single command, execute custom scripts on all of them, monitor their patch compliance (i.e. do all your instances of interest have latest updates) and so on.
aws cloudwatch is primary used as a central location for storing variety of logs, from your applications (e.g. lambda execution logs), aws services and so on. It also allows you to monitor performance metrics of your instances (e.g. CPU utilization) as well as other resources. Other functionality is to respond to live events from resources (e.g. execute lambda whenever an instance is terminated)
In short, AWS System Manger is a centralized tool to automate management of AWS resources.
Whereas AWS Cloudwatch is centralized tool for monitoring AWS resource logs.
These short video resources might help -
AWS System Manager -
https://www.youtube.com/watch?v=MK4ZoCs-muo&ab_channel=AmazonWebServices
AWS Cloudwatch -
https://www.youtube.com/watch?v=a4dhoTQCyRA&ab_channel=AmazonWebServices
We have couple of application running on AWS. Currently we are redirecting all our logs to single bucket. However for ease of access to users, I am thinking to install ELK Stack on EC2 instance.
Would want to check if there is alternate way available where I don't have to maintain this stack.
Scaling won't be an issue, as this is only for logs generated through application running on AWS, so not ingestion or processing is required. mostly log4j logs.
You can go for either the managed Elasticsearch available in AWS or setup your own in an EC2 instance
It usually comes down to the price involved and the amount of time you have in hand in setting up and maintaining your own setup
With your own setup, you can do a lot more configurations than that provided by the managed service and also helps in reducing the cost
You can find more info on this blog
I tried searching for this topic on Google and after many failed attempts I decided to post this as questions here.
What I want to achieve: Monitoring my aws limits using Nagios.
As I have understood aws cli can be used to get the limits of only few aws services, for more in depth cost management and service limit management one has to opt for trusted advisor. Unfortunately it's quite expensive.
So I was wondering if there's a much simpler way with Nagios in which I could get notified if any of the aws services for my account is hitting a limit?
What kind of service limit notification strategy is used by organizations(That can't afford to buy a subscription of trusted advisor) that use Aws?
You're right: only few services can show their limit (and current usage) through CLI or API. I don't like it either :) We have three options here:
Create a parser that grabs information from AWS Console (there is an example code here: https://forrestbrazeal.com/2015/07/20/adventures-in-aws-automating-service-limit-checks/).
Buy Trusted Advisor (btw, you can get a Trusted Advisor report with API call).
Try using awslimitchecker. Cause someone already tried to solve this problem.
https://awslimitchecker.readthedocs.io/en/latest/
I have deployed a LAMP stack application on AWS. I need to monitor that using CloudWatch.
Can someone guide me on how to use the CloudWatch API for GetMetrics for CPU utilization? The AWS documentation is very scarce.
I see that the putmetrics call will let me create my own metrics.
My requirement is that I need to display those metric results in a mobile app.
My app monitors a project deployed on AWS. The alerts and metrics that come in must stream into the app.
I don't want just the metrics data in the AWS console,
I want it viewable in my mobile app. The app is developed in MEAN stack.
I must also add that the app is deployed on AWS and the application that is
being monitored is also in there(its a LAMP stack). I have managed to set 2 endpoints(HTTP and DB) and I have written
simple scripts in Javascript to monitor them. But ideally they should happen via Cloudwatch.
Providing a piece of code that replicates the issue that you are seeing normally allows who sees the question to help you better than guessing what you're doing.
Are you using an SDK to do this? What language/version?
here are links to the API docs:
http://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html
http://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html
The pattern is to list the metrics and after that use the result and feed it into getmetricsstatistics.
In your specific case, googling the issue a bit before might answer the question before you ask it on SO. For example:
https://forums.aws.amazon.com/thread.jspa?messageID=295740
This can happen when you are hitting the wrong endpoint. Check if you are hitting endpoint of the right AWS service.
For example, trying to hit DynamoDB's endpoint when you want to access CloudWatch APIs.