I'm trying to monitor memory on an EC2 Amazon Linux 2 instance. I'm using these instructions for reference and I'm seeing the error ERROR: Cannot obtain instance id from EC2 meta-data.. I disabled IMDVs1 in my instance which I'm guessing is the way the CloudWatch agent is trying to get my instance id. Does anyone know if there're updated docs on this or a way to fix this? I looked at the AWS script here and I think I could figure out how to have it get this instance ID with IMDSv2 but I'd be surprised if they didn't have a way to do this already. I think I'm missing something though.
I figured it out...
TLDR: Don't use the amazon script to monitor memory, use the CloudWatch Agent
I clicked the first link that showed up when looking at how to monitor memory on EC2 and unfortunately that link is to an old way to monitor memory using a script.
They've updated how to monitor memory and the CloudWatch agent can be configured to do this starting here. There's an automated way to set it up from this documentation and a manual way. This will create a Custom Namespace in the CloudWatch metric view.
Related
I am stuck on one point I have created one EC2 Linux based instance in Aws.
Now I want to send the EC2 metrics data to the managed Elasticsearch domain for monitoring purposes in Kiban, I go through the cloud watch console and check the metric is present of instance but didn't get how to connect with the Elasticsearch domain that I have created.
Can anyone please help me with this situation?
There is no build in mechanism for extraction/streaming of metrics data points in real time. You have to develop a custom solution for that. For example, by having a lambda function which is invoked every minute and which reads data points using get_metric_data. The the lambda would inject the points into your ES.
To invoke a lambda function periodically, e.g. every 1 minute you would have to setup CloudWatch Event rule with schedule Expressions. Lambda function would also need to have permissions granted to interact with CloudWatch metrics.
Welcome to SO :)
An alternative to the solution suggested by Marcin is to install metricbeat on the EC2 Instance and configure the metricbeat config file to send metrics to your Managed AWS ES Domain.
This is pretty simple and you should be able to do this fairly quickly.
I am trying to autoscale gcp instances based on memory metrics but I am unable to find the way how this can be done. I have tried to setup this through "stackdriver monitoring metrics" but no luck. Can someone help here how this can be done.
This is similar problem like posted on google forum but no proper answer here as well.
https://groups.google.com/forum/#!topic/gce-discussion/X6LA0-8mFak
It's required to install the Stackdriver Monitoring Agent by following this documentation.
Once installed, you will get more options to configure your autoscaler from your instance group page
I have multiple instances.
So i want to monitor server and application log for each instances on cloudwatch.
Can i use aws cloudwatch log on multiple instances?
Yes, you can do that.
Simply install the Cloudwatch log agent on each of the instance that you have and you are good to go
For instructions about installing and configuring Cloudwatch log agent click here
You can, but beware that if you pick the same log group / log stream for every instance you'll get comingled messages (which are painful to debug).
It's easy to change the configuration (I recommend using log streams named after the instance ID), but you have to do this when you provision the instance (which means that you actually need a provisioning step, rather than just spinning up an AMI).
I am able to monitor a Windows instance's memory usage using custom metrics in CloudWatch.
I have followed the following blog to achieve that :
http://blog.krishnachaitanya.ch/2016/03/monitor-ec2-memory-usage-using-aws.html
Using that, I am able to monitor only one instance. I am now doing the process in every instance launched.
Can I do it at once for all instances instead of changing .json file and enabling cloud watch integration in every instance?
If the instances are already launched, you have to do it for each instance. Else you can take an AMI of the first instance, then launch other instances from that AMI and you do not have to do it for each instance.
If you have to do it manually, consider something like Ansible to do it for you. There is a bit of learning but not difficult.
BTW, adding custom metrics is straightforward for Linux instances. Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
For Windows instance: Sending Performance Counters to CloudWatch and Logs to CloudWatch Logs Using Amazon EC2 Simple Systems Manager
If your instances have the appropriate instance profile and are running the SSM agent (which they probably are if you launched from an Amazon provided AMI), you can use SSM run command to run arbitrary powershell against an instance or a set of instances (using tags). There is even a Amazon managed SSM document called AWS-ConfigureCloudWatch that is built specifically for this use case.
See http://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html
I am programming a Jersey service on Tomcat via EBS with LoadBalancer. I am finding getting the EC2's S3 catalina files very cumbersome. Currently I need to determine the EC2 instance(s) then work my way to each of the S3 locations, download the files, then I can diagnose.
The snapshot doesn't help due to the amount of requests that come in, it doesn't hold enough info and by the time I get the snapshot, it has "rolled" off the snapshot.
Two questions:
1) Is there an easier approach to logs files via AWS? (Increase time before rotation which I don't believe is supported as of now, scripts, etc)
2) Is there any software or scripts to access all the logs under load balancer? I am basically wanting to say "give me all logs for this EBS" and have it get all logs for that day under all servers for that load balancer (up or down)". The clincher is down. Problem becomes more complex when the load balancer takes down an instance right when the issue occurs.
Thanks!
As an immediate solution to your problem you can follow the approach suggested in this answer. Essentially you can modify the logrotate configuration to rotate for a bigger log size using ebextensions.
Then snapshot logs should work for you.
Let me know if you need more clarifications on this approach.
AWS has released CloudWatch Logs just last week, which enables you to to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files:
You can send your existing system, application, and custom log files to CloudWatch Logs and monitor these logs in near real-time. [...] you can store your logs using highly durable, low-cost storage for later access.
See the introductory blog post Store and Monitor OS & Application Log Files with Amazon CloudWatch for an illustrated walk through, which touches on using Elastic Beanstalk and CloudWatch Logs already - this is further detailed in Using AWS Elastic Beanstalk with Amazon CloudWatch Logs.