As far as I understand there are special logs (like for EC2 Instances, RDS) which can be enabled to collect metrics and are not enabled automatically. Is there an OVERVIEW somewhere stating for which resources do special AWS Logs exist and how they can be enabled (via CloudFormation)?
If not I would suggest to create such an Overview since AWS DOCUMENTATION doesn’t have it.
Merci A
Amazon CloudWatch collects metrics from AWS services that you use. (Or, more accurately, the services you use send metrics to Amazon CloudWatch.)
The documentation for each service shows the metrics that are collected.
The only logs that go beyond this are special ones sent from within an Amazon EC2 instance. By default, the Amazon EC2 service can only examine your virtual machine ('instance') from outside of the instance. This provides basic metrics for CPU, disk and network. It is not possible to see inside your instance.
However, if you run additional software inside the instance that communicates with the operating system, then the software can send additional metrics to Amazon CloudWatch. This is typically done with the CloudWatch Agent.
See: Collect Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent - Amazon CloudWatch
There are additional Metrics Collected by the CloudWatch Agent that can provide details such as free memory and free disk space because the agent communicates with the Operating System running inside the instance. These metrics are then sent to Amazon CloudWatch as custom metrics.
The CloudWatch Agent can also send application logs from an Amazon EC2 instance to Amazon CloudWatch Logs, which retains the individual log entries and can trigger alarms based on their content.
The CloudWatch Agent must be installed into the Amazon EC2 instance, either via a startup script ('User Data' script) or via AWS Systems Manager.
Similarly, Amazon RDS can send database logs to Amazon CloudWatch Logs. However, I couldn't see an option in Amazon CloudFormation to activate these log exports.
Related
I am kind of confused with the difference between aws system manager and aws cloudwatch?
Could someone help me to get clear with the difference?
Thank you very much.
They have different purposes.
aws system manager in the core of its functionality allows you to manage a fleet of instances as well as on-premise servers. Using the manger you can updated hundreds of instances with just a single command, execute custom scripts on all of them, monitor their patch compliance (i.e. do all your instances of interest have latest updates) and so on.
aws cloudwatch is primary used as a central location for storing variety of logs, from your applications (e.g. lambda execution logs), aws services and so on. It also allows you to monitor performance metrics of your instances (e.g. CPU utilization) as well as other resources. Other functionality is to respond to live events from resources (e.g. execute lambda whenever an instance is terminated)
In short, AWS System Manger is a centralized tool to automate management of AWS resources.
Whereas AWS Cloudwatch is centralized tool for monitoring AWS resource logs.
These short video resources might help -
AWS System Manager -
https://www.youtube.com/watch?v=MK4ZoCs-muo&ab_channel=AmazonWebServices
AWS Cloudwatch -
https://www.youtube.com/watch?v=a4dhoTQCyRA&ab_channel=AmazonWebServices
Looking into adding autoscaling of a portion of our application using AWS simple message queuing which would launch EC2 on-demand or spot instances based on queue backlog.
One question I had, is how do you deal with collecting logs from autoscaled instances? New instances are spun up based on an image, but then they are shut down when complete. Currently, if there is an issue with one of our services, which causes it to crash, we have a system to automatically restart the service, but the logs and core dump files are there to review. If we switch to an autoscaling system, where new instance are spun up, how do you get logs and core dump files when there is a failure? Particularly if the instance is spun down.
Good practice is to ship these logs and aggregate them somewhere else, and there are many services such as DataDog and Rapid7 which will do this for you at a cost.
AWS however provides CloudWatch logs, which gives you a central place to store and view logs. It also allows you then to give users access to logs on the AWS console without them having to ssh onto a server.
Shipping your logs to CloudWatch logs requires the installation of the CloudWatch agent on your server and specifying in the config which logs to ship.
You could install the CloudWatch agent once and create an AMI of that server to use in your autoscaling group, or install and configure the CloudWatch agent in userdata for every time a server is spun up.
All the information you need to get started can be found here:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
Say I set up aws cloudwatch logging on an ec2 instance to centralize logs from various files. If I have auto-scaling and a new machine gets started up due to high traffic, will the new copied machine start sending logs then too? Does logging work with auto-scale?
As long as the CloudWatch Logs agent is installed and configured on the AMI that is used for auto-scaling, the logs for the new instance(s) will be sent to CloudWatch. You can use the Instance ID when configuring the CloudWatch Logs agent to be able to identify which instance originated the event in the logs.
Also, make sure the instances have the necessary IAM role policy to publish the logs to CloudWatch.
I dont have access to the Amazon ec2 instance on which my servers are installed.
My question is how do i get server metrics from ec2 instance into nagios and then monitor it with amazon cloudwatch itself.which means pushing monitoring data back to cloudwatch but this without access id and access key.
You cannot get metrics form a machine you do not have access to. But once you access the machine, you can configure its monitoring agent so that it sends the metrics to CloudWatch.
I have an AWS account, and have several EC2 servers and an ElasticSearch domain set up to take the syslogs from these servers. However, in Cloudwatch and when investigating a specific server instance in the EC2 control panel, I see specific metrics and graphs for things like CPU, memory load, storage use, etc. Is there some way I can pipe this information into my ElasticSearch as well?
Set up Logstash and use this plugin https://github.com/EagerELK/logstash-input-cloudwatch
Or go the other way and use AWS Logs agent to put your syslogs into Cloudwatch and stop using ElasticSearch