CloudWatch can be used to perform system monitoring(AWS resources) & application monitoring on AWS resources.
Does CloudWatch rely on SNMP mib(of AWS instance) to monitor the metrics?
No. CloudWatch collects all these metrics from the hypervisor on the physical host that is running your instance. It does not communicate with any processes running inside your instance, SNMP or otherwise.
No definitely not. CloudWatch standard EC2metrics are hypervisor-level and not reported by the instance. If you put the instance in a Security Group that blocks ALL outbound network traffic, the metrics still update. That also explains why CloudWatch metrics don’t include any OS-level stuff like memory utilization, free disk space, etc.
You can install a CloudWatch agent on the instance that pushes additional metrics (like memory utilization and free disk space), but that communicates over HTTPS like all the other AWS APIs. The CloudWatch API is well documented here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/Welcome.html
Furthermore, SNMP is typically a LAN protocol. So if CloudWatch was using SNMP it would have an IP address in your VPC. CloudWatch does not have a VPC endpoint by default.
No, CloudWatch doesn't support SNMP monitoring. But you can use some external tools, which will read SNMP metrics and it will put metrics to the CloudWatch.
For example, you can use customized collectd with the SNMP plugin.
No, CloudWatch can´t get SNMP directly, but you can use other solutions to "collect" traditional SNMP data into CloudWatch.
Follow reference:
https://aws.amazon.com/blogs/mt/snmp-monitoring-using-amazon-cloudwatch-and-elastic-logstash/
Related
I am planning to migrate our app to AWS Fargate and so want to set up logging for the same as well and store all the logs in cloudwatch. I could see we have two options in Fargate - either use default awslogs log driver or use AWS Firelens to gather logs. I read the AWS documentation but unfortunately still not able to figure out which option to use and when precisely. Also, can someone advise on the cost side as well- which option costs how much between using awslogs driver vs aws firelens to send logs to cloudwatch in the same account? [I am looking for easy and efficient and cost effective option]
Is it fair to say in general,we use AWS Firelnes when you want to send logs data to non-AWS tools like elastic stack or datadog etc vs use awslogs driver when sending logs to cloudwatch
Can someone please advise?
Using Fargate launch type and want to use CloudWatch: You have to use awslog driver in your task definition. You can find more information about the CloudWatch pricing here. CloudWatch has a free tier and anything after the free tier cap (metrics, dashboards, alarms, logs, events, etc) has a different pricing calculation. For example, first 10K metrics cost 0.3$ in most regions, but the next 240K will cost 0.1$ while events are priced at 1$ for 1 million
Using Fargate and don't want to use CloudWatch: Use AWS FireLens to push container logs to third party logstorage system. The cost of 3rd party log storage system would come in play here. Data Dog/AppDynamics and others usually offer membership packages (Free/Premium/Enterprise etc). Unlike CloudWatch, each package will give you different capabilities. For example, on the free tier in DataDog you do not have alerts. Also non-AWS native monitoring tools are priced per host/CPU core for a specific amount of hours.
Firelens would also make sense if you want to ship to CloudWatch but want to do upfront filtering versus sending everything to CloudWatch
If you simply want all the log output from your ECS Fargate tasks to go to AWS CloudWatch Logs, then use the awslogs driver. This basically works "out of the box" with no further configuration needed on your part. This is the easiest solution. The only additional cost will be the cost of CloudWatch Logs, detailed in the "Logs" tab here.
If you want to send logs to some other logging service, like Splunk, then use the Firelens driver, and provide a Firelens configuration file that tells Firelens where to send your logs. There is no added cost for using the Firelens driver, but of course there is the added cost of whatever target services you configure Firelens to send your logs to.
I am a bit confused in monitoring of EC2 with and without Cloudwatch agent. As far as I know, Cloudwatch agent does not get installed by default on EC2 linux but some basic system metrics like CPU usage can still be monitored and shown in Cloudwatch.
My questions
If I need to monitor memory usage which is not being monitored by default in EC2 now, should I just setup Cloudwatch agent and memory usage can be published to CloudWatch metrics?
What about I don't setup Cloudwatch agent but just enable detailed monitoring? Can memory usage be monitored by just enabling detailed monitoring without cloudwatch agent?
If I need to monitor memory usage which is not being monitored by
default in EC2 now, should I just setup Cloudwatch agent and memory
usage can be published to CloudWatch metrics?
Yes, this is the correct way to monitor OS level metrics on your EC2 instances.
What about I don't setup Cloudwatch agent but just enable detailed
monitoring? Can memory usage be monitored by just enabling detailed
monitoring without cloudwatch agent?
Detailed monitoring just changes the monitoring interval from 5 minutes to 1 minute, it doesn't enable additional metrics. CloudWatch can't reach into the EC2 operating system to see things like memory usage, so you have to install the CloudWatch agent on the server to monitor memory usage.
I am trying to create a graph for memory usage of an autoscaling group but I discovered that there is no such metric. Although there is Memory usage metric but it is for individual instances. It is useless since instances keep on changing in autoscaling group. I want to know the technical reason why AWS cloudwatch didn't provide it. Moreover I want to know the work around to achieve it.
The metrics that AWS provides can be collected at the hypervisor level. But memory metrics (like disk metrics) is from the OS level. So it is a custom metric that you have to periodically push to CloudWatch.
Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
shows how to push your metrics to CloudWatch. Install the scripts (along with credentials if you are not using IAM role) before creating your AMI and you are set. Each instance in AS will start pushing its memory metric to CloudWatch. Not sure how useful it will be for you.
Just wondering if the AWS cloudwatch runs on the same VPC where i have all my applications are running?
Is there any chance that AWS cloudwatch might go down and we may loose the monitoring capability?
Do we need to have a monitoring mechanism to check the Cloudwatch health?
Thanks
AWS Cloudwatch isn't run on your instances. Its infrastructure is fully managed by Amazon and independent from your VPC. You can see it as a SaaS (Software as a Service).
So you don't have to worry about that. For more informations, please see: https://aws.amazon.com/cloudwatch/
Cloudwatch collects data from the host OS, where your VMs are actually running.
If the physical server had a significant issue both cloudwatch and your VM would go down but in that case the VM would get started automatically on another physical server. In such a case, recovery would be usually quite quickly.
You don't need to check Cloudwatch at all because AWS handles that but you could add alerts for things such as CPU usage on your VMs.
Because Cloudwatch doesn't run on your machines it can't know some things such as memory usage, disk space usage or others so if you need more advanced monitoring capabilities you might consider running something like collectd inside your virtual machine.
Just wondering if the AWS cloudwatch runs on the same VPC where i have all my applications are running?
If you chose to install CloudWatch Agent on your EC2 then only it runs in your EC2 and thus in the VPC your EC2 is provisioned.
CloudWatch service that publishes/maintain logs, metrics, alarms etc is managed by AWS and runs outside your VPC.
CloudWatch has a SLA of 99.9%
https://aws.amazon.com/cloudwatch/sla/
Is there any chance that AWS cloudwatch might go down and we may loose the monitoring capability?
CloudWatch like any other service can have outages and it did have some in the past but I have never seen any data getting lost, only temporarily not being available or slow to retrieve during the outage.
Do we need to have a monitoring mechanism to check the Cloudwatch health?
SLA is already 99.9% for CloudWatch Service so chances of catching a blip is very rare on your own monitoring mechanism.
If you are using CloudWatch Agent then consider checking health of agent to make sure it is in running state (you can use AWS System Manager Run command).
Suppose I have an EC2 instance, which I understood is a VM instance. So if I enable CloudWatch for this EC2 instance, should this monitoring capabilitiy offered by CloudWatch added into my EC2 instance, or it is just running in the hypervisor like XEN?
Thanks.
CloudWatch monitoring is always enabled by default for every EC2 instance at 5-min granularity. What you can enable is detailed monitoring which means you get 1-min observation granularity and aggregate metrics. Default monitoring at 5-min level is free, but detailed monitoring costs money.
Out-of-the-box CloudWatch metrics are measured at hypervisor level and you do not need to do anything to turn them on. See more info on what metrics are available here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch.html
Things like memory utilization and disk space can't be measured at hypervisor level so CloudWatch distributes a simple package with scripts that can be installed on the instance (Linux or Windows.) Those scripts report the data as custom metrics which also costs money. See http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts.html
It is monitored at the hypervisor layer. Aamazon generally will not look into the instance at VM layer so they can't monitor some feature such as memroy usage at the VM.