I was wondering if it is possible to automatically monitor the usage percentage on a EBS volume in aws (the volume I wish to monitor is attached to a instance). Perhaps this can be done with alarms in cloudwatch? For example, I need to be alerted if the volume usage percentage reaches 95%. Any ideas?
Amazon won't do this for you - from their point of view an EBS volume is just a bunch of blocks
In the past I've done this by writing a script (run via a cronjob) that checked the amount of free space on the volume and posted it to cloudwatch (which was setup to trigger an alarm past a certain threshold).
Amazon also provide such a script
AWS provide a perl script which can be used to create CW alerts/metrics as detailed here
https://serverfault.com/questions/439928/making-alarm-in-disk-space-using-cloudwatch
An update on this question.
All the answers are now outdated and the links posted show deprecated procedures.
The new way to get disk usage in EC2, is to use the unified cloudwatch agent which has pre-built capabilities to extract metrics from EC2, if configured correctly.
You can follow the instruction from these docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
Now you can actually create a cloudwatch alarm and find the EBS Volume and create a metric using freeDiskSpace and have it send notifications to the SNS.
Related
I want to monitor free disk space on an EC2 instance using CloudWatch but can´t find any good tutorials. That instance is an Ubuntu Linux 18.
Any helps?
Start here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html - this will install the cloudwatch agent.
The https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html - this shows the how to configure the agent. You can get it to send various metrics to CloudWatch including Disk usage.
One you have the metrics in CloudWatch you can setup alarms on the metric also in CloudWatch
I'm trying to setup the Cloudwatch Alarm for memory on all instances of an AWS Elastic Beanstalk environment. I've setup capability to get Memory usage on Cloudwatch using the following tutorial:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html
Now I want to setup an alarm that would trigger if the MemoryUtilization of any of these instances go beyond a certain threshold. I can select all and setup alert on each of those separately, but I want to make sure that even if Beanstalk scales up the cluster or swaps an instance, the alert doesn't have to be reconfigured.
Is there a way I can setup alarm for a condition where Instance Name = "env-name" and Metric is MemoryUtilization?
What I understand from your question are the following requirements:
You have multiple metrics and want to use a logical OR condition when configuring an alarm, e.g. (avg metric1 > x || avg metric2 > y) ==> set alarm state to ALARM
You want the alarm to consider new metrics as they become available when new instances are launched by elastic beanstalk during scale out.
You want old metrics to not be considered as soon as elastic beanstalk scales in.
I think this is currently not possible.
There is an ongoing discussion on aws discussion forums [1] which reveals that at least (1) is possible using Metric Math. The Metric Math feature supports max. 10 metrics.
Solution
What you need to do is, to create a single metric which transports the information whether the alarm should be triggered or not ('computed metric'). There are multiple ways to achieve this:
For complex metrics you could write a bash script and run it on an EC2 instance using cron. The script would first query existing metrics using a dimension filter ('list-metrics'), then gather each metric ('get-metric-data'), aggregate it and then push the computed metric data point ('put-metric-data').
If the metric is rather simple, you could try the aggregate option of the AWS put-metric-data script [2]:
option_settings:
"aws:elasticbeanstalk:customoption" :
CloudWatchMetrics : "--mem-util --mem-used --mem-avail --disk-space-util --disk-space-used --disk-space-avail --disk-path=/ --auto-scaling --aggregated"
The documentation for the aggregated option says:
Adds aggregated metrics for instance type, AMI ID, and overall for the region.
References
[1] https://forums.aws.amazon.com/thread.jspa?threadID=94984
[2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html#put-metric-data
In the Elastic Beanstalk console for your environment:
Click the Monitoring link in the left-hand side navigation links.
Underneath the Overview, in the Monitoring section, click the Edit button.
Choose AWSEBAutoScalingGroup for the Resource.
Choose MemoryUtilization under CloudWatch Metric.
Modify Statistic and Description as desired.
Click the Add button, and then click the Save button in the Monitoring section.
Scroll down to find the new panel that was added. Click the bell icon in the upper right hand corner of the panel. This will take you to the settings to set up a new alarm.
If you do not see the MemoryUtilization metric available, verify that you have correctly set up the collection of the memory metrics.
Cloudwatch cannot create alarms in a generic way. There are only 2 ways to accomplish the task.
1) Create a startup script in your AMI. When a new instance is launched, it is responsible for its own Cloudwatch alarms. I used this a long time ago, and the approach is solid. However, running scripts on termination isn't reliable, so you'll have to periodically clean out the old alarms.
2) Use a tool that has decent capabilities (ahem.... not Cloudwatch). I recommend Blue Matador. With them, you don't even have to setup the alarms or thresholds, the machine learning automatically baselines your resources and creates alerts for you.
If you got here and don't know Beanstalk or Cloudwatch well enough to contribute, start here: How to Monitor AWS Elastic Beanstalk with CloudWatch
I am using AWS EC2 for deployment of the dropwizard server code. We recently came across a case where the instance was stopped automatically. On investigation we found that it's whole memory was consumed we saw what consumed the memory mostly were backup files & log files. We removed those & restarted the server it is working well.
To avoid such behavior in future we thought of making use of ClaudWatch alarms provided by the AWS, but the parameters for alarm for EC2 are more of disk throughput, Network related not the diskspace related.
EC2 monitoring.
They suggest having cloud watch agent to be installed on actual instance.
In RDS there is memory alarm type which gets triggered on if memory remains less than some predefined criterion.
For EC2 instance, S3 bucket, Do we have any cloud watch alarm type or any other tool which will trigger a notification on emails when instance is low on memory?
Note: S3 provides object count alarms but couldn't find any specific to low memory.
Update:
Comment by Michael suggests that "There are two problems, here. EC2 instances do not stop when they run out of "memory" (nor storage, which is what you are actually describing). This does not happen. Also, bucket storage is unlimited."
There could be two possibilities instance may have stopped due to other reasons, but when we investigated the added storage to an instance which was 8GB got full. For S3 as he suggested there no limit how much can be stored [ Hence the couldwatch dashboard may be showing object count, not actual memory consume but is there any way that when S3 consumes a certain amount of files ( In MBs or GBs ) on the AWS we may get notifications ]
As you mentioned you need to put monitoring script or CloudWatch agent on EC2 instance to export memory usage or disk usage and attach alert to it - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html
S3 is about storing blob files - you don't need to care about memory usage in that service as that's handled under-hood by AWS so that's why you don't have access to any memory related metric.
Depending on your EC2 instance type you can configure those kinds of alarms.
Memory, Disk-space, CPU utilization relates to OS metrics.
There are various tools too to monitor those for example : Nagios
Or else you can setup your own custom monitoring via email if instances are Unix/AMI
Set up cron jobs and execute the monitoring shell scripts --> disk specific, CPU etc and set up email notification targeting your email addresses.
I am trying to create a graph for memory usage of an autoscaling group but I discovered that there is no such metric. Although there is Memory usage metric but it is for individual instances. It is useless since instances keep on changing in autoscaling group. I want to know the technical reason why AWS cloudwatch didn't provide it. Moreover I want to know the work around to achieve it.
The metrics that AWS provides can be collected at the hypervisor level. But memory metrics (like disk metrics) is from the OS level. So it is a custom metric that you have to periodically push to CloudWatch.
Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
shows how to push your metrics to CloudWatch. Install the scripts (along with credentials if you are not using IAM role) before creating your AMI and you are set. Each instance in AS will start pushing its memory metric to CloudWatch. Not sure how useful it will be for you.
I am interesting in adding CloudWatch for two specific stats on just a few named EBS volumes. The metrics specifically are VolumeReadOps and VolumeWriteOps. These do not show up by default under the volume graphs. My question is as follows:
Is it possible to have those graphs show up by default underneath all volumes graphs tab?
Is it possible to search volumes by tag/name when going to CloudWatch -> EBS? Do I really have to remember volumes by volume-id if I want to see some metrics?
You cannot add a custom metrics to cloud watch. But you can still monitor the VolumeReadIops and VolumeWriteIops using munin. I will give many other information like your app server performance, web server performance etc etc.