Can we stop an AWS windows server EC2 instance of a development environment when there is no activity in it, say after 2 hours of inactivity? I am having trouble identifying whether any user is connected to the server virtually.
I can easily start/stop the EC2 at a fixed time, programmatically, but in order to cut the cost of my server, I am trying to stop the EC2 when it is not being used.
My intent(or use case) is: If no user is using the EC2 till a specified amount of time, it will automatically stop. Developers can restart it as and when needed.
Easiest solution probably would be to set up an Alert with CloudWatch.
Have a read at the documentation, which basically describes your use case perfectly:
You can create an alarm that stops an Amazon EC2 instance when a
certain threshold has been met
A condition could be the average CPU utilisation, e.g. CPU utilisation is below a certain point (which most probably correlates with no logged in users / no developer actually utilising the machine).
This is not a simple task.
The Amazon EC2 service provides a virtual computer that has RAM, CPU and Disk. It can view the amount of activity on the CPU, Network traffic and disk access but it cannot see into the Operating System.
So, the problem becomes how to detect 'inactivity'. This really comes down to the operating system and making some hard decisions. For example, your home computer screen turns off after a defined time of no mouse/keyboard input but the operating system is still doing activity in the background. If the system is running an application such as a web server, and there are no web requests, it is hard to know whether this is 'inactive' because there are no requests, or 'active' because the web server is running.
Bottom line: There is no out-of-the-box feature to do this. You would need to find your own definition of 'inactivity' and then trigger a shutdown in the Operating System.
If you wish to do it via schedule, this might help: Auto-Stop EC2 instances when they finish a task - DEV Community
UPDATE: Lambda's aren't needed anymore, see tpschmidt's answer.
Create a Lambda to turn off the EC2 that will be triggered by a Cloud Watch Alarm when for example the CPU goes under 20% average for an hour. This is fine when you're coding as you will be using more than 20%, and when you have a break for over an hour that's when you want it turned off.
Be sure to set auto save in your IDE's.
Example Python Lambda:
import boto3
region = 'eu-west-3'
instances = ['i-05be5c0c4039881ed']
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
#TODO getInstanceIDFromCloudWatch = event["instanceid"]
ec2.stop_instances(InstanceIds=instances)
print('stopped your instances: ' + str(instances))
Ref: https://www.howtoforge.com/aws-lambda-function-to-start-and-stop-ec2-instance/
In AWS Console:
Goto EC2, select the EC2 instance and copy the Instance ID
Goto Cloud Watch and select Metrics
Under AWS Namespaces click EC2
Paste the Instance ID to find it
Select EC2 > Per-Instance Metrics
Choose the first metric CPU utilisation
Select the second tab called Graphed Metric
Click the Bell icon under Actions
Set a threshold, also this is the hard part, leave the default of Statistic: Average over 1 hour
Set the Condition Lower/Equal and put the value as 20% (you'll need to use the machine more than 1/5th of the hour over 20% CPU otherwise it'll turn off).
Next create an alarm, setup a notification if you like or remove it
Once the Alarm is created
In Cloud Watch select Event > Rules
Add a Rule
Select EC2 as the Service Name and All Event
Click Target and select your Lambda.
When the Alarm goes off the Lambda will turn off the instance ID
You can set up an AWS Cloudwatch alarm that monitors activity. Different parameters like ComparisonOperator, Period, and Threshold can be modified according to how you want to monitor your Ec2 instance.
Then, you can set up an SQS queue and set a Python Lambda function as its target. Within the lambda function, you can use boto3 to turn off the ec2 instance. You can read more details here: https://medium.com/geekculture/automatically-turn-off-ec2-instances-upon-inactivity-31fedd363cad
Terraform setup:
https://medium.com/geekculture/terraform-setup-for-automatically-turning-off-ec2-instances-upon-inactivity-d7f414390800
You are looking for adding stop action to your ec2 instance, this can be easily achieved using CloudWatch alarms.
You can do this from the console using the following steps:
Open the Amazon EC2 console
In the navigation pane, choose Instances.
Select the instance and choose Actions, Monitor and troubleshoot,
Manage CloudWatch alarms.
Alternatively, you can choose the plus sign ( ) in the Alarm status
column.
On the Manage CloudWatch alarms page, do the following:
Choose to Create an alarm.
To receive an email when the alarm is triggered, for Alarm
notification, choose an existing Amazon SNS topic. You first need to
create an Amazon SNS topic using the Amazon SNS console. For more
information, see Using Amazon SNS for application-to-person (A2P)
messaging in the Amazon Simple Notification Service Developer Guide.
Toggle on the Alarm action, and choose Stop.
For Group samples by and Type of data to sample, choose a statistic
and a metric. In this example, choose Average and CPU utilization.
For Alarm When and Percent, specify the metric threshold. In this
example, specify <= and 10 percent.
For the Consecutive period and Period, specify the evaluation period for
the alarm. In this example, specify 1 consecutive period of 5
Minutes.
Amazon CloudWatch automatically creates an alarm name for you. To
change the name, for the Alarm name, enter a new name. Alarm names must
contain only ASCII characters.
Choose to Create.
Note You can adjust the alarm configuration based on your own
requirements before creating the alarm, or you can edit them later.
This includes the metric, threshold, duration, action, and
notification settings. However, after you create an alarm, you
cannot edit its name later.
Check this link from the documentation for terminating the instance using the same way.
You are looking for adding stop action to your ec2 instance, this can be easily achieved using CloudWatch alarms.
Here, I will show how to do that using Terraform:
resource "aws_cloudwatch_metric_alarm" "ec2_cpu" {
alarm_name = "StopTheInstanceAfterInactivity"
metric_name = "CPUUtilization"
comparison_operator = "LessThanOrEqualToThreshold"
statistic = "Average"
threshold = var.threshold
evaluation_periods = var.evaluation_periods # The number of periods over which data is compared to the specified threshold
period = var.period # Evaluation Period (seconds)
namespace = "AWS/EC2"
alarm_description = "This metric monitors ec2 cpu utilization and stop the instance if it is inactive"
actions_enabled = "true"
alarm_actions = ["arn:aws:automate:${var.region}:ec2:stop"]
ok_actions = [] # do nothing
insufficient_data_actions = [] # do nothing
dimensions = {InstanceId = aws_instance.ec2_instance.id}
}
Related
I see that AWS RDS provides a FreeStorageSpace metric for monitoring disk usage. Now I am trying to create a generic pre-emptive alert for all my RDS but setting up an ideal threshold on FreeStorageSpace is not making sense.
For example, 20G might be a good threshold with RDS having total disk space as 100G but might be misleading for a RDS with total disk space of 40G.
So I was wondering if there is a way to get TotalStorageSpace or UsedStorageSpace metric from RDS (directly or indirectly).
Update
Since the fact is established that FreeStorageSpace is the only metric RDS provides related to disk storage, any ideas on if / how we can we build a custom metric for TotalStorageSpace or UsedStorageSpace?
p.s.: Creating separate alarms for each RDS for evaluating disk usage percentage seems such waste of time and resource.
If you enable Enhanced Monitoring, then the RDSOSMetrics log group in Cloudwatch Logs will have detailed JSON log messages which include filesystem statistics. I ended up creating a Cloudwatch Logs metric filter to parse out the usedPercent value from the fileSys attribute for the root filesystem. At least for Postgresql, these detailed logs include both / and /rdsdbdata filesystems; the latter is the one that is of interest in terms of storage space.
You can create a metric filter of the form {$.instanceID = "My_DB_Instance_Name" && $.fileSys[0].mountPoint = "/rdsdbdata"} and a corresponding metric value $.fileSys[0].usedPercent to get the used storage percentage for a given instance. This would then be available as a Log Metric that you could use to trigger an alarm. You probably need to create another metric replacing filesystem[0] with filesystem[1] since ordering is unknown for that array. You'd probably want to create these for each RDS instance you have so you know which one is running out of space, but you question seems to indicate you don't want a per-instance alarm.
I suppose you could exclude the $.instanceID from the metric filter and just get all values written to a single metric. When it reached a threshold and triggered an alarm, you'd need to start checking to see which instance is responsible.
According to the doc FreeStorageSpace is the only StorageSpace metrics you can get.
I can only assume that their logic is that you know what is your total space and having the FreeStorageSpace value you can also calculate how much is used.
First, you can check storage-related info in the monitoring section of AWS RDS.
Now I am trying to create a generic pre-emptive alert for all my RDS
but setting up an ideal threshold on FreeStorageSpace is not making
sense.For example, 20G might be a good threshold with RDS having total
disk space as 100G but might be misleading for a RDS with total disk
space of 40G.
If there is the different storage size then you need to configure multiple alarm based on size. A generic one will not work, as it does not accept percentage.
How can I create CloudWatch alarms to monitor the Amazon RDS free storage space and prevent storage full issues?
Short Description
Create alarms in the CloudWatch console or use the AWS Command Line
Interface (AWS CLI) to create alarms that monitor free storage space.
By creating CloudWatch alarms that notify you when the
FreeStorageSpace metric reaches a defined threshold, you can prevent
storage full issues. This can prevent downtime that occurs when your
RDS DB instance runs out of storage.
Resolution
Open the CloudWatch console, and choose Alarms from the navigation pane.
- Choose Create alarm, and choose Select metric.
From the All metrics tab, choose RDS.
Choose Per-Database Metrics.
Search for the FreeStorageSpace metric.
For the instance that you want to monitor, choose the DB instance Identifier FreeStorageSpace metric.
In the Conditions section, configure the threshold. For example, choose Lower/Equal, and then specify the threshold value.
Note: You must specify the value for the parameter in bytes. For example, 10 GB is 10737418240 bytes.
Fore more details you can check storage-full-rds-cloudwatch-alarm
I have recently come across this. I am setting up CloudWatch alarms for a wide mix of various RDS instances. As you note, creating a static threshold does not make much sense when the allocated storage varies.
I am creating the alarms using Powershell. I have a for-each loop that iterates through the RDS instances I need to create the alarm for. The criteria to raise an alarm is for the disk to only have 10 % free space, or 100 GB, which ever is less. Here is the important part of the script:
$AWSAccountName = "aws-account-name"
$Region = "us-east-1"
$DBInstanceIdentifier = "rds-name"
$DBInstance = Get-RDSDBInstance -DBInstanceIdentifier $DBInstanceIdentifier -Region $Region
$MetricName = "FreeStorageSpace"
$ThresholdPerCent = 0.10 * $DBInstance.AllocatedStorage * 1.074e+9 # 10% free disk space in bytes
$Threshold = ($ThresholdPerCent, 107374182400| Measure-Object -Minimum).Minimum
I specify several other variables that a splatted into an object, and then create the alarm:
# Specify Parameters
$params = #{"AlarmName" = $AlarmName ;
"AlarmDescription" = $AlarmDesc;
"ActionsEnabled" = $true;
"AlarmAction" = $AlarmAction;
"ComparisonOperator" = "LessThanOrEqualToThreshold";
"Dimensions" = $dimensions;
"EvaluationPeriod" = 1;
"MetricName" = $MetricName;
"Namespace" = "AWS/RDS";
"Period" = 300;
"Statistic" = "Minimum";
"DatapointsToAlarm" = 1;
"Threshold" = $Threshold;
"TreatMissingData" = "missing";
"Region" = $Region
}
# Create Rule
Write-CWMetricAlarm #params -Force
If the allocated storage for the instance is increased, you can re-run this and it will update the threshold.
What is the best way to check the EC2 instance uptime and possibly send alerts if uptime for instance is more then N hours? How can it be organized with default AWS tools such as CloudWatch, Lambda ?
Here's another option which can be done just in CloudWatch.
Create an alarm for your EC2 instance with something like CPUUtilization - you will always get a value for this when the instance is running.
Set the alarm to >= 0; this will ensure that whenever the instance is running, it matches.
Set the period and consecutive periods to match the required alert uptime, for example for 24 hours you could set the period to 1 hour and the consecutive periods to 24.
Set an action to send a notification when the alarm is in ALARM state.
Now, when the instance has been on less than the set time, the alarm will be in INSUFFICIENT DATA state. Once it has been on for the uptime, it will go to ALARM state and the notification will be sent.
One option is to use AWS CLI and get the launch time. From that calculate the uptime and send it to Cloudwatch:
aws ec2 describe-instances --instance-ids i-00123458ca3fa2c4f --query 'Reservations[*].Instances[*].LaunchTime' --output text
Output
2016-05-20T19:23:47.000Z
Another option is to periodically run a cronjob script that:
calls uptime -p command
converts the output to hours
sends the result to Cloudwatch with dimension Count
After adding the cronjob:
add a Cloudwatch alarm that sends an alert when this value exceeds a threshold or if there is INSUFFICIENT DATA
INSUFFICIENT DATA means the machine is not up
I would recommend looking into an "AWS" native way of doing this.
If it is basically sending OS level metrics (e.g. Free Memory, Uptime, Disk Usage etc...) to Cloudwatch then this can be achieved by following the guide:
This installs the Cloudwatch Logs Agent on your EC2 instances.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
The great thing about this is you then get the metrics show up in Cloudwatch logs (see attached picture which shows the CW Logs interface in AWS Console.).
Suppose I have an ec2 instance with service /etc/init/my_service.conf with contents
script
exec my_exec
end script
How can I monitor that ec2 instance such that if my_service stopped running I can act on it?
You can publish a custom metric to CloudWatch in the form of a "heart beat".
Have a small script running via cron on your server checking the
process list to see whether my_service is running and if it is, make
a put-metric-data call to CloudWatch.
The metric could be as simple as pushing the number "1" to your custom metric in CloudWatch.
Set up a CloudWatch alarm that triggers if the average for the metric falls below 1
Make the period of the alarm be >= the period that the cron runs e.g. cron runs every 5 minutes, make the alarm alarm if it sees the average is below 1 for two 5 minute periods.
Make sure you also handle the situation in which the metric is not published (e. g. cron fails to run or whole machine dies). you would want to setup an alert in case the metric is missing. (see here: AWS Cloudwatch Heartbeat Alarm)
Be aware that the custom metric will add an additional cost of 50c to your AWS bill (not a big deal for one metric - but the equation changes drastically if you want to push hundred/thousands of metrics - i.e. good to know it's not free as one would expect)
See here for how to publish a custom metric: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/publishingMetrics.html
I am not sure if CloudWatch is the right route for checking if the service is running - it would be easier with Nagios kind of solution.
Nevertheless, you may try the CloudWatch Custom metrics approach. You add Additional lines of code which publishes say an integer 1 to CloudWatch Custom Metrics every 5 mins. Your can then configure CloudWatch alarms to do a SNS Notification / Mail Notification for the conditions like Sample Count or sum deviating your anticipated value.
script
exec my_exec
publish cloudwatch custom metrics value
end script
More Info
Publish Custom Metrics - http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/publishingMetrics.html
Sometimes if there are conditions that prevent the app from starting, say a bad config, the auto scaler will continue to start up instances one after the other.
Anybody know of a good way to alert on this?
Most of our servers receive network traffic so we put a CloudWatch monitor on the NetworkIn metric.
I would suggest configuring the start-up script to Terminate/Shutdown the instance upon failure and sending an alert using CloudWatch custom metrics or any other service like NewRelic.
I don't think that there is a way to alert auto-scaling-group to stop spanning up instances. You could set the max instances limit and have an alert upon reaching this number.
You could alert based on the CloudWatch metric:
Auto Scaling / Group Metrics / GroupTerminatingInstances
See the doc page for more details
I've got an app running on AWS. How do I set up Amazon CloudWatch to notify me when the EC2 instance fails or is no longer responsive?
I went through the CloudWatch screens, and it appears that you can monitor certain statistics, like CPU or disk utilization, but I didn't see a way to monitor an event like "the instance got an http request and took more than X seconds to respond."
Amazon's Route 53 Health Check is the right tool for the job.
Route 53 can monitor the health and performance of your application as well as your web servers and other resources.
You can set up HTTP resource checks in Route 53 that will trigger an e-mail notification if the server is down or responding with an error.
http://eladnava.com/monitoring-http-health-email-alerts-aws/
To monitor an event in CloudWatch you create an Alarm, which monitors a metric against a given threshold.
When creating an alarm you can add an "action" for sending a notification. AWS handles notifications through SNS (Simple Notification Service). You can subscribe to a notification topic and then you'll receive an email for you alarm.
For EC2 metrics like CPU or disk utilization this is the guide from the AWS docs: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/US_AlarmAtThresholdEC2.html
As answered already, use an ELB to monitor HTTP.
This is the list of available metrics for ELB:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_MonitoringLoadBalancerWithCW.html#available_metrics
To answer your specific question, for monitoring X seconds for the http response, you would set up an alarm to monitor the ELB "Latency".
CloudWatch monitoring is just like you have discovered. You will be able to infer that one of your instances is frozen by taking a look at the metrics, but CloudWatch won't e.g. send you an email when your app is down or too slow, for example.
If you are looking for some sort of notification when your app or instance is down, I suggest you to use a monitoring service. Pingdom is a good option. You can also set up a new instance on AWS and install a monitoring tool, like Nagios, which would be my preferred option.
Good practices that are always worth, in the long road: using load balancing (Amazon ELB), more than one instance running your app, Autoscaling (when an instance is down, Amazon will automatically start a new one and maintain your SLA), and custom monitoring.
My team has used a custom monitoring script for a long time, and we always knew of failures as soon as they occurred. Basically, if we had two nodes running our app, node 1 sent HTTP requests to node 2 and node 2 to 1. If any request took more than expected, or returned an unexpected HTTP status or response body, the script sent an email to the system admins. Nowadays, we rely on more robust approaches, like Nagios, which can even monitor operating system stuff (threads, etc), application servers (connection pools health, etc) and so on. It's worth every cent invested in setting it up.
CloudWatch recently added "status check" metrics that will answer one of your questions on whether an instance is down or not. It will not do a request to your Web server but rather a system check. As previous answer suggest, use ELB for HTTP health checks.
You could always have another instance for tools/testing, that instance would try the http request based on a schedule and measure the response time, then you could publish that response time with CloudWatch and set an alarm when it goes over a certain threshold.
You could even do that from the instance itself.
As Kurst Ursan mentioned above, using "Status Check" metrics is the way to go. In some cases you won't be able to browse that metrics (i.e if you;re using AWS OpsWorks), so you're going to have to report that custom metric on your own. However, you can set up an alarm built on a metric that always matches (in an OK sate) and have the alarm trigger when the state changes to "INSUFFICIENT DATA" state, this technically means CloudWatch can't tell whether the state is OK or ALARM because it can't reach your instance, AKA your instance is offline.
There are a bunch of ways to get instance health info. Here are a couple.
Watch for instance status checks and EC2 events (planned downtime) in the EC2 API. You can poll those and send to Cloudwatch to create an alarm.
Create a simple daemon on the server which writes to DynamoDB every second (has better granularity than Cloudwatch). Have a second process query the heartbeats and alert when missing.
Put all instances in a load balancer with a dummy port open that that gives a TCP response. Setup TCP health checks on the ELB, and alert on unhealthy instances.
Unless you use a product like Blue Matador (automatically notifies you of production issues), it's actually quite heinous to set something like this up - let alone maintain it. That said, if you're going down the road, and want some help getting started using Cloudwatch (terminology, alerts, logs, etc), start with this blog: How to Monitor Amazon EC2 with CloudWatch
You can use CloudWatch Event Rule to Monitor whenever any EC2 instance goes down. You can create an Event rule from CloudWatch console as following :
In the CLoudWatch Console choose Events -> rule
For Event Pattern, In service Name Choose EC2
For Event Type, Choose EC2 Instance State-change Notification
For Specific States, Choose Stopped
In targets Choose any previously created SNS topic for sending a notification!
Source : Create a Rule - https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatch-Events-Input-Transformer-Tutorial.html#input-transformer-create-rule
This is not exactly a CloudWatch alarm, however this serves the purpose of monitoring/notification.