I've created aws cloudwatch alarm based on ASG's group metrics cpuutilization. It sends an email alert email whenever cpuutilization exceeds more than 99% for more than an hour.
Is there a way to execute an event/action that will terminate specific ec2 instances that triggered the alarm? These instances hang and has to be terminated.
I would create an additional alarm that would terminate any instance that reaches 99% cpu for an hour. This is directly supported by CloudWatch.
From Create Alarms to Stop, Terminate, Reboot, or Recover an Instance:
Using Amazon CloudWatch alarm actions, you can create alarms that automatically stop, terminate, reboot, or recover your EC2 instances. You can use the reboot and recover actions to automatically reboot those instances or recover them onto new hardware if a system impairment occurs.
See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/UsingAlarmActions.html
I feel possible solution for this requirement is to write AWS Cli script which would run probably every 15 mins and get list of all EC2 instances running and then terminate if needed. Also, need historical info for ec2's w/c cpu is at 100% for more than 45mins
Related
I am using CloudWatch alarm to stop ec2 instance. For my case I am pushing log information from my ec2 instance to CloudWatch via log grope. And I filter those information via filter with specific pattern that detect error messages due to failed authentication. Unlike standard ways to stop ec2 instance using CPU utilization. I am using a custom metric (figure). Then I am configuring the action to stop the ec2 instance (figure).
But my alarm appears with state "Insufficient data" all the time. can any one help me to solve the problem and stop my ec2 instance once it is in alarm (means that the logs match the pattern of the filter), Thanks a lot!
Sometimes I'm experiencing that an AWS instance is using 100% of the CPU resources - because of a conflict between the hardware and software virtualization. More exact it is the process called "System interrupts" which consumes all CPU resources. This is simply solved by stopping the instance, wait a while and then start the instance again (hereby it will startup on other hardware - just a restart would not work).
My question is now: What is the easiest way to stop the AWS instance and after 1 min start the instance up again automatically when the system is none-responsive? Can this somehow be done within AWS ecosystem itself?
The not so easy way would be to get another system to ping the server and if it doesn't respond - then I could execute a custom script with the needed actions. But I'm hoping an easier solution exists?
Create a CloudWatch Alarm for when the instance reaches a certain percentage in CPU.
Add a trigger for during AlertState to trigger a Lambda. The Lambda would call the AWS CLI to run the StopInstance method, then sleep for 1 minute, then run the StartInstance method.
Example function: https://www.howtoforge.com/aws-lambda-function-to-start-and-stop-ec2-instance/
You can create an Amazon CloudWatch Alarm in the Amazon EC2 management console:
Select the EC2 instance
Go to the Monitoring tab
Click Create Alarm
Configure Take the action: Reboot this instance
This will attempt a graceful restart of the operating system, but will force the restart if necessary.
You can configure the alarm to trigger after a given period of 100% CPU. Just be careful that it doesn't trigger when the instance is simply doing "real" work. You might need to play around with the alarm settings to get it just right.
I just figured out about AWS CloudWatch that would let you terminate your EC2 instance after some time in inactivity. So I created an alarm that would terminate the instance when the CPU usage is less the 1% for 2 hours. Which ultimately ended up putting my instance into an alarm state right away and it prevented me from starting it up to test the feature out.
I then deleted the CloudWatch alarm again in order to be able to launch the EC2 instance gain but even after I deleted the CloudWatch, the state is set to Terminated but the but the Start option from the Actions drop down button is still disabled.
How do I get the instance to start again?
You can't restart a terminated instance. The instance no longer exists. It is just listed as "terminated" in your web console for a little bit so you can see that it was deleted. You have to create a new instance now.
I have created an alarm in Cloudwatch to stop my EC2 instance, if the CPU < 5% for more than five minutes.
And after five minutes I get the email but the instance continues running. The instance is using an EBS volume and nothing in the history indicates a problem.
Can someone please tell me why the alarm I setup is not stopping the instance?
Thanks
According to the documentation, you need to create an IAM role so that AWS can automatically stop the instance on your behalf when the alarm is triggered. Also note that you cannot assign an IAM role to an existing instance.
Update:
You can now attach to or modify an IAM role of your existing instance. Read more here.
I am new to aws part,hope this may help...not sure...
Amazon EBS may not send metric data for an available volume that is not attached to an Amazon EC2 instance, because there is no metric activity to be monitored for that volume. If you have an alarm set for such a metric, you may notice its state change to Insufficient Data. This may simply be an indication that your resource is inactive, and may not necessarily mean that there is a problem.
While working with a g2.2xlarge spot instance, I have tried to set up an alarm that will notify me when the average CPU usage over a two hour period has dropped below 5% and will then automatically stop the instance. Here's a link to a nice article Amazon wrote up on how to use the stop/start instance feature. The AWS alarms seem to allow you to do this however after the trigger goes off I get this reply:
Dear AWS customer,
We are unable to execute the 'Stop' action on Amazon EC2 instance i-e60e21ec that you specified in the Amazon CloudWatch alarm awsec2-i-e60e21ec-Low-CPU-Utilization.
You may want to check the alarm configuration to ensure that it is compatible with your instance configuration. You can also attempt to execute the action manually.
These are some possible reasons for this failure and steps you can try to resolve it:
Incompatible action selected:
Your instance’s configuration may not be compatible with the selected action.
To execute the 'Terminate' action, your instance may have Termination Protection enabled. Disable this feature if you want to terminate your instance. Once you do that, the alarm will execute the action after the next applicable alarm state change.
To execute the 'Stop' action, your instance’s root device type must be an EBS volume. If the root device type is the instance store, select the 'Terminate' action instead. Once you do that, the alarm will execute the action after the next applicable alarm state change.
Temporary service interruption: There may have been an issue with Amazon CloudWatch or Amazon EC2. We have retried the action without success. You can try to execute the action manually, or wait for the next applicable alarm state change.
Sincerely, Amazon Web Services
Stop seems to be an option for the free micro instance but not for these other instances. When I try to change the shutdown behavior to stop in actions it says:
An error occurred while changing the shutdown behavior of this instance.
Modifying 'instanceInitiatedShutdownBehavior is not supported for spot instances.
Is there another way to get around this problem or will we have to wait until Amazon makes this feature available?
Use standard instances instead of spot instances. Spot instances allow you to bid on extra capacity within ec2. However, they may automatically shut down if the spot price exceeds your bid.
Its not really intended for an always on instance.