I just figured out about AWS CloudWatch that would let you terminate your EC2 instance after some time in inactivity. So I created an alarm that would terminate the instance when the CPU usage is less the 1% for 2 hours. Which ultimately ended up putting my instance into an alarm state right away and it prevented me from starting it up to test the feature out.
I then deleted the CloudWatch alarm again in order to be able to launch the EC2 instance gain but even after I deleted the CloudWatch, the state is set to Terminated but the but the Start option from the Actions drop down button is still disabled.
How do I get the instance to start again?
You can't restart a terminated instance. The instance no longer exists. It is just listed as "terminated" in your web console for a little bit so you can see that it was deleted. You have to create a new instance now.
Related
I would like to use AWS EC2 spot requests for machine learning that I do in spots.
If you don't configure any settings for spot requests, they will be set to "fleet" and an EC2 instance will be created that cannot be paused (it can only be terminated).
If you know how to create an instance with a spot request that can be paused, please let me know.
Spot instances launched in a fleet can’t be paused or stopped. To stop a Spot instance, it must be launched by a Spot instance request for a single instance, you can create such requests using Launch Instance wizard, you may follow steps at this link. Make sure to select "Persistent request" in the wizard.
This request will launch a Spot instance that can be Stopped but still can’t be Paused. Hibernate/Pause option is only available as a behavior at the event of interruption and can't be requested by the user.
I've created aws cloudwatch alarm based on ASG's group metrics cpuutilization. It sends an email alert email whenever cpuutilization exceeds more than 99% for more than an hour.
Is there a way to execute an event/action that will terminate specific ec2 instances that triggered the alarm? These instances hang and has to be terminated.
I would create an additional alarm that would terminate any instance that reaches 99% cpu for an hour. This is directly supported by CloudWatch.
From Create Alarms to Stop, Terminate, Reboot, or Recover an Instance:
Using Amazon CloudWatch alarm actions, you can create alarms that automatically stop, terminate, reboot, or recover your EC2 instances. You can use the reboot and recover actions to automatically reboot those instances or recover them onto new hardware if a system impairment occurs.
See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/UsingAlarmActions.html
I feel possible solution for this requirement is to write AWS Cli script which would run probably every 15 mins and get list of all EC2 instances running and then terminate if needed. Also, need historical info for ec2's w/c cpu is at 100% for more than 45mins
I'm having an issue with AWS boxes (EC2) where I terminate the box and it re-spawns. To give context, there is no autoscaling group. Anywhere I can search for some config that might be triggering the launch?
I would make sure you don't have a persistent spot request active in your account, and also check to see if you perhaps installed the AWS Instance scheduler - either or both of those could be starting instances on your behalf - (double check the autoscaling group, that is the most obvious reason though)
If you terminate a running Spot Instance that was launched by a
persistent Spot request, the Spot request returns to the open state so
that a new Spot Instance can be launched. To cancel a persistent Spot
request and terminate its Spot Instances, you must cancel the Spot
request first and then terminate the Spot Instances. Otherwise, the
persistent Spot request can launch a new instance.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
https://aws.amazon.com/solutions/instance-scheduler/
So I found out the culprit, maybe it can help more people. Apparently, there is a service from AWS called OpsWorks that automates things like Cheff of Puppet, which my company had configured some time ago. That would be checking for instances running and triggering re-provisioning when it didn't see the instance running. OpsWorks is here
I wrote a terraform script to spin up an EC2 spot instance, and every time I terminate it, it automatically spins up a new instance. I don't understand this behavior. Why is this happening and how do I stop it?
When you requested your spot instance, you probably didn't specify the spot-type as 'one-time' so the request persists beyond instance termination. Or in other words, when your instance is terminated the request begins anew and launches a new instance based on the same bid price, launch specification, etc.
From the terraform documentation:
spot_type - (Optional; Default: "persistent") If set to "one-time",
after the instance is terminated, the spot request will be closed.
Also, Terraform can't manage one-time spot requests, just launch them.
From the AWS documentation:
This will continue until you cancel the spot instance request. You can do this via the AWS Console, CLI (cancel-spot-instance-requests), or via terraform by destroying the resource.
Further Reading
AWS Documentation - Cancelling a Spot Instance Request
While working with a g2.2xlarge spot instance, I have tried to set up an alarm that will notify me when the average CPU usage over a two hour period has dropped below 5% and will then automatically stop the instance. Here's a link to a nice article Amazon wrote up on how to use the stop/start instance feature. The AWS alarms seem to allow you to do this however after the trigger goes off I get this reply:
Dear AWS customer,
We are unable to execute the 'Stop' action on Amazon EC2 instance i-e60e21ec that you specified in the Amazon CloudWatch alarm awsec2-i-e60e21ec-Low-CPU-Utilization.
You may want to check the alarm configuration to ensure that it is compatible with your instance configuration. You can also attempt to execute the action manually.
These are some possible reasons for this failure and steps you can try to resolve it:
Incompatible action selected:
Your instance’s configuration may not be compatible with the selected action.
To execute the 'Terminate' action, your instance may have Termination Protection enabled. Disable this feature if you want to terminate your instance. Once you do that, the alarm will execute the action after the next applicable alarm state change.
To execute the 'Stop' action, your instance’s root device type must be an EBS volume. If the root device type is the instance store, select the 'Terminate' action instead. Once you do that, the alarm will execute the action after the next applicable alarm state change.
Temporary service interruption: There may have been an issue with Amazon CloudWatch or Amazon EC2. We have retried the action without success. You can try to execute the action manually, or wait for the next applicable alarm state change.
Sincerely, Amazon Web Services
Stop seems to be an option for the free micro instance but not for these other instances. When I try to change the shutdown behavior to stop in actions it says:
An error occurred while changing the shutdown behavior of this instance.
Modifying 'instanceInitiatedShutdownBehavior is not supported for spot instances.
Is there another way to get around this problem or will we have to wait until Amazon makes this feature available?
Use standard instances instead of spot instances. Spot instances allow you to bid on extra capacity within ec2. However, they may automatically shut down if the spot price exceeds your bid.
Its not really intended for an always on instance.