I wrote a terraform script to spin up an EC2 spot instance, and every time I terminate it, it automatically spins up a new instance. I don't understand this behavior. Why is this happening and how do I stop it?
When you requested your spot instance, you probably didn't specify the spot-type as 'one-time' so the request persists beyond instance termination. Or in other words, when your instance is terminated the request begins anew and launches a new instance based on the same bid price, launch specification, etc.
From the terraform documentation:
spot_type - (Optional; Default: "persistent") If set to "one-time",
after the instance is terminated, the spot request will be closed.
Also, Terraform can't manage one-time spot requests, just launch them.
From the AWS documentation:
This will continue until you cancel the spot instance request. You can do this via the AWS Console, CLI (cancel-spot-instance-requests), or via terraform by destroying the resource.
Further Reading
AWS Documentation - Cancelling a Spot Instance Request
Related
I'm aware that the spot service will manage a spot instance and stop or start it based on whether the price matches or capacity is available as per https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#specifying-spot-interruption-behavior
And https://aws.amazon.com/about-aws/whats-new/2020/01/amazon-ec2-spot-instances-stopped-started-similar-to-on-demand-instances/
mentions now it's possible to manually stop or start a spot instance.
I'm confused if I stop a spot instance manually, will the spot service start it again whenever the price/capacity requirements are met? Or would it stop monitoring the instance till I manually start the spot instance again?
To answer your question, there are 2 kinds of request. 1) one time request 2) persistent request.
one-time-request:- manually you can only terminate instance, you cannot stop it.Once you terminate instance the request goes away.
persistent-request:- In case of persistent request, it will automatically launch an instance when you manually terminate the instance, but if you stop it manually then you need to manually start it again. spot instance will not start for you automatically.docs
If the request is persistent, the request is opened again after your Spot Instance is interrupted( interruption by aws). If the request is persistent and you stop your Spot Instance, the request only opens after you start your Spot Instance.
Please NOTE
You can only stop/ start instance which is launched from persistent request and which not a part of fleet or launch group, Availability Zone group, or Spot block. docs
when you stop the instance the request goes into disabled state.
I would like to use AWS EC2 spot requests for machine learning that I do in spots.
If you don't configure any settings for spot requests, they will be set to "fleet" and an EC2 instance will be created that cannot be paused (it can only be terminated).
If you know how to create an instance with a spot request that can be paused, please let me know.
Spot instances launched in a fleet can’t be paused or stopped. To stop a Spot instance, it must be launched by a Spot instance request for a single instance, you can create such requests using Launch Instance wizard, you may follow steps at this link. Make sure to select "Persistent request" in the wizard.
This request will launch a Spot instance that can be Stopped but still can’t be Paused. Hibernate/Pause option is only available as a behavior at the event of interruption and can't be requested by the user.
I just figured out about AWS CloudWatch that would let you terminate your EC2 instance after some time in inactivity. So I created an alarm that would terminate the instance when the CPU usage is less the 1% for 2 hours. Which ultimately ended up putting my instance into an alarm state right away and it prevented me from starting it up to test the feature out.
I then deleted the CloudWatch alarm again in order to be able to launch the EC2 instance gain but even after I deleted the CloudWatch, the state is set to Terminated but the but the Start option from the Actions drop down button is still disabled.
How do I get the instance to start again?
You can't restart a terminated instance. The instance no longer exists. It is just listed as "terminated" in your web console for a little bit so you can see that it was deleted. You have to create a new instance now.
I'm having an issue with AWS boxes (EC2) where I terminate the box and it re-spawns. To give context, there is no autoscaling group. Anywhere I can search for some config that might be triggering the launch?
I would make sure you don't have a persistent spot request active in your account, and also check to see if you perhaps installed the AWS Instance scheduler - either or both of those could be starting instances on your behalf - (double check the autoscaling group, that is the most obvious reason though)
If you terminate a running Spot Instance that was launched by a
persistent Spot request, the Spot request returns to the open state so
that a new Spot Instance can be launched. To cancel a persistent Spot
request and terminate its Spot Instances, you must cancel the Spot
request first and then terminate the Spot Instances. Otherwise, the
persistent Spot request can launch a new instance.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
https://aws.amazon.com/solutions/instance-scheduler/
So I found out the culprit, maybe it can help more people. Apparently, there is a service from AWS called OpsWorks that automates things like Cheff of Puppet, which my company had configured some time ago. That would be checking for instances running and triggering re-provisioning when it didn't see the instance running. OpsWorks is here
While working with a g2.2xlarge spot instance, I have tried to set up an alarm that will notify me when the average CPU usage over a two hour period has dropped below 5% and will then automatically stop the instance. Here's a link to a nice article Amazon wrote up on how to use the stop/start instance feature. The AWS alarms seem to allow you to do this however after the trigger goes off I get this reply:
Dear AWS customer,
We are unable to execute the 'Stop' action on Amazon EC2 instance i-e60e21ec that you specified in the Amazon CloudWatch alarm awsec2-i-e60e21ec-Low-CPU-Utilization.
You may want to check the alarm configuration to ensure that it is compatible with your instance configuration. You can also attempt to execute the action manually.
These are some possible reasons for this failure and steps you can try to resolve it:
Incompatible action selected:
Your instance’s configuration may not be compatible with the selected action.
To execute the 'Terminate' action, your instance may have Termination Protection enabled. Disable this feature if you want to terminate your instance. Once you do that, the alarm will execute the action after the next applicable alarm state change.
To execute the 'Stop' action, your instance’s root device type must be an EBS volume. If the root device type is the instance store, select the 'Terminate' action instead. Once you do that, the alarm will execute the action after the next applicable alarm state change.
Temporary service interruption: There may have been an issue with Amazon CloudWatch or Amazon EC2. We have retried the action without success. You can try to execute the action manually, or wait for the next applicable alarm state change.
Sincerely, Amazon Web Services
Stop seems to be an option for the free micro instance but not for these other instances. When I try to change the shutdown behavior to stop in actions it says:
An error occurred while changing the shutdown behavior of this instance.
Modifying 'instanceInitiatedShutdownBehavior is not supported for spot instances.
Is there another way to get around this problem or will we have to wait until Amazon makes this feature available?
Use standard instances instead of spot instances. Spot instances allow you to bid on extra capacity within ec2. However, they may automatically shut down if the spot price exceeds your bid.
Its not really intended for an always on instance.