I'm aware that the spot service will manage a spot instance and stop or start it based on whether the price matches or capacity is available as per https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#specifying-spot-interruption-behavior
And https://aws.amazon.com/about-aws/whats-new/2020/01/amazon-ec2-spot-instances-stopped-started-similar-to-on-demand-instances/
mentions now it's possible to manually stop or start a spot instance.
I'm confused if I stop a spot instance manually, will the spot service start it again whenever the price/capacity requirements are met? Or would it stop monitoring the instance till I manually start the spot instance again?
To answer your question, there are 2 kinds of request. 1) one time request 2) persistent request.
one-time-request:- manually you can only terminate instance, you cannot stop it.Once you terminate instance the request goes away.
persistent-request:- In case of persistent request, it will automatically launch an instance when you manually terminate the instance, but if you stop it manually then you need to manually start it again. spot instance will not start for you automatically.docs
If the request is persistent, the request is opened again after your Spot Instance is interrupted( interruption by aws). If the request is persistent and you stop your Spot Instance, the request only opens after you start your Spot Instance.
Please NOTE
You can only stop/ start instance which is launched from persistent request and which not a part of fleet or launch group, Availability Zone group, or Spot block. docs
when you stop the instance the request goes into disabled state.
Related
Imagine I have a persistent spot request of size 1. If I manually stop this instance, it will not restart until I start it again.
However, if the instance is interrupted by AWS because of a lack of spot capacity, it will restart automatically when capacity is available (as I understand it).
How can I put an instance in the stopped-because-interrupted state into a state where it will not restart automatically if capacity becomes available.
According to AWS documentation only EC2 can start interrupted spot instances so I think that the only option would be monitoring events/alerts which notify when instances restart and then stop them again with automation. Never tried this si this is only an educated guess/opinion.
I would like to use AWS EC2 spot requests for machine learning that I do in spots.
If you don't configure any settings for spot requests, they will be set to "fleet" and an EC2 instance will be created that cannot be paused (it can only be terminated).
If you know how to create an instance with a spot request that can be paused, please let me know.
Spot instances launched in a fleet can’t be paused or stopped. To stop a Spot instance, it must be launched by a Spot instance request for a single instance, you can create such requests using Launch Instance wizard, you may follow steps at this link. Make sure to select "Persistent request" in the wizard.
This request will launch a Spot instance that can be Stopped but still can’t be Paused. Hibernate/Pause option is only available as a behavior at the event of interruption and can't be requested by the user.
I just figured out about AWS CloudWatch that would let you terminate your EC2 instance after some time in inactivity. So I created an alarm that would terminate the instance when the CPU usage is less the 1% for 2 hours. Which ultimately ended up putting my instance into an alarm state right away and it prevented me from starting it up to test the feature out.
I then deleted the CloudWatch alarm again in order to be able to launch the EC2 instance gain but even after I deleted the CloudWatch, the state is set to Terminated but the but the Start option from the Actions drop down button is still disabled.
How do I get the instance to start again?
You can't restart a terminated instance. The instance no longer exists. It is just listed as "terminated" in your web console for a little bit so you can see that it was deleted. You have to create a new instance now.
I'm having an issue with AWS boxes (EC2) where I terminate the box and it re-spawns. To give context, there is no autoscaling group. Anywhere I can search for some config that might be triggering the launch?
I would make sure you don't have a persistent spot request active in your account, and also check to see if you perhaps installed the AWS Instance scheduler - either or both of those could be starting instances on your behalf - (double check the autoscaling group, that is the most obvious reason though)
If you terminate a running Spot Instance that was launched by a
persistent Spot request, the Spot request returns to the open state so
that a new Spot Instance can be launched. To cancel a persistent Spot
request and terminate its Spot Instances, you must cancel the Spot
request first and then terminate the Spot Instances. Otherwise, the
persistent Spot request can launch a new instance.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
https://aws.amazon.com/solutions/instance-scheduler/
So I found out the culprit, maybe it can help more people. Apparently, there is a service from AWS called OpsWorks that automates things like Cheff of Puppet, which my company had configured some time ago. That would be checking for instances running and triggering re-provisioning when it didn't see the instance running. OpsWorks is here
I wrote a terraform script to spin up an EC2 spot instance, and every time I terminate it, it automatically spins up a new instance. I don't understand this behavior. Why is this happening and how do I stop it?
When you requested your spot instance, you probably didn't specify the spot-type as 'one-time' so the request persists beyond instance termination. Or in other words, when your instance is terminated the request begins anew and launches a new instance based on the same bid price, launch specification, etc.
From the terraform documentation:
spot_type - (Optional; Default: "persistent") If set to "one-time",
after the instance is terminated, the spot request will be closed.
Also, Terraform can't manage one-time spot requests, just launch them.
From the AWS documentation:
This will continue until you cancel the spot instance request. You can do this via the AWS Console, CLI (cancel-spot-instance-requests), or via terraform by destroying the resource.
Further Reading
AWS Documentation - Cancelling a Spot Instance Request