I'm confused about when you can or cannot hibernate spot instances.
(1.) Under "Hibernation prerequisites," the documentation says that hibernation "is available for On-Demand Instances and Reserved Instances. It is not available for Spot Instances. For more information, see Hibernating interrupted Spot Instances." https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Hibernate.html#hibernating-prerequisites
(2.) But the following article: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#hibernate-spot-instances
... says that you can set instances to automatically hibernate upon interruption.
What explains this apparent contradiction?
My guess would be that you cannot manually initiate hibernation from a spot instance, but you can configure a spot to hibernate automatically upon interruption. Is that right?
You cannot manually hibernate a Spot Instance.
However, a Spot Instance can be hibernated automatically if the Spot Instance is interrupted and hibernation has been configured.
See: Preparing for instance hibernation
I think it is supported, but with some exceptions. I suspect this sentence is missing the word all:
It is not available for all Spot Instances.
The page that is linked to in the next sentence, Hibernating interrupted Spot Instances, describes the requirements imposed on spot instances.
Related
Imagine I have a persistent spot request of size 1. If I manually stop this instance, it will not restart until I start it again.
However, if the instance is interrupted by AWS because of a lack of spot capacity, it will restart automatically when capacity is available (as I understand it).
How can I put an instance in the stopped-because-interrupted state into a state where it will not restart automatically if capacity becomes available.
According to AWS documentation only EC2 can start interrupted spot instances so I think that the only option would be monitoring events/alerts which notify when instances restart and then stop them again with automation. Never tried this si this is only an educated guess/opinion.
I'm having an issue with AWS boxes (EC2) where I terminate the box and it re-spawns. To give context, there is no autoscaling group. Anywhere I can search for some config that might be triggering the launch?
I would make sure you don't have a persistent spot request active in your account, and also check to see if you perhaps installed the AWS Instance scheduler - either or both of those could be starting instances on your behalf - (double check the autoscaling group, that is the most obvious reason though)
If you terminate a running Spot Instance that was launched by a
persistent Spot request, the Spot request returns to the open state so
that a new Spot Instance can be launched. To cancel a persistent Spot
request and terminate its Spot Instances, you must cancel the Spot
request first and then terminate the Spot Instances. Otherwise, the
persistent Spot request can launch a new instance.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
https://aws.amazon.com/solutions/instance-scheduler/
So I found out the culprit, maybe it can help more people. Apparently, there is a service from AWS called OpsWorks that automates things like Cheff of Puppet, which my company had configured some time ago. That would be checking for instances running and triggering re-provisioning when it didn't see the instance running. OpsWorks is here
I read myself pretty much through all the online docs of EC2.
Still one question is unanswered to me:
What is the maximum runtime of a spot instance (uninterrupted) ?
everything is well documented for < 6h runtime. But what happens afterwards ? is the instance automatically terminated or does is keep running until the ressources are no more available ? somehow i didn`t find anything clear on this.
Thanks !
As mentioned in the AWS documentation under How Spot Instances Work:
Spot Instances run until you terminate them or until Amazon EC2 must interrupt them (known as a Spot Instance interruption).
Therfore in theory an EC2 spot instance will run indefinitely until you terminate it or AWS interrupts it because your spot bid price is too low, or similar.
Spot instances will not terminate until you do or it is interrupted from AWS side (Eg. The Spot price is greater than your maximum price). So your application should be able to handle these terminations gracefully.
More details : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html
Also if you terminate it you will be charge for that hour. If AWS terminates it you are not charged for that hour.
Is there a way to elegantly Script/Configure Spot instances request, if Spot not available in some specified duration, just use OnDemand. And if Spot instance gets terminated just shift to OnDemand.
Spot Fleet does not do this (it just manages only Spot), EMR fleets have some logic around this. You can have auto scaling with Spot or on Demand not both (even though you can have 2 separate ASGs simulate this behavior).
This should be some kind of a base line use case.
Also does an Event get triggered when a Spot instance is launched or when it is Terminated. I am only seeing CLIs to check Spot status, not any CloudWatch metric/event.
Cloudwatch Instance State events can fire when any event changes states.
They can fire for any event in the lifecycle of an instance:
pending (launching), running (launch complete), shutting-down, stopped, stopping, and terminated, for any instance (or for all instances, which is probably what you want -- just disregard any instance that isn't of interest), and this includes both on-demand and spot.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#ec2_event_type
http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/LogEC2InstanceState.html
You could use this to roll your own solution -- there's not a built in mechanism for marshaling mixed fleets.
I used to do this from the ELB with health checks. You can make two groups, one with spot instances and one with reserved or on demand. Create a CW alarm when spot group contains zero healthy hosts, and scale up the other group when it fires. And the other way around, when it has enough healthy hosts scale down the other group. Use 30 sec health checks on alarm you use to scale up and 30-60 minute cooldown on scale down.
There is also Spotml which allows you to always keep a spotInstance or an onDemand instance up and running.
In addition to simply spawning the instance it also allows you to
Preserve data via persistent storage
And configure a startup script each time a new instance is spawned.
Disclosure: I'm also the creator of SpotML, it's primarily useful for ML/DataScience workflows that can largely just run on spot instances.
A new AWS ELB feature, connection draining, was recently announced.
http://aws.amazon.com/about-aws/whats-new/2014/03/20/elastic-load-balancing-supports-connection-draining/
Apparently this works with Auto Scaling Groups - instances are drained before being removed, but does that also apply to spot instances that are being terminated by AWS due to a rising spot price?
Nothing definitive I could find, but from my reading on this, I think the answer is almost definitely no. Spot instances are different animals than regular instances, and the way the connection draining works you can specify upto 60 minute delay before your connection-drained enabled instance gets terminated when it becomes unhealthy - if AWS was to allow this added layer of safety to spot instances, it would completely up-end the way spot instances are used and how they are positioned.
The trade-off for using spot instances has always been, "you can pay a fraction of the cost, but you risk being terminated at any instant without warning"...if they added an up to 60 minute 'warning' to spot instances, while it would be fantastic from the end-users point of view, I think it would severely eat into AWS's on-demand and reserved instance pricing model and thus they probably won't support this anytime soon (unless forced to by competitive pressure).
EDIT 1/6/2015: now, almost a year later, AWS has indeed added a 'two minute warning' for EC2 spot instance termination. https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termination-notices/