I understand that spot instances are terminated if the bid price is exceeded, and that they can't launch if there is no capacity to do so or the instance limit is reached. I've not seen them terminated due to over-subscription before. Used AWS CLI:
aws ec2 request-spot-instances --spot-price "0.8" --instance-count 1 --type "one-time" --launch-specification c:\path\to\spot-instance-spec.json
The r4.4xlarge linux spot launched successfully in (my) eu-west-1b, then terminated and the status in the spot request is:
instance-terminated-capacity-oversubscribed: Your Spot instance was terminated as there is no more unused capacity available in this pool.
Does anyone know what this means specifically? Is a pool just the AZ? Is it down to more expensive spots? Is it due to on-demands being launched?
The AWS documentation now has a definition of Spot instance pool:
Spot Instance pool – A set of unused EC2 instances with the same instance type, operating system, Availability Zone, and network platform.
Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html
Your Spot Instances continue to run as long as your maximum price is
at or above the Spot price, there is available capacity for your
instance type, and you don't terminate the instance. If a change in
the Spot price or available capacity requires Amazon EC2 to terminate
your Spot Instances, the Spot request goes into a terminal state. For
example, if your price equals the Spot price but Spot Instances are
not available, the status code is
instance-terminated-capacity-oversubscribed. A request also goes into
the terminal state if you cancel the Spot request or terminate the
Spot Instances.
Its mentioned here in AWS Doc, and also check for limitation of spot request.For some instance there is a limit beyond that it cant be launched.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-bid-status.html
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html
Related
As we know, when a spot instance is about to be terminated , we get notified 2 minutes in advance.Now i want my services to be running by replacing the spot instance that is notified to be deleted, by a new spot instance.
Are there any existing ways to automatically replace an EC2 spot instance by a new spot instance?.
From Spot Fleet - Amazon Elastic Compute Cloud:
A Spot Fleet is a set of Spot Instances and optionally On-Demand Instances that is launched based on criteria that you specify. The Spot Fleet selects the Spot capacity pools that meet your needs and launches Spot Instances to meet the target capacity for the fleet. By default, Spot Fleets are set to maintain target capacity by launching replacement instances after Spot Instances in the fleet are terminated. You can submit a Spot Fleet as a one-time request, which does not persist after the instances have been terminated. You can include On-Demand Instance requests in a Spot Fleet request.
I have suddenly started getting no capacity available error while trying to create defined duration based spot instance in us-west-2. Other region works fine for me. I do not see any option to specify a price while trying to create this type of spot instance.
I do not have any other spot instance or any other ec2 instance running for me in this AZ as well.
My hypothesis is that there is no ec2 capacity available in whole us-west-2 region right now to create a spot instance. But I am finding that hard to believe. Is that correct?
Also I went through doc but it doesn't point out that by capacity it means for the whole of the aws us-west-2 region or just for my account
The no Spot capacity available error occurs when there isn't enough spare capacity to fulfill your Spot Instance or Spot Fleet request.
As capacity becomes available, Amazon EC2 fulfills requests in the following order:
Reserved Instances ->
On-Demand Instances ->
Spot Instances
The Spot request continues to automatically make the launch request until capacity becomes available. When capacity becomes available, Amazon EC2 fulfills the Spot request.
Its important to take a note about Lifecyle of spot request ->
Also according to lifecycle of spot request
If one or more request constraints are valid but can't be met yet, or if there is not enough capacity, the request goes into a holding state waiting for the constraints to be met. The request options affect the likelihood of the request being fulfilled. For example, if you specify a maximum price below the current Spot price, your request stays in a holding state until the Spot price goes below your maximum price. If you specify an Availability Zone group, the request stays in a holding state until the Availability Zone constraint is met.
In the event of an outage of one of the Availability Zones, there is a chance that the spare EC2 capacity available for Spot Instance requests in other Availability Zones can be affected
I'm having an issue with AWS boxes (EC2) where I terminate the box and it re-spawns. To give context, there is no autoscaling group. Anywhere I can search for some config that might be triggering the launch?
I would make sure you don't have a persistent spot request active in your account, and also check to see if you perhaps installed the AWS Instance scheduler - either or both of those could be starting instances on your behalf - (double check the autoscaling group, that is the most obvious reason though)
If you terminate a running Spot Instance that was launched by a
persistent Spot request, the Spot request returns to the open state so
that a new Spot Instance can be launched. To cancel a persistent Spot
request and terminate its Spot Instances, you must cancel the Spot
request first and then terminate the Spot Instances. Otherwise, the
persistent Spot request can launch a new instance.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
https://aws.amazon.com/solutions/instance-scheduler/
So I found out the culprit, maybe it can help more people. Apparently, there is a service from AWS called OpsWorks that automates things like Cheff of Puppet, which my company had configured some time ago. That would be checking for instances running and triggering re-provisioning when it didn't see the instance running. OpsWorks is here
I read myself pretty much through all the online docs of EC2.
Still one question is unanswered to me:
What is the maximum runtime of a spot instance (uninterrupted) ?
everything is well documented for < 6h runtime. But what happens afterwards ? is the instance automatically terminated or does is keep running until the ressources are no more available ? somehow i didn`t find anything clear on this.
Thanks !
As mentioned in the AWS documentation under How Spot Instances Work:
Spot Instances run until you terminate them or until Amazon EC2 must interrupt them (known as a Spot Instance interruption).
Therfore in theory an EC2 spot instance will run indefinitely until you terminate it or AWS interrupts it because your spot bid price is too low, or similar.
Spot instances will not terminate until you do or it is interrupted from AWS side (Eg. The Spot price is greater than your maximum price). So your application should be able to handle these terminations gracefully.
More details : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html
Also if you terminate it you will be charge for that hour. If AWS terminates it you are not charged for that hour.
We have basic plan Amazon configuration. Also we have created Windows instance which is terminated frequently and new Ip address is assigned. I don't know how to check and why its terminated frequently.
Under standard conditions, Amazon EC2 instances are never terminated on their own. You may be accidentally using EC2 Spot instances which works on a bidding model. AWS has the power to terminate these instances with a warning time of ~2 minutes if there is a a higher bid for the spot instance.
If your instances get terminated immediately after you launch, refer this page: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_InstanceStraightToTerminated.html
For more information on EC2 spot instances, refer this page:
http://aws.amazon.com/ec2/purchasing-options/spot-instances/