I'm confused about setting maximum price. For example, I'm approved to use g2.8xlarge spot instances in us-west region. In Configure Instance Details I am shown the following:
Availability Zone Current price
us-west-2a $1.261 USD
us-west-2b $1.500 USD
us-west-2c $26.00 USD
So I set max price to $7 thinking that should easily get me an instance in us-west-2a or us-west-2b. However, on submitting the request I get
Status: capacity-not-available: There is no Spot capacity
available that matches your request.
Furthermore, the request description specifies:
Availability Zone: us-west-2c
...which suggests to me that my request is only being considered within the vastly more expensive of the three us-west sub-regions I was quoted prices from before. Thanks for any guidance you can offer me!
When looking at spot instances, look into several different instance types. Not all instance types are available at any given time. Availability varies by region. If the region does not matter, then you will have more choices.
Note: just because a spot price is specified, does not mean that there are any hosts with available instances. They could all be taken. Some companies will bid on thousands of instances which will keep utilization high. This typically happens with the very large instances, which by their nature are expensive and powerful and spot pricing makes these instances much cheaper. If you select middle of the road instances (in performance) you may have better luck.
For bidding price, look at the history of the instance by region. If the history price is close to On-Demand, move on. If the history is less the 50% (better 30%) of On-Demand and there are not too many price spikes per week or month going above 2x On-Demand, then look at bidding 2x On-Demand pricing. If you don't care when and how often your instances run, then bid 1.5x On-Demand.
Remember to make your instances stateless (e.g. store everything in a database, S3, etc.). Your instances may be terminated at any time. You can use ASG to keep a specific number of instances running.
[EDIT]
I just noticed that you are bidding on G2 instances. G2 instances are previous generation instances, which means that they are being phased out making spot bidding harder and harder. Bid instead on current generation instances such as the G3. Also unless you need Windows instances, bid on Linux instances.
Related
I am trying to use aws spot instances (m5.large in eu-west-2 region) with a maximum bid equal to the price of on demand instances. According to https://aws.amazon.com/ec2/spot/instance-advisor/ these instances should have a < 5% frequency of interruption, however, after launching 40 such instances this morning, I have found that within the hour 34 of them were evicted by aws ("instance-terminated-no-capacity" according to the spot requests page on the ec2 dashboard).
This eviction rate looks much too high compared to both amazon's own advisor and other users experiences. Does anybody know what could be causing this behaviour, if there is any better way to debug it or predict it, or if this is just what I should expect from spot instances?
Thank you!
Actually, for m5.large instance in eu-west-2 region(Oregon) it's 5%-10% frequency of interruption, so you can expect a max of 10%. I'm not saying that issue you are facing is because of this.
AWS terminates your spot instances because of any of these reasons,
The Spot price is above the maximum price.
There isn't enough capacity.
Amazon EC2 can't meet the constraints you placed on your Spot request.
In your case, since you are seeing instance-terminated-no-capacity message it is definitely because of the second reason. Since you've asked for 40 such instances, the amazon spot instance pool might not have enough capacity at that time.
The capacity of available spot instances pool depends on the demand for regular instances, and when users ask for regular on-demand instances, AWS will start terminating spot instances to fulfil those requests if there is not enough capacity
In my experience spot interruption is highly variable, and for some instances more or less likely at different times of the day.
If you need 40 instances and they do not need to be in the same availability zone (AZ) to each other you might reduce the chance of a mass interruption of all/most instances if you spread the machines across different AZs within the region as each availability zone has its own pool of machines. Although you will likely increase the chance that some machines will be interrupted.
Note this is not an option if you are using EMR, then they have to be in the same AZ.
i want to the difference between spot and on demand instances. I know there is a price difference between these two but other than this i want to know the differences. Please help me
Actually there are three allocation types:
on demand - kind of "default" mode. You request an instance, if there is free capacity, you will get the instance. No long term commitment, but once you get an instance, it's yours. It may happen that you will get a message that there is no free capacity for the specific instance type and AZ (so far it happened to me only once with AWS).
reserved - AWS reserves the capacity for you. You have guarantee that you will get the instance type in the selected region or AZ.
spot instance - it's kind of auction / bidding of unused capacity. You ask for an instance, you provide your maximum price and if there is free capacity and your price is at the current price or higher, you will get an instance. The difference is - if the free capacity is exhausted, or the current price is higher than your maximum bid price, your spot instance is terminated . You can get a termination warning event upfront.
The resources for both are the same, spot instances utilise the spare compute capacity within the AWS availability zone (those that are not reserved or launched on-demand).
Depending on the demand for that instance class in the availability zone the spot price will increase or decrease (even surpassing the on-demand price).
When you use a spot instance you are taking a risk that if demand increases you will lose access to the spot instance (you are given a 2 minute warning before termination). For his reason it is common to use a mixture of on-demand/reserved instances and spot instances so that you can withstand instance terminations.
Commonly in EC2 applications you would use an autoscaling group with a configured proportion between on-demand/reserved nodes and spot instances.
For more information take a look at the Requesting Spot Instances for fault-tolerant and flexible applications documentation.
I need to attach a fixed number of spot block instances as core nodes to the EMR cluster at my job. The reason we're going with spot block instances is because our Spark jobs are pretty much deterministic in terms of execution time. I'm using the boto3 EMR client apis for spawning and killing EMRs. The only unknown part for me is how the bidding happens for spot blocks. AWS docs have a price chart here for those instance types, but I can't find any information or apis for the accessing the bidding prices, similar to the ones present for normal spot instances.
The end goal is to find out the optimal bidding price, but I don't have any info rather than the static price chart. For the time being, I've set the bidding price to be 70% of the on-demand price using BidPriceAsPercentageOfOnDemandPrice. Any help is appreciated.
I haven't seen any data feeds for Spot Blocks. I suspect they are heavily dependent upon the current workloads and probably aren't used as much as normal Spot instances. The price would also vary based upon duration.
Please note that, at the end of a Spot Block period, the instances are terminated.
An alternative is to use normal Spot instances, but include a mix of instance types to reduce the likelihood of losing all of them.
These days, Spot instance can be terminated if capacity is reduced, even if the Spot price doesn't increase. This has resulted in a much smoother Spot price, but there is no guarantee of spot capacity even at the current Spot price.
Since Spot Blocks are more expensive than normal Spot instances, I'd suggest simply going for normal Spot on a couple of different instance types.
I have a queue of jobs and running AWS EC2 instances which process the jobs. We have an AutoScaling groups for each c4.* instance type in spot and on-demand version.
Each instance has power which is a number equal to number of instances CPUs. (for example c4.large has power=2 since it has 2 CPUs).
The the exact power we need is simply calculated from the number of jobs in the queue.
I would like to implement an algorithm which would periodically check the number of jobs in the queue and change the desired value of the particular AutoScaling groups by AWS SDK to save as much money as possible and maintain the total power of instances to keep jobs processed.
Especially:
I prefer spot instances to on-demand since they are cheaper
EC2 instances are charged per hour, we would like to turn off the instance only at the very last minute of its 1hour uptime.
We would like to replace on-demand instance by spot instances when possible. So, at 55min increase spot-group, at 58 check the new spot instance is running and if yes, decrease on-demand-group.
We would like to replace spot instances by on-demand if the bid would be too high. Just turn off the on-demand one and turn on the spot one.
Seems the problem is really difficult to handle. Anybody have any experience or a similar solution implemented?
You could certainly write your own code to do this, effectively telling your Auto Scaling groups when to add/remove instances.
Also, please note that a good strategy for lowering costs with Spot Instances is to appreciate that the price for a spot instance varies by:
Region
Availability Zone
Instance Type
So, if the spot price for a c4.xlarge goes up in one AZ, it might still be the same cost in another AZ. Also, the price of a c4.2xlarge might then be lower than a c4.xlarge, with twice the power.
Therefore, you should aim to diversity your spot instances across multiple AZs and multiple instance types. This means that spot price changes will impact only a small portion of your fleet rather than all at once.
You could use Spot Fleet to assist with this, or even third-party products such as SpotInst.
It's also worth looking at AWS Batch (not currently available in every region), which is designed to intelligently provide capacity for batch jobs.
Autoscaling groups allow you to use alarms and metrics that are defined outside of the autoscaling group.
If you are using SNS, you should be able to set up an alarm on your SNS queue and use that to scale up and scale down your scaling group.
If you are using a custom queue system, you can push metrics to cloudwatch to create a similar alarm.
You can determine how often scaling actions do occur, but it may be difficult to get the run time to exactly one hour.
Edit: This question is NOT to ask about "Spot Instances"; this question is to ask regular "On Demand Instance". I think I need to clarify this, after reading comments below.
Basically, my question is about whether I should consider the risk that when I need to launch an EC2 instance, but that EC2 region has run out of capacity and can't fulfill my request.
I understand the chance for the above situation is extremely low, but I'd like to understand if AWS has any SLA to make sure that situation/risk won't happen.
There are protective controls in place to make the unavailability of a particular type of instance in a particular availability zone at a particular time unlikely... but it's possible, and there is no assurance provided by AWS that a given type of EC2 instance will be available for launch, on-demand, at any particular time, in any particular Availability Zone unless you purchased reserved instances of that type, specifically in that availability zone. In that case, there is supposed to always be sufficient hardware available so that you can have the number of paid reserved instances running, at minimum, including the ability to launch enough new instances to bring the total up to that minimum.
Reserved instances are commonly discussed in the context of their associated discount, but they have two purposes:
Reserved Instances are not physical instances, but rather a billing discount applied to the use of On-Demand Instances in your account. These On-Demand Instances must match certain attributes in order to benefit from the billing discount.
When you purchase Reserved Instances in a specific Availability Zone, they provide a capacity reservation. (emphasis added)
For example, if you purchased 4 reserved t2.2xlarge instances in us-east-2a, the assurance is that you will always be able to launch enough to bring the total running instances of that type in that zone to 4. If you already have 4, there's not an assurance of being able to start more, but there is an assurance that if you stop them, you will be able to start them again.
Pricing models for reserved instances has changed over the years, such that reserved instances are generally billed at the same rate whether they're running or not, so you can look at it one of two ways:
If you need the capacity all the time, you're getting a substantial discount... or, if you don't need the capacity all the time, you're technically paying all the time for capacity that is largely unneeded, but you are still paying less than you'd pay for on-demand instances without reservations, and you can either leave it running or launch it when you need it.
Should you consider the risk that an entire region has widespread capacity issues? You should consider it, but there are, historically speaking, other significant outage scenarios that are more likely... EBS and S3 have both had failures that impaired the ability to launch instances, even though the capacity was idle in EC2.
Yes. I've had API calls to create EC2 resources fail many times due to lack of available AWS resources. I most commonly see this when attempting to create a new EC2 instance with Dedicated Tenancy in a specific Availability Zone.
Yes. It is possible your instance request cannot be fulfilled. On-Demand instance does not guarantee you an instance. In particular,t2.small instances are more likely to be not fulfilled based on my experience. It is possible, AWS has only limited number of t2.small instances.
How can you make sure it is always fulfilled?
Reserve the instances for you so that it is not given to anyone else. But there is a cost associated with it. You have pay for the instance irrespective of whether you use it not. I am talking in general terms. Reserved instance is a complex topic, but that is the route you should take if you want AWS to guarantee you an instance.
The answer is yes, your launch request can fail because there is no available capacity in the relevant Availability Zone. I would say that it's a rare occurrence, but certainly possible.
You can mitigate by using multiple zones in the same region, other regions, or by using Reserved Instances.
Today it happened in an AWS account that I manage. If I am not wrong it was the family r4, exactly an r4.xlarge instance (4 hours ago) in Virgina Region. I had to choose a different AZ. That is why AWS always recommends working with more than one AZ.
But for that reason, I started to use Scheduled Reserved Instances
You always will have for a period of time a reserved capacity of a family type of instance(s).
Sure, it works if you have a defined schedule or workflow.
Hope it helps you.