I need to attach a fixed number of spot block instances as core nodes to the EMR cluster at my job. The reason we're going with spot block instances is because our Spark jobs are pretty much deterministic in terms of execution time. I'm using the boto3 EMR client apis for spawning and killing EMRs. The only unknown part for me is how the bidding happens for spot blocks. AWS docs have a price chart here for those instance types, but I can't find any information or apis for the accessing the bidding prices, similar to the ones present for normal spot instances.
The end goal is to find out the optimal bidding price, but I don't have any info rather than the static price chart. For the time being, I've set the bidding price to be 70% of the on-demand price using BidPriceAsPercentageOfOnDemandPrice. Any help is appreciated.
I haven't seen any data feeds for Spot Blocks. I suspect they are heavily dependent upon the current workloads and probably aren't used as much as normal Spot instances. The price would also vary based upon duration.
Please note that, at the end of a Spot Block period, the instances are terminated.
An alternative is to use normal Spot instances, but include a mix of instance types to reduce the likelihood of losing all of them.
These days, Spot instance can be terminated if capacity is reduced, even if the Spot price doesn't increase. This has resulted in a much smoother Spot price, but there is no guarantee of spot capacity even at the current Spot price.
Since Spot Blocks are more expensive than normal Spot instances, I'd suggest simply going for normal Spot on a couple of different instance types.
Related
i want to the difference between spot and on demand instances. I know there is a price difference between these two but other than this i want to know the differences. Please help me
Actually there are three allocation types:
on demand - kind of "default" mode. You request an instance, if there is free capacity, you will get the instance. No long term commitment, but once you get an instance, it's yours. It may happen that you will get a message that there is no free capacity for the specific instance type and AZ (so far it happened to me only once with AWS).
reserved - AWS reserves the capacity for you. You have guarantee that you will get the instance type in the selected region or AZ.
spot instance - it's kind of auction / bidding of unused capacity. You ask for an instance, you provide your maximum price and if there is free capacity and your price is at the current price or higher, you will get an instance. The difference is - if the free capacity is exhausted, or the current price is higher than your maximum bid price, your spot instance is terminated . You can get a termination warning event upfront.
The resources for both are the same, spot instances utilise the spare compute capacity within the AWS availability zone (those that are not reserved or launched on-demand).
Depending on the demand for that instance class in the availability zone the spot price will increase or decrease (even surpassing the on-demand price).
When you use a spot instance you are taking a risk that if demand increases you will lose access to the spot instance (you are given a 2 minute warning before termination). For his reason it is common to use a mixture of on-demand/reserved instances and spot instances so that you can withstand instance terminations.
Commonly in EC2 applications you would use an autoscaling group with a configured proportion between on-demand/reserved nodes and spot instances.
For more information take a look at the Requesting Spot Instances for fault-tolerant and flexible applications documentation.
I have a multi-day analysis problem that I am running on a 72 cpu c5n EC2 instance. To get spot pricing, I made my code interruption-resilient and am launching a spot request of one instance. It works great, but this seems like overkill given that Spot can handle thousands of instances. Is this the correct way to solve my problem or am I using a sledgehammer to squash a fly?
I've tried normal EC2 launching, which works great, except that it is four times the price. I don't know of any other way to approach this except for these two ways. I thought about Fargate or containers or something, but I am running a 72 cpu c5n node, and those other options won't let me use that kind of horsepower (that I know of, hence my question).
Thanks!
Amazon EC2 Spot Instances are an excellent way to get cheaper compute (up to 90% discount). The only downside is that the instances might be stopped/terminated (your choice) if there is insufficient capacity.
Some strategies to improve your chance of obtaining spot instances:
Use instances across different Instance Types and Availability Zones because they each have different availability pools (EC2 Spot Fleet can assist with this)
Use resources on weekends and in evenings (even in different regions!) because these tend to be times of lower usage
Use Spot Instances with a specified duration (also known as Spot blocks), but this is at a higher price and a maximum duration of 6 hours
If your software permits it, you could split your load between multiple instances to get the job done faster and to be more resilient against any stoppages of your Spot instances.
Hopefully your application is taking advantage of all the CPUs, otherwise you'd be better-off with smaller instances.
I'm confused about setting maximum price. For example, I'm approved to use g2.8xlarge spot instances in us-west region. In Configure Instance Details I am shown the following:
Availability Zone Current price
us-west-2a $1.261 USD
us-west-2b $1.500 USD
us-west-2c $26.00 USD
So I set max price to $7 thinking that should easily get me an instance in us-west-2a or us-west-2b. However, on submitting the request I get
Status: capacity-not-available: There is no Spot capacity
available that matches your request.
Furthermore, the request description specifies:
Availability Zone: us-west-2c
...which suggests to me that my request is only being considered within the vastly more expensive of the three us-west sub-regions I was quoted prices from before. Thanks for any guidance you can offer me!
When looking at spot instances, look into several different instance types. Not all instance types are available at any given time. Availability varies by region. If the region does not matter, then you will have more choices.
Note: just because a spot price is specified, does not mean that there are any hosts with available instances. They could all be taken. Some companies will bid on thousands of instances which will keep utilization high. This typically happens with the very large instances, which by their nature are expensive and powerful and spot pricing makes these instances much cheaper. If you select middle of the road instances (in performance) you may have better luck.
For bidding price, look at the history of the instance by region. If the history price is close to On-Demand, move on. If the history is less the 50% (better 30%) of On-Demand and there are not too many price spikes per week or month going above 2x On-Demand, then look at bidding 2x On-Demand pricing. If you don't care when and how often your instances run, then bid 1.5x On-Demand.
Remember to make your instances stateless (e.g. store everything in a database, S3, etc.). Your instances may be terminated at any time. You can use ASG to keep a specific number of instances running.
[EDIT]
I just noticed that you are bidding on G2 instances. G2 instances are previous generation instances, which means that they are being phased out making spot bidding harder and harder. Bid instead on current generation instances such as the G3. Also unless you need Windows instances, bid on Linux instances.
I have a queue of jobs and running AWS EC2 instances which process the jobs. We have an AutoScaling groups for each c4.* instance type in spot and on-demand version.
Each instance has power which is a number equal to number of instances CPUs. (for example c4.large has power=2 since it has 2 CPUs).
The the exact power we need is simply calculated from the number of jobs in the queue.
I would like to implement an algorithm which would periodically check the number of jobs in the queue and change the desired value of the particular AutoScaling groups by AWS SDK to save as much money as possible and maintain the total power of instances to keep jobs processed.
Especially:
I prefer spot instances to on-demand since they are cheaper
EC2 instances are charged per hour, we would like to turn off the instance only at the very last minute of its 1hour uptime.
We would like to replace on-demand instance by spot instances when possible. So, at 55min increase spot-group, at 58 check the new spot instance is running and if yes, decrease on-demand-group.
We would like to replace spot instances by on-demand if the bid would be too high. Just turn off the on-demand one and turn on the spot one.
Seems the problem is really difficult to handle. Anybody have any experience or a similar solution implemented?
You could certainly write your own code to do this, effectively telling your Auto Scaling groups when to add/remove instances.
Also, please note that a good strategy for lowering costs with Spot Instances is to appreciate that the price for a spot instance varies by:
Region
Availability Zone
Instance Type
So, if the spot price for a c4.xlarge goes up in one AZ, it might still be the same cost in another AZ. Also, the price of a c4.2xlarge might then be lower than a c4.xlarge, with twice the power.
Therefore, you should aim to diversity your spot instances across multiple AZs and multiple instance types. This means that spot price changes will impact only a small portion of your fleet rather than all at once.
You could use Spot Fleet to assist with this, or even third-party products such as SpotInst.
It's also worth looking at AWS Batch (not currently available in every region), which is designed to intelligently provide capacity for batch jobs.
Autoscaling groups allow you to use alarms and metrics that are defined outside of the autoscaling group.
If you are using SNS, you should be able to set up an alarm on your SNS queue and use that to scale up and scale down your scaling group.
If you are using a custom queue system, you can push metrics to cloudwatch to create a similar alarm.
You can determine how often scaling actions do occur, but it may be difficult to get the run time to exactly one hour.
Correct me if I'm wrong, but there seem to be some inconsistencies between creating sport block requests using EC2 console and AWS SDK (Boto3, namely). When requesting a spot block using AWS Management console, the only pricing option is "Use automated bidding".
However when doing the same via Boto3, SpotPrice parameter is marked as required with no indication that it might represent, say, a percentage of on-demand price.
Is there any option to use automated bidding programmatically without hard-coding on-demand prices in the requests?
The console is merely trying to present a simplified process. I think it is simply setting SpotPrice to the on-demand price. That's a much cleaner interface than asking for a different price per selected Instance Type.
You always only pay the current Spot Price. Bidding is always automatic up to your Spot price, which represents the maximum you are willing to pay.
If you want to make an equivalent bid without hard-coding the On-Demand Price, you could use the AWS PriceList API, which is really just some downloadable JSON/CSV files with pricing information. Pricing doesn't change very often, so you could cache that information and occasionally refresh it.
Because you choose Reserved for duration. Automatic bidding is the only way you can do this.
IMHO, you should consider your SPOT requirement before jump into reserved for duration. If your application is spot instance ready , then you should specify a fleet of instance with your desire minimum price. Because AWS Spot always has under utilised spare instance, in fact, this minimise the interruption without even the need of reservation.
Perhaps due to c4.* pricing and cause many people move from c3.* to c4., it seems c3. pricing is all time low (e.g. us-east-1* show a price below $0.02)