To Developers,
Can an AWS Batch job use Reserved Instances?
I would end up using a base of RI's + (On-Demand or Spot Instances) per Client.
I would like the cost savings from Reserved Instances but need more reliability than spot.
Thanks,
Marc
Yes, Batch compute resources are just ec2 instances - and reserved instances are just a billing mechanism, and are not assigned to a particular instance - i.e. you don't tell AWS which EC2 instance in particular is a RI and which is an OnDemand - it doesn't matter.
So if you have 20 RI, and need 40 running during the month, so you fill the gap with 20 on-demand or spot instances, then 20 will be billed as RI and 20 will be OD/or spot. If you kill any 25 instances (for example), you will still be billed for 20 RI in this example and no spot or OD instances.
My guess is that maximizing the use of spot instances is going to save you some money - but it is sound logic to have a certain number of RI around to provide a base level of service - and you can't count on there being ANY spot or ANY OD instances at any given time.
Related
Our organization has many EC2 on-demand instances. We never looked at the reserved instance pricing because we thought we had to pay up-front. But I see that now there are reserved instance with no up-front costs, but monthly.
This looks like what we need to keep costs down, but I am unclear as to how the reserved instance pricing is applied.
For example, we have 10 t3.medium ec2 instance. Let's say that there are 6 always running 24/7, but the other 4 are not running 24/7, they are turned off when not in use.
If I buy a reserved instance for t3.medium, how does AWS know to apply it to the instances running 24/7 and not part time?
For example, we have 10 t3.medium ec2 instance. Let's say that there
are 6 always running 24/7, but the other 4 are not running 24/7, they
are turned off when not in use.
If I buy a reserved instance for t3.medium, how does AWS know to apply
it to the instances running 24/7 and not part time?
Amazon doesn't apply reserved pricing to a specific instance at all. It basically just applies reserved pricing to your bill. It's like a discount at the time your bill is processed. If you have reserved pricing for N instances, and you have at least N instances running 24/7 reflected in your bill, then the reserved pricing gets applied to those N instances.
Amazon doesn't really care if you are running a single specific instance 24/7, or deleting and recreating instances once a minute. In the end it's just the total number of seconds you have a specific type of instance running each month that they care about, and bill you for.
Be aware that when you setup instance reservations it is actually a capacity reservation. You are telling Amazon you are committing to running this type of instance 24/7 for either 1 year or 3 years. By letting Amazon know this, it helps with their capacity planning, and in return they give you a cost discount. But it also means that they are going to charge you for that reserved capacity even if you don't actually have that instance running 24/7.
I am trying to use aws spot instances (m5.large in eu-west-2 region) with a maximum bid equal to the price of on demand instances. According to https://aws.amazon.com/ec2/spot/instance-advisor/ these instances should have a < 5% frequency of interruption, however, after launching 40 such instances this morning, I have found that within the hour 34 of them were evicted by aws ("instance-terminated-no-capacity" according to the spot requests page on the ec2 dashboard).
This eviction rate looks much too high compared to both amazon's own advisor and other users experiences. Does anybody know what could be causing this behaviour, if there is any better way to debug it or predict it, or if this is just what I should expect from spot instances?
Thank you!
Actually, for m5.large instance in eu-west-2 region(Oregon) it's 5%-10% frequency of interruption, so you can expect a max of 10%. I'm not saying that issue you are facing is because of this.
AWS terminates your spot instances because of any of these reasons,
The Spot price is above the maximum price.
There isn't enough capacity.
Amazon EC2 can't meet the constraints you placed on your Spot request.
In your case, since you are seeing instance-terminated-no-capacity message it is definitely because of the second reason. Since you've asked for 40 such instances, the amazon spot instance pool might not have enough capacity at that time.
The capacity of available spot instances pool depends on the demand for regular instances, and when users ask for regular on-demand instances, AWS will start terminating spot instances to fulfil those requests if there is not enough capacity
In my experience spot interruption is highly variable, and for some instances more or less likely at different times of the day.
If you need 40 instances and they do not need to be in the same availability zone (AZ) to each other you might reduce the chance of a mass interruption of all/most instances if you spread the machines across different AZs within the region as each availability zone has its own pool of machines. Although you will likely increase the chance that some machines will be interrupted.
Note this is not an option if you are using EMR, then they have to be in the same AZ.
I have a fresh AWS account with Reserved Capacity of 15 EC2 instances of the same instance type in the same region primarily intended for EMR.
If I spawn a EMR cluster with 10 instances for 15 days and scale it up to 20 instances for the remaining 15 days, will I get charged $0?
If not, how will the final bill for EC2 for the month be calculated?
The reason why I am confused is that based on my understanding EC2 charges based on capacity consumed (in instance hours) and not on actual instances allocated. Since I'm reserving 15 instances worth of capacity, I would expect it to cost me a total of $0 since my overall monthly average consumption is worth 15 instances.
The Reserved EC2 Pricing documentation and most of the AWS price-calculation related material on the internet don't mention the use-case of dynamic scaling of instances in accounts with reserved capacity.
Reserved instances are applied to running instances in real time, not averaged over any longer period of time. They are baseline, not average.
For example, if you own three Reserved Instances with the same instance attributes and region (or Availability Zone if applicable), the billing system checks each hour to see how many total instances you have running that match those parameters. If it is three or less, you will be charged the Reserved Instance rate for each matching instance running that hour. If more than three are running, you will be charged the On-Demand rate for the additional instances.
https://aws.amazon.com/ec2/pricing/reserved-instances/buyer/
At any instant in time when you lack enough reserved capacity to cover your running workload, you're billed for the excess.
If I have 2 equal EC2 instances running simultaneously for 12 h a day and I have only 1 reserved instance, does this reserved instance cover the cost of both instances since the sum of daily hours is 24h? Or however, since the 2 works simultaneously, would not the reserved one cover me and would they apply extra expenses? Thanks in advance.
According to this document from rackspace, you would be billed for 1 reserved instance, and 1 on-demand instance when it is running:
Example 4: Oversubscribing your Reserved Instances for half the time:
Assume that you have purchased 2 Reserved Instances. You decide to
launch 4 instances for only 12 hours per day, and Terminate or Stop
them for the other 12 hours. In this scenario, you will be charged an
additional 24 hours (12 hours x 2 instances) at the On-Demand rate, as
you already have 2 instances consuming your Reserved Instance
reservations and any additional instances running at the same time are
charged at the On-Demand rate. You cannot store unused Reserved
Instance hours and use them to oversubscribe at other times. If you
have workloads that need predictable scaled capacity, you may consider
want to consider a purchase of Scheduled Reserved instances instead:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-scheduled-instances.html
from: https://manage.rackspace.com/aws/docs/product-guide/reserved_instances.html
Can AWS Reserved Instances be used in OpsWorks?
If so, how? Any documentation would be great.
Thanks.
Yes, because a reserved instance is just a billing mechanism, not a specific machine - when you buy a reserved instance you are just agreeing ahead of time to pay a certain amount for a certain type of machine (whether you use it or not).
If you already have a reserved instance 'under contract' and have not yet spun it up, if you use Opsworks to spin it up, you will be charge your reserved price while that instance is running.
If you have 3 reserved instances, and you spin up 4 in Opsworks, you will be charge 3 instances at the reserved price, and one at the on demand price.
If you buy 5 reserved instances and only spin up 2 in Opsworks, you still are on the hook for all 5 reserved instances.