Correct me if I'm wrong, but there seem to be some inconsistencies between creating sport block requests using EC2 console and AWS SDK (Boto3, namely). When requesting a spot block using AWS Management console, the only pricing option is "Use automated bidding".
However when doing the same via Boto3, SpotPrice parameter is marked as required with no indication that it might represent, say, a percentage of on-demand price.
Is there any option to use automated bidding programmatically without hard-coding on-demand prices in the requests?
The console is merely trying to present a simplified process. I think it is simply setting SpotPrice to the on-demand price. That's a much cleaner interface than asking for a different price per selected Instance Type.
You always only pay the current Spot Price. Bidding is always automatic up to your Spot price, which represents the maximum you are willing to pay.
If you want to make an equivalent bid without hard-coding the On-Demand Price, you could use the AWS PriceList API, which is really just some downloadable JSON/CSV files with pricing information. Pricing doesn't change very often, so you could cache that information and occasionally refresh it.
Because you choose Reserved for duration. Automatic bidding is the only way you can do this.
IMHO, you should consider your SPOT requirement before jump into reserved for duration. If your application is spot instance ready , then you should specify a fleet of instance with your desire minimum price. Because AWS Spot always has under utilised spare instance, in fact, this minimise the interruption without even the need of reservation.
Perhaps due to c4.* pricing and cause many people move from c3.* to c4., it seems c3. pricing is all time low (e.g. us-east-1* show a price below $0.02)
Related
I need to attach a fixed number of spot block instances as core nodes to the EMR cluster at my job. The reason we're going with spot block instances is because our Spark jobs are pretty much deterministic in terms of execution time. I'm using the boto3 EMR client apis for spawning and killing EMRs. The only unknown part for me is how the bidding happens for spot blocks. AWS docs have a price chart here for those instance types, but I can't find any information or apis for the accessing the bidding prices, similar to the ones present for normal spot instances.
The end goal is to find out the optimal bidding price, but I don't have any info rather than the static price chart. For the time being, I've set the bidding price to be 70% of the on-demand price using BidPriceAsPercentageOfOnDemandPrice. Any help is appreciated.
I haven't seen any data feeds for Spot Blocks. I suspect they are heavily dependent upon the current workloads and probably aren't used as much as normal Spot instances. The price would also vary based upon duration.
Please note that, at the end of a Spot Block period, the instances are terminated.
An alternative is to use normal Spot instances, but include a mix of instance types to reduce the likelihood of losing all of them.
These days, Spot instance can be terminated if capacity is reduced, even if the Spot price doesn't increase. This has resulted in a much smoother Spot price, but there is no guarantee of spot capacity even at the current Spot price.
Since Spot Blocks are more expensive than normal Spot instances, I'd suggest simply going for normal Spot on a couple of different instance types.
I need to make a cost model for AWS vs GCP. Currently, our organization is using AWS. Our biggest services used are:
EC2
RDS
Labda
AWS Gateway
S3
Elasticache
Cloudfront
Kinesis
I have very limited knowledge of cloud platforms. However, I have access to:
AWS Simple Monthly Calculator
Google Cloud Platform Pricing Calculator
MAP AWS services to GCP products
I also have access to CloudHealth so that I can get a breakdown of costs per services within our organization.
Of the 8 major services listed above are main usage and costs go to EC2, S3, and RDS.
Our director of engineering mentioned that I should be most concerned with vCPU and memory.
I would appreciate any insight (big or small) that people have into how I can go about creating this model, any other factors I should consider, which functionalities of the two providers for the services are considered historically "better" or cheaper, etc.
Thanks in advance, and any questions people may have, I am more than happy to answer.
-M
You should certainly cost-optimize your resources. It's so easy to create cloud resources that people don't always think about turning things off or right-sizing them.
Looking at your Top 5...
Amazon EC2
The simplest way to save money with Amazon EC2 is to turn off unused resources. You can even stop instances overnight and on the weekend. If they are only used 8 hours per workday, then that is only 40 out of 168 hours, so you can save 75% by turning them off when unused! For example, Dev and Test instances. People have written various types of automated utilities to turn instances on and off based on tags. Try search the Internet for AWS Stopinator.
Another way to save money on Amazon EC2 is to use spot instances. They are a fraction of the price, but have a risk that they might be turned off when demand increases. They are great where it is okay for systems to be terminated sometimes, such as automated testing systems. They are also a great way to supplement existing capacity at a fraction of the price.
If you definitely need the Amazon EC2 instances to keep running all the time, purchase Amazon EC2 Reserved Instances, which also offer a price saving.
Chat with your AWS Account Manager for help with the above options.
Amazon Relational Database Service (RDS)
Again, Amazon RDS instances can be stopped overnight/on weekends and turned on again when needed. You only pay while the instance is running (plus storage costs).
Examine the CloudWatch metrics for your RDS instances and determine whether they can be downsized without impacting applications. You can even resize them when they are used less (eg over weekends). Everything can be scripted, so you could trigger such downsizing and upsizing on a schedule.
Also look at the Engine used with RDS. Commercial offerings such as Oracle and Microsoft SQL Server are more expensive than open-source offerings like MySQL and PostgreSQL. Yes, your applications might need some changes, but the cost savings can be significant.
AWS Lambda
It is most unusual that Lambda is #3 in your list. In fact, some customers never get a charge for Lambda because it falls in the monthly free usage tier. Having high charges means you're making good use of Lambda (which is saving you EC2 costs), but take a look at which applications are using it the most and see whether they are using it wisely.
When correctly used, a Lambda function should only ever run for a few seconds, so check whether any application seem to be using it outside this pattern.
AWS API Gateway
Once again, these costs tend to be low ($3.50/million calls) so again I'd recommend trying to figure out how this is being used. If you really need that many calls, it would also explain the high Lambda costs. It would probably be more expensive if you were providing such functionality via Amazon EC2.
Amazon S3
Consider using different Storage Classes to reduce your costs. Costs can be reduced by:
Moving infrequently-accessed data to a different storage class
Moving data to One-Zone (if you have a copy of the data elsewhere, so don't need the redundancy)
Archiving infrequently-accessed data to Amazon Glacier, which offers much cheaper storage but does not have instant access
With GCP, you can benefit by receiving discounts such as the Committed Use Discount and the Sustained Use Discount.
With a Committed Use Discount, you can receive a discount of up to 70% if your usage is predictable.
With the Sustained Use Discount, there is an incremental discount if you reach certain usage thresholds.
On your concern with vCPU and memory, you may use predefined machine types. They are cheaper than custom machine types.
Lastly, you can also test the charges by trying out the Google Cloud Platform Free Tier.
A long time ago, there was the most useful spot price comparison graph that I have ever used, but it stopped working, as far as I know, because the creator ran out of time to maintain it. The website is still active ec2price.com and the code is on Github. Does anyone know if anyone has replicated this? or any way to do it myself? As I said it was really useful to decide which spot instance to choose.
You can see this information in the EC2 console by browsing to Spot Requests and selecting the Pricing History button.
If you want to select the cheapest ec2 instance type automatically you can create a spot fleet request; select all the instance types you might want to use and an allocation strategy of lowestPrice. Deploy this to a VPC with a subnet in all availability zones in your region to get the lowest price possible.
Besides codes, somebody must pay to maintain server that polling the information.
Check out How Spot Fleet Works. Spot fleet is way better than price monitoring. You can make request base on pricing for a fleet of instance type than limit yourself to specific instance type. You can kick start instances from a large fleet base on maximum instance price or vCPU price.
If you are using a SPOT ready batch application, after submit your bidding for fleet of different instance type and set the maximum per vCPU price, Spotfleet will automatically launch available instance with the best price. So you don't need to compete with limited popular instance(for example, c4.* SPOT instance is scarce for most region).
This is a win-win for both AWS and customer, as AWS able to spread the usage to underutilized instance type. IMHO, There is no point of keep raising the bid for particular type if those instance exhausted, while there is still many idle AWS instances in alternate zone that are not fully utilised for grab.
I am currently using spot instances managed with auto-scaling groups. However, ASG has a number of shortcomings for use with spot instances. For example, it cannot launch instances of a different instance type if the current type is experiencing a price spike across all availability zones. It can't even re-distribute the number of running instances across zones (if one zone has a price spike, you're down 30% in the number of running instances.)
Are there any software solutions that I could run which would replace built-in AWS Auto-Scaling Groups? I've heard of SpotInst and Batchly, but I do not trust them. Basically, I think their business plan involves being bought out and killed by Amazon, like what happened to ClusterK. The evidence for this is the bizarre pricing policies and other red flags. I need something that I can self-host and depend on.
AWS recently released Auto Scaling for Spot Fleets which seems to fit your use case pretty well. You can define the cluster capacity in terms of vCPU that you need, choose the instance types you'd like to use and their weights and let AWS manage the rest.
They will provision spot instances at their current market price up to a limit you can define per instance type (as before), but integrating Auto Scaling capabilities.
You can find more information here.
https://aws.amazon.com/blogs/aws/new-auto-scaling-for-ec2-spot-fleets/
It's unlikely that you're going to find something that takes into account everything you want. But because everything in Amazon is an API, so you can write that yourself. There are lots of ways to do that.
For example, you could write a small script (bash, ruby, python etc) that shells out the AWS CLI to get the price, then shells out to launch boxes. For bonus points, use the native AWS SDK library instead of shelling out. (That will be slightly easier to handle errors, etc.) For even more bonus points, open source it, and hope that other people to improve on it!
This script can run on your home computer, or on a t1.micro for $5/month. Or you could write it in node.js, and run it on Lambda for pennies per month.
Here at Spotinst, these are exactly the problems we built Elastigroup to solve.
Elastigroup enables running simultaneously on as many instance types and availability zones (within a region) as you’d like. This is coupled with several things to maintain production availability:
Our algorithm makes live choices for the best Spot markets in terms of price and availability.
When an interruption happens, we predict it about 15 minutes in advance and take all the necessary steps to ensure (and insure) the capacity of your group.
In the extreme case that none of the markets have Spot availability, we simply fall back to an on-demand instance.
We have a great relationship with AWS and work closely with both their technical and business teams to provide our joined customers with the best experience possible. As we manage resources inside your own AWS account, I wouldn’t put the relationship between us as a concern, to begin with.
I have a micro instance of EC2 that I am currently paying for using pay-as-you-go. I am thinking about transferring this instance to a reserved instance (upfront fee + lower hourly charge). I read through the documentation here, but am still unclear about a couple of things.
First, will I be able to seamlessley 'transfer' my current pay-as-you-go instance onto the reserved instance platform without having to move my snapshot onto a new server? Also, does the 'Reserved Instance Type' only refer to the down payment + hourly rate options you have and not have anything to do with the actual performance of the box? From what I've read, I should choose the 'Heavy Utilization' option since the box will be running a site that is up 24/7.
You do not have to move your server or make any change when buying a reserved instance. You simply need to select an RI that is the same size and in the same region and availability zone as your current system. Heavy utilization is the best choice for you if your system runs 24/7. If you don't think you'll need it for a full year, consider buying from the marketplace where you may be able to buy in some increment lower than 1 year.
Reserved instances are just a billing feature. All you need to do is create a reservation that matches the parameters of your current instance.
It breaks down like this:
You just purchase an RI, and if it happens to match a running instance (same Region, Zone and Instance Type), you get a discount when running the instance.
Multiple RIs will match multiple instances.
If you're not running a "matching" instance, you don't get a discount.
The RI type (Heavy/Medium/Light) is just a trade-off on up-front cost vs discount cost. If you're running 24x7, get the heavy, it will save you the most money. If you don't want to invest a lot now, you can get the light and pay a little more, but loose less if you stop your instance.