I would like to know the best solution.
I have an instance of t3.medium type running 6 hours a day.
Does it make sense for me to buy t3.nano type reserved instance, if so, how many instances? Or does it not pay to buy a reserved instance?
From a purely mathematical viewpoint, in US regions a t3.medium Linux instance would cost:
On-Demand: $0.0416 per Hour x 6 hours per day x 5 days per week x 52 weeks = $64.896 per year (Or ~$90 if 7 days per week)
1-Year upfront Reserved instance: $213 per year
3-Year upfront Reserved instance: $412 = $137 per year
So, the cheapest option is On-Demand.
An alternative is a Scheduled Reserved Instance, which "are a good choice for workloads that do not run continuously, but do run on a regular schedule." However, it seems that this option has been removed from the Management Console in some regions.
A Reserved Instance also includes a capacity reservation in case of capacity constraints, which makes it attractive beyond merely price.
The maximum discount you can get is ~62% for 3 year, 100% upfront. Since you running the instance for only 6 hours / day, it makes no financial sense to reserve your instance.
Related
Introduction
We are trying to "measure" the cost of usage of a specific use case on one of our Aurora DBs that is not used very often (we use it for staging).
Yesterday at 18:18 hrs. UTC we issued some representative queries to it and today we were examining the resulting graphs via Amazon CloudWatch Insights.
Since we are being billed USD 0.22 per million read/write IOs, we need to know how many of those there were during our little experiment yesterday.
A complicating factor is that in the cost explorer it is not possible to group the final billed costs for read/write IOs per DB instance! Therefore, the only thing we can think of to estimate the cost is from the read/write volume IO graphs on CLoudwatch Insights.
So we went to the CloudWatch Insights and selected the graphs for read/write IOs. Then we selected the period of time in which we did our experiment. Finaly, we examined the graphs with different options: "Number" and "Lines".
Graph with "number"
This shows us the picture below suggesting a total billable IO count of 266+510=776. Since we have choosen the "Sum" metric, this we assume would indicate a cost of about USD 0.00017 in total.
Graph with "lines"
However, if we choose the "Lines" option, then we see another picture, with 5 points on the line. The first and last around 500 (for read IOs) and the last one at approx. 750. Suggesting a total of 5000 read/write IOs.
Our question
We are not really sure which interpretation to go with and the difference is significant.
So our question is now: How much did our little experiment cost us and, equivalently, how to interpret these graphs?
Edit:
Using 5 minute intervals (as suggested in the comments) we get (see below) a horizontal line with points at 255 (read IOs) for a whole hour around the time we did our experiment. But the experiment took less than 1 minute at 19:18 (UTC).
Wil the (read) billing be for 12 * 255 IOs or 255 ... (or something else altogether)?
Note: This question triggered another follow-up question created here: AWS CloudWatch insights graph — read volume IOs are up much longer than actual reading
From Aurora RDS documentation
VolumeReadIOPs
The number of billed read I/O operations from a cluster volume within
a 5-minute interval.
Billed read operations are calculated at the cluster volume level,
aggregated from all instances in the Aurora DB cluster, and then
reported at 5-minute intervals. The value is calculated by taking the
value of the Read operations metric over a 5-minute period. You can
determine the amount of billed read operations per second by taking
the value of the Billed read operations metric and dividing by 300
seconds. For example, if the Billed read operations returns 13,686,
then the billed read operations per second is 45 (13,686 / 300 =
45.62).
You accrue billed read operations for queries that request database
pages that aren't in the buffer cache and must be loaded from storage.
You might see spikes in billed read operations as query results are
read from storage and then loaded into the buffer cache.
Imagine AWS report these data each 5 minutes
[100,150,200,70,140,10]
And you used the Sum of 15 minutes statistic like what you had on the image
F̶i̶r̶s̶t̶,̶ ̶t̶h̶e̶ ̶"̶n̶u̶m̶b̶e̶r̶"̶ ̶v̶i̶s̶u̶a̶l̶i̶z̶a̶t̶i̶o̶n̶ ̶r̶e̶p̶r̶e̶s̶e̶n̶t̶ ̶o̶n̶l̶y̶ ̶t̶h̶e̶ ̶l̶a̶s̶t̶ ̶a̶g̶g̶r̶e̶g̶a̶t̶e̶d̶ ̶g̶r̶o̶u̶p̶.̶ ̶I̶n̶ ̶y̶o̶u̶r̶ ̶c̶a̶s̶e̶ ̶o̶f̶ ̶1̶5̶ ̶m̶i̶n̶u̶t̶e̶s̶ ̶a̶g̶g̶r̶e̶g̶a̶t̶i̶o̶n̶,̶ ̶i̶t̶ ̶w̶o̶u̶l̶d̶ ̶b̶e̶ ̶(̶7̶0̶+̶1̶4̶0̶+̶1̶0̶)̶
Edit: First, the "number" visualization represent the whole selected duration, aggregated with would be the total of (100+150+200+70+140+10)
The "line" visualization will represent all the aggregated groups. which would in this case be 2 points (100+150+200) and (70+140+10)
It can be a little bit hard to understand at first if you are not used to data points and aggregations. So I suggest that you set your "line" chart to Sum of 5 minutes you will need to get value of each points and devide by 300 as suggested by the doc then sum them all
Added images for easier visualization
I ran a query which resulted in the below stats.
Elapsed time: 12.1 sec
Slot time consumed: 14 hr 12 min
total_slot_ms: 51147110 ( which is 14 hr 12 min)
We are on an on-demand pricing plan. So the max slots would be 2000. That being said, if I used 2000 slots for the whole 12.1 seconds span then I should end up with total_slot_ms as 24200000 ( which is 2000x12.1x1000). However, the total_slot_ms is 51147110. Average number of slots used are 51147110/121000 = 4225 ( which is way above 2000). Can some explain to me how I ended up using more than 2000 slots?
In a course of Google, there is an example where a query shows 13 "elapsed time" seconds and 50 minutes of "slot time consumed". They says:
Hey, across all of our workers, we did essentially 50 minutes of work massively in parallel, 50 minutes so that your query could be returned back in 13 seconds. Best of all for you, you don't need to worry about spinning up those workers, moving data in-between them, making sure they're sharing all their results between their aggregations. All you care about is writing the SQL, finding the insights, and then running that query in a very fast turnaround. But there is abstracted from you a lot of distributed parallel processing that's happening.
Increasing Bigquery slot capacity significantly improves overall query performance, despite the fact that slots amount is actually the subject for Quotas restriction along Bigquery on-demand pricing plan, exceeding slots limit does not charge you for additional costs:
BigQuery slots are shared among all queries in a single project.
BigQuery might burst beyond this limit to accelerate your queries.
To check how many slots you're using, see Monitoring BigQuery using
Cloud Monitoring.
BigQuery on-demand supports limited bursting. https://cloud.google.com/bigquery/docs/release-notes#December_10_2019
You might want to check execution plan for the query and understand all different slot_time_ms for wait, read, write activities at each stage. Since this is on-demand slots, you may see lots of wait time, that will add up into total time.
Besides bursting, each stage of explain pan will help you understand that total time is not necessarily actual slot consumption but equivalent slot consumption.
I am using a T2.medium instance. A third of the day I am doing intensive statistical calculations and figured that the rest 2/3 of the time I would "earn" credits at a rate at 24 per hour.
But that is not happening. This is my usage the last two days:
And this is my credit account:
I hadn´t used it for (more than) a day until yesterday 6 pm. I use it intensive for five hours. Then I would expect my "account" to acummulate 24 credits per hour but for 9-10 hours almost nothing happens, then it acummulate as expected for 9 hours and then goes flat again.
I am unable to figure out what is going on and if it is a fault. Do anyone have a good explanation?
EDIT: I have included a week of activity below. I still can´t figure out the algoritm:
Update: The rules used to calculate t2 CPU credit balances appear to have changed such that the issue prompting this question should no longer have an impact.
Based on customer feedback, we’ve updated T2 instances with a new CPU Credit allocation policy that is the same as or better than the previous policy in all cases.
...
Now, earned CPU Credits do not expire until the instance is terminated or stopped. A T2 instance can still earn up to the same maximum level allowed by the instance size. The CPUCreditBalance will now increase anytime the current CPUCreditUsage is below the baseline and can grow to the maximum allowed for the instance size
https://forums.aws.amazon.com/ann.jspa?annID=5196
h/t: Last Week in AWS for the update.
The original answer follows.
This question has caused me quite a bit of mental anguish over the last few hours, because the graphs almost make sense, based on what I know about t2 instances. Almost, but not quite, and I couldn't put my finger on the problem. That's the worst kind. Particularly being a huge fan of the value proposition offered by t2 machines.
But I did finally figure out what's going on here.
There's one concept of CPU credits the documentation doesn't seem to explain, but the math works out, and the explanation holds up nicely under real-world observations:
The most recently earned CPU credits are spent first, not last.
Does order matter? It does.
For testing, I used a t2.micro (primarily because I had an idle one that had been running for several days, and needed something to do, and I didn't want the extra "initial" credits of a new instance to cloud up the observations) but all instance types in the t2 class have similar behavior.
By way of background: in the t2 class, CPU credits are earned at different rates, but CPU credits are used at the same rate for all instance types in the class:
A CPU Credit provides the performance of a full CPU core for one minute.
The t2.micro and t2.small have only one core, so they can burn up to 1 credit per minute or 60 credits per hour, at 100% CPU utilization. The t2.medium and t2.large are dual core, so they can burn up to 2 credits per minute, or 120 credits per hour, at 100% CPU utilization on both cores.
If 1 credit = 100% of 1 core for 1 minute, then 1 credit is also equal to 20% of 1 core for 5 minutes. Since the Cloudwatch graph interval is in 5 minute increments, I set up the following test:
On a t2.micro that has been running for several weeks with essentially no load, I installed lookbusy, a handy utility that allows you to make a machine "look busy" with parameters you specify -- e.g, keep the CPU at 20% utilization.
$ screen -S eat_cpu
$ ./lookbusy -v -c 20 -r fixed
This does exactly what you'd expect, burning 1 CPU credit every 5 minutes. The "CPU Credit Usage" graph confirms this, showing 1 credit being used every 5 minutes. (The CPU Utilization graph, and top, both confirm the 20%.)
But what's happening to my credit balance? It's being depleted by 1 credit every 5 minutes. That seems wrong, doesn't it? I mean, yes, I just said that's how many I'm using, but... I'm also supposed to be earning 6 credits per hour, so I should only be depleting by balance by a net of 0.5 credits every 5 minutes, right?
Hold on... checking the numbers, again: I'm earning 6 per hour, spending 12 per hour, so, yes... that seems like it should be a net decrease of only 6 per hour, not 12... right? Clearly, something doesn't add up the way I expected, because my balance is definitely going down by 12 per hour, and my CPU is definitely only running at 20%.
I seem to be earning no credits to offset my usage. How is that possible?
Unless...
Unused earned credits from a given 5 minute interval expire 24 hours after they are earned
Well, 24 hours ago, my instance was completely idle. During that hour, I earned 6 credits that I... didn't (?) use. Am I not using them now? Shouldn't I be?
any expired credits are removed from the CPU credit balance at that time, before any newly earned credits are added
Crud. Could this be related? This hour, I earned 6 new credits. But right before that, I lost 6 credits from 24 hours ago. Then I spent 12 credits this hour... so my balance when down by 6, up by 6, and down by another 12. Well, that explains the -12 change for the hour, but...
Can that be the reason?
I'm a voracious reader of documentation, so I knew about the expiring credits aspect... but I assumed all along that this was nothing more than the reason an idle instance hovers near its maximum balance, and did not have any other significance. How could it? If I have less than the maximum (6 x 24 = 144 for a t2.micro) then how could I have credits the need to expire?
If my credits from 24 hours ago are always counting against me, wouldn't my balance tend toward zero, regardless of what I do?
Unless...
After tossing and turning most of the night while contemplating sliding around piles of imaginary tokens (representing CPU credits) on an imaginary table top (representing time)... I realized that the "expiration" rule would cause exactly the behavior we observe if, counter-intuitively, credits are not spent in the order in which they are earned (FIFO), but rather in the reverse order (LIFO).
Following that line of reasoning, the explanation for what my 20% CPU test is actually doing is this, where the first hour of my test was "hour 0" --
| spends 6+6 credits | expire 6 credits
test | earned this many | earned this many
hour | hours before hour 0 | hours before hour 0
-----+---------------------+--------------------
0 -1, -2 -24
1 -3, -4 -23
2 -5, -6 -22
3 -7, -8 -21
4 -9, -10 -20
5 -11, -12 -19
6 -13, -14 -18
7 -15, -16 -17
And they meet in the middle.
Is this genuine, or am I guessing? I'm not guessing, and here's the evidence:
After 8 hours, my CPU credit usage graph remains solid, still holding steady at 1 credit per 5 minutes, but after the same 8 hours, my CPU credit balance finally begins to deplete at the (slower) rate I originally expected: 0.5 credits every 5 minutes.
Apparently, as I worked backward in time, spending previously earned credits "newest first," I caught up with my old credits that were about to expire, finally reaching the point where I was using them before they had a chance to expire. Now, I have no credits that are 24 hours old, and so no credits are expiring -- so I am no longer losing credits before new credits are earned. I am now able to keep the 6 that I earn per hour, because I used up the old ones, decreasing the net impact to my credit balance to the expected level.
This explains the only reservation I had about the graphs in the question: why, when utilization drops off, does it take so long for the balance to rebound?
The TL;DR answer is this: the balance doesn't rebound immediately, after a burst of heavy utilization, because you still have unused credits from 24 hours prior, which are canceling out your newly-earned credits, until you reach the point in time when you don't have any 24-hour-old unused credits. When that happens, your credit balance increases again.
Leave the instance completely idle for 24 hours and you will eventually see the balance steadily (for the most part) rise to the maximum again, as expected. Anything less than 24 hours completely idle will cause your balance to remain perpetually be somewhere below the max.
My test script eventually depleted my credit balance almost all the way down. When I killed the process eating the CPU, the credit balance began to recover immediately, at the expected rate of 6 credits per hour.
Conversely, when I took a different machine that had seen low utilization for 24 hours, and ran it's CPU to 100% for a few minutes, then took it back to idle, the credits did not begin to accumulate immedately... being offset by old, expiring ones.
Quotes are from http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html.
I am using a T2.medium instance. A third of the day I am doing intensive statistical calculations and figured that the rest 2/3 of the time I would "earn" credits at a rate at 24 per hour.
But that is not happening. This is my usage the last two days:
And this is my credit account:
I hadn´t used it for (more than) a day until yesterday 6 pm. I use it intensive for five hours. Then I would expect my "account" to acummulate 24 credits per hour but for 9-10 hours almost nothing happens, then it acummulate as expected for 9 hours and then goes flat again.
I am unable to figure out what is going on and if it is a fault. Do anyone have a good explanation?
EDIT: I have included a week of activity below. I still can´t figure out the algoritm:
Update: The rules used to calculate t2 CPU credit balances appear to have changed such that the issue prompting this question should no longer have an impact.
Based on customer feedback, we’ve updated T2 instances with a new CPU Credit allocation policy that is the same as or better than the previous policy in all cases.
...
Now, earned CPU Credits do not expire until the instance is terminated or stopped. A T2 instance can still earn up to the same maximum level allowed by the instance size. The CPUCreditBalance will now increase anytime the current CPUCreditUsage is below the baseline and can grow to the maximum allowed for the instance size
https://forums.aws.amazon.com/ann.jspa?annID=5196
h/t: Last Week in AWS for the update.
The original answer follows.
This question has caused me quite a bit of mental anguish over the last few hours, because the graphs almost make sense, based on what I know about t2 instances. Almost, but not quite, and I couldn't put my finger on the problem. That's the worst kind. Particularly being a huge fan of the value proposition offered by t2 machines.
But I did finally figure out what's going on here.
There's one concept of CPU credits the documentation doesn't seem to explain, but the math works out, and the explanation holds up nicely under real-world observations:
The most recently earned CPU credits are spent first, not last.
Does order matter? It does.
For testing, I used a t2.micro (primarily because I had an idle one that had been running for several days, and needed something to do, and I didn't want the extra "initial" credits of a new instance to cloud up the observations) but all instance types in the t2 class have similar behavior.
By way of background: in the t2 class, CPU credits are earned at different rates, but CPU credits are used at the same rate for all instance types in the class:
A CPU Credit provides the performance of a full CPU core for one minute.
The t2.micro and t2.small have only one core, so they can burn up to 1 credit per minute or 60 credits per hour, at 100% CPU utilization. The t2.medium and t2.large are dual core, so they can burn up to 2 credits per minute, or 120 credits per hour, at 100% CPU utilization on both cores.
If 1 credit = 100% of 1 core for 1 minute, then 1 credit is also equal to 20% of 1 core for 5 minutes. Since the Cloudwatch graph interval is in 5 minute increments, I set up the following test:
On a t2.micro that has been running for several weeks with essentially no load, I installed lookbusy, a handy utility that allows you to make a machine "look busy" with parameters you specify -- e.g, keep the CPU at 20% utilization.
$ screen -S eat_cpu
$ ./lookbusy -v -c 20 -r fixed
This does exactly what you'd expect, burning 1 CPU credit every 5 minutes. The "CPU Credit Usage" graph confirms this, showing 1 credit being used every 5 minutes. (The CPU Utilization graph, and top, both confirm the 20%.)
But what's happening to my credit balance? It's being depleted by 1 credit every 5 minutes. That seems wrong, doesn't it? I mean, yes, I just said that's how many I'm using, but... I'm also supposed to be earning 6 credits per hour, so I should only be depleting by balance by a net of 0.5 credits every 5 minutes, right?
Hold on... checking the numbers, again: I'm earning 6 per hour, spending 12 per hour, so, yes... that seems like it should be a net decrease of only 6 per hour, not 12... right? Clearly, something doesn't add up the way I expected, because my balance is definitely going down by 12 per hour, and my CPU is definitely only running at 20%.
I seem to be earning no credits to offset my usage. How is that possible?
Unless...
Unused earned credits from a given 5 minute interval expire 24 hours after they are earned
Well, 24 hours ago, my instance was completely idle. During that hour, I earned 6 credits that I... didn't (?) use. Am I not using them now? Shouldn't I be?
any expired credits are removed from the CPU credit balance at that time, before any newly earned credits are added
Crud. Could this be related? This hour, I earned 6 new credits. But right before that, I lost 6 credits from 24 hours ago. Then I spent 12 credits this hour... so my balance when down by 6, up by 6, and down by another 12. Well, that explains the -12 change for the hour, but...
Can that be the reason?
I'm a voracious reader of documentation, so I knew about the expiring credits aspect... but I assumed all along that this was nothing more than the reason an idle instance hovers near its maximum balance, and did not have any other significance. How could it? If I have less than the maximum (6 x 24 = 144 for a t2.micro) then how could I have credits the need to expire?
If my credits from 24 hours ago are always counting against me, wouldn't my balance tend toward zero, regardless of what I do?
Unless...
After tossing and turning most of the night while contemplating sliding around piles of imaginary tokens (representing CPU credits) on an imaginary table top (representing time)... I realized that the "expiration" rule would cause exactly the behavior we observe if, counter-intuitively, credits are not spent in the order in which they are earned (FIFO), but rather in the reverse order (LIFO).
Following that line of reasoning, the explanation for what my 20% CPU test is actually doing is this, where the first hour of my test was "hour 0" --
| spends 6+6 credits | expire 6 credits
test | earned this many | earned this many
hour | hours before hour 0 | hours before hour 0
-----+---------------------+--------------------
0 -1, -2 -24
1 -3, -4 -23
2 -5, -6 -22
3 -7, -8 -21
4 -9, -10 -20
5 -11, -12 -19
6 -13, -14 -18
7 -15, -16 -17
And they meet in the middle.
Is this genuine, or am I guessing? I'm not guessing, and here's the evidence:
After 8 hours, my CPU credit usage graph remains solid, still holding steady at 1 credit per 5 minutes, but after the same 8 hours, my CPU credit balance finally begins to deplete at the (slower) rate I originally expected: 0.5 credits every 5 minutes.
Apparently, as I worked backward in time, spending previously earned credits "newest first," I caught up with my old credits that were about to expire, finally reaching the point where I was using them before they had a chance to expire. Now, I have no credits that are 24 hours old, and so no credits are expiring -- so I am no longer losing credits before new credits are earned. I am now able to keep the 6 that I earn per hour, because I used up the old ones, decreasing the net impact to my credit balance to the expected level.
This explains the only reservation I had about the graphs in the question: why, when utilization drops off, does it take so long for the balance to rebound?
The TL;DR answer is this: the balance doesn't rebound immediately, after a burst of heavy utilization, because you still have unused credits from 24 hours prior, which are canceling out your newly-earned credits, until you reach the point in time when you don't have any 24-hour-old unused credits. When that happens, your credit balance increases again.
Leave the instance completely idle for 24 hours and you will eventually see the balance steadily (for the most part) rise to the maximum again, as expected. Anything less than 24 hours completely idle will cause your balance to remain perpetually be somewhere below the max.
My test script eventually depleted my credit balance almost all the way down. When I killed the process eating the CPU, the credit balance began to recover immediately, at the expected rate of 6 credits per hour.
Conversely, when I took a different machine that had seen low utilization for 24 hours, and ran it's CPU to 100% for a few minutes, then took it back to idle, the credits did not begin to accumulate immedately... being offset by old, expiring ones.
Quotes are from http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html.
I have a bit of a unique problem here. I currently have two warehouses that I ship items out of for selling on Amazon, my primary warehouse and my secondary warehouse. Shipping out of the secondary warehouse takes significantly longer than shipping from the main warehouse, hence why it is referred to as the "secondary" warehouse.
Some of our inventory is split between the two warehouses. Usually this is not an issue, but we keep having a particular issue. Allow me to explain:
Let's say that I have 10 red cups in the main warehouse, and an additional 300 in the secondary warehouse. Let's also say it's Christmas time, so I have all 310 listed. However, from what I've seen, Amazon only allows one shipping time to be listed for the inventory, so the entire 310 get listed as under the primary warehouse's shipping time (2 days) and doesn't account for the secondary warehouse's ship time, rather than split the way that they should be, 10 at 2 days and 300 at 15 days.
The problem comes in when someone orders an amount that would have to be split across the two warehouses, such as if someone were to order 12 of said red cups. The first 10 would come out of the primary warehouse, and the remaining two would come out of the secondary warehouse. Due to the secondary warehouse's shipping time, the remaining two cups would have to be shipped out at a significantly different date, but Amazon marks the entire order as needing to be shipped within those two days.
For a variety of reasons, it is not practical to keep all of one product in one warehouse, nor is it practical to increase the secondary warehouse's shipping time. Changing the overall shipping date for the product to the longest ship time causes us to lose the buy box for the listing, which really defeats the purpose of us trying to sell it.
So my question is this: is there some way in MWS to indicate that the inventory is split up in terms of shipping times? If so, how?
Any assistance in this matter would be appreciated.
Short answer: No.
There is no way to specify two values for FulfillmentLatency, in the same way as there is no way to specify two values for Quantity in stock. You can only ever have one inventory with them (plus FBA stock)
Longer answer: You could.
Sign up twice with Amazon:
"MySellerName" has an inventory of 10 and a fulfillment latency of 2 days
"MySellerName Overseas Warehouse" has an inventory of 300 and a fulfillment latency of 30 days
I haven't tried by I believe Amazon will then automatically direct the customer to the best seller for them, which should be "MySellerName" for small orders and "MySellerName Overseas Warehouse" for larger quantities.