I have recently explored AWS DynamoDB and was reading about Read/Write Capacity Units. I understood that they are as under:
WCU: A write capacity unit represents one write per second, for
an item up to 1 KB in size.
RCU: A read capacity unit represents
one strongly consistent read per second, or two eventually consistent
reads per second, for an item up to 4 KB in size.
So, my table has 1 WCU and 1 RCU. This means I would be charged for every read and every write I do to my table. Good so far. Plus I would be charged additionally for data storage.
However, when I look at this link (scroll down to DynamoDB detailed feature pricing/Read and Write Requests), it shows me WCU and RCU in Price per hour which is $0.00065 per WCU or $0.00013 per RCU.
What is the meaning or Price per hour?
Would I be wrong to assume $0.00065 per WCU or $0.00013 per RCU and ignore the hour part completely? Meaning, per write would cost me $0.00065 and per read would cost me $0.00013.
There are two pricing models for DynamoDB: Provisioned and On-Demand.
Provisioned capacity means that you will be charged each hour for the provisioned unit whether you consume it or not -- and your requests will be throttled if you go over provisioned capacity in a given second. This model allows for considerable savings when you have stable demand and you have a method within your solution to deal with throttles gracefully.
If you are in the dev mode, or your application has unpredictable peaks, you should consider On-Demand mode where you will be billed a flat rate per request irrespectively of the rate of requests.
Hope this helps.
Would I be wrong to assume $0.00065 per WCU or $0.00013 per RCU and ignore the hour part completely? Meaning, per write would cost me $0.00065 and per read would cost me $0.00013.
Yes, for this configuration you would be charged as follows:
Writes
$0.00065 * 730 = $0.4745 per month
Reads
$0.00013 * 730 = $0.0949 per month
In provisioned mode you pay for the capacity regardless of how many requests you make. Having 1WCU and 1RCU will allow you to write 1 * 1KB items per second and read 1 * 4KB per second. Moreover, DynamoDB free tier allows you 25WCU and 25RCU per region per month. You can apply this capacity to a single table or multiple tables as you wish. This is enough capacity to provide 200M requests per month.
In contrast, On-demand mode allows you to pay per request, however it's not as cost effective as provisioned mode. It simplifies using DynamoDB without needing to think about setting capacity or auto-scaling. It does not have a free tier.
DynamoDB free tier also provides 25GB of storage per month per region.
Have a look over the Docs… it's filled with valuable information to help you understand the difference in capacity modes.
Related
I created an on-demand DynamoDb table, and as I know that Dynamodb automatically scales the write/read capacity on on-demand mode.
But AWS Glue job gives error as "An error occurred while calling o201.pyWriteDynamicFrame. DynamoDB write exceeds max retry 10" because of the write capacity. How is this possible if the table is on on-demand mode? I didn't set any read/write capacity and and the table isn't even on the provisioned mode.
Dynamodb Table:
AWS Glue job output:
Dynamodb Tabble Throttled:
Thanks.
Here is what you need to know about On-Demand mode tables
On Demand
If you recently switched an existing table to on-demand capacity mode for the first time, or if you created a new table with on-demand capacity mode enabled, the table has the following previous peak settings, even though the table has not served traffic previously using on-demand capacity mode:
Following are examples of possible scenarios.
A provisioned table configured as 100 WCU and 100 RCU. When this table is switched to on-demand for the first time, DynamoDB will ensure it is scaled out to instantly sustain at least 4,000 write units/sec and 12,000 read units/sec.
A provisioned table configured as 8,000 WCU and 24,000 RCU. When this table is switched to on-demand, it will continue to be able to sustain at least 8,000 write units/sec and 24,000 read units/sec at any time.
A provisioned table configured with 8,000 WCU and 24,000 RCU, that consumed 6,000 write units/sec and 18,000 read units/sec for a sustained period. When this table is switched to on-demand, it will continue to be able to sustain at least 8,000 write units/sec and 24,000 read units/sec. The previous traffic may further allow the table to sustain much higher levels of traffic without throttling.
A table previously provisioned with 10,000 WCU and 10,000 RCU, but currently provisioned with 10 RCU and 10 WCU. When this table is switched to on-demand, it will be able to sustain at least 10,000 write units/sec and 10,000 read units/sec.
Important
If you need more than double your previous peak on table, DynamoDB automatically allocates more capacity as your traffic volume increases to help ensure that your workload does not experience throttling. However, throttling can occur if you exceed double your previous peak within 30 minutes. For example, if your application’s traffic pattern varies between 25,000 and 50,000 strongly consistent reads per second where 50,000 reads per second is the previously reached traffic peak, DynamoDB recommends spacing your traffic growth over at least 30 minutes before driving more than 100,000 reads per second.
Above information is directly from AWS Docs src
Glue Workers
Now, when you begin to write using AWS Glue, you will very quickly exceed the 4000 WCU limit, which means you have exceeded the rule which is double your previous peak (4000) within 30 minutes.... So what now??
Pre-warming your table
DynamoDB provides you capacity in the form of partitions, where each partition is capable of providing you 1000 WCU and 3000 RCU. DynamoDB only ever scales partitions out, never merging in.
For that reason, we can "pre-warm" our DynamoDB tables by creating them in Provisioned-mode and allocating our peak WCU. For eg. let's imagine we expect Glue to consume 40,000 WCU, then we will be sure our table can handle that following these steps:
Create table in provisioned mode
No Autoscaling
40,000 WCU
40,000 RCU
When table is marked as Active (1-2 mins)
Switch capacity mode to On-Demand
Now, you have a new DynamoDB table in On-Demand which is capable of providing 40,000 WCU out of the gates, not the 4,000 WCU provided by default. This will eliminate throttling from Glue.
DynamoDB sets read/write capacity for its on-demand tables in order to balance performance and cost. The read/write capacity units determine the rate at which DynamoDB can read and write data to the table, with a larger number of units allowing for a higher rate of read/write operations. By setting these values, users can control the performance of their DynamoDB table and ensure that it meets the demands of their application. Additionally, setting the capacity units helps DynamoDB automatically manage the distribution of data and traffic, ensuring low latency and high reliability.
I create a dynamodb table with On-demand capacity mode. I can use ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits metrics to view consumed unites of the table or GSI. But how can I know what the capacity the table is provisioned at a given time?
I think you are confusing between on-demand pricing and auto-scaling:
In on-demand, you pay per request, and there is no "provisioned capacity" at all - in theory (although perhaps not guaranteed in practice) your workload can do 10 requests a second one minute, and 10,000 a second in the next minute. In auto-scaling, there is a provisioned capacity for which you pay, but DynamoDB helps you by periodically figuring out the "best" provisioned capacity - something only slightly higher than your current needs.
Amazon have a nice blog post explaining the two features - on-demand (with no provisioned capacity) and auto-scaling (changing the provisioned capacity automatically) - and how they can even be combined to reduce your bills.
Right now I am using on-demand mode for my DynamoDB tables, as I didn't know how much data to expect. But now that the application has run a while, I can see the metrics for ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits for my tables in CloudWatch.
In on-demand mode I pay per request, whereas in provisioned capacity mode I have to pay for the provisioned capacity. If I simply take the metrics for (max) consumed capacity units and compare the prices of those in provisioned capacity mode to my current costs, I believe provisioned capacity mode would be a lot cheaper for me.
My question is, can I simply take the metrics and take the max (plus some buffer) of the consumed capacity units and configure them as provisioned capacity, or is that an error in reasoning on my part?
There are two other things you need to consider:
How 'bursty' is your throughput?
Are you using SDKs to connect to your database?
Setting your provision to the maximum throughput you ever see will ensure you don't get throttled requests, however you will probably be setting the provision too high. Dynamodb can actually consume more provision than you have set using Burst Capacity. This will accomodate short bursts of high throughput over the space of 5 minutes. If you see sustained peaks, for example your database is busy in the day but not the night, you might consider setting your tables to Autoscale. In this case you can set the provisioned throughput lower, and Dyanmodb will automatically scale up provision as required. Note that autoscale is good for workloads that vary over the course of hours (e.g. for handling daily peak hours). It's not good for reacting to events that occur in less than about 30 minutes.
If you are using official SDKs, they will handle throttle responses, and retry any failed requests. This gives Dynamodb some time to scale without your application failing requests.
I'm newbie to dynamodb, I have just 10 items and 1 global secondary Index with Read/write capacity units 5(minimum) which cost around $6/2days which is unacceptable for me because I have used only 0.01% of my actual needs I have gone through some documentation in aws dynamodb price reduction non helped me, because AWS recommend to avoid sudden spike read using query or scan, which is impossible to get more than one item with Partition key alone.
Tables as follows
Add_Employee
Add_Stocks
Add_vendor
All above table have read/write capacity units 1 and each have one global secondary index read/write capacity units 1. All tables are config within specified region Asia Pacific (Mumbai)
Here is my billing for reference
$0.00 per hour for 25 units of read capacity for a month (free tier)18,600 ReadCapacityUnit-Hrs $0.00
$0.00 per hour for 25 units of write capacity for a month (free tier)18,600 WriteCapacityUnit-Hrs $0.00
$0.000148 per hour for units of read capacity beyond the free tier 6,723 ReadCapacityUnit-Hrs $1.00
$0.00074 per hour for units of write capacity beyond the free tier 6,723 WriteCapacityUnit-Hrs $4.98
Thanks in advance
You're not just paying for actual throughput, you're paying for provisioned throughput.
Looking at The Dynamo cost page, this means you are paying $0.0065 per throughput-hour each table exists per month, minus the free-tier hours.
Based on your table names, I'm guessing you are not following the best practice of using 1 de-normalized table table for everything. You may be better off using an RDS instance, which will not charge by the table, but by the hour (it's an EC2 instance behind the scenes).
Cost Breakdown
The default is 5 provisioned read/write units, and there are 720 hours in a 30-day month
$0.0065 * 5 * 720 = $24.37 a month per table
The free tier generally allows one table for free a month.
Per AWS docs you must have at least 1 provisioned unit.
How to Save
Make sure you're following the best practice of using 1 de-normalized table
For any dev work, make sure both read and write provisions are set to 1 ($0.0065 * 1 * 720 = $4.68 a month per table)
If you know you're going to be away for a while, remove the stack from AWS. You're only charged while the table(s) exists.
By limiting read/write units you should be able to bring the cost down to ~$5.00 a table per dev.
DO NOT TURN ON AUTO-SCALING
A commenter suggested auto-scaling. Per docs, you'll be charged for at least 5 units, which is what you are paying now.
This AWS forum link is about the same thing.
In the answer to "How is Amazon DynamoDB throughput calculated and limited?" it's been suggested, that DynamoDB throttles request whenever you exceed provisioned throughput on per second basis. However, this contradicts my experience.
I've table where I post multiple rows, often the number of rows way exceeding provisioned write capacity. This happens in short bursts. At one point I've even got 5 minutes average above provisioned capacity. OTOH, 15 minutes average is below capacity. I haven't got any throttled request in that period.
5 minutes average peaks at 8.053 with provisioned capacity of 6:
15 minutes average peaks well below provisioned capacity:
So when does DynamoDB throttle requests? What kind of average does it take in account? How high above provisioned capacity can the burst be before it gets throttled?
DynamoDB is designed to ensure that your provisioned capacity is available on a per-second basis. If you provision a table for ten 1kB reads per second then DynamoDB will give you enough capacity to handle that throughput rate. In addition, DynamoDB will sometimes allow you to achieve limited bursting above your provisioned throughput for a short period of time. This is intended to absorb natural variations in customer workloads. This bursting is not guaranteed and it is not always available (and the nature of the available bursting may change over time). As is currently described in the best practices documentation, in order to get the best performance you should have an evenly distributed workload that does not exceed your provisioned capacity and distributes the load evenly over the key space. However, if the reality of production behavior for your application deviates from an evenly distributed workload then DynamoDB may absorb some of the bursts.
As for how much to provision your table, it depends a lot on your workload. You could start with provisioning to something like 80% of your peaks and then adjust your table capacity depending on how many throttles you receive (which you can see in your CloudWatch graphs) and your application’s tolerance for latency induced by retries. Keep in mind that DynamoDB does not allow unlimited bursts above your provisioned capacity. You may be able to absorb short bursts but you cannot sustain a throughput rate above your provisioned capacity level for an extended period of time. The general guidance we can give is to provision for something close to your peaks and then dial down while watching for throttles.
This answer was posted in AWS forums
Disclaimer: I work for Amazon, DynamoDB team.
There's a hint in the DynamoDB documentation that explains how bursting works:
When you are not fully utilizing a partition's throughput, DynamoDB retains a portion of your unused capacity for later bursts of throughput usage. DynamoDB currently retains up five minutes (300 seconds) of unused read and write capacity.
But it also says that you cannot rely on this behavior:
However, do not design your application so that it depends on burst capacity being available at all times: DynamoDB can and does use burst capacity for background maintenance and other tasks without prior notice.
At least that would explain why it was possible to have a 5 minute average above the provisioned capacity. With the explanation above, it would even be possible to have 15 minute averages (or longer timespans) to be above the provisioned capacity, if you have a spike in the very beginning of the interval and less usage within the 300 seconds before the start of the interval.
DynamoDB provides some flexibility in your per-partition throughput provisioning by providing burst capacity. Whenever you're not fully using a partition's throughput, DynamoDB reserves a portion of that unused capacity for later bursts of throughput to handle usage spikes.
DynamoDB currently retains up to 5 minutes (300 seconds) of unused read and write capacity. During an occasional burst of read or write activity, these extra capacity units can be consumed quickly—even faster than the per-second provisioned throughput capacity that you've defined for your table.
DynamoDB can also consume burst capacity for background maintenance and other tasks without prior notice.