I am new to DynamoDB, and I have a small in house application, which will be used by my parents for their small business. I just have to keep records of 10 - 20 rows daily, and will have a few edits close to 5 - 10 at max.
WIll I be able to use the Free Tier of Dynamo DB for the same?
I am using Heroku to host my LWC OSS (Node JS) application, which is again a free version. If not then any heads up to any particular type of Database which can fulfil my need.
Will I be able to use the Free Tier of Dynamo DB for the same?
Yes, dependent on the size of the data you want to be inputting & the rate at which you want to input.
Amazon DynamoDB offers a free tier with the following provisions, which is enough to handle up to 200M requests per month:
25 GB of Storage
25 provisioned Write Capacity Units (WCU)
25 provisioned Read Capacity Units (RCU)
Just be aware of the fact that:
25 WCU is 25 writes per second for an item up to 1KB or 5 writes per second for an item up to 5KB etc.
25 RCU is 50 reads per second for an item up to 4KB or 10 reads per second for an item up to 20KB etc.
If your API calls fall within the above criteria, you'll be within the free tier.
The main costed aspects of DynamoDB are how much you read and write to the tables. AWS call them "Read capacity units" (RCU) and "Write capacity units" (WCU).
When you create a DynamoDB table there are many options to choose from, but it's roughly accurate to say that:
One RCU gives you one strongly consistent read request per second
One WCU gives you one standard write request per second
So if you create a standard class table with 1 RCU and 1 WCU (the lowest possible) that would already easily accomodate what you predict you will need.
According to the AWS DynamoDB pricing page you can get 25 WCUs and 25 RCUs in the free tier.
So I would say choose DynamoDB standard class table, with Provisioned Capacity, no Auto Scaling, and customized to 1 RCU and 1 WCU like below, and your usage will remain well within the free tier.
Related
I have created a DynamoDB table with 1 RCU (manual provisioned capacity).
I have inserted some items to read in that table.
I can launch a scan on my table (which consumes 82 RCUs according to the response).
I understand this is possible because of the burst capacity.
What I don't understand though, is why am I able to keep consuming huge numbers of RCUs for long periods of time.
As you can see on this screenshot, despite the RCU being 1, I have been
consuming around 150 or 200 RCU per minute for more than 1 hour (we can barely see the 1 RCU red line at the bottom).
Why is that? (some of the requests are of course throttled but why so little ?)
How much data do you have in that table?
When you try scan operation from console, it will read items from the table that will consume RCUs.
There are options to configure baseline read/write capacity units and enable autoscaling if you expect variable reads/write requests. If the load starts to increase, dynamo db service will gradually scale to fulfil those requests instead of throttling.
https://aws.amazon.com/blogs/database/amazon-dynamodb-auto-scaling-performance-and-cost-optimization-at-any-scale/
After following the dataflow tutorial, I used the pub/sub topic to big query template to parse a JSON record into a table. The Job has been streaming for 21 days. During that time I have ingested about 5000 JSON records, containing 4 fields (around 250 bytes).
After the bill came this month I started to look into resource usage. I have used 2,017.52 vCPU hr, memory 7,565.825 GB hr, Total HDD 620,407.918 GB hr.
This seems absurdly high for the tiny amount of data I have been ingesting. Is there a minimum amount of data I should have before using dataflow? It seems over powered for small cases. Is there another preferred method for ingesting data from a pub sub topic? Is there a different configuration when setting up a Dataflow Job that uses less resources?
It seems that the numbers you mentioned, correspond to not customizing the job resources. By default streaming jobs use a n1-standar-4 machine:
3 Streaming worker defaults: 4 vCPU, 15 GB memory, 400 GB Persistent Disk.
4 vCPU x 24 hrs x 21 days = 2,016
15 GB x 24 hrs x 21 days = 7,560
If you really need streaming in Dataflow, you will need to pay for resources allocated even if there is nothing to process.
Options:
Optimizing Dataflow
Considering that the number and size of the JSON string you need to process are really small, you can reduce the cost to aprox 1/4 of current charge. You just need to set the job to use a n1-standard-1 machine, which has 1vCPU and 3.75GB memory. Just be careful with max nodes, unless you are planning increase the load, one node may be enough.
Your own way
If you don't really need streaming (not likely), you can just create a function that pulls using Synchronous Pull, and add the part that writes to BigQuery. You can schedule according to your needs.
Cloud functions (my recommendation)
You can create a serverless Event-Driven Cloud Function with a Cloud Pub/Sub trigger. This way, considering your low volume, you can take advantage of the Free Tier and keep the real time processing:
"Cloud Functions provides a perpetual free tier for compute-time resources, which includes an allocation of both GB-seconds and GHz-seconds. In addition to the 2 million invocations, the free tier provides 400,000 GB-seconds, 200,000 GHz-seconds of compute time and 5GB of Internet egress traffic per month."[1]
[1] https://cloud.google.com/functions/pricing
I am trying to understand how dynamo DB provisioned throughput (RCU/WCU) works.
I tried 2 scenarios where i made a change in WCU ( 1,000 & 10,000), but the WCU consumed figures which i am getting is same i.e. 809.63.
In a nutshell, i have 123 records distributed in 5 files, each record is of 400 KB ( according to dynamo db limit rule). When executing these cases there was no throttling, and strange thing is script execution time is same i.e. 6 sec, even though i have changed WCU count to 1k & 10k respectively.
My question is why does it behave like this. I would like to know your comments on this.
My assumption is if i decrease/increase WCU count, i should see changes in script execution time, which is not in my case.
Dynamo DB Scenario tests:
WCU/RCU do not increase the speed of a DynamoDB SDK response time, they only set an upper limit for capacity usage.
Read and Write Capacity Units are, as the name suggests, capacity units. They indicate the upper limit of how much capacity your table can handle in terms of read/write. What this means is, in your case since you are using 809.63 WCU, if your WCU is set to above 810 then you won't get any throttled requests. However, if you lower your WCU to 800, you will start seeing your requests being throttled.
If you have consistent TPS and know how many capacity units you will be using, then set just the amount that you will require. In your case, 1k WCU seems sufficient and will not make any difference compared to 10k in terms of performance, unless you use more than 1k WCU, in which case you can provision more capacity or implement auto-scaling to handle it.
See here for more information: Documentation
Edit: As discussed in below comments, if you use more capacity than is provisioned, DynamoDB will temporarily allow a burst of capacity to support it for up to 5 minutes, which could lead to varying results in terms of throttling
Before answering, many Thanks to Deiv & Stu for finding this evidence.
DynamoDB can consume up to 300 seconds of unused throughput in burst capacity.
The maximum item size in DynamoDB is 400KB and 1 RCU gives you a read of up to 4KB.
Lets say you want to read an item that is 400KB in size and you have 1 RCU on your table. You could retrieve that item once every 100 seconds.
Because of burst capacity there will always be a time you can read that item, because in fact you can use up to 300 RCUs in one go, not just 1.
Imagine starting the table with that 400KB item. You need to wait 100 seconds without spending any RCUs so that you've earned enough burst capacity to get the item. After 101 seconds you make the request, spend 100 RCUs and get the item. After another 5 seconds you make the request again, but get denied with a Throttling Exception.
So no, DynamoDB will not increase request latency to meet your RCU provision. It either returns your results as fast as possible, or throws an exception.
Input data
DynamoDB free tier provides:
25 GB of Storage
25 Units of Read Capacity
25 Units of Write Capacity
Capacity units (SC/EC - strongly/eventually consistent):
1 RCU = 1 SC read of 4kB item/second
1 RCU = 2 EC reads of 4kB item/second
1 WCU = 1 write of 1kB item/second
My application:
one DynamoDB table 5 RCU, 5 WCU
one lambda
runs each 1 minute
writes 3 items ~8kB each to the DynamoDB
lambda execution takes <1 second
The application works ok, no throttling so far.
CloudWatch
In my CloudWatch there are some charts (ignore the part after 7:00):
the value on this chart is 22 WCU
on this chart it is 110 WCU - actually figured it out - this chart resolution is 5min - 5*22=110 (leaving it here in case my future self gets confused)
Questions
We have 3 writes of ~8kB items/second - that's ~24 WCU. That is consistent with what we see in the CloudWatch (22 WCU). But the table is configured to have only 5 WCU. I've read some other questions and as far as I understand I'm safe from paying extra if the sum of WCUs in my tables configurations is below 25.
Am I overusing the write capacity for my table?
Should I expect throttling or extra charges?
As far as I can tell my usage is still within the free tier limits, but it is close (22 of 25). Am I to be charged extra if my usage gets over 25 on those charts?
The configured provisioned capacity is per second, while the data you see in CloudWatch is per minute. So your configured 5 WCU per second translate to 300 WCU per minute (5 WCU * 60 seconds), which is well above the consumed 22 WCU per minute.
That should already answer your question, but to elaborate a bit on some details:
A single write of 7KB with a configured amount of 5 WCU would in theory never succeed and cause throttling, as 7KB would require 7 WCU to write, while you only have 5 WCU configured (and we can safely assume that your write would occur within one second). Fortunately the DynamoDB engineers thought about that and implemented burst capacity. While you're not using provisioned capacity you'll save them up for up to 5 minutes to use them when you need more than the provisioned capacity. That's something to keep in mind when increasing the utilization of your capacity.
I am using the AWS free tier and I have 5 DynamoDB tables, each with 5 read/write capacities, although I am not sure I need 5 - I just use them for simple Lambda functions and there is definitely not much written and read.
What happens when I create a sixth table? Will the free 25 read/write capacities be automatically divided? Or will I have to lower the capacities on the other tables to be able to stay within the 25er capacity?
And what exactly happens when I lower the capacity (lets say to just 1)? There is definitely not mich happening, maybe 5-10 reads an hour.
As stated in AWS DynamoDB Pricing, you have 25 RCU and 25 WCU for free. It is up to you to decide the distribution among your tables. For example,
... you may have one table with 25 RCU and 25 WCU for free.
... you may have 25 tables, with 1 RCU and 1 WCU each, for free.
Each supplementary capacity unit will be paid, all it meters is the cumulative sum of all capacities.
If you have rare access to tables, I strongly advise you to lower your table's capacities as much as possible. 1 WCU and 1 RCU seems to work for you. If you notice any throttling, try 2 capacity units :)