Amazon DynamoDB doesn’t provide inbuild capabilities to auto tune throughput based on Dynamic Load.
It provide API to increase or Decrease throughput. Customers are being charges hourly basis for provisioned read & write throughput.
What are the different ways to alter throughput of dynamodb and achieve cost saving benefits ?
The answer from Chris is an accurate answer. Just to add a few points from prior experience using DynamoDB …
The situation with DynamoDB is different from EC2. The elastic compute service has an API supported directly as a web service by Amazon to allow you to program how to scale up or down according some logic such as how much demand exists. You program this by defining a monitoring threshold and automatically triggering creation or deletion of instances in a group.
Data servers do not work in the same way with triggers to adjust their capacity. But the capacity of DynamoDB is very flexible and may be controlled as Chris has pointed out. The API to provide this is good enough to make one off changes. Or equivalent manual changes from the console.
The different language bindings to program create and update actions with DynamoDB is here …
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/index.html
The important operation to modify capacity is here …
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/update-table.html
So this gives you the ability to make an increase or decrease in the ReadCapacityUnits or WriteCapacityUnits of ProvisionedThroughput.
Which is fine for a predicted or one-off change. But that is not the same thing as a flexibility tool to allow you to trigger the change automatically.
Programmatically, what you are most likely to want to do is to adjust capacity in response to change in utilization in the preceding time interval. In particular you may need to scale up rapidly in response to a surge in demand by defining an appropriate time slot and a lower and upper threshold to trigger.
A more complete solution to achieve this is described here …
https://aws.amazon.com/blogs/aws/auto-scale-dynamodb-with-dynamic-dynamodb/
The solution is maintained by Sebastian Dahlgren and may be found with all instructions at …
https://github.com/sebdah/dynamic-dynamodb
I see that the current release is 1.18.5 which is more recent than when I last used it.
Judging from earlier releases it is simple to configure by means of a dynamodb.conf properties style file …
Having provided credentials and region, the most crucial settings are
check-interval — to test throughput in seconds
min-provisioned-reads, max-provisioned-reads; reads-upper-threshold, reads-lower-threshold; increase-reads-with, decrease-reads-with — These are all percentages
min-provisioned-writes, max-provisioned-writes; writes-upper-threshold, writes-lower-threshold; increase-writes-with, decrease-writes-with — These are all percentages
Is this information up to date?
Well if you look at http://aws.amazon.com/new/ you will see just one additional recent change affecting DynamoDB which affects the documents stored. The entry for Dynamic DynamoDB is the last published entry dealing with scaling actions. So this is the best maintained DynamoDB automatic scaling capability at this time.
Amazon just added autoscaling for dynamodb, see the details here
I just discovered this project that will autoscale up and down your Dynamodb and looks better than Dynamic Dynamo, because it uses Lambda functions rather than EC2 instances:
https://github.com/channl/dynamodb-lambda-autoscale
5 minute setup process
Serverless design
Flexible code over configuration style
Autoscale table and global secondary indexes
Autoscale multiple tables
Autoscale by fixed settings
Autoscale by provisioned capacity utilisation
Autoscale by throttled event metrics
Optimised for large spikes in usage and hotkey issues by incorporating throttled event metrics
Optimised performance using concurrent queries
RateLimitedDecrement as imposed by AWS
Statistics via 'measured'
AWS credential configuration via 'dotenv'
Optimised lambda package via 'webpack'
ES7 code
100% Flow static type checking coverage
You can manage throughput programmatically through the updateTable API or manually through the console.
There's also tools like Dynamic DynamoDB, though you could roll your own version as well: you'd use the updateTable API and have some background process running to detect those circumstances and call updateTable as necessary.
Some things to watch out for when changing the scale of DynamoDB:
You get billed for allocated throughput, whether you're actually using it or not.
Wen you scale up, Dynamo may allocate new partitions for you - but it won't remove them when it scales down. This can result in unexpected hot hash key problem where you have a lot of partitions but very low throughput on each of them.
Jeff Bar recently wrote a blog in AWS official blog: "Auto Scale DynamoDB With Dynamic DynamoDB":
https://aws.amazon.com/blogs/aws/auto-scale-dynamodb-with-dynamic-dynamodb/
He introduced Dynamic DynamoDB, an open source tool built by independent developer to handle this automatically with CloudFormation template.
I think other answers have done a great job but I have a different approach to autoscale DynamoDB in an event driven fashion by leveraging CloudWatch alarms and DynamoDB's UpdateTable operation to change provisioned capacity. The following approach not only helps to reduce costs, but to scale up capacity for unexpected loads.
Summary:
Configure CloudWatch alarms on DynamoDB metrics to alert you based on thresholds and push the alerts to an SQS queue via SNS topic. A daemon process which polls SQS queue can process those alerts and change table provisioned capacity using DynamoDB's UpdateTable operation and update CloudWatch alarm thresholds.
Detailed version:
Please be advised that this approach would require 1. Understanding of AWS services like CloudWatch, SNS, SQS 2. Good amount of time for implementing in your favorite programming language 3. Maintaining a daemon to process SQS messages and change the provisioned capacity.
One time setup:
Create CloudWatch alarms on ConsumedWriteCapacityUnits and ConsumedReadCapacityUnits metrics of your DynamoDB table. You can use this documentation.
Configure the CloudWatch alarms to alert a SNS topic. Create an AWS SQS queue and subscribe the queue to receive alerts from SNS topic.
Write a daemon in any programming language to poll SQS queue and process all alerts. AWS has SDKs in multiple languages so choosing any of those languages would avoid writing lot of code to communicate with AWS services.
Daemon algorithm:
For every SQS message it receives, calculate the new provisioned capacity to be used and issue an UpdateTable operation with the new value.
Update the CloudWatch alarm with the new thresholds, if required.
You can use above approach to either scale up or down. For example, maintain CloudWatch alarm threshold at 80% of ProvisionedWriteCapacityUnits and every time the usage crosses 80%, increase the capacity and set alarm threshold to 80% of new value. Similarly you can scale down when the consumption falls below x%.
Though this is the crux, there would be lot of points to be considered in a production quality solution.
Understand about DynamoDB partitions and hot key problems.
Be aware of all DynamoDB limits.
Constraints on no.of scale downs per UTC day.
Batching the multiple UpdateTable operations.
Finally, Neptune.io provides a packaged SaaS solution to autoscale DynamoDB by using this architecture. See http://blog.neptune.io/one-click-autoscaling-of-dynamodb/ and http://blog.neptune.io/dos-and-donts-of-dynamodb-autoscaling/ for some reading on that.
P.S: I work for Neptune. And, I can help you if you need more details of implementation.
I added new features to Rockeee Dynamic DynamoDB Lambda. You can see this project:
https://github.com/touchvie/dynamic-dynamodb-lambda
Global Secondary Index support
Enable/Disable read/write autoscaling in config json file
Throttle Events in CloudWatch support
Enable/Disable throttle-read/throttle-write checking in config json file
Added test to lambda
I hope that it can help you.
AWS added native auto scaling support for DynamoDB in June 2017. See the announcement here.
You can configure this using code (Java SDK example), but if you have just a few tables, you can use the Management Console. Click in your table configuration and select the Capacity tab. The following image shows what are your options:
Guidelines for DynamoDB Auto Scaling Script :
Customers are being charged on hourly basis for provisioned read & write throughput. Below is Amazon Dynamo DB Pricing for EU (Ireland Region).
• Write Throughput: $0.00735 per hour for every 10 units of Write Capacity
• Read Throughput: $0.00735 per hour for every 50 units of Read Capacity
Amazon Dynamo DB doesn’t provide in-build capabilities to auto tune throughput based on Dynamic Load. It provides API to increase or Decrease throughput with some restrictions like throughput can be decreased twice in a day and increased any time in a day.
What will be the monthly bill of a Production Table for fixed read capacity 2,000 read/second and 2,000 write/second for 24 hours?
Calculation: $0.00735 X 24hrs X 200 X 30days {write cost for month} + $0.00735X 24hrs X 40 X 30 days {read cost for month} = 1058.4+ 211.68 = Fixed 1270 $/month.
Guidelines for writing utility {amazon supported programming languages} which adjust throughput of table and reduces monthly bills.
(A) Initial Value: Basically, here you have to watch and decide read & write throughput for table as an initialization value after analyzing average usage considering 15 days or 1 month load and add X% extra for read and Y% extra for write on the top to withstand unexpected load.
Initial read/write throughput = calculate read throughput based on average usage +X {read} % or Y {write} %
X & Y can be anything between 10% and 30% based on observation.
(B) Peak Load Shaping: Alert on tables can be set as when load reaches to 50% to 60 % of provisioned throughput, necessary action can be taken like calling throughput increment API to increase throughput anything between 30 % to 50% of provision throughput.*
(C) Manual Shaping: For known heavy load like batch load/festival season, throughput should be set manually to 200% - 300% extra of normal daily operations until load is complete*
* Once business working hours or load is over. Throughput should reduce down to initial value.
Note: Reader can calculate monthly saving considering 1,000 read/write for 16 hrs. + 2,000 read/write for 8 hours, provided utility in place.
Now that AWS has announced scheduled execution of lambda services, these seem a great fit to do time-based auto scaling. I wrote up an example of how to use this on medium. Example code is on github.
AWS added native auto scaling support for DynamoDB in June 2017. The following code (source) provides an example of how to configure auto scaling using the Java SDK:
package com.amazonaws.codesamples.autoscaling;
import com.amazonaws.services.applicationautoscaling.AWSApplicationAutoScalingClient;
import com.amazonaws.services.applicationautoscaling.model.DescribeScalableTargetsRequest;
import com.amazonaws.services.applicationautoscaling.model.DescribeScalableTargetsResult;
import com.amazonaws.services.applicationautoscaling.model.DescribeScalingPoliciesRequest;
import com.amazonaws.services.applicationautoscaling.model.DescribeScalingPoliciesResult;
import com.amazonaws.services.applicationautoscaling.model.MetricType;
import com.amazonaws.services.applicationautoscaling.model.PolicyType;
import com.amazonaws.services.applicationautoscaling.model.PredefinedMetricSpecification;
import com.amazonaws.services.applicationautoscaling.model.PutScalingPolicyRequest;
import com.amazonaws.services.applicationautoscaling.model.RegisterScalableTargetRequest;
import com.amazonaws.services.applicationautoscaling.model.ScalableDimension;
import com.amazonaws.services.applicationautoscaling.model.ServiceNamespace;
import com.amazonaws.services.applicationautoscaling.model.TargetTrackingScalingPolicyConfiguration;
public class EnableDynamoDBAutoscaling {
static AWSApplicationAutoScalingClient aaClient = new AWSApplicationAutoScalingClient();
public static void main(String args[]) {
ServiceNamespace ns = ServiceNamespace.Dynamodb;
ScalableDimension tableWCUs = ScalableDimension.DynamodbTableWriteCapacityUnits;
String resourceID = "table/TestTable";
// Define the scalable target
RegisterScalableTargetRequest rstRequest = new RegisterScalableTargetRequest()
.withServiceNamespace(ns)
.withResourceId(resourceID)
.withScalableDimension(tableWCUs)
.withMinCapacity(5)
.withMaxCapacity(10)
.withRoleARN("SERVICE_ROLE_ARN_GOES_HERE");
try {
aaClient.registerScalableTarget(rstRequest);
} catch (Exception e) {
System.err.println("Unable to register scalable target: ");
System.err.println(e.getMessage());
}
// Verify that the target was created
DescribeScalableTargetsRequest dscRequest = new DescribeScalableTargetsRequest()
.withServiceNamespace(ns)
.withScalableDimension(tableWCUs)
.withResourceIds(resourceID);
try {
DescribeScalableTargetsResult dsaResult = aaClient.describeScalableTargets(dscRequest);
System.out.println("DescribeScalableTargets result: ");
System.out.println(dsaResult);
System.out.println();
} catch (Exception e) {
System.err.println("Unable to describe scalable target: ");
System.err.println(e.getMessage());
}
System.out.println();
// Configure a scaling policy
TargetTrackingScalingPolicyConfiguration targetTrackingScalingPolicyConfiguration =
new TargetTrackingScalingPolicyConfiguration()
.withPredefinedMetricSpecification(
new PredefinedMetricSpecification()
.withPredefinedMetricType(MetricType. DynamoDBWriteCapacityUtilization))
.withTargetValue(50.0)
.withScaleInCooldown(60)
.withScaleOutCooldown(60);
// Create the scaling policy, based on your configuration
PutScalingPolicyRequest pspRequest = new PutScalingPolicyRequest()
.withServiceNamespace(ns)
.withScalableDimension(tableWCUs)
.withResourceId(resourceID)
.withPolicyName("MyScalingPolicy")
.withPolicyType(PolicyType.TargetTrackingScaling)
.withTargetTrackingScalingPolicyConfiguration(targetTrackingScalingPolicyConfiguration);
try {
aaClient.putScalingPolicy(pspRequest);
} catch (Exception e) {
System.err.println("Unable to put scaling policy: ");
System.err.println(e.getMessage());
}
// Verify that the scaling policy was created
DescribeScalingPoliciesRequest dspRequest = new DescribeScalingPoliciesRequest()
.withServiceNamespace(ns)
.withScalableDimension(tableWCUs)
.withResourceId(resourceID);
try {
DescribeScalingPoliciesResult dspResult = aaClient.describeScalingPolicies(dspRequest);
System.out.println("DescribeScalingPolicies result: ");
System.out.println(dspResult);
} catch (Exception e) {
e.printStackTrace();
System.err.println("Unable to describe scaling policy: ");
System.err.println(e.getMessage());
}
}
}
This code requires that you supply an ARN for a valid Application Auto Scaling service role. Replace SERVICE_ROLE_ARN_GOES_HERE with the actual ARN.
Related
Is there a quick way to check how many data (volume wise, GBs, TBs etc) did my specific DMS task transfered for example within last month?
I can't find any note in the documentation on that, I could probably try with boto3 but want to double check first. Thanks for help!
Even with Boto3, you can check the API - DescribeReplicationTasks but likely, there is no information about your data transfer.
Reference: https://docs.aws.amazon.com/dms/latest/APIReference/API_DescribeReplicationTasks.html
If you have only 1 data replication task that is associated with only 1 replication instance, you can check that replication instance's network metric via CloudWatch metric. From CloudWatch metrics, AWS DMS namespace, there will be several network metrics such as NetworkTransitThroughput or NetworkReceiveThroughput. You can choose one and try as below:
Statistic: Sum
Period: 30 days (or up to you)
And you have a 30DAYS_THROUGHPUT.
Is there a way to allow creation of a resource like a DynamoDB table only if the table to be created was PAY_PER_REQUEST or was provisioned with capacity below a certain amount?
I initially looked at IAM condition keys, but they appear to only be available for interactions with the table data operations (scan, update, put operations etc.) but not creation operations for the table.
Alternatively, are there ways to reduce service quotas for an account?
Ideally, I'm wondering if it is possible to scope down the ability to create DynamoDB table resources beyond a certain capacity and I’m not sure how to do it proactively instead of retroactively processing CloudTrail logs or listing existing table properties.
AWS Config
You can use AWS Config to retrospectively query AWS resources and their properties, and then determine if they are compliant or not. There are rules already available out of the box, but I can't see one which matches your use case. You will need to then write a Lambda function to implement this yourself. Here is an example.
After your rule is working you can either create a remediation action to
Delete the Table
Scale the Table Down
Send a Notification
Adjust Autoscaling (i.e. reduce max)
AWS Budgets
(My Preference)
For determining if an account is using too much DynamoDB, probably the easiest is to setup a budget for the DynamoDB Service. That would have a couple of benefits:
Auto-Scaling: Developers would be free to use high amounts of capacity (such as load tests) for short periods of time.
Potentially Cheaper: what I have found is that if you put restrictions on projects often developers will allocate 100% of the maximum, as opposed to using only what they need, in fear for another developer coming along and taking all the capacity.
Just like before with AWS Config you can setup Billing Alarms to take action and notify developers that they are using too much DynamoDB, also when the Budget is at 50%, 80% ... and so on.
CloudWatch
You could also create CloudWatch Alarms as well for certain DynamoDB metrics, looking at the capacity which has been used and again responding to excessive use.
Conclusion
You have a lot of flexibility how to approach this, so make sure you have gathered up your requirements and then the appropriate response will be easier to see. AWS Config requires a bit more work than budgets so if you can get what you want out of Budgets I would do that.
I was studying about DynamoDb where I am stuck on a question for which I can't find any common solution.
My question is: if I have an application with dynamodb as db with initial write capacity of 100 writes per second and there is heavy load during peak hours suppose 300 writes per sec. In order to reduce load on the db which service should I use?
My take says we should go for auto-scaling but somewhere I studied that we can use sqs for making queue for data and kinesis also if order of data is necessary.
In the old days, before DynamoDB Auto-Scaling, a common use pattern was:
The application attempts to write to DynamoDB
If the request is throttled, the application stores the information in an Amazon SQS queue
A separate process regularly checks the SQS queue and attempts to write the data to DynamoDB. If successful, it removes the message from SQS
This allowed DynamoDB to be provisioned for average workload rather than peak workload. However, it has more parts that need to be managed.
These days, DynamoDB can use adaptive capacity and burst capacity to handle temporary changes. For larger changes, you can implement DynamoDB Auto Scaling, which is probably easier to implement than the SQS method.
The best solution depends on the characteristics of your application. Can it tolerate asynchronous database writes? Can it tolerate any throttling on database writes?
If you can handle some throttling from DynamoDB when there’s a sudden increase in traffic, you should use DynamoDB autoscaling.
If throttling is not okay, but asynchronous writes are okay, then you could use SQS in front of DynamoDB to manage bursts in traffic. In this case, you should still have autoscaling enabled to ensure that your queue workers have enough throughout available to them.
If you must have synchronous writes and you can not tolerate any throttling from DynamoDB, you should use DynamoDB’s on demand mode. (However, do note that there can still be throttling if you exceed 1k WCU or 3k RCU for a single partition key.)
Of course cost is also a consideration. Using DynamoDB with autoscaling will be the most cost effective method. I’m not sure how On Demand compares to the cost of using SQS.
We have a service running in aws ecs that we want to scale in and out based on 2 metrics.
Scale out when: cpu > 80% or connection_count > 9500
Scale in when: cpu < 50% and connection_count < 5000
We have access to both the cpu and connection count metrics and alarms in cloud watch. However, we can't figure out how to setup a dynamic scaling policy like this based on both of them.
Using the standard aws console interface for creating the auto scaling rules I don't see any options for multiple. Any links to a tutorial or aws docs on this would be appreciated.
Based on the responses posted in the support aws forums, nothing can be done for AND/OR/IF conditions. (https://forums.aws.amazon.com/thread.jspa?threadID=94984)
It does mention however that they already put a feature request to the cloudwatch team.
The following is mentioned as a workaround:
"In the meantime, a possible workaround can be to create a custom metric using a custom script which would run after every five minutes and get the data points from the CloudWatch metrics, then perform the AND or OR operation and then push the output to a custom metric. You can then create a CloudWatch alarm which would monitor this custom metric and then trigger actions accordingly."
I have messages being put into SQS on a cron job at a rate of about 1,000 per minute.
I am looking to run a lambda function periodically, that will grab some of the messages and out them into dynamoDB with regards to the throughout which will changeover time.
You can go with 'OnDemand' pricing for your use-case. AWS link The pricing is different that the provisioned capacity method.
With on-demand capacity mode, you pay per request for the data reads and writes your application performs on your tables. You do not need to specify how much read and write throughput you expect your application to perform as DynamoDB instantly accommodates your workloads as they ramp up or down.
With this approach, you don't need to configure WCUs (or RCUs).