Accoring to the link [1]
Amazon DynamoDB has built-in fault tolerance, automatically and synchronously
replicating your data across three Availability Zones in a Region for high
availability and to help protect your data against individual machine, or even
facility failures.
So can I assume that, at the time I get result for a success write, it is already replicated into three Availability zones?
[1] http://aws.amazon.com/dynamodb/
I think it depends on how you do the read:
from http://aws.amazon.com/dynamodb/faqs/
Q: What is the consistency model of Amazon DynamoDB?
When reading data from Amazon DynamoDB, users can specify whether they want the read to be eventually consistent or strongly consistent:
Eventually Consistent Reads (Default) – the eventual consistency option maximizes your read throughput. However, an eventually consistent read might not reflect the results of a recently completed write. Consistency across all copies of data is usually reached within a second. Repeating a read after a short time should return the updated data.
Strongly Consistent Reads — in addition to eventual consistency, Amazon DynamoDB also gives you the flexibility and control to request a strongly consistent read if your application, or an element of your application, requires it. A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read.
Yes, you can rely on the data being there if PutItem succeeded.
automatically and synchronously replicating your data across three Availability Zones
The keyword is synchronously, meaning at the same time. At the same time it accepts your data, it's writing to all three Availability Zones. If PutItem returned before copmleting those writes, DynamoDB wouldn't have the consistency and durability guarantees advertised.
Related
Is there a limit on the number of simultaneous/concurrent read/write operations on a single file stored in AWS S3? I'm thinking of designing a solution which requires parallel processing of some large amount of data stored in a single file which means there will be multiple read/write operations at any given point on time. I want to understand if there is a limit for this. Consistency of data is not required for me.
S3 doesn't sounds like the ideal service for your requirement. S3 is object storage. This is an oversimplification, but it basically means that you're dealing with the entire file. Clients don't go into S3 and read/write directly into it.
When a client "reads" a file on S3, it essentially has to retrieve a copy of that file into the memory of the client device and read it from there. This way, it can handle thousands of parallel reads (up to 5,500 requests according to this announcement).
Similarly, writing to S3 means creating a new copy/version of a file. There is no way to write to a file in-place. So while S3 can support a large number of concurrent writes in general, multiple clients can't write to the same copy of a file.
Maybe EFS might fit your requirement, but I think a database designed for this sort of performance would be a better option.
Reads: Multiple concurrent reads OK. The request limit is 5,500 GET/HEAD requests per second per partitioned prefix.
Writes: For object PUT and DELETE (3,500 RPS), strong read-after-write consistency. Last writer wins, no locking:
If two PUT requests are simultaneously made to the same key, the request with the latest timestamp wins.
The docs have several concurrent application examples. See also Best practices design patterns: optimizing Amazon S3 performance.
EFS is a file-based storage option for concurrent access uses cases. See the Comparing Amazon Cloud Storage table from the docs.
I have a Lambda with an Event source pointed to a Kinesis Stream Consumer. The stream has 30 shards.
I can see requests coming in on the lambda console, and I can see metrics in the Enhanced Fan Out section of the kinesis console so it appears everything is configured right.
However, my concurrent executions of the lambda is capped at 10 for some reason and I can't figure out why. Most of the documentation suggests that when Enhanced Fan Out is used and a lambda is listening to a stream consumer, then one lambda per shard should be running.
Can anyone explain how concurrency with lambda stream consumers work?
I have a couple of pointers just in case. The first thing is to make sure your lambda concurrency limit is actually over 10. It should be, as it defaults to 1000, but it doesn't hurt to check.
About the explanation of how lambda stream consumers work, you have the details at the lambda docs.
One thing I've seen often with Kinesis Data Streams, is having trouble with the Partition Key of the records. As you probably know, Kinesis Data Streams will send all the records with the same partition key to the same shard, so they can be processed in the right order. If records were sent to any shard (for example using a simple round-robin) then you couldn't have any guarantee they would be processed in order, as different shards are read by different processors.
It's important then to make sure you are distributing your keys as evenly as possible. If most records have the same partition key, then one of the shards will be very busy while the others are not getting traffic. It might be the case you are using only 10 different values for your partition keys, in which case you would be sending data to only 10 shards, and since a lambda function execution will be connected to only one shard, you have only 10 concurrent executions.
You can know the shard Id you are using by checking the output of PutRecord. You can also force a shard ID by overriding the Hashing mechanism. There is more information about partition keys processing and record sorting at the SDK docs.
Also make sure you read the troubleshooting guide, as sometimes you can get records processed by two processors concurrently and you might want to be prepared for that.
I don't know if your issue will be related to these pointers, but the Key Partitioning is a recurrent issue, so I thought I would comment on it. Good luck!
Terraform offers a few different backend types for saving its state. AWS S3 is probably the most popular one, but it only offers eventual read-after-write consistency for overriding objects. This means that when two people apply a terraform change at approx. the same time, they might create a resource twice or get errors because a resource was deleted in the meantime.
Does Terraform solve that using DynamoDB? WRITES in DynamoDB are strongly consistent. READS, by default, are only eventually consistent, though.
So the question is whether there is strong consistency when working with S3 as a backend for Terraform.
tl;dr: Using DynamoDB to lock state provides a guarantee of strongly consistent reads or at least erroring if the read is not consistent. Without state locking you have a chance of eventual consistency biting you but it's unlikely.
Terraform doesn't currently offer DynamoDB as an option for remote state backends.
When using the S3 backend it does allow for using DynamoDB to lock the state so that multiple apply operations cannot happen concurrently. Because the lock is naively attempted as a put with a condition that that the lock doesn't already exist this gives you the strongly consistent action you need to make sure that it won't write twice (while also avoiding a race condition from making a read of the table followed by the write).
Because you can't run a plan/apply while a lock is in place this allows the first apply in a chain to complete before the second one is allowed to read the state. The lock table also holds an MD5 digest of the state file so if during plan time the state hasn't been updated it won't match the MD5 digest and so will fail hard with the following error:
Error refreshing state: state data in S3 does not have the expected content.
This may be caused by unusually long delays in S3 processing a previous state
update. Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: 9081e134e40219d67f4c63f4fef9c875
If, for some reason, you aren't using state locking then Terraform does read back the state from S3 to check that it's what it expects it is (and currently retries every 2 seconds for 10 seconds until they match or fails if that timeout is exceeded) but I think that it is still technically possible in an eventually consistent system for a read to show the update only for a second read to not show the update when it hits another node. In my experience this certainly happens in IAM which is a global service with eventual consistency, leading to much slower eventual consistency times.
All that said I have never seen any issues caused by the eventual consistency on the S3 buckets and would expect to see lots of orphaned resources because of things like that, particularly in a previous job where we were executing huge amounts of Terraform jobs concurrently and on a tight schedule.
If you wanted to be more certain of this you could probably test this by having Terraform create an object with a key of a UUID/timestamp that Terraform generates so that every apply will delete the old object and create a new one and then run that in a tight loop, checking the amount of objects in the bucket and exiting if you ever have 2 objects in the bucket.
I am planning to build a mini Content Management System. Checking the possibility of storing the Content on a DynamoDB, Will the services be able to access the content while Updating the same content? (Scenario of updating the content on CMS and publishing)
Or CloudSearch will be the better solution instead of DynamoDB in such use case?
Thanks in advance!
Please think about your use case and decide whether it requires eventually consistent read or strongly consistent read.
Read Consistency
Eventually Consistent Reads
When you read data from a DynamoDB table, the response might not reflect the results of a recently completed write operation. The response might include some stale data. If you repeat your read request after a short time, the response should return the latest data.
Strongly Consistent Reads
When you request a strongly consistent read, DynamoDB returns a response with the most up-to-date data, reflecting the updates from all prior write operations that were successful. A strongly consistent read might not be available in the case of a network delay or outage.
Note:-
DynamoDB uses eventually consistent reads, unless you specify otherwise. Read operations (such as GetItem, Query, and Scan) provide a ConsistentRead parameter: If you set this parameter to true, DynamoDB will use strongly consistent reads during the operation.
AWS DynamoDB tool (Java) for transaction management:-
Out of the box, DynamoDB provides two of the four ACID properties:
Consistency and Durability. Within a single item, you also get
Atomicity and Isolation, but when your application needs to involve
multiple items you lose those properties. Sometimes that's good
enough, but many applications, especially distributed applications,
would appreciate some of that Atomicity and Isolation as well.
Fortunately, DynamoDB provides the tools (especially optimistic
concurrency control) so that an application can achieve these
properties and have full ACID transactions.
You can use this tool if you are using Java or AWS SDK Java for DynamoDB. I am not sure whether similar tool is available for other languages.
One of the features available on this library is Isolated Reads.
Isolated reads: Read operations to multiple items are not interfered with by other transactions.
Dynamodb transaction library for Java
Transaction design
Most of the nosql solution only use eventually consistency, and given that DynamoDB replicate the data into three datacenter, how does read after write consistency is being maintained?
What would be generic approach to this kind of problem? I think it is interesting since even in MySQL replication data is replicated asynchronously.
I'll use MySQL to illustrate the answer, since you mentioned it, though, obviously, neither of us is implying that DynamoDB runs on MySQL.
In a single network with one MySQL master and any number of slaves, the answer seems extremely straightforward -- for eventual consistency, fetch the answer from a randomly-selected slave; for read-after-write consistency, always fetch the answer from the master.
even in MySQL replication data is replicated asynchronously
There's an important exception to that statement, and I suspect there's a good chance that it's closer to the reality of DynamoDB than any other alternative here: In a MySQL-compatible Galera cluster, replication among the masters is synchronous, because the masters collaborate on each transaction at commit-time and a transaction that can't be committed to all of the masters will also throw an error on the master where it originated. A cluster like this technically can operate with only 2 nodes, but should not have less than three, because when there is a split in the cluster, any node that finds itself alone or in a group smaller than half of the original cluster size will roll itself up into a harmless little ball and refuse to service queries, because it knows it's in an isolated minority and its data can no longer be trusted. So three is something of a magic number in a distributed environment like this, to avoid a catastrophic split-brain condition.
If we assume the "three geographically-distributed replicas" in DynamoDB are all "master" copies, they might operate with logic along same lines of synchronous masters like you'd find with Galera, so the solution would be essentially the same since that setup also allows any or all of the masters to still have conventional subtended asynchronous slaves using MySQL native replication. The difference there is that you could fetch from any of the masters that is currently connected to the cluster if you wanted read-after-write consistency, since all of them are in sync; otherwise fetch from a slave.
The third scenario I can think of would be analogous to three geographically-dispersed MySQL masters in a circular replication configuration, which, again, supports subtended slaves off of each master, but has the additional problems that the masters are not synchronous and there is no conflict resolution capability -- not at all viable for this application, but for purposes of discussion, the objective could still be achieved if each "object" had some kind of highly-precise timestamp. When read-after-write consistency is needed, the solution here might be for the system serving the response to poll all of the masters to find the newest version, not returning an answer until all masters had been polled, or to read from a slave for eventual consistency.
Essentially, if there's more than one "write master" then it would seem like the masters have no choice but to either collaborate at commit-time, or collaborate at consistent-read-time.
Interestingly, I think, in spite of some whining you can find in online opinion pieces about the disparity in pricing among the two read-consistency levels in DynamoDB, this analysis -- even as divorced from the reality of DynamoDB's internals as it is -- does seem to justify that discrepancy.
Eventually-consistent read replicas are essentially infinitely scalable (even with MySQL, where a master can easily serve several slaves, each of which can also easily serve several slaves of its own, each of which can serve several... ad infinitum) but read-after-write is not infinitely scalable, since by definition it would seem to require the involvement of a "more-authoritative" server, whatever that specifically means, thus justifying a higher price for reads where that level of consistency is required.
I'll tell you exactly how DynamoDB does this. No guessing.
In order for a write request to be acknowledged to the client, the write must be durable on two of the three storage nodes for that partition. One of the two storage nodes MUST be the leader node for that partition. The third storage node is probably updated as well, but on the off chance something happened, it may not be. DynamoDB will get that one updated as soon as it can.
When you request a strongly consistent read, that read comes from the leader storage node for the partition the item(s) are stored in.
I know I'm answering this question long after it has been asked, but I thought a could contribute some helpful information...
In a distributed database the concept of a "master" is not particularly relevant anymore (at least for reads/writes). Each node should be able to perform reads and writes, so that read/write performance increases as the # of machines increases. If you want reads to be correct immediately after a write, the number of machines you write to and then read from must be greater than the total number of machines in the system.
Example: if you only write to 1 machine, then you must read from all of them to ensure that your data is not stale. Or if you write to 2 machines (in this case, quorum) you can perform reads at quorum and guarantee that your data is recent.
NOTE: these assumptions change when a subset of nodes in the system crash.