I want to update the Sort Key of the following DynamoDB item from FILE#789 to FILE#790. From DynamoDB docs as well as some StackOverflow answers, the right way to do this is to delete the existing item (DeleteItem) and recreate it (PutItem) with the updated primary key, performed with TransactWriteItems.
Item: {
"PK": {
"S": "USER#123"
},
"SK": {
"S": "FILE#789"
},
"fileName": {
"S": "Construction"
}
}
DeleteItem is straightforward, since I know the composite Primary Key's values. But to re-create the item, I need the most current values of the item's attributes and then perform a PutItem. Reading the item separately, and then performing a DeleteItem and PutItem within a transaction does not guarantee that the most current values of the item's attributes are used to re-create the item. What is the recommended way to handle an update of an item's Primary Key in this scenario?
I'd use a variation of Optimistic Locking here. I've written a blog about the concept a while ago if you want to get into details, but here's the basic summary:
Each item gets a version counter that's incremented once you update it. When you want to update an item, you first read it and store the current version number. Then you perform your update and increment the version number. Now you need to write it back to the table and this is when you add a condition. The condition is that the current version of the item needs to be the same as it was when you read the item originally. If that's not the case the write should fail, because somebody else modified the item in the mean time.
Applied to your use case that means:
Read the item and keep track of the version
Update the item locally with the new key and version
Prepare your transaction:
The DeleteItem should be conditional with a check to make sure the version number is identical to 1)
The PutItem can be without conditions
You use TransactWrite to store the data
This way the transaction will fail if the original item has been updated while you were changing it. If it fails, you start at 1) again.
Related
We want to run a query, in which all the items that are returned, are deleted. More clearly, what we want to do exactly is run a query, in which if an item matches the condition, it should be included in the response, and be deleted from Amazon DynamoDB. And then the query should go with the second option.
So, after the query would respond, there would no such orders exist in database, since they were deleted on the go.
An example workflow with 5 items (items sample img. below) would look like -
A Query runs checking if From = Kartik.
The query comes on 1st item (1000) & finds that it matches the condition.
It captures the item, and deletes it from the Table. Now, only the response contains this item, not the table.
The query moves onto further items (1001 & 1002) and finds that they don't fit under the condition, so it doesn't even capture them, and does not delete too.
The query finds the 4th item (1003) matching the condition. So, it captures it in the response, and deletes it from the table.
Same as above for the 5th item (1004).
Now, the query completes, and returns a response containing ONLY the 1st, 4th & 5th Item. Now if I go and look for them in DynamoDB, it would return an error because they were deleted from there.
So, that's how I want the flow to be. Any chances of this being possible to do?
Any help is appreciated! Thanks!
You can perform a deleteItem operation for an item, and get its old value (before delete) by setting: "ReturnValues": "ALL_OLD" in the request params.
To delete an item you must specify its primary key. so you can only delete one item. (in your case From doesn't seem to be the primary key)
DeleteItem doc
You can perform a delete within a batchWriteItem operation to deal with multiple items at once. But note that batchWriteItem is not atomic i.e some delete ops may fail, and you can find them in batchWriteItem's response.
BatchItem doc
As an additional detail to reda la's answer, as doc mentions:
The individual PutItem and DeleteItem operations specified in
BatchWriteItem are atomic; however BatchWriteItem as a whole is not.
So if your only will is to delete multiple items in dynamodb, you can do it with BatchWriteItem atomically.
Two days to delete Item
1) Enable DynamoDB Streams and perform additional activities using lambda
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
2) Use DeleteItem or BatchWriteItem api methods to perform search and deletion.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SQLtoNoSQL.DeleteData.html#SQLtoNoSQL.DeleteData.DynamoDB
In both of the cases, it requires to find the item by PartitionKey , PartitionKey+Range sort key.
OrderID is not suitable primary key, since you want to find item by From Column. Also, Scan is not option here to do that.
So to do Query on those items,
There are ways to make it efficient:
1) suggest to make FROM as partition key and OrderID as sort/range key (Composite Key)
FROM | OrderDetails
kartik | kartik#1000
kartik | kartik#1003
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html
How can I Use begins_with method on primary key in DynamoDB?
https://aws.amazon.com/blogs/database/using-sort-keys-to-organize-data-in-amazon-dynamodb/
Then you can use KeyConditionExpression operators like begins_with on Sort key and EQ operator on Partition key and get all records.
I want to add an item only if it does not exist. I am not sure how to do it. Currently I am adding successfully without checking the condition (it adds regardless if the item exists). The code is:
const params = {
TableName: MY_TABLE,
Item: myItem
};
documentClient.put(params).promise()
.catch(err => { Logger.error(`DB Error: put in table failed: ${err}`) });
}
What do I need to add in order to make the code check if the item exists and if it does, just return?
Note: I do not want to use the database mapper. I want the code to the be written using the AWS.DynamoDB class
DynamoDB supports Conditional Writes, allowing you to define a check which needs to be successfull, for the item to be inserted/updated.
DynamoDB is not an SQL database (hopefully you know this...) and does not offer the full set of ACID guarantees. One of the things you can't do with it is an atomic "check and write" - all you can do is fetch the item, see if it exists and then put the item. However, if done other process wires the item to the table between your "get" and your "write", you won't know anything about it.
If you absolutely need this kind of behaviour, DynamoDB is not the right solution.
I have an item (number) in DynamoDB table. This value is read in a service, incremented and updated back to the table. There are multiple machines with multiple threads doing this simultaneously.
My problem here is to be able to read the correct consistent value, and update with the correct value.
I tried doing the increment and update in a java synchronized block.
However, I still noticed inconsistencies in the count in the end. It doesn't seem to be updating in a consistent manner.
"My problem here is to be able to read the correct consistent value, and update with the correct value."
To read/write the correct consistent value
Read Consistency in dynamodb (you can set it in your query as ConsistentRead parameter):
There are two types of read.
Eventually Consistent Read: if you read data after changes in table, that might be stale and should wait a bit to be consistent.
Strongly Consistent Data: it returns most up-to-date data, so, should not be worried about the stale data
ConditionExpression (specify in your query):
in your query you can specify that update the value if some conditions are true (for example current value in db is the same as the value you read before. meaning no one updated it in between) otherwise it returns ConditionalCheckFailedException and you need to handle it in your code to redo, ...
So, to answer your question, first you need to ready strongly consistent to get the current counter value in db. Then, to update it, your query should look like this (removed unnecessary parameters) and you should handle ConditionalCheckFailedException in your code:
"TableName": "counters",
"ReturnValues": "UPDATED_NEW",
"ExpressionAttributeValues": {
":a": currentValue,
":bb": newValue
},
"ExpressionAttributeNames": {
"#currentValue": "currentValue"
},
**// current value is what you ve read
// by Strongly Consistent **
"ConditionExpression": "(#currentValue = :a)",
"UpdateExpression": "SET #currentValue = :bb", // new counter value
With every record store a uuid (long random string) sort of value, whenever you are trying to update the record send with update request which should update only if uuid is equal the the value you read. And update the uuid value.
synchronised block will not work if you are trying to write from multiple machines together.
I want to check whether id(primary-key) of new item is uniq or not before adding into dynamoDB
what could be best option for both performance and cost wise.
Possible options to check uniqueness of primary-key can be...
1) Get (if empty array returns, it means there are no matching data. which also means it is uniq)
2) Scan (obvious, worst idea for both performance and cost)
3) Query
++ my another thought is, if there has any way to forcibly ignore incoming request in DynamoDB settings(discard incoming request or send error message), logic could be much simpler.
In normal RDB, if we try to add new item with existing primary key, Database will return error message without changing original data stored in database.
however, in DynamoDB, whether we Put item or Update item with existing primary key, it just silently changes original data stored in database.
have any idea?
As you mentioned, DynamoDB will update an item with the primary key you provide if it already exists. The article below shows you how you can make a conditional PUT request which will fail upon trying to insert an item that already exists (based on the primary key).
http://docs.amazonwebservices.com/amazondynamodb/latest/developerguide/API_PutItem.html
To prevent a new item from replacing an existing item, use a conditional expression that contains the attribute_not_exists function with the name of the attribute being used as the partition key for the table. Since every record must contain that attribute, the attribute_not_exists function will only succeed if no matching item exists.
Scenario: We have a Dynamo DB table supporting Optimistic Locking with Version Number. Two concurrent threads are trying to save two different entries with the same primary key value to that Table.
Question: Will ConditionalCheckFailedException be thrown for the latter save action?
Yes, the second thread which tries to insert the same data would throw ConditionalCheckFailedException.
com.amazonaws.services.dynamodbv2.model.ConditionalCheckFailedException
As soon as the item is saved in database, the subsequent updates should have the version matching with the value on DynamoDB table (i.e. server side value).
save — For a new item, the DynamoDBMapper assigns an initial version
number 1. If you retrieve an item, update one or more of its
properties and attempt to save the changes, the save operation
succeeds only if the version number on the client-side and the
server-side match. The DynamoDBMapper increments the version number
automatically.
We had a similar use case in past but in our case, multiple threads reading first from the dynamoDB and then trying to update the values.
So finally there will be change in version by the time they read and they try to update the document and if you don't read the latest value from the DynamoDB then intermediate update will be lost(which is known as update loss issue refer aws-docs for more info).
I am not sure, if you have this use-case or not but if you have simply 2 threads trying to update the value and then if one of them get different version while their request reached to DynamoDB then you will get ConditionalCheckFailedException exception.
More info about this error can be found here http://grepcode.com/file/repo1.maven.org/maven2/com.michelboudreau/alternator/0.10.0/com/amazonaws/services/dynamodb/model/ConditionalCheckFailedException.java