When using ConditionExpression in a DynamoDB request, and the condition is not met, the entire request will fail. I am using conditional updates, and the fact that ConditionCheckFailedException doesn't contain any information about which condition failed is giving me a hard time.
For example consider this scenario: There's an item in a table like this:
{
state: 'ONGOING'
foo: 'FOO',
bar: 'BAR'
}
I then want to update this item, changing both foo and state:
ExpressionAttributeValues: {
:STATE_FINISHED: 'FINISHED',
:FOO: 'NEW FOO'
},
UpdateExpression: 'SET state=:STATE_FINISHED, foo=:FOO',
However, my application has a logical transition order of states, and to prevent concurrency issues where two requests concurrently modify an item and causing an inconsistent state, I add a condition to make sure only valid transitions of state are accepted:
ExpressionAttributeValues: {
:STATE_ONGOING: 'ONGOING'
},
ConditionExpression: 'state = :STATE_ONGOING'
This e.g. prevents two concurrent requests from modifying state into FINISHED and CANCELLED at the same time.
This is all fine when there's only one condition; if the request fails I know it was because an invalid state transition and I can choose whether to just fail the request, or to make a new request that only modifies FOO, whatever makes sense in my application. But if I have multiple conditions in one request, it seems impossible to find out which particular condition failed, which means I need to fail the entire request or divide it into multiple separate requests, updating one conditional value at a time. This can however raise new concurrency issues.
Has anyone found a decent solution to a similar problem?
Ideally what I'd want is to be able to make a UpdateExpression that modifies a certain attribute conditionally, otherwise ignoring it, or by using a custom function that returns the new value based on the old value and the suggested updated value, similar to an SQL UPDATE with an embedded SELECT .. CASE .... Is anything like this possible?
Or, is it at least possible to get more information out of a ConditionalCheckFailedException (such as which particular condition failed)?
As you mentioned, the DynamoDB doesn't provide granular level error message if there are multiple fields available on ConditionalExpression. I am not addressing this part of the question in my answer.
I would like to address the second part i.e. returning the old/new value.
The ReturnValues parameter can be used to get the desired values based on your requirement. You can set one of these values to get the required values.
New value - should already be available
Old value - To get old value, you can either use UPDATED_OLD or ALL_OLD
ReturnValues: NONE | ALL_OLD | UPDATED_OLD | ALL_NEW | UPDATED_NEW,
ALL_OLD - Returns all of the attributes of the item, as they appeared
before the UpdateItem operation.
UPDATED_OLD - Returns only the
updated attributes, as they appeared before the UpdateItem operation.
ALL_NEW - Returns all of the attributes of the item, as they appear
after the UpdateItem operation.
UPDATED_NEW - Returns only the updated
attributes, as they appear after the UpdateItem operation.
Related
Is the following operation implementable in DynamoDB:
On providing the key,
SET status as "active".
If the attribute additional_info starts_with "xyz"
Then REMOVE additional_info
Else do nothing to additional_info
Basically, an update where we override value of status regardless of additional_info. And optionally unset additional_info as an attribute if a condition is satisfied?
It cannot be done in one call. It also can't be done in a single transaction wrapping the two calls into one, because a transaction can't repeatedly reference the same item.
If you can just make two calls, that's the way to go. If you need it to be somewhat transactional, you'll need to control things in your app. The details depend on your requirements.
I'm having trouble updating a single item many times at once. If I try to update an item with new attributes many times like so:
UpdateExpression: 'SET attribute.#uniqueId = :newAttribute'
not all of the updates go through. I tried sending 20 updates with unique ids and this resulted in only 15 new attributes. This also occurs in my local dynamodb instance. I assume that the updates are somehow overwriting each other in a "last update wins" scenario but I'm not sure. How can I solve this?
DynamoDB is eventually consistent on update, so "race conditions" are possible. If you want more strict logic in writes, take a look at transactions
Items are not locked during a transaction. DynamoDB transactions
provide serializable isolation. If an item is modified outside of a
transaction while the transaction is in progress, the transaction is
canceled and an exception is thrown with details about which item or
items caused the exception.
Your observation is very interesting, and contradicts observations made in the past in Are DynamoDB "set" values CDRTs? and Concurrent updates in DynamoDB, are there any guarantees? - in those issues people observed that concurrent writes to different set items or to different top-level attributes seem to not get overwritten. Neither case is exactly the same as what you tested (nested attributes), though, so it's not a definitive proof there was something wrong with your test, but it's still surprising.
Presentations made in the past by the DynamoDB developers suggested that in DynamoDB writes happen on a single node (the designated "leader" of the partition), and that this node can serialize the concurrent writes. This serialization is needed to allow conditional updates, counter increments, etc., to work safely with concurrent writes. Presumably, the same serialization could have also allowed multiple sub-attributes to be modified concurrently safely. If it doesn't, it might mean that this serialization is deliberately disabled for certain updates, perhaps all unconditional updates (without a ConditionExpression). This is very surprising, and should have been documented by Amazon...
I'm trying to update an item where I need to edit about different 150 attributes on the item and I'm getting Invalid UpdateExpression: Expression size has exceeded the maximum allowed size. I assume it's because my UpdateExpression string is very long. AWS docs say that the maximum length of an expression parameters is 4kb.
How would I go about reconciling this error? I guess I could break the request down into multiple but that seems dirty.
There are some other questions on here about the Item size limit but I haven't found anything about the UpdateExpression limit.
Solution:
I ended up limiting the request to 50 attribute updates and that along with transactWrite and ConsistentRead fixed my issue.
You have indeed reached the limit for expression parameters.
Technically you can get around this if you construct another PutItem request which will replace the item that was there previously.
By doing any writes (including updates) you will need to wait for result to be propagated to the nodes your DynamoDB table uses (because of eventual consistency), or use strong consistent reads when you try to perform a GetItem, Query or Scan request that must return this item immediately after the write.
I didn't run to a very long update query for DynamoDB, but for me anything more than 50 updates, I would split it to 2 update requests!
I have an item (number) in DynamoDB table. This value is read in a service, incremented and updated back to the table. There are multiple machines with multiple threads doing this simultaneously.
My problem here is to be able to read the correct consistent value, and update with the correct value.
I tried doing the increment and update in a java synchronized block.
However, I still noticed inconsistencies in the count in the end. It doesn't seem to be updating in a consistent manner.
"My problem here is to be able to read the correct consistent value, and update with the correct value."
To read/write the correct consistent value
Read Consistency in dynamodb (you can set it in your query as ConsistentRead parameter):
There are two types of read.
Eventually Consistent Read: if you read data after changes in table, that might be stale and should wait a bit to be consistent.
Strongly Consistent Data: it returns most up-to-date data, so, should not be worried about the stale data
ConditionExpression (specify in your query):
in your query you can specify that update the value if some conditions are true (for example current value in db is the same as the value you read before. meaning no one updated it in between) otherwise it returns ConditionalCheckFailedException and you need to handle it in your code to redo, ...
So, to answer your question, first you need to ready strongly consistent to get the current counter value in db. Then, to update it, your query should look like this (removed unnecessary parameters) and you should handle ConditionalCheckFailedException in your code:
"TableName": "counters",
"ReturnValues": "UPDATED_NEW",
"ExpressionAttributeValues": {
":a": currentValue,
":bb": newValue
},
"ExpressionAttributeNames": {
"#currentValue": "currentValue"
},
**// current value is what you ve read
// by Strongly Consistent **
"ConditionExpression": "(#currentValue = :a)",
"UpdateExpression": "SET #currentValue = :bb", // new counter value
With every record store a uuid (long random string) sort of value, whenever you are trying to update the record send with update request which should update only if uuid is equal the the value you read. And update the uuid value.
synchronised block will not work if you are trying to write from multiple machines together.
I need to have a table with the structure
{
userId : 123,
Tracking: true
}
It is possible that the user does not exist for the first operation. So, by default a false should be set. The next request makes this value true, 3rd request makes it false again and so on. Similar to NOT(Tracking) i.e writing a negation to the value.
I could do this by reading the table, negating the value in the my lambda function and updating the table with new attributes.
This would mean a GET and UPDATE request for the DB. I am looking for a way to send a negation flag instead. In this way I just write a false if the user is not existing. If the user is existing, I would toggle between true and false depending on the already existing boolean value.
Just wondering if there is a way to do this. It would be then a single update request to the DB. Any pointers would be helpful.
I did not find much help from the documentation https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#update-property
Instead of a boolean you can use an integer. Increment it by 1 each time and then use even as true, odd as false (or reversed)