Update list items in dynamodb - amazon-web-services

My data structure in AWS DynamoDB looks like this:
{ key: 'roomNameOne',
value: {
attendees: ['A', 'B', 'C'] // this is a set,
wsConnections: [{ connectiondId: 'foo', domain: 'xyz.com' }, { connectiondId: 'bar', domain: 'xyz.com' }]
}
}
{ key: 'roomNameTwo',
value: {
attendees: ['X', 'Y', 'Z'],
wsConnections: [{ connectiondId: 'foo', domain: 'xyz.com' }, { connectiondId: 'bar', domain: 'xyz.com' }]
}
}
Now when I get a request that connectionId: foo is lost, I want to remove that entry from all the items.
So after DynamoDB update operation my list should look like this:
{ key: 'roomNameOne',
value: {
attendees: ['A', 'B', 'C'] // this is a set,
wsConnections: [{ connectiondId: 'bar', domain: 'xyz.com' }]
}
}
{ key: 'roomNameTwo',
value: {
attendees: ['X', 'Y', 'Z'],
wsConnections: [{ connectiondId: 'bar', domain: 'xyz.com' }]
}
}
Can you please help me with the query for update? The trick here is I don't know the room names, but while connection, I am aware of what all room names a connection is interested in.

Unfortunately, DynamoDB does not allow for this type of operation on a complex attribute (e.g. list of maps).
Modeling one-to-many relationships using complex attributes is a useful pattern. However, one of the drawbacks of this approach is that you won't be able to perform the types of operations you're describing.
If you have access patterns that require you to update wsConnections, you might consider modeling the relationship by making each entry of the wsConnections list it's own item in DynamoDB. For example
Storing your data in this way would make it easier for you to remove connections. For example, if you wanted to remove bar from your connections, you could perform the following operation
ddbClient.delete({
TableName: "YOUR_TABLE_NAME",
Key: {PK: "roomNameOne", SK: "wsConnection#bar"}
})
EDIT: If you don't have access to the PK, your only option is a scan operation.
ddbClient.scan({
"TableName": "YOUR TABLE NAME",
"FilterExpression": "contains(#key, :value)",
"ExpressionAttributeValues": {
":value": {
"S": "foo"
}
},
"ExpressionAttributeNames": {
"#key": "connections"
}
})
This will scan the entire database looking for items whose connections attribute contains "foo". This will let you fetch the list of items, which you can then update and persist back to DDB.
This approach is not ideal. The scan operation will search the entire database, which can be horribly inefficient. You'd also have to issue multiple requests to DDB; one to fetch and one to update. multiple roundtrips aren't the end of the world, but again, not ideal.
To unlock more flexible and efficient access patterns, it would be ideal to get the data out of the wsConnections list attribute. As long a the data is buried in a complex attribute, your options will be limited.

Related

Trying to update DynamoDB item, what does this error mean? (I'm using Python/Boto3) [duplicate]

I'm trying to update an Item in my Dynamodb Table +Users+. I have tried many different ways but I always received the same error message:
The provided key element does not match the schema
The creation of an Item works, as well as a query but not the update. When I check on DynamoDB the user is well created:
{
"email": "test#email.com",
"password": "123",
"registration": 1460136902241,
"verified": false
}
Here is the table information:
Table name: Users
Primary partition key: email (String)
Primary sort key: registration (Number)
Here is the code (called from lambda):
exports.handler = function(event, context)
{
var AWS = require("aws-sdk");
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "Users",
Item:{
email: "test#email.com",
password: "123",
verified: false,
registration: (new Date()).getTime(),
}
};
// Create the user.
docClient.put(params, function(err, data)
{
if (err)
{
context.fail("Put failed...");
return;
}
var params = {
TableName: "Users",
Key: { email : "test#email.com" },
AttributeUpdates: {
verified: {
Action: "PUT",
Value: true
}
}
};
// Update the user.
docClient.update(params, function(err, data)
{
if (err)
{
console.log(JSON.stringify(err));
context.fail(JSON.stringify(err));
return;
}
context.succeed("User successfully updated.");
});
});
};
Do you have any idea of what could be wrong in my code?
You are only providing half of your primary key. Your primary key is a combination of the partition key and range key. You need to include the range key in your Key attribute in the update parameters.
For others who have faced the same challenge and the issue is not fixed by above answers, it is always better to double check the data type of the value being updated, in my case the primary key was expecting a Number and I was trying to update with a string. Silly me
My issue was with the Node SDK for deletes, where the documentation says to provide in format:
... {Key: {'id': {S: '123'}}} ...
Which does not appear to work with the aws-sdk ^2.1077.0. This seems to work:
... {Key: {'id': '123'}} ...
My checklist when facing this issue:
Check that the name and type of your key correspond to what you have in the database.
Use corresponding attributes to make it explicit. E.g. use #DynamoDBHashKey(attributeName = "userId") in Java to indicate the partition key named userId.
Ensure that only one field or getter marked as partition key in your class.
Please, add more if you know in the comments.
I was doing BatchGetItem, then streamed it to BatchWriteItem (Delete). DeleteItem didn't like it got all attributes from the object instead of only partition and sort key.
Gathering all answers:
mismatch in an attribute name
mismatch in attribute type
half key provided
unnecessary additional keys

How to Search Nested Array of Object in DynamoDB

I am new to dynamoDb, i would like to search nested array properties. For ex my table has sample data given below
[{
id: '123',
name: 'test',
subShops: [
{
shopId: '234',
shopName: 'New Shop'
},
{
shopId: '345',
shopName: 'New Shop 2'
}
]
},
{
id: '1234',
name: 'test2',
subShops: [
{
shopId: '2345',
shopName: 'New Shop 3'
},
{
shopId: '3456',
shopName: 'New Shop 4'
}
]
}
]
I want to search where name : ['test', 'test2', 'test3'] or subShops[].shopeName where ['New Shop', 'New Shop 2', ''New Shop 3].
I have existing code for only name : ['test', 'test2', 'test3']
const params: AWS.DynamoDB.DocumentClient.ScanInput = {
TableName: VENDOR_TABLE_INFO.Name,
ExpressionAttributeNames: { "#Id": "name" },
FilterExpression: `#Id in (${Object.keys(keyValues).toString()}) or contains (subShops, :category2)`,
ExpressionAttributeValues: {
...keyValues,
':category2': {
...keyValues
}
}
};
Please notice that DynamoDB (DDB) is mainly a hyperscale key-value serverless datastore with very limited query pattern and flexibility, you need to be ok with that to use it.
In each DDB table you can only define one hash key (pk), and up to 5 local secondary index (sort key) for querying. And you can have up to 20 Global Secondary Index (GSI)
In you example, you have hash key of "id", and then if you need to query by "name" only, you need to build a GSI with name as hash key, and included the needed fields in the projection. There is no way to query by "shopname" in sub shop array unless you "flaten" the JSON tree structure.
In short, if you want JSON tree level data query/manipulation and all of your data is JSON documents, i would suggest you to use Amazon DocumentDB which is MongoDB 4 compatible, or directly use MongoDB itself.

How to get a response of BatchWriteItem DocumentClient?

I'm going to put an item in two dynamodb tables. This is my request params of BatchWriteItem operation.
RequestItems: {
first_table: [{
PutRequest: {
Item: {
employee_id: '123',
company_id: '123',
job_position: 'manager'
}
}
}],
second_table: [{
PutRequest: {
Item: {
facility_id: '123',
company_id: '123',
job_position: 'manager'
}
}
}]
},
ReturnConsumedCapacity: "TOTAL"
My item is updated succefully but I get this response -
UnprocessedItems: {}
How can I get response with updated data? Thanks
Its not possible to get a response containing the items you have put using BatchWriteItem. PutItem can return overwritten values, but not new ones.
You might consider:
1) Using the data you already have. Afterall, you know the items have been written and you already have them.
2) If you want some statistics on your batchwrite you could use
"ReturnItemCollectionMetrics": "SIZE"
3) Query for the items after you have written them.

Apollo: Refetch queries that have multiple variable permutations after mutation

Let's say I have a table that lists a bunch of Posts using a query like:
const PostsQuery = gql`
query posts($name: string) {
posts {
id
name
status
}
}
`;
const query = apolloClient.watchQuery({query: PostsQuery});
query.subscribe({
next: (posts) => console.log(posts) // [ {name: "Post 1", id: '1', status: 'pending' }, { name: "Paul's Post", id: '2', status: 'pending'} ]
});
Then later my user comes along and enters a value in a search field and calls this code:
query.setVariables({name: 'Paul'})
It fetches the filtered posts and logs it out fine.
// [ { name: "Paul's Post", id: '2', status: 'pending'} ]
Now, in my table there is a button that changes the status of a post from 'Pending' to 'Active'. The user clicks that and it calls code like:
const PostsMutation = gql`
mutation activatePost($id: ID!) {
activatePost(id: $id) {
ok
object {
id
name
status
}
}
}
`;
apolloClient.mutate({mutation: PostsMutation});
All is well with the mutation, but now I want to refetch the table data so it has the latest, so I make a change:
apolloClient.mutate({
mutation: PostsMutation,
refetchQueries: [{query: PostsQuery, variables: {name: 'Paul'}]
});
Hurray, it works!
// [ { name: "Paul's Post", id: '2', status: 'active'} ]
But... now my user clears the search query, expecting the results to update.
query.setVariables({});
// [ {name: "Post 1", id: '1', status: 'pending' }, { name: "Paul's Post", id: '2', status: 'pending'} ]
Oh no! Because the data was not refetched in our mutation with our "original" variables (meaning none), we are getting stale data!
So how do you handle a situation where you have a mutation that may affect a query that could have many permutations of variables?
I had a similar issue, I am using Apollo with Angular, so I am not sure if this method will work with React Client, but it should.
If you look closely at refetchQueries properties on the mutate method, you will see that the function can also return a string array of query names to refetch. By returning just the query name as a string, you do not need to worry about the variables. Be advised that this will refetch all the queries matching the name. So if you had a lot queries with different variables it could end up being a large request. But, in my case it is worth the trade off. If this is a problem, you could also get access to the queryManager through apolloClient.queryManager which you could use to do some more fine grained control of what to refetch. I didn't implement it, but it looked very possible. I found the solution below fits my needs fine.
In your code, what you need to do is:
apolloClient.mutate({
mutation: PostsMutation,
refetchQueries: (mutationResult) => ['PostQueries']
});
This will refetch any query with the name 'PostQueries'. Again, it is possible to only refetch a subset of them if you dig into the queryManager and do some filtering on the active watch queries. But, that is another exercise.

DynamoDB - Key element does not match the schema

I'm trying to update an Item in my Dynamodb Table +Users+. I have tried many different ways but I always received the same error message:
The provided key element does not match the schema
The creation of an Item works, as well as a query but not the update. When I check on DynamoDB the user is well created:
{
"email": "test#email.com",
"password": "123",
"registration": 1460136902241,
"verified": false
}
Here is the table information:
Table name: Users
Primary partition key: email (String)
Primary sort key: registration (Number)
Here is the code (called from lambda):
exports.handler = function(event, context)
{
var AWS = require("aws-sdk");
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "Users",
Item:{
email: "test#email.com",
password: "123",
verified: false,
registration: (new Date()).getTime(),
}
};
// Create the user.
docClient.put(params, function(err, data)
{
if (err)
{
context.fail("Put failed...");
return;
}
var params = {
TableName: "Users",
Key: { email : "test#email.com" },
AttributeUpdates: {
verified: {
Action: "PUT",
Value: true
}
}
};
// Update the user.
docClient.update(params, function(err, data)
{
if (err)
{
console.log(JSON.stringify(err));
context.fail(JSON.stringify(err));
return;
}
context.succeed("User successfully updated.");
});
});
};
Do you have any idea of what could be wrong in my code?
You are only providing half of your primary key. Your primary key is a combination of the partition key and range key. You need to include the range key in your Key attribute in the update parameters.
For others who have faced the same challenge and the issue is not fixed by above answers, it is always better to double check the data type of the value being updated, in my case the primary key was expecting a Number and I was trying to update with a string. Silly me
My issue was with the Node SDK for deletes, where the documentation says to provide in format:
... {Key: {'id': {S: '123'}}} ...
Which does not appear to work with the aws-sdk ^2.1077.0. This seems to work:
... {Key: {'id': '123'}} ...
My checklist when facing this issue:
Check that the name and type of your key correspond to what you have in the database.
Use corresponding attributes to make it explicit. E.g. use #DynamoDBHashKey(attributeName = "userId") in Java to indicate the partition key named userId.
Ensure that only one field or getter marked as partition key in your class.
Please, add more if you know in the comments.
I was doing BatchGetItem, then streamed it to BatchWriteItem (Delete). DeleteItem didn't like it got all attributes from the object instead of only partition and sort key.
Gathering all answers:
mismatch in an attribute name
mismatch in attribute type
half key provided
unnecessary additional keys