I have a dynamo db table with InvId (Primary Partition Key) and PgNo (Primary Sort Key). There is an item in the table called Details which is a list of maps and every map has an attribute called ChargeId. How can I query the map having a particular ChargeId? Can someone help me with a solution how can I design the table so that I can pass the InvId and ChargeId to fetch the particular item from the Details list?
{
"Anytime": 0,
"Details": [
{
"AccNum": "ACCZ4402255319",
"Amt": 49.67,
"ChargeId": 1652849999
},
{
"AccNum": "ACCZ4402255319",
"Amt": 50,
"ChargeId": 1652849991
},
{
"AccNum": "ACCZ4402255319",
"Amt": 49.67,
"ChargeId": 1652849992
},
{
"AccNum": "ACCZ4402255319",
"Amt": 50,
"ChargeId": 1652849993
}
],
"ExpTime": 253402300800,
"InvId": "305_40225614",
"PgNo": 1,
"SubsId": "406890"
}
You need to use a filter expression. It won't be index optimized so be careful.
See DynamoDB: How to use a query filter to check for conditions in a MAP for a code sample.
Related
I have a scenario where I want to create an item if it doesn't exist, or update an item - incrementing a total, if it already exists.
I was running into problems splitting the two operations, so I am now trying to do both using UpdateItem in a single command.
I've tried 3 different approaches none work, and they have different errors listed below, the problem it seems is creating the map and trying to update it in a single command - what should my update params look like?
Attempt one:
{
TableName: TableName,
Key: {
'key': key
},
UpdateExpression: `
ADD #total :change
, mapname.#type.#total :one
`,
ExpressionAttributeValues: {
':change': change,
':one': 1
},
ExpressionAttributeNames: {
'#type': 'dynamicstring',
'#total': 'total'
}
};
With an error of: ValidationException: The document path provided in the update expression is invalid for update
Attempt two:
{
TableName: TableName,
Key: {
"key": key
},
UpdateExpression: `
SET custommap = if_not_exists(custommap, :emptyMap)
SET #total = #total + :change,
custommap.#type.#total = custommap.#type.#total + :one
`,
ExpressionAttributeValues: {
':change': change,
':one': 1,
':emptyMap': {
'M': {
'dynamicstring': {
'M': {
'total': {
'N': 0
}
}
}
}
}
},
ExpressionAttributeNames: {
'#type': 'dynamicstring',
'#total': 'total'
}
}
With an error of: ValidationException: Invalid UpdateExpression: The "SET" section can only be used once in an update expression;
So when I use UpdateItem to create or update (increment) a map within an Item, what syntax is correct?
Thanks
SET will only stop you overwriting an attribute, not an item.
They way to achieve this is:
Use GetItem with your key to see if the item already exists
If the item exists, then do an UpdateItem and increment the counter
If the item does not exist, then use PutItem
I am little confused about retrieving data from dynamodb ... connecting is not an issue cause I am getting The provided key element does not match the schema
:the example provided from AWS
var table = new AWS.DynamoDB({params: {TableName: 'MY_TABLE'}});
var key = 'UNIQUE_KEY_ID';
var itemParams = {Item: {id: {S: key}, data: {S: 'data'}}};
table.getItem({Key: {id: {S: key}}}, function(err, data) {
console.log(data.Item); // print the item data
});
in my case the unique key is "time" and what I want to do is retrieve by key (not unique)
getItem only works on the primary key. From the docs: The GetItem operation returns a set of attributes for the item with the given primary key. See docs.
To solve this, create a Global Secondary Index that has "key" as HASH and "time" as RANGE. Then do a query operation using that index as IndexName:
var params = {
IndexName: 'your-new-GSI-index',
KeyConditionExpression: '#key = :key',
ExpressionAttributeNames: { '#key': 'key },
ExpressionAttributeValues: { ':key': { S: yourKeyVar } }
}
table.query(params, callback);
(did not test this code, but should work)
I have movie database with different fields. the Genre field contains a comma separated string like :
{genre: 'Action, Adventure, Sci-Fi'}
I know I can use regular expression to find the matches. I also tried:
{'genre': {'$in': genre}}
the problem is the running time. it take lot of time to return a query result. the database has about 300K documents and I have done normal indexing over 'genre' field.
Would say use Map-Reduce to create a separate collection that stores the genre as an array with values coming from the split comma separated string, which you can then run the Map-Reduce job and administer queries on the output collection.
For example, I've created some sample documents to the foo collection:
db.foo.insert([
{genre: 'Action, Adventure, Sci-Fi'},
{genre: 'Thriller, Romantic'},
{genre: 'Comedy, Action'}
])
The following map/reduce operation will then produce the collection from which you can apply performant queries:
map = function() {
var array = this.genre.split(/\s*,\s*/);
emit(this._id, array);
}
reduce = function(key, values) {
return values;
}
result = db.runCommand({
"mapreduce" : "foo",
"map" : map,
"reduce" : reduce,
"out" : "foo_result"
});
Querying would be straightforward, leveraging the queries with an multi-key index on the value field:
db.foo_result.createIndex({"value": 1});
var genre = ['Action', 'Adventure'];
db.foo_result.find({'value': {'$in': genre}})
Output:
/* 0 */
{
"_id" : ObjectId("55842af93cab061ff5c618ce"),
"value" : [
"Action",
"Adventure",
"Sci-Fi"
]
}
/* 1 */
{
"_id" : ObjectId("55842af93cab061ff5c618d0"),
"value" : [
"Comedy",
"Action"
]
}
Well you cannot really do this efficiently so I'm glad you used the tag "performance" on your question.
If you want to do this with the "comma separated" data in a string in place you need to do this:
Either with a regex in general if it suits:
db.collection.find({ "genre": { "$regex": "Sci-Fi" } })
But not really efficient.
Or by JavaScript evaluation via $where:
db.collection.find(function() {
return (
this.genre.split(",")
.map(function(el) {
return el.replace(/^\s+/,"")
})
.indexOf("Sci-Fi") != -1;
)
})
Not really efficient and probably equal to above.
Or better yet and something that can use an index, the separate to an array and use a basic query:
{
"genre": [ "Action", "Adventure", "Sci-Fi" ]
}
With an index:
db.collection.ensureIndex({ "genre": 1 })
Then query:
db.collection.find({ "genre": "Sci-Fi" })
Which is when you do it that way it's that simple. And really efficient.
You make the choice.
I am using an Amazon DynamoDB database, and I have a list of items with various strings as the key. I want to query for items whose key contains a substring. For example, if some of the keys are:
"abcd_aaa"
"abcd_bbb"
"abcd_ccc"
I want to query where a key contains "abcd" and these 3 items will be returned. Is this possible?
You can only query the hashKey using the equality operator (EQ). That being said if those values ("abcd_aaa", "abcd_bbb", "abcd_ccc") belong to your hashKey then you have to provide them entirely. On the other hand, the Query operation does allow partial matching on the rangeKey with the option of a few additional comparison operators:
EQ | LE | LT | GE | GT | BEGINS_WITH | BETWEEN
See the Query documentation for more details.
One possibility would be to use a hashKey and rangeKey where the first part of your code would be the hashKey and the last the rangeKey, example:
hashKey : abcd
rangeKey : aaa
By doing this when you query by hashKey (abcd), you would receive all three records sorted by the rangeKey
Scan will work
something like this
var params = {
TableName: "TABLE",
ScanFilter: {
"id": {
ComparisonOperator: "CONTAINS",
AttributeValueList: ["abcd"]
}
}
};
var template = null;
ddb.scan(params, function (err, data) {
if (err) {
console.error("Unable to query. Error:", JSON.stringify(err, null, 2));
} else {
//console.log("Query succeeded.");
data.Items.forEach(function (item) {
console.log(item);
});
}
});
I am looking to find all duplicates in my collection by flagging duplicates based on the date. The following was my attempt but I am not sure how to use cmdResult within update. Any clues?
//filter duplicates
bson::bo cmdResult;
bool ok = c.runCommand(dbcol, BSON("distinct" << "date"), cmdResult);
c.update(dbcol,Query("date"<<cmdResult<<NOT<<"_id"), BSON("$set"<<BSON("noise"<<"true")), false, true);
The "distinct" command will return you a list of all unique "date" values there are in the collection. But what you need is a list of "date" values that occur more than once.
You can get this list using the aggregate command, by grouping by "date" and counting the entries, then matching for counts > 1:
aggregate([
{ $group: { "_id": "$name", count: {$sum:1} } },
{ $match: { $gt: [ count, 1 ] } }
])
You would then update your collection (multi:true) by querying for "date" IN that list, setting the "noise" field:
update( {"name": {$in: [<list>]} },{$set: {"noise": true} }, true, false )
For help on aggregation, see http://docs.mongodb.org/manual/reference/aggregation/