"IN" statement in dynamodb - amazon-web-services

I have a "Users" table, here is a sample :
{
username:"haddox",
formattedPhoneNumber:"676767676",
verified: 0,
}
My wish is to retrieve all users whose formattedPhoneNumber is contained in an array of phone numbers (retrieved from my contacts). I created a secondary index, with verified as HASH and formattedPhoneNumber as RANGE. Here is my try :
var params = {
TableName: "Users",
IndexName: "FormattedPhoneSecondaryIndex",
KeyConditionExpression: "verified = :v AND formattedPhone IN :n",
ExpressionAttributeValues: {
":v":1,
":n": ["672053916", "642117296"]
},
ProjectionExpression: "username, formattedPhoneNumber"
};
dynamodb.query(params, function(err, data) {
if (err)
console.log(JSON.stringify(err, null, 2));
else
console.log(JSON.stringify(data, null, 2));
});
But I get the following error : Invalid KeyConditionExpression: Syntax error; token: \":n\", near: \"IN :n\"",
Is there something wrong with the IN keyword ?
Maybe there is another way to achieve this ?

KeyConditionExpression's cannot use the "IN" operator (see http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#FilteringResults). The idea with KeyConditions/KeyConditionExpression in a query operation is to more efficiently read pages of items from DynamoDB, since items with the same hash key but different range keys are stored contiguously and in sorted order. The IN operator would require extracting small portions of certain pages, which makes the Query operation less efficient, so it is not allowed in KeyConditions. You would want to add that as a FilterExpression instead, which is a convenience parameter to reduce the number of items returned from DynamoDB, but does not impact how the data is read from DynamoDB.

This is how we solved.
-(AWSDynamoDBScanExpression *) prepareScanExpressionWithName:(NSString*)name andValues:(NSArray *)vals {
AWSDynamoDBScanExpression *scanExpression = [AWSDynamoDBScanExpression new];
NSMutableString* filterExpression = [NSMutableString string];
NSMutableDictionary* expression = [NSMutableDictionary dictionary];
for(int i = 0; i < vals.count; i++)
NSString *val = vals[i];
NSString* key = [NSString stringWithFormat:#":val%i",i];
[filterExpression appendString:key];
[expression setObject:val forKey:key];
if (i < vals.count) {
[filterExpression appendString:#","];
}
}
scanExpression.filterExpression = [NSString stringWithFormat:#"#P IN (%#)", filterExpression];
scanExpression.expressionAttributeNames = #{#"#P": name};
scanExpression.expressionAttributeValues = expression;
return scanExpression;
}

Related

AWS Lambda function to scan/query DynamoDB table using array values as FilterExpression

here's my case: I'm trying to make a query on a table (table name HCI.LocCatApp) using a value sent by API as KeyConditionExpression, and I'm storing the results (which must be numbers not strings) in an array, and I want to use each value from this array as a FilterExpression to scan another table (table name HCI.Category) .. So what I need is to loop on the array values, take each of them as FilterExpression and perform the scan operation. I'm currently trying to use IN but I'm not sure if it's even supported or not.
And keep in mind that the array is being filled during the runtime. And the callback can be performed only once.
here's my code:
'use strict'
var AWS = require('aws-sdk');
var mydocumentClient = new AWS.DynamoDB.DocumentClient();
exports.handler = function (event, context, callback) {
var params = {
TableName: 'HCI.LocCatApp',
KeyConditionExpression : 'LocID = :lid',
ExpressionAttributeValues: {
":lid": event.LocID
},
ProjectionExpression: 'CatID'
};
var catIDs = [];
var catIDsObject = {};
var index = 0;
mydocumentClient.query(params, function (err, data){
if (err) {
callback(err, null);
}else{
data.Items.forEach(function(item){catIDs.push(item.CatID)});
//callback(null, catIDs);
}
})
catIDs.forEach(function(value){
index ++;
var catIDsKey = ":catID"+index;
catIDsObject[catIDsKey] = value;
})
var params2 = {
TableName: 'HCI.Category',
FilterExpression : "CatID IN (:cIDs)",
ExpressionAttributeValues : {
':cIDs' : catIDs
}
};
mydocumentClient.scan(params2, function (err, data){
if (err) {
callback(err, null);
}else{
callback(null, data);
}
})
}
For some reason, the current code runs successfully but it doesn't find any matches, even if I fill in the values manually in the array, there's still no results, the IN operation doesn't seem to work.
And many thanks in advance
In your code catIds is an array of IDs (strings probably).
When you pass it to FilterExpression, you are assuming that it will be converted to a) string b) to a string in correct format.
FilterExpression : "CatID IN (:cIDs)",
ExpressionAttributeValues : {
':cIDs' : catIDs
}
I cannot try this myself at the moment, but I'm assuming this is where the query fails. IN operator expects a comma separated list of values to compare to, in parenthesis. So, after the array is inserted to query, it should be like this
FilterExpression : "CatID IN (cat1, cat2, cat2)",
But most probably it contains extra set of [ and ], and maybe even the array to string conversion causes it to something like [Object object] etc.
One solution would be to use Array.join to concatenate all the elements from the array to single string before passing it to FilterExperession. Something like this
FilterExpression : "CatID IN (:cIDs)",
ExpressionAttributeValues : {
':cIDs' : catIDs.join()
}

how to define attribute types in Dynamodb?

I want to store data attributes are threadId , threadType (sms, livechat ,fb) , createat and updateat how I define the threadType I follow this procedure but output displays all of the types of threadType?
and how to fix the time? this is manual input is there any method to get system time?
var doClient = new AWS.DynamoDB.DocumentClient();
var DynamoDB = new AWS.DynamoDB({});
var table = "thread";
var threadId = "3";
var threadType = "livechat";
var createDate = "11:38";
var updateDate = "12:00";
var channelName = "three";
var params = {
TableName : table,
Item:
{
"threadId" : threadId,
"threadType" : { "SS": ["sms", "livechat" ,"fb"] },
"createDate" : { 'S' : createDate },
"updateDate" : { 'S' : updateDate },
"channelName" :{'S' : channelName }
}
};
console.log("Adding a new item...");
doClient.put(params, function(err, data) {
if (err) {
console.error("Unable to add item. Error JSON:", JSON.stringify(err, null, 2));
} else {
console.log("Added item:", JSON.stringify(data, null, 2));
}
});
I need that only livechat display in a result of threadtype because in this insertion I need livechat
It looks like you are modelling threadType as a StringSet, which means that it can contain multiple values. If you just want to store 'livechat' then just store this value as a String. DynamoDB doesn't have the concept of server side validation, or support for Enums, though some SDK's do - in node.js there is no Enum type. As for the system time, there is no way to ask for DynamoDB to insert the system time for you. Most people will insert epoch seconds or milliseconds as a Number type for sorting purposes, and based on the client's timestamp, which should use NTP and when hosted within EC2 will be super super close to the DDB fleet timestamp.

How to search each document field individually for specific value?

I have a search bar so that when the user presses enter, the string in the search bar is sent to my ExpressJS server. The server then needs to look through every document in the MongoDB; a document is found if any of its fields matches what was in the search bar.
My current code technically works, but it seems very redundant and probably very inefficient. I use the find() method on each field, saving the matches in an array. After searching each field individually, I prune the array of found matches, removing any duplicates.
Is there a better way to do this? See my current code below:
router.get('/', function(req, res) {
var regSearch = new RegExp('.*'+searchdata+'.*', 'i'); //Create regular expression of search data -> (text in search bar)
var arr = [];
InventoryObject.find({productId: {$regex: regSearch}}).limit(100).exec(function (err, data) { //Get all docs with a matching productId
InventoryObject.find({scannerIn: {$regex: regSearch}}).limit(100).exec(function (err, data1) { //Get all docs with a matching scannerIn
InventoryObject.find({scannerOut: {$regex: regSearch}}).limit(100).exec(function (err, data2) { //Get all docs with a matching scannerOut....
InventoryObject.find({dateIn: {$regex: regSearch}}).limit(100).exec(function (err, data3) {
InventoryObject.find({dateOut: {$regex: regSearch}}).limit(100).exec(function (err, data4) {
InventoryObject.find({productName: {$regex: regSearch}}).limit(100).exec(function (err, data5) {
InventoryObject.find({purchaseOrder: {$regex: regSearch}}).limit(100).exec(function (err, data6) {
InventoryObject.find({productDestination: {$regex: regSearch}}).limit(100).exec(function (err, data7) {
InventoryObject.find({productCost: parseFloat(searchdata)}).limit(100).exec(function (err, data8) {
//Concatenate all matched documents into single array
arr = arr.concat(data, data1, data2, data3, data4, data5, data6, data7, data8);
//Remove undefined last element...
arr.splice(arr.length-1, 1);
//Iterate through array and remove any documents that are duplicates
for (var i = 0; i < arr.length; i++) {
for (var j = i+1; j < arr.length; j++) {
if (arr[i]._id.toString() === arr[j]._id.toString()) {
arr.splice(j, 1);
j--;
}
}
}
//Sort the documents by their _id property
arr.sort(function (a, b) {
if (a._id < b._id) return +1;
if (a._id > b._id) return -1;
return 0;
});
//If the array is longer than 100, truncate it.
if (arr.length > 100)
arr.length = 100; //truncate to 100 elements sorted by the order they were inputted
//console.log(arr);
res.render('index', {'inventoryObjects': arr});
searchdata = ''; //Clear search data
});
});
});
});
});
});
});
});
});
});
Here is my Schema for reference:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var InventoryObject = new Schema({
productId: String,
scannerIn: String,
scannerOut: String,
dateIn: String,
dateOut: String,
productName: String,
purchaseOrder: String,
productDestination: String,
productCost: Number
});
mongoose.model('InventoryObject', InventoryObject);
Unfortunately that's not possible in current Mongo DB versions.
You should optimise your query like this:
InventoryObject.find({
$or:[
{productId: {$regex: regSearch}},
{scannerIn: {$regex: regSearch}},
...
]
});
But if you really need to optimise speed of such queries, you should change your schema to something like:
{
attributes: [
{key: 'productId', value: 'product ID'},
{key: 'scannerId', value: 'scanner ID'},
...
]
}

how to sort embedded records in ember without using ArrayController

I want to sort embedded records in my content. Initially I had separated the embedded record as an ArrayController and did sorting on it - it was pretty straight forward, but now I am told that I should use just embedded records without ArrayController. I followed http://www.javascriptkit.com/javatutors/arraysort2.shtml to sort the array objects and the content is getting sorted but the view is not getting updated accordingly. My function looks like :
setSort: function (sort) {
var sortedContent = this.get('content.analyticsRunParameters');
sortedContent.sort(function(a, b){
var colA=a.get(sort).toLowerCase(), colB=b.get(sort).toLowerCase();
if (colA < colB) //sort string ascending
return -1;
if (colA > colB)
return 1;
return 0; //default return value (no sorting)
});
this.set('content.analyticsRunParameters',sortedContent);
console.log(sortedContent);//is sorted
console.log(this.get('content.analyticsRunParameters'));//is sorted
}
Is there a way to update the view when my content is sorted? Or using ArrayController the only way around? Thanks.
I found the solution in another post here : Ember.ArrayProxy changes not triggering handlebars #each update
I don't know if this is a best solution either. It seems like I should be just able to sort the array objects without the help of array proxy. I had to make minor modification in the transform I was using to return ArrayProxy instead of normal array of objects like :
AS.AnalyticsRunParameterTransform = DS.Transform.extend({
//return array of ember objects
deserialize: function (serialized) {
var objects = [];
for (var key in serialized) {
objects.push(Ember.Object.create({
"name": serialized[key].name,
"description": serialized[key].description,
"default": serialized[key]["default"],
"value": serialized[key].value,
"category": serialized[key].category
}));
}
//return objects;
return Ember.ArrayProxy.create({ content: objects });
},
//return JSON object
serialize: function (deserialized) {
var analyticsTemplate = {}, object;
for (var i = 0, len = deserialized.length; i < len; i++) {
object = deserialized[i];
analyticsTemplate[object.get('name')] = {"name": object.get('name'),
"description": object.get('description'),
"default": object.get('default'),
"value": object.get('value'),
"category": object.get('category')};
}
return analyticsTemplate;
}
});

How do you sort results of a _View_ by value in the in Couchbase?

So from what I understand in Couchbase is that one can sort keys* by using
descending=true
but in my case I want to sort by values instead. Consider the Twitter data in json format, my question is What it the most popular user mentioned?
Each tweet has the structure of:
{
"text": "",
"entities" : {
"hashtags" : [ ... ],
"user_mentions" : [ ...],
"urls" : [ ... ]
}
So having used MongoDB before I reused the Map function and modified it slightly to be usable in Couchbase as follows:
function (doc, meta) {
if (!doc.entities) { return; }
doc.entities.user_mentions.forEach(
function(mention) {
if (mention.screen_name !== undefined) {
emit(mention.screen_name, null);
}
}
)
}
And then I used the reduce function _count to count all the screen_name occurrences. Now my problem is How do I sort by the count values, rather than the key?
Thanks
The short answer is you cannot sort by value the result of you view. You can only sort by key.
Some work around will be to either:
analyze the data before inserting them into Couchbase and create a counter for the values you are interested by (mentions in your case)
use the view you have to sort on the application size if the size of the view is acceptable for a client side sort.
The following JS code calls a view, sorts the result, and prints the 10 hottest subjects (hashtags):
var http = require('http');
var options = {
host: '127.0.0.1',
port: 8092,
path: '/social/_design/dev_tags/_view/tags?full_set=true&connection_timeout=60000&group=true',
method: 'GET'
}
http.request(
options,
function(res) {
var buf = new Buffer(0);
res.on('data', function(data) {
buf += data;
});
res.on('end', function() {
var tweets = JSON.parse(buf);
var rows = tweets.rows;
rows.sort( function (a,b){ return b.value - a.value }
);
for ( var i = 0; i < 10; i++ ) {
console.log( rows[i] );
}
});
}
).end();
In the same time I am looking at other options to achieve this
I solved this by using a compound key.
function (doc, meta) {
emit([doc.constraint,doc.yoursortvalue]);
}
url elements:
&startkey=["jim",5]&endkey=["jim",10]&descending=true