How to search each document field individually for specific value? - regex

I have a search bar so that when the user presses enter, the string in the search bar is sent to my ExpressJS server. The server then needs to look through every document in the MongoDB; a document is found if any of its fields matches what was in the search bar.
My current code technically works, but it seems very redundant and probably very inefficient. I use the find() method on each field, saving the matches in an array. After searching each field individually, I prune the array of found matches, removing any duplicates.
Is there a better way to do this? See my current code below:
router.get('/', function(req, res) {
var regSearch = new RegExp('.*'+searchdata+'.*', 'i'); //Create regular expression of search data -> (text in search bar)
var arr = [];
InventoryObject.find({productId: {$regex: regSearch}}).limit(100).exec(function (err, data) { //Get all docs with a matching productId
InventoryObject.find({scannerIn: {$regex: regSearch}}).limit(100).exec(function (err, data1) { //Get all docs with a matching scannerIn
InventoryObject.find({scannerOut: {$regex: regSearch}}).limit(100).exec(function (err, data2) { //Get all docs with a matching scannerOut....
InventoryObject.find({dateIn: {$regex: regSearch}}).limit(100).exec(function (err, data3) {
InventoryObject.find({dateOut: {$regex: regSearch}}).limit(100).exec(function (err, data4) {
InventoryObject.find({productName: {$regex: regSearch}}).limit(100).exec(function (err, data5) {
InventoryObject.find({purchaseOrder: {$regex: regSearch}}).limit(100).exec(function (err, data6) {
InventoryObject.find({productDestination: {$regex: regSearch}}).limit(100).exec(function (err, data7) {
InventoryObject.find({productCost: parseFloat(searchdata)}).limit(100).exec(function (err, data8) {
//Concatenate all matched documents into single array
arr = arr.concat(data, data1, data2, data3, data4, data5, data6, data7, data8);
//Remove undefined last element...
arr.splice(arr.length-1, 1);
//Iterate through array and remove any documents that are duplicates
for (var i = 0; i < arr.length; i++) {
for (var j = i+1; j < arr.length; j++) {
if (arr[i]._id.toString() === arr[j]._id.toString()) {
arr.splice(j, 1);
j--;
}
}
}
//Sort the documents by their _id property
arr.sort(function (a, b) {
if (a._id < b._id) return +1;
if (a._id > b._id) return -1;
return 0;
});
//If the array is longer than 100, truncate it.
if (arr.length > 100)
arr.length = 100; //truncate to 100 elements sorted by the order they were inputted
//console.log(arr);
res.render('index', {'inventoryObjects': arr});
searchdata = ''; //Clear search data
});
});
});
});
});
});
});
});
});
});
Here is my Schema for reference:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var InventoryObject = new Schema({
productId: String,
scannerIn: String,
scannerOut: String,
dateIn: String,
dateOut: String,
productName: String,
purchaseOrder: String,
productDestination: String,
productCost: Number
});
mongoose.model('InventoryObject', InventoryObject);

Unfortunately that's not possible in current Mongo DB versions.
You should optimise your query like this:
InventoryObject.find({
$or:[
{productId: {$regex: regSearch}},
{scannerIn: {$regex: regSearch}},
...
]
});
But if you really need to optimise speed of such queries, you should change your schema to something like:
{
attributes: [
{key: 'productId', value: 'product ID'},
{key: 'scannerId', value: 'scanner ID'},
...
]
}

Related

How to delete duplicates of a List<MyDataModel> (Dart/Flutter)

I have a futurebuilder that builds the UI based on a List, it does the job, however I get duplicates due to the UI being built again and again whenever I navigate. My question is, is there a innate method in Dart that can remove duplicates from a list? I've tried this StackOverflow question however it doesn't work.
Here is my custom model:
class HomeList {
Widget navigateScreen;
String imagePath;
PatientInfo patientInfo;
HomeList({
this.navigateScreen,
this.imagePath = '',
this.patientInfo,
});
static List<HomeList> homeList = [];
}
Here is my function for the futureBuilder i'm getting the data from my cloud_firestore:
_getPatients() async {
if (didLoadpatients == 0) {
print('this is didloadpatients at start of func $didLoadpatients');
var document = await db
.collection('users')
.document(mUser.uid)
.collection('patients');
document.getDocuments().then((QuerySnapshot query) async {
query.documents.forEach((f) {
uids.add(f.data['uID']);
});
didLoadpatients++;
print('this is didloadpatients at end of func $didLoadpatients');
for (var i = 0; i < uids.length; i++) {
var userDocuments = await db.collection('users').document(uids[i]);
userDocuments.get().then((DocumentSnapshot doc) {
print(doc.data);
homeList.add(HomeList(
imagePath: 'assets/fitness_app/fitness_app.png',
patientInfo: new PatientInfo.fromFbase(doc.data)));
});
print(homeList);
}
});
} else
print('I am leaving the get patient function');
}
Future<bool> getData() async {
_getCurrentUser();
await Future.delayed(const Duration(milliseconds: 1500), () async {
_getPatients();
});
return true;
}
Any help would be appreciated thank you!
To remove duplicates you can use Set Data Structure instead of List.
Just use Set instead of List to get unique values only.
Before Adding you can Remove Element from model this will Work
dummymodel.removeWhere((m) => m.id == id);
dummymodel.add(dummymodel.fromJson(data));
To Remove Duplicates from Data Model simply use Set (Data structure),
Original List with Duplicate Entries:
List<MyDataModel> mList = [MyDataModel(1), MyDataModel(2), MyDataModel(1), MyDataModel(3)];
New List that removes duplicate Entries from your List<MyDataModel>:
List<MyDataModel> mNewList = list.toSet().toList();
Output:
The result will be like
MyDataModel(1), MyDataModel(2), MyDataModel(3)
To remove the duplicate elements from custom object list, you need to override == and hashcode methods in your POJO class and then add the items in Set and again convert set to list to remove duplicate objects. Below is the working code:-
class TrackPointList {
double latitude;
double longitude;
String eventName;
Time timeZone;
TrackPointList({
this.latitude,
this.longitude,
this.eventName,
this.timeZone,
});
#override
bool operator==(other) {
// Dart ensures that operator== isn't called with null
// if(other == null) {
// return false;
// }
if(other is! TrackPointList) {
return false;
}
// ignore: test_types_in_equals
return eventName == (other as TrackPointList).eventName;
}
int _hashCode;
#override
int get hashCode {
if(_hashCode == null) {
_hashCode = eventName.hashCode;
}
return _hashCode;
}
factory TrackPointList.fromJson(Map<String, dynamic> json) => TrackPointList(
latitude: json["latitude"].toDouble(),
longitude: json["longitude"].toDouble(),
eventName: json["eventName"],
timeZone: timeValues.map[json["timeZone"]],
);
Map<String, dynamic> toJson() => {
"latitude": latitude,
"longitude": longitude,
"eventName": eventName,
"timeZone": timeValues.reverse[timeZone],
};
}
Above is the POJO class. Now below is the method which helps you to filter the objects according to the eventName data member.
List<TrackPointList> getFilteredList(List<TrackPointList> list){
final existing = Set<TrackPointList>();
final unique = list
.where((trackingPoint) => existing.add(trackingPoint))
.toList();
return unique;
}
This will work definitely.
Please +1 if it helps you.
I've come up with quite a brute force solution. Instead of
_getPatients() async {
if (didLoadpatients == 0) {
print('this is didloadpatients at start of func $didLoadpatients');
var document = await db
.collection('users')
.document(mUser.uid)
.collection('patients');
document.getDocuments().then((QuerySnapshot query) async {
query.documents.forEach((f) {
uids.add(f.data['uID']);
});
didLoadpatients++;
print('this is didloadpatients at end of func $didLoadpatients');
for (var i = 0; i < uids.length; i++) {
var userDocuments = await db.collection('users').document(uids[i]);
userDocuments.get().then((DocumentSnapshot doc) {
print(doc.data);
homeList.add(HomeList(
imagePath: 'assets/fitness_app/fitness_app.png',
patientInfo: new PatientInfo.fromFbase(doc.data)));
});
print(homeList);
}
});
} else
print('I am leaving the get patient function');
}
I've done what #Jay Mungara says and clear my Set everytime my UI rebuilds:
_getPatients() async {
homeList.clear();
if (didLoadpatients == 0) {
print('this is didloadpatients at start of func $didLoadpatients');
var document = await db
.collection('users')
.document(mUser.uid)
.collection('patients');
document.getDocuments().then((QuerySnapshot query) async {
query.documents.forEach((f) {
uids.add(f.data['uID']);
});
didLoadpatients++;
print('this is didloadpatients at end of func $didLoadpatients');
for (var i = 0; i < uids.length; i++) {
var userDocuments = await db.collection('users').document(uids[i]);
userDocuments.get().then((DocumentSnapshot doc) {
print(doc.data);
homeList.add(HomeList(
imagePath: 'assets/fitness_app/fitness_app.png',
patientInfo: new PatientInfo.fromFbase(doc.data)));
});
print(homeList);
}
});
} else
print('I am leaving the get patient function');
}
Thank you for all your answers!
this is a small examples to remove duplicate element
removeDuplicate() {
List<dynamic> demoList = [
{"userId": 1, "id": 1, "name": "thappu1"},
{"userId": 2, "id": 2, "name": "appu"},
{"userId": 1, "id": 1, "name": "thappu1"},
{"userId": 2, "id": 2, "name": "appu"},
{"userId": 2, "id": 2, "name": "appu"},
{"userId": 2, "id": 2, "name": "appu"},
{"userId": 2, "id": 2, "name": "appu"},
];
var toRemove = {};
demoList.forEach((e) {
toRemove.putIfAbsent("$e", () => e);
});
print(toRemove.keys.toList());
}
output is
[{userId: 1, id: 1, name: thappu1}, {userId: 2, id: 2, name: appu}]

AWS Lambda function to scan/query DynamoDB table using array values as FilterExpression

here's my case: I'm trying to make a query on a table (table name HCI.LocCatApp) using a value sent by API as KeyConditionExpression, and I'm storing the results (which must be numbers not strings) in an array, and I want to use each value from this array as a FilterExpression to scan another table (table name HCI.Category) .. So what I need is to loop on the array values, take each of them as FilterExpression and perform the scan operation. I'm currently trying to use IN but I'm not sure if it's even supported or not.
And keep in mind that the array is being filled during the runtime. And the callback can be performed only once.
here's my code:
'use strict'
var AWS = require('aws-sdk');
var mydocumentClient = new AWS.DynamoDB.DocumentClient();
exports.handler = function (event, context, callback) {
var params = {
TableName: 'HCI.LocCatApp',
KeyConditionExpression : 'LocID = :lid',
ExpressionAttributeValues: {
":lid": event.LocID
},
ProjectionExpression: 'CatID'
};
var catIDs = [];
var catIDsObject = {};
var index = 0;
mydocumentClient.query(params, function (err, data){
if (err) {
callback(err, null);
}else{
data.Items.forEach(function(item){catIDs.push(item.CatID)});
//callback(null, catIDs);
}
})
catIDs.forEach(function(value){
index ++;
var catIDsKey = ":catID"+index;
catIDsObject[catIDsKey] = value;
})
var params2 = {
TableName: 'HCI.Category',
FilterExpression : "CatID IN (:cIDs)",
ExpressionAttributeValues : {
':cIDs' : catIDs
}
};
mydocumentClient.scan(params2, function (err, data){
if (err) {
callback(err, null);
}else{
callback(null, data);
}
})
}
For some reason, the current code runs successfully but it doesn't find any matches, even if I fill in the values manually in the array, there's still no results, the IN operation doesn't seem to work.
And many thanks in advance
In your code catIds is an array of IDs (strings probably).
When you pass it to FilterExpression, you are assuming that it will be converted to a) string b) to a string in correct format.
FilterExpression : "CatID IN (:cIDs)",
ExpressionAttributeValues : {
':cIDs' : catIDs
}
I cannot try this myself at the moment, but I'm assuming this is where the query fails. IN operator expects a comma separated list of values to compare to, in parenthesis. So, after the array is inserted to query, it should be like this
FilterExpression : "CatID IN (cat1, cat2, cat2)",
But most probably it contains extra set of [ and ], and maybe even the array to string conversion causes it to something like [Object object] etc.
One solution would be to use Array.join to concatenate all the elements from the array to single string before passing it to FilterExperession. Something like this
FilterExpression : "CatID IN (:cIDs)",
ExpressionAttributeValues : {
':cIDs' : catIDs.join()
}

"IN" statement in dynamodb

I have a "Users" table, here is a sample :
{
username:"haddox",
formattedPhoneNumber:"676767676",
verified: 0,
}
My wish is to retrieve all users whose formattedPhoneNumber is contained in an array of phone numbers (retrieved from my contacts). I created a secondary index, with verified as HASH and formattedPhoneNumber as RANGE. Here is my try :
var params = {
TableName: "Users",
IndexName: "FormattedPhoneSecondaryIndex",
KeyConditionExpression: "verified = :v AND formattedPhone IN :n",
ExpressionAttributeValues: {
":v":1,
":n": ["672053916", "642117296"]
},
ProjectionExpression: "username, formattedPhoneNumber"
};
dynamodb.query(params, function(err, data) {
if (err)
console.log(JSON.stringify(err, null, 2));
else
console.log(JSON.stringify(data, null, 2));
});
But I get the following error : Invalid KeyConditionExpression: Syntax error; token: \":n\", near: \"IN :n\"",
Is there something wrong with the IN keyword ?
Maybe there is another way to achieve this ?
KeyConditionExpression's cannot use the "IN" operator (see http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html#FilteringResults). The idea with KeyConditions/KeyConditionExpression in a query operation is to more efficiently read pages of items from DynamoDB, since items with the same hash key but different range keys are stored contiguously and in sorted order. The IN operator would require extracting small portions of certain pages, which makes the Query operation less efficient, so it is not allowed in KeyConditions. You would want to add that as a FilterExpression instead, which is a convenience parameter to reduce the number of items returned from DynamoDB, but does not impact how the data is read from DynamoDB.
This is how we solved.
-(AWSDynamoDBScanExpression *) prepareScanExpressionWithName:(NSString*)name andValues:(NSArray *)vals {
AWSDynamoDBScanExpression *scanExpression = [AWSDynamoDBScanExpression new];
NSMutableString* filterExpression = [NSMutableString string];
NSMutableDictionary* expression = [NSMutableDictionary dictionary];
for(int i = 0; i < vals.count; i++)
NSString *val = vals[i];
NSString* key = [NSString stringWithFormat:#":val%i",i];
[filterExpression appendString:key];
[expression setObject:val forKey:key];
if (i < vals.count) {
[filterExpression appendString:#","];
}
}
scanExpression.filterExpression = [NSString stringWithFormat:#"#P IN (%#)", filterExpression];
scanExpression.expressionAttributeNames = #{#"#P": name};
scanExpression.expressionAttributeValues = expression;
return scanExpression;
}

How do you sort results of a _View_ by value in the in Couchbase?

So from what I understand in Couchbase is that one can sort keys* by using
descending=true
but in my case I want to sort by values instead. Consider the Twitter data in json format, my question is What it the most popular user mentioned?
Each tweet has the structure of:
{
"text": "",
"entities" : {
"hashtags" : [ ... ],
"user_mentions" : [ ...],
"urls" : [ ... ]
}
So having used MongoDB before I reused the Map function and modified it slightly to be usable in Couchbase as follows:
function (doc, meta) {
if (!doc.entities) { return; }
doc.entities.user_mentions.forEach(
function(mention) {
if (mention.screen_name !== undefined) {
emit(mention.screen_name, null);
}
}
)
}
And then I used the reduce function _count to count all the screen_name occurrences. Now my problem is How do I sort by the count values, rather than the key?
Thanks
The short answer is you cannot sort by value the result of you view. You can only sort by key.
Some work around will be to either:
analyze the data before inserting them into Couchbase and create a counter for the values you are interested by (mentions in your case)
use the view you have to sort on the application size if the size of the view is acceptable for a client side sort.
The following JS code calls a view, sorts the result, and prints the 10 hottest subjects (hashtags):
var http = require('http');
var options = {
host: '127.0.0.1',
port: 8092,
path: '/social/_design/dev_tags/_view/tags?full_set=true&connection_timeout=60000&group=true',
method: 'GET'
}
http.request(
options,
function(res) {
var buf = new Buffer(0);
res.on('data', function(data) {
buf += data;
});
res.on('end', function() {
var tweets = JSON.parse(buf);
var rows = tweets.rows;
rows.sort( function (a,b){ return b.value - a.value }
);
for ( var i = 0; i < 10; i++ ) {
console.log( rows[i] );
}
});
}
).end();
In the same time I am looking at other options to achieve this
I solved this by using a compound key.
function (doc, meta) {
emit([doc.constraint,doc.yoursortvalue]);
}
url elements:
&startkey=["jim",5]&endkey=["jim",10]&descending=true

Sencha Touch - List with Search-Field (XMLStore)

I have a external XML-file which I use to filling my list. This works great.
But now I want to filter(search) the XML-data with a search-field on top of the list.
My List looks like this:
ToolbarDemo.views.Beitrage = Ext.extend(Ext.List, {
title: "Beiträge",
iconCls: "btnbeitraege",
id: 'disclosurelist',
store: storeXML,
itemTpl: '<div class="contact"><img src="{bild}" width="96" height="52" border="0"/> {titel}</div>',
grouped: true,
onItemDisclosure: function(record, btn, index) {
Ext.Msg.alert('', '<video width="200" height="200" x-webkit-airplay="allow" poster="'+ record.get('bild') +'" controls="controls" id="video_player" style="" tabindex="0"><source src="'+ record.get('video') +'"></source></video>', Ext.emptyFn);
} });storeXML.load();
And my XML-input looks like this:
Ext.regModel('beitrag', {fields: ['datum', 'titel', 'video', 'bild']});
var storeXML = new Ext.data.Store({
model: 'beitrag',
sorters: [
{
property : 'Datum',
direction: 'DESC'
}],
getGroupString : function(record) {
var month = record.get('datum').split('-');
return month[2] + '.' + month[1] + '.' + month[0];
},
method: 'GET',
proxy: {
url: 'beitraege.xml',
type: 'ajax',
reader: {
type: 'xml',
record: 'beitrag',
root: 'beitraege'
},
}});
I know it's an old question, but I have managed to filter my list using a filter function in it's store. Here is how I did:
In my view I have a text field (xtype: 'searchfield').
In the controller for this view I have registered for 2 events by using the 'control' property
control: {
'searchfield': {
clearicontap: 'onSearchClearIconTap',
keyup: 'onSearchKeyUp'
}
}
onSearchKeyUp function looks like this (note: the field I'm going to filter is 'docName')
onSearchKeyUp: function(field)
{
var value = field.getValue(),
store = this.getMaster().getStore();
store.clearFilter();
if (value)
{
var searches = value.split(' '),
regexps = [],
i;
for (i = 0; i < searches.length; i++)
{
//if it is nothing, continue
if (!searches[i]) continue;
//if found, create a new regular expression which is case insenstive
regexps.push(new RegExp(searches[i], 'i'));
}
store.filter(function(record)
{
var matched = [];
//loop through each of the regular expressions
for (i = 0; i < regexps.length; i++)
{
var search = regexps[i],
didMatch = record.get('docName').match(search);
//if it matched the first or last name, push it into the matches array
matched.push(didMatch);
} //if nothing was found, return false (dont so in the store)
if (regexps.length > 1 && matched.indexOf(false) != -1) {
return false;
} else {
//else true true (show in the store)
return matched[0];
}
});
}
}
The 'onSearchClearIconTap' function instead is called when the user taps on the clear icon that is the 'X' included in the searchfield component, that clears the text, so the only thing we want to do is to reset the filter for our list:
onSearchClearIconTap: function()
{
this.getMaster().getStore().clearFilter();
}