CouchDB - aggregate multiple values in the reduce step - mapreduce

I am trying to aggregate multiple values emitted from my map in a custom reduce function( rereduce enabled) but I get null values in the result except for few datapoints where I see 0s., Any help is much appreciated. Thanks
My data emitted is in this format:
Key:"dateX" Values:"{'a':435, 'b':5645}"
Key:"dateX" Values:"{'a':8451, 'b':9245}"
Key:"dateX" Values:"{'a':352, 'b':450}"
Key:"dateY" Values:"{'a':5675, 'b':1245}"
Key:"dateY" Values:"{'a':4455, 'b':620}"
I want to aggregate/sum the values of both a & b for dateX and dateY, my map-reduce is:
"map": "function(doc){emit(doc.logDate, {'a': doc.a, 'b': doc.b} );}",
"reduce": "function(key, values, rereduce) {
var total = {tA:0, tB:0};
if(rereduce){
for(i=0; i<values.length; i++)
{
total.tA += values[i].a;
total.tB += values[i].b;
}
return total;
}
total.tA = sum(values.a);
total.tB = sum(values.b);
return total; }"
------------------------
Results:
dateX {tA: 0, tB: 0}
dateY {tA: null, tB: null}
dateZ {tA: null, tB: null}
.
.
.

First of all function sum expects an array as an argument instead of values.a which is undefined, so you can try
total.tA = sum(values.map(function (value) {return value.a}));
total.tB = sum(values.map(function (value) {return value.b}));
Or something like
values.reduce(function (t, value) {
t.tA += value.a;
t.tB += value.b;
return t;
}, total);
Also, when rereduce flag is enabled means that the reduce function works with already reduced values, so
for (i=0; i < values.length; i++) {
total.tA += values[i].tA; //not a
total.tB += values[i].tB; //not b
}
Also, you should add else to prevent double execution in rereduce case
if (rereduce) {
//
} else {
//
}
http://guide.couchdb.org/editions/1/en/views.html#reduce

Related

Initializing Lists to Avoid Potential Errors

This function
List<int> _calculateTrips() {
List<int> trips = [];
trips = List.generate(
30,
(index) {
var counter = 0;
var aDay = DateTime.now().subtract(Duration(days: index));
for (var aWalk in walks) {
if ((aDay.month == aWalk.month) && (aDay.day == aWalk.day)) {
counter++;
}
}
trips.add(counter);
},
);
return trips;
}
creates the error The body might complete normally, causing null to be returned, but the return type is a potentially non-nullable type.Try adding either a return or a throw statement at the end. I'm struggling a bit to understand the message because (a) I thought I initialized the list at the beginning of the function and (b) I thought I had a return statement at the end.
The issue is with the function you pass to List.generate(). It expects a E Function(int), where E is the type of the element, for example:
final evenNumbers = List.generate(10, (index) {
return index * 2;
});
Your issue comes from trips.add(counter):
List<int> trips = [];
trips = List.generate(30, (index) {
final trip = calculateTrip(index);
trips.add(trip);
})
The inner function needs to be an int Function(int) (i.e. a function that takes an int, and returns an int), because your list is a List<int>.
However, your inner function never returns anything.
Simply replace trips.add(counter); with return counter; and it should solve this error. You may also want to refactor your function a little:
List<int> _calculateTrips() => List.generate(30, (index {
var counter = 0;
var aDay = DateTime.now().subtract(Duration(days: index));
for (var aWalk in walks) {
if ((aDay.month == aWalk.month) && (aDay.day == aWalk.day)) {
counter++;
}
}
return counter;
});

How to update length of list in dart with default value?

In my app, at many places I have used Lists like this:-
List<int> nums = [];
// initializing list dynamically with some values.
nums.length = 12; // increasing length of list
// setting these values afterward using nums[i] at different places.
Now after migrating to null-safety obviously nums.length = 4 is giving me a runtime error, so I was wondering is there any method to set the length of the list with default values such that, after if the length of the list was smaller than before then with new length extra elements are added with some default value.
Note: Of course I know we can use for loop, but I was just wondering if there is any easier and cleaner method than that.
var num = List<int>.generate(4, (i) => i);
You can read this.
Another approach:
extension ExtendList<T> on List<T> {
void extend(int newLength, T defaultValue) {
assert(newLength >= 0);
final lengthDifference = newLength - this.length;
if (lengthDifference <= 0) {
return;
}
this.addAll(List.filled(lengthDifference, defaultValue));
}
}
void main() {
var list = <int>[];
list.extend(4, 0);
print(list); // [0, 0, 0, 0];
}
Or, if you must set .length instead of calling a separate method, you could combine it with a variation of julemand101's answer to fill with a specified default value instead of with null:
class ExtendableList<T> with ListMixin<T> {
ExtendableList(this.defaultValue);
final T defaultValue;
final List<T> _list = [];
#override
int get length => _list.length;
#override
T operator [](int index) => _list[index];
#override
void operator []=(int index, T value) {
if (index >= length) {
_list.extend(index + 1, defaultValue);
}
_list[index] = value;
}
#override
set length(int newLength) {
if (newLength > length) {
_list.extend(newLength, defaultValue);
} else {
_list.length = newLength;
}
}
}
(I also made its operator []= automatically grow the ExtendableList if the specified index is out-of-bounds, similar to JavaScript.)
Your problem is that the List in Dart does not have the concept of adding more space while you promise that you are not going to use this new capacity before it is set.
But you can easily make your own List implementation which does this:
import 'dart:collection';
void main() {
List<int> nums = ExtendableList();
nums.length = 3;
nums[2] = 1;
nums[0] = 1;
nums[1] = 1;
print(nums); // [1, 1, 1]
nums.add(2);
print(nums); // [1, 1, 1, 2]
print(nums.runtimeType); // ExtendableList<int>
}
class ExtendableList<T> with ListMixin<T> {
final List<T?> _list = [];
#override
int get length => _list.length;
#override
T operator [](int index) => _list[index] as T;
#override
void operator []=(int index, T value) => _list[index] = value;
#override
set length(int newLength) => _list.length = newLength;
}
As you can see we are using a null type behind the scene but from the outside it will work like the list contains non-nullable. This only works because we assume the [] operator will not be called while a null value are in the list (which happens if we extend the list and does not set the value).
I should add that using such a List implementation does comes with great risk since you don't get any warning/error from the analyzer if you are using it wrongly.
You have to use a list of nullable element to make it longer.
List<int?> nums = [];
nums.length = 4; // OK
print(nums); // [null, null, null, null]
You can also use filled method. Here growable is false by default.
void main() {
var a = List<int>.filled(3, 0, growable: true);
print(a);
// [0, 0, 0]
}
Refer: https://api.flutter.dev/flutter/dart-core/List/List.filled.html

CouchDB view reduce one doc per key

I'm trying to solve what seems like a fairly simple problem with a couchDb view, but I'm not even getting close to the target with my result set.
Rather than updating documents, I'm creating a new document every time as my versioning strategy, and tying those documents together with a versioning field called ver. The very first document in a version chain will see the ver field and the _id field having the same value. All subsequent documents in the chain will have the same ver field as previous docs in the chain, but will have a unique _id field. These documents also have a createdTime field which is my way of knowing which document is the latest.
Here's my documents:
{
"_id": "abcd-1234-efgh-9876",
"ver": "abcd-1234-efgh-9876",
"createdTime": "2020-01-12 01:15:00 PM -0600",
...
},
{
"_id": "uopa-3849-pmdi-1935",
"ver": "abcd-1234-efgh-9876",
"createdTime": "2020-02-16 02:39:00 PM -0600",
...
}
Here's my map function:
function (doc) {
emit(doc.ver, doc);
}
Here's my reduce function:
function(keys, values, rereduce) {
var latestVersions = {};
for (var i = 0; i < keys.length; i++) {
var found = latestVersions[keys[i][0]];
if (!found || found.createdTime < values[i].createdTime) {
latestVersions[keys[i][0]] = values[i];
}
}
return latestVersions;
}
And finally, here's my desired output from the view (just the doc that I want):
{
"_id": "uopa-3849-pmdi-1935",
"ver": "abcd-1234-efgh-9876",
"createdTime": "2020-02-16 02:39:00 PM -0600",
...
}
What am I missing here? The reduce function is returning both records, which is not what I want. Is what I'm trying to achieve possible or is there a better way to go about this?
Update
I was able to get this to work when a single key is used to access the view, which is one of my use cases.
function (keys, values, rereduce) {
var toReturn = values[0];
for (var i = 1; i < values.length; i++) {
if (values[i].createdTime > toReturn.createdTime) {
toReturn = values[i];
}
}
return toReturn;
}
I have another use case that will be returning all of the data in the view, however. The desired result there is the same as above, but the function I'm using for single keys will only ever return one result. How do I filter multiple values with a shared key such that 1 "shared" key:n values -> 1 key:1 value.
I was finally able to resolve this when I stumbled upon this couchbase article. It was much more articulate than some of the other dry computer-science documentation.
I still do not understand why certain items are grouped in a reduce method and other ones are not. For example, reduce was called 5 times for 6 items that shared an identical key; only one of the keys had actually grouped anything -- an array of two documents. It probably has something to do with those dry computer-science B-tree documents I glossed over.
Anyway, I was able to determine that all I needed to do was group the values by the ver field in both scenarios (the only difference being that rereduce had a 2 dimensional array). Here's what my reduce function ended up looking like:
function (keys, values, rereduce) {
var toValues = function(myMap) {
return Object.keys(myMap).map(function(key) {
return myMap[key];
});
}
if (rereduce) {
// values should look like [[{...}, {...}], [{...}]]
var outputMap = {};
for (var i = 0; i < values.length; i++) {
for (var j = 0; j < values[i].length; j++) {
var currentEl = values[i][j];
var found = outputMap[currentEl.ver];
if ((found && found.createdDate < currentEl.createdDate) || !found) {
outputMap[currentEl.ver] = currentEl;
}
}
}
return toValues(outputMap);
} else {
var outputMap = {};
for (var i = 0; i < values.length; i++) {
var found = outputMap[values[i].ver];
if ((found && found.createdDate < values[i].createdDate) || !found) {
outputMap[values[i].ver] = values[i];
}
}
return toValues(outputMap);
}
}

Puzzling "reduction too large" error

I have a document bucket with the following format:
{
"a": 1,
"b": 2,
"c": 3,
"d": 4,
"e": 5
}
This is my mapping function:
function (doc, meta) {
var summary = {
b: doc.b,
c: doc.c,
history: [{
d: doc.d,
e: doc.e
}]
};
emit(doc.a, summary);
}
And here's my reduce function:
function(key, values, rereduce) {
for (i = 1; i < values.length; ++i)
Array.prototype.push.apply(values[0].history, values[i].history);
return values[0];
}
And finally, here are my view query parameters:
?stale=false&group=true&reduce=true
This combination leads to a reduction too large error. For now, I have at most 2 of the same keys in my dataset, so I can change the reduce function to:
function(key, values, rereduce) {
if (values[1])
Array.prototype.push.apply(values[0].history, values[1].history);
return values[0];
}
Keeping everything else the same, I get the exact result that I want:
{
"key":1,
"value":{
"b":2,
"c":3,
"history":[
{
"d":4,
"e":5
},
{
"d":6,
"e":7
}
]
}
}, many more like this
However, I expect the number of keys to grow over time, so I need to be able to loop over values.
I also changed my reduce function to simply return values.length in order to check if it was returning unreasonable values, but everything I get is either 1 or 2.
What is going on?
UPDATE:
When I change my reduce function to:
function(key, values, rereduce) {
if (rereduce) return "whatisgoingon";
for (i = 1; i < values.length; ++i)
Array.prototype.push.apply(values[0].history, values[i].history);
return values[0];
}
I get the results that I desire, however whatisgoingon is nowhere to be found in the view result. I don't want to "discard" the rereduce logic since I might actually use it in the future, so I need to understand the cause of the issue.

Fuzzy Matches on dijit.form.ComboBox / dijit.form.FilteringSelect Subclass

I am trying to extend dijit.form.FilteringSelect with the requirement that all instances of it should match input regardless of where the characters are in the inputted text, and should also ignore whitespace and punctuation (mainly periods and dashes).
For example if an option is "J.P. Morgan" I would want to be able to select that option after typing "JP" or "P Morgan".
Now I know that the part about matching anywhere in the string can be accomplished by passing in queryExpr: "*${0}*" when creating the instance.
What I haven't figured out is how to make it ignore whitespace, periods, and dashes. I have an example of where I'm at here - http://jsfiddle.net/mNYw2/2/. Any help would be appreciated.
the thing to master in this case is the store fetch querystrings.. It will call a function in the attached store to pull out any matching items, so if you have a value entered in the autofilling inputfield, it will eventually end up similar to this in the code:
var query = { this.searchAttr: this.get("value") }; // this is not entirely accurate
this._fetchHandle = this.store.query(query, options);
this._fetchHandle.then( showResultsFunction );
So, when you define select, override the _setStoreAttr to make changes in the store query api
dojo.declare('CustomFilteringSelect', [FilteringSelect], {
constructor: function() {
//???
},
_setStoreAttr: function(store) {
this.inherited(arguments); // allow for comboboxmixin to modify it
// above line eventually calls this._set("store", store);
// so now, 'this' has 'store' set allready
// override here
this.store.query = function(query, options) {
// note that some (Memory) stores has no 'fetch' wrapper
};
}
});
EDIT: override queryEngine function as opposed to query function
Take a look at the file SimpleQueryEngine.js under dojo/store/util. This is essentially what filters the received Array items on the given String query from the FilteringSelect. Ok, it goes like this:
var MyEngine = function(query, options) {
// create our matching query function
switch(typeof query){
default:
throw new Error("Can not query with a " + typeof query);
case "object": case "undefined":
var queryObject = query;
query = function(object){
for(var key in queryObject){
var required = queryObject[key];
if(required && required.test){
if(!required.test(object[key])){
return false;
}
}else if(required != object[key]){
return false;
}
}
return true;
};
break;
case "string":
/// HERE is most likely where you can play with the reqexp matcher.
// named query
if(!this[query]){
throw new Error("No filter function " + query + " was found in store");
}
query = this[query];
// fall through
case "function":
// fall through
}
function execute(array){
// execute the whole query, first we filter
var results = arrayUtil.filter(array, query);
// next we sort
if(options && options.sort){
results.sort(function(a, b){
for(var sort, i=0; sort = options.sort[i]; i++){
var aValue = a[sort.attribute];
var bValue = b[sort.attribute];
if (aValue != bValue) {
return !!sort.descending == aValue > bValue ? -1 : 1;
}
}
return 0;
});
}
// now we paginate
if(options && (options.start || options.count)){
var total = results.length;
results = results.slice(options.start || 0, (options.start || 0) + (options.count || Infinity));
results.total = total;
}
return results;
}
execute.matches = query;
return execute;
};
new Store( { queryEngine: MyEngine });
when execute.matches is set on bottom of this function, what happens is, that the string gets called on each item. Each item has a property - Select.searchAttr - which is tested by RegExp like so: new RegExp(query).test(item[searchAttr]); or maybe a bit simpler to understand; item[searchAttr].matches(query);
I have no testing environment, but locate the inline comment above and start using console.debug..
Example:
Stpre.data = [
{ id:'WS', name: 'Will F. Smith' },
{ id:'RD', name:'Robert O. Dinero' },
{ id:'CP', name:'Cle O. Patra' }
];
Select.searchAttr = "name";
Select.value = "Robert Din"; // keyup->autocomplete->query
Select.query will become Select.queryExp.replace("${0]", Select.value), in your simple queryExp case, 'Robert Din'.. This will get fuzzy and it would be up to you to fill in the regular expression, here's something to start with
query = query.substr(1,query.length-2); // '*' be gone
var words = query.split(" ");
var exp = "";
dojo.forEach(words, function(word, idx) {
// check if last word
var nextWord = words[idx+1] ? words[idx+1] : null;
// postfix 'match-all-but-first-letter-of-nextWord'
exp += word + (nextWord ? "[^" + nextWord[0] + "]*" : "");
});
// exp should now be "Robert[^D]*Din";
// put back '*'
query = '*' + exp + '*';