Puzzling "reduction too large" error - mapreduce

I have a document bucket with the following format:
{
"a": 1,
"b": 2,
"c": 3,
"d": 4,
"e": 5
}
This is my mapping function:
function (doc, meta) {
var summary = {
b: doc.b,
c: doc.c,
history: [{
d: doc.d,
e: doc.e
}]
};
emit(doc.a, summary);
}
And here's my reduce function:
function(key, values, rereduce) {
for (i = 1; i < values.length; ++i)
Array.prototype.push.apply(values[0].history, values[i].history);
return values[0];
}
And finally, here are my view query parameters:
?stale=false&group=true&reduce=true
This combination leads to a reduction too large error. For now, I have at most 2 of the same keys in my dataset, so I can change the reduce function to:
function(key, values, rereduce) {
if (values[1])
Array.prototype.push.apply(values[0].history, values[1].history);
return values[0];
}
Keeping everything else the same, I get the exact result that I want:
{
"key":1,
"value":{
"b":2,
"c":3,
"history":[
{
"d":4,
"e":5
},
{
"d":6,
"e":7
}
]
}
}, many more like this
However, I expect the number of keys to grow over time, so I need to be able to loop over values.
I also changed my reduce function to simply return values.length in order to check if it was returning unreasonable values, but everything I get is either 1 or 2.
What is going on?
UPDATE:
When I change my reduce function to:
function(key, values, rereduce) {
if (rereduce) return "whatisgoingon";
for (i = 1; i < values.length; ++i)
Array.prototype.push.apply(values[0].history, values[i].history);
return values[0];
}
I get the results that I desire, however whatisgoingon is nowhere to be found in the view result. I don't want to "discard" the rereduce logic since I might actually use it in the future, so I need to understand the cause of the issue.

Related

How to remove or erase a "key":"value" pair in a Json::Value object using "key"?

I am working in C++ language, visual studio 2022, and using jsoncpp library for working with Json.
To Give you an Idea, here is an example of Json data I am working with
[
{
"name":"Regina Eagle",
"job":"Biologist",
"salary":"728148120",
"email":"Regina_Eagle6155#y96lx.store",
"city":"Nashville"
},
{
"name":"Julius Baker",
"job":"Fabricator",
"salary":"299380360",
"email":"Julius_Baker9507#voylg.center",
"city":"Las Vegas"
},
{
"name":"Rocco Sawyer",
"job":"Chef Manager",
"salary":"223764496",
"email":"Rocco_Sawyer4620#qu9ml.club",
"city":"San Francisco"
},
{
"name":"Chad Murray",
"job":"Project Manager",
"salary":"43031808",
"email":"Chad_Murray6940#jcf8v.store",
"city":"Bridgeport"
},
{
"name":"Rocco Parker",
"job":"Lecturer",
"salary":"322089172",
"email":"Rocco_Parker202#ag5wi.solutions",
"city":"Indianapolis"
}
]
It's a Json array of objects (with key:value pairs).
I have a set of column heads for eg: {"name","job","salary"}, and I want to sort the json data in a way that each object will have only columns that are in the given set.
This is my approach:
Store json data in a Json::Value object (let us say records).
Iterate through records (as it is an array).
Creating another loop to Iterate through object stored at each index.
Extract the key of the key:value pair and check if it's present in the set or not.
If it's present then continue, else if it isn't then remove that key:value entry from there.
This way we can delete unwanted column while looping through the object.
Here, is the code snippet:
set<string> col {"name","job","salary"};
Json::Value records = [
{
"name":"Regina Eagle",
"job":"Biologist",
"salary":"728148120",
"email":"Regina_Eagle6155#y96lx.store",
"city":"Nashville"
},
{
"name":"Julius Baker",
"job":"Fabricator",
"salary":"299380360",
"email":"Julius_Baker9507#voylg.center",
"city":"Las Vegas"
},
{
"name":"Rocco Sawyer",
"job":"Chef Manager",
"salary":"223764496",
"email":"Rocco_Sawyer4620#qu9ml.club",
"city":"San Francisco"
},
{
"name":"Chad Murray",
"job":"Project Manager",
"salary":"43031808",
"email":"Chad_Murray6940#jcf8v.store",
"city":"Bridgeport"
},
{
"name":"Rocco Parker",
"job":"Lecturer",
"salary":"322089172",
"email":"Rocco_Parker202#ag5wi.solutions",
"city":"Indianapolis"
}
];
for (int i = 0; i<records.size(); i++)
{
for (auto j = records[i].begin(); j != records[i].end(); j++)
{
string key = j.key().asString();
if (col.find(key) != col.end())
{
continue;
}
else
{
records[i].removeMember(key);
}
}
}
It works fine until the 'removeMember' function get to run, and throws an error saying can't increment the value of iterator.
Expression: cannot increment value-initialized map/set iterator
Am I doing something wrong?
Or there is another/better way of doing this ?
Please advice.
Don't remove or add elements in a container you're currently iterating.
The JSON objects are stored in a std::map and removeMember calls std::map::erase. It invalidates the current iterator and it can't be incremented anymore. j++ causes the error.
One approach is to first only store the keys of properties you want to delete, and then to delete the properties in a separate loop.
set<string> col {"name","job","salary"};
Json::Value records = [
{
"name":"Regina Eagle",
"job":"Biologist",
"salary":"728148120",
"email":"Regina_Eagle6155#y96lx.store",
"city":"Nashville"
},
{
"name":"Julius Baker",
"job":"Fabricator",
"salary":"299380360",
"email":"Julius_Baker9507#voylg.center",
"city":"Las Vegas"
},
{
"name":"Rocco Sawyer",
"job":"Chef Manager",
"salary":"223764496",
"email":"Rocco_Sawyer4620#qu9ml.club",
"city":"San Francisco"
},
{
"name":"Chad Murray",
"job":"Project Manager",
"salary":"43031808",
"email":"Chad_Murray6940#jcf8v.store",
"city":"Bridgeport"
},
{
"name":"Rocco Parker",
"job":"Lecturer",
"salary":"322089172",
"email":"Rocco_Parker202#ag5wi.solutions",
"city":"Indianapolis"
}
];
for (int i = 0; i<records.size(); i++)
{
std::vector<std::string> toRemove;
for (auto j = records[i].begin(); j != records[i].end(); j++)
{
string key = j.key().asString();
if (col.find(key) != col.end())
{
continue;
}
else
{
// records[i].removeMember(key);
toRemove.push_back(key);
}
}
for (const auto &key : toRemove)
{
records[i].removeMember(key);
}
}

CouchDB view reduce one doc per key

I'm trying to solve what seems like a fairly simple problem with a couchDb view, but I'm not even getting close to the target with my result set.
Rather than updating documents, I'm creating a new document every time as my versioning strategy, and tying those documents together with a versioning field called ver. The very first document in a version chain will see the ver field and the _id field having the same value. All subsequent documents in the chain will have the same ver field as previous docs in the chain, but will have a unique _id field. These documents also have a createdTime field which is my way of knowing which document is the latest.
Here's my documents:
{
"_id": "abcd-1234-efgh-9876",
"ver": "abcd-1234-efgh-9876",
"createdTime": "2020-01-12 01:15:00 PM -0600",
...
},
{
"_id": "uopa-3849-pmdi-1935",
"ver": "abcd-1234-efgh-9876",
"createdTime": "2020-02-16 02:39:00 PM -0600",
...
}
Here's my map function:
function (doc) {
emit(doc.ver, doc);
}
Here's my reduce function:
function(keys, values, rereduce) {
var latestVersions = {};
for (var i = 0; i < keys.length; i++) {
var found = latestVersions[keys[i][0]];
if (!found || found.createdTime < values[i].createdTime) {
latestVersions[keys[i][0]] = values[i];
}
}
return latestVersions;
}
And finally, here's my desired output from the view (just the doc that I want):
{
"_id": "uopa-3849-pmdi-1935",
"ver": "abcd-1234-efgh-9876",
"createdTime": "2020-02-16 02:39:00 PM -0600",
...
}
What am I missing here? The reduce function is returning both records, which is not what I want. Is what I'm trying to achieve possible or is there a better way to go about this?
Update
I was able to get this to work when a single key is used to access the view, which is one of my use cases.
function (keys, values, rereduce) {
var toReturn = values[0];
for (var i = 1; i < values.length; i++) {
if (values[i].createdTime > toReturn.createdTime) {
toReturn = values[i];
}
}
return toReturn;
}
I have another use case that will be returning all of the data in the view, however. The desired result there is the same as above, but the function I'm using for single keys will only ever return one result. How do I filter multiple values with a shared key such that 1 "shared" key:n values -> 1 key:1 value.
I was finally able to resolve this when I stumbled upon this couchbase article. It was much more articulate than some of the other dry computer-science documentation.
I still do not understand why certain items are grouped in a reduce method and other ones are not. For example, reduce was called 5 times for 6 items that shared an identical key; only one of the keys had actually grouped anything -- an array of two documents. It probably has something to do with those dry computer-science B-tree documents I glossed over.
Anyway, I was able to determine that all I needed to do was group the values by the ver field in both scenarios (the only difference being that rereduce had a 2 dimensional array). Here's what my reduce function ended up looking like:
function (keys, values, rereduce) {
var toValues = function(myMap) {
return Object.keys(myMap).map(function(key) {
return myMap[key];
});
}
if (rereduce) {
// values should look like [[{...}, {...}], [{...}]]
var outputMap = {};
for (var i = 0; i < values.length; i++) {
for (var j = 0; j < values[i].length; j++) {
var currentEl = values[i][j];
var found = outputMap[currentEl.ver];
if ((found && found.createdDate < currentEl.createdDate) || !found) {
outputMap[currentEl.ver] = currentEl;
}
}
}
return toValues(outputMap);
} else {
var outputMap = {};
for (var i = 0; i < values.length; i++) {
var found = outputMap[values[i].ver];
if ((found && found.createdDate < values[i].createdDate) || !found) {
outputMap[values[i].ver] = values[i];
}
}
return toValues(outputMap);
}
}

CouchDB - aggregate multiple values in the reduce step

I am trying to aggregate multiple values emitted from my map in a custom reduce function( rereduce enabled) but I get null values in the result except for few datapoints where I see 0s., Any help is much appreciated. Thanks
My data emitted is in this format:
Key:"dateX" Values:"{'a':435, 'b':5645}"
Key:"dateX" Values:"{'a':8451, 'b':9245}"
Key:"dateX" Values:"{'a':352, 'b':450}"
Key:"dateY" Values:"{'a':5675, 'b':1245}"
Key:"dateY" Values:"{'a':4455, 'b':620}"
I want to aggregate/sum the values of both a & b for dateX and dateY, my map-reduce is:
"map": "function(doc){emit(doc.logDate, {'a': doc.a, 'b': doc.b} );}",
"reduce": "function(key, values, rereduce) {
var total = {tA:0, tB:0};
if(rereduce){
for(i=0; i<values.length; i++)
{
total.tA += values[i].a;
total.tB += values[i].b;
}
return total;
}
total.tA = sum(values.a);
total.tB = sum(values.b);
return total; }"
------------------------
Results:
dateX {tA: 0, tB: 0}
dateY {tA: null, tB: null}
dateZ {tA: null, tB: null}
.
.
.
First of all function sum expects an array as an argument instead of values.a which is undefined, so you can try
total.tA = sum(values.map(function (value) {return value.a}));
total.tB = sum(values.map(function (value) {return value.b}));
Or something like
values.reduce(function (t, value) {
t.tA += value.a;
t.tB += value.b;
return t;
}, total);
Also, when rereduce flag is enabled means that the reduce function works with already reduced values, so
for (i=0; i < values.length; i++) {
total.tA += values[i].tA; //not a
total.tB += values[i].tB; //not b
}
Also, you should add else to prevent double execution in rereduce case
if (rereduce) {
//
} else {
//
}
http://guide.couchdb.org/editions/1/en/views.html#reduce

Can I get {{link-to}} to specify which animation to use with ember fire?

I have left/right arrows on a page and I want to pick the animation without having to define a relationship between all the routes. Is it possible to set it on the {{link-to}}? Right now it's pretty brittle.
I have been looking, and I don't think it's possible to know what link was clicked from the transition it caused. However, I can think of two different ways to tackle your use case.
Solution 1: metaprogramming
Make a list of your routes and generate transitions dynamically from it. Something like this:
// app/transitions.js
export default function() {
const orderedRoutes = [
'left-route',
'center-route',
'right-route',
];
// See https://github.com/coleww/each-cons
// where I pinched this from
function eachCons(a, n) {
var r = []
for (var i = 0; i < a.length - n + 1; i++) {
r.push(range(a, i, n))
}
return r
}
function range (a, i, n) {
var r = []
for (var j = 0; j < n; j++) {
r.push(a[i + j])
}
return r
}
eachCons(orderedRoutes, 2).forEach(pair => {
// `pair` will be each pair of consecutive routes
// on our `orderedRoutes` list
const left = pair[0];
const right = pair[1];
// For each pair, define a transition
this.transition(
this.fromRoute(left),
this.toRoute(right),
this.use('toLeft'),
this.reverse('toRight')
);
});
}
Note that I only define transitions for adjacent routes. If you want to define a transition between left-route and center-route, you'll need to alter the algorithm to define new combinations.
Solution 2: callback to fromRoute
The function fromRoute can not only take a string, but also a function. This function receives two parameters: the names of the initial and the final routes of a transition. In this function you can return true if the transition should apply, and false otherwise. See here:
http://ember-animation.github.io/liquid-fire/#/transition-map/route-constraints
You can use this function to decide whether you should be going left or right (as per your use case). See this:
// app/transitions.js
export default function() {
// Names of the routes in the left-right succession
const orderedRoutes = [
'left-route',
'center-route',
'right-route',
];
function isLeft(initial, destination) {
const i0 = orderedRoutes.indexOf(initial);
const i1 = orderedRoutes.indexOf(destination);
if (i0 === -1 || i1 === -1) {
// This is not one of the transitions
// in the left-right succession
return false;
}
if (i0 === i1) {
// They are the same route
return false;
}
// This will be `true` if the initial route
// is "to the left" of the destination route
return i0 < i1;
}
this.transition(
this.fromRoute(isLeft),
this.use('toLeft')
this.reverse('toRight')
);
}
In this example, for each transition we check the initial and the destination route. We see if they belong to the left-right succession, and whether the transition corresponds to a "left" or a "right". If it's a "left" we return true in the "toLeft" case. If it's a "right", we return true in the "toRight" case.

State machine: Use transition matrix instead of nested switch cases C++

I am having difficulties with my state machine. I use a function that returns the new state based on input parameters oldState and two input parameters.
In this function I have a lot of nested switch cases. I'd rather use a 2x2 transition matrix but have no idea how to use it. I did make a transition table from the state diagram with sates and inputs.
But how exaclty do I use the 2 dim. array transition_table[3][4]?
You stated you currently have something like this:
StateType transition (StateType old, InputType one, InputType two) {
//... nested switch statements
return new_state;
}
So, it seems what you need is a 3-dimensional array:
#define MAX_STATES 12
#define MAX_INPUT_VAL 2
StateType transitionTable[MAX_STATES][MAX_INPUT_VAL][MAX_INPUT_VAL] = {
{ { StateA, StateB },
{ StateC, StateD } },
{ { StateE, StateF },
{ StateG, StateH } },
{ { StateI, StateJ },
{ StateK, StateL } },
//...
};
Then you would transition like this:
new_state = transitionTable[StateIndex(old)][one][two];
So, assuming that StateIndex(StateC) returns 2, then:
old = StateC;
new_state = transitionTable[StateIndex(old)][1][0];
assert(new_state == StateK);
would result in new_state holding StateK.
Given a matrix like this:
state1_input1 state1_input2 state1_input3
state2_input1 state2_input2 state2_input3
state3_input1 state3_input2 state3_input3
When you are in state n and receive input m, you look at row n, column m to find out the new state. Assuming you have 3 possible states and 4 possible inputs, all you need to do is:
state = transition_table[state][input]
Based on your description, you don't need a 2-dimentional array, 1 dimension is fine. It should be made this way:
void foo()
{
int States[2] = {1,2};
int currentState = 1;///initial state, let's say
int oldState;///prev. state
while(true)
{
if(currentState == 1 && *add any other condition that you need*)
{
<...>do something<...>
oldState = currentState;//saving the old state, in case you need it.
currentState = states[currentState]; //changing the state
}
else if( currentState == 2 && *add any other condition that you need*)
{
<...>some other code<...>
}
}
So you have an array of states. You then calculate the index of that array based on your input parameters (you said you use the old state and something else for it). After that you simply get the new state from the array by that index.
My explanation is a bit messy, so leave a comment if you need a clarification of some part.