Customized Reordering/Sorting of a List in Perl in O(n) - list

I am re-ordering a list.
My OrignalList, INPUT: #lineItems
[
{id=> 'id1', ..},
{id=> 'id2', 'groupId'=>45D,.. },
{id=> 'id3', 'groupId'=>56A, .. },
{id=> 'id4', 'groupId'=>45D, 'isParent'=>1 .. },
{id=> 'id5', ..},
{id=> 'id6', 'groupId'=>56A, 'isParent'=>1.. },
]
In above list, groupId signifies that the item is a part of bundle. GroupId uniquely determines the bundle group. If group Id is not present then its a non bundle item.
Aim - To re-order the list such that all the bundle items should be grouped together with parent item in the starting of every bundle and the incoming order of the bundle and non bundle items (when groupId not present) should remain unchanged.
To sort the list in O(n)
Expected Output:
[
{id=> 'id1', ..},
{id=> 'id4', 'groupId'=>45D, 'isParent'=>1 .. },
{id=> 'id2', 'groupId'=>45D,.. },
{id=> 'id6', 'groupId'=>56A, 'isParent'=>1.. },
{id=> 'id3', 'groupId'=>56A, .. },
{id=> 'id5', ..},
]
Here my Algo:
Create a sortedList of Ids = #sortedLineitemsIds
Use sortedIdsList to form the final sorted list
Code for #1
my $grouIdToLineItemIdMap;
foreach my $lineItem (#$lineItems) {
if(!$lineItem->{'groupID'}) { #non bundle item, add as it is
push #sortedLineitemsIds, $lineItem->{'id'};
} else {
if($lineItem->{'IsParent'} eq 1) {
unshift #{$grouIdToLineItemIdMap->{$groupId}}, $lineItem->{'id'};
} else {
push #{$grouIdToLineItemIdMap->{$groupId}}, $lineItem->{'id'};
}
}
}
push #sortedLineitemsIds, $grouIdToLineItemIdMap; # **[[Question 1]]** This will always add bundle items at the end irrespective of whether it was in starting or end.
Now this will yield sortedLineitemsIds =>
$VAR1 = [
'id1',
'id5',
{
'45D' => [
'id4:',
'id2:'
],
'56A' => [
'id6:',
'id3:'
]
}
];
Code for #2
foreach my $Id (#sortedLineitemsIds) {
if(determineIfSingleIdOrMapOfGroupId) { #**[[Question 2]]**
my $lineItem = grep #lineItems with $Id; #**[[Question 3]]**
push #sortedLineItems, $lineItem;
} else {
my $listOfLineItemsForGroupId = $sortedLineitemsIds->{$Id};
foreach groupLineItemId (#$listOfLineItemsForGroupId) {
my $lineItem = grep #lineItems with groupLineItemId; #**[[Question 3]]**
push #sortedLineItems, $lineItem;
}
}
}
I have now 3 questions marked above at different places in the code:
Question 1 -> Here I dont want to change the incoming order of
items. Just group them. But what I am doing is it is pushing all the
lineItems of the group in map, which I am appending at end after the
loop. How can I can do that in the loop to preserve that order?
Question 2 -> How can I determine whether it is a single Id (non
bundle id) or a groupID (basically a ref containing the
lineItemIds)?
Question 3 -> How can I grep the orginal list based on
the 'id' and get the corresponding lineItem?

You say you don't want to change the order of the items, but that's clearly not true. I'm going to assume you meant this:
I want to preserve the relative order of items which are either independent or the first of their group.
This can indeed be done in O(N).
We're going to build this:
my #grouped = (
[ $lineItem_id1 ],
[ $lineItem_id4, $lineItem_id2 ],
[ $lineItem_id6, $lineItem_id3 ],
[ $lineItem_id5 ],
);
To achieve that, we're going to use the following algorithm:
For each item,
If the item is independent,
Add it to #grouped.
Else,
Lookup if we've encountered the item's group before.
If the item is part of a group we haven't encountered before,
If it's the parent,
Add it to start of the existing group.
Else,
Add it to end of the existing group.
Else,
Create a new group from the item.
Add the new group to #grouped.
Add the new group to the lookup hash.
At the end of this, we'll end up with the following:
my $group_45D = [ $lineItem_id4, $lineItem_id2 ];
my $group_56A = [ $lineItem_id6, $lineItem_id3 ];
my %groups = (
'45D' => $group_45D,
'56A' => $group_56A,
);
my #grouped = (
[ $lineItem_id1 ],
$group_45D,
$group_56A,
[ $lineItem_id5 ],
);
Solution:
my #grouped;
{
my %groups;
for my $lineItem (#$lineItems) {
if ( my $groupId = $lineItem->{groupId} ) {
if (!$groups{$groupId}) {
push #grouped, $groups{$groupId} = [];
}
if ($lineItem->{isParent}) {
unshift #{ $groups{$groupId} }, $lineItem;
} else {
push #{ $groups{$groupId} }, $lineItem;
}
} else {
push #grouped, [ $lineItem ];
}
}
}
Finally, we simply need to flatten the list.
my #ordered = map { #$_ } #grouped;
Tested.

Related

Ember-table shows index on the header

I am using ember-table but have a weird behavior. It automatically adds index next to my header title.
But after I click one of the header to sort the table, the index will disappear as I want. How do I get rid of the index in the first place. Plus, without the sorting function, the table was normal.
After I click anyone of the header to sort the column, the index will go away.
Here is my sort object
sorts = [
{ valuePath: 'username' },
{ valuePath: 'total_assignment_count' },
{ valuePath: 'accepted_assignment_count' },
{ valuePath: 'accepted_rate' },
{ valuePath: 'acl_name' },
{ valuePath: 'repo_name'}
];
template
<EmberTable as |t|>
<t.head
#columns={{this.tableColumns}}
#sorts={{this.sorts}}
#onUpdateSorts={{action (mut this.sorts)}}
/>
<t.body #rows={{this.tableData}} />
</EmberTable>
I found the answer.
Use only one sorting key.
sorts = [ { valuePath: 'username' } ];
Moreover, if you specify more than 1 key, then ember-table will help sort the table by the sequence.
EX:
sorts = [
{ valuePath: 'username' },
{ valuePath: 'score' },
];
The table will sort username first and then score

MongoDB query to find text in third level array of objects

I have a Mongo collection that contains data on saved searches in a Vue/Laravel app, and it contains records like the following:
{
"_id" : ObjectId("6202f3357a02e8740039f343"),
"q" : null,
"name" : "FCA last 3 years",
"frequency" : "Daily",
"scope" : "FederalContractAwardModel",
"filters" : {
"condition" : "AND",
"rules" : [
{
"id" : "awardDate",
"operator" : "between_relative_backward",
"value" : [
"now-3.5y/d",
"now/d"
]
},
{
"id" : "subtypes.extentCompeted",
"operator" : "in",
"value" : [
"Full and Open Competition"
]
}
]
},
The problem is the value in the item in the rules array that has the decimal.
"value" : [
"now-3.5y/d",
"now/d"
]
in particular the decimal. Because of a UI error, the user was allowed to enter a decimal value, and so this needs to be fixed to remove the decimal like so.
"value" : [
"now-3y/d",
"now/d"
]
My problem is writing a Mongo query to identify these records (I'm a Mongo noob). What I need is to identify records in this collection that have an item in the filters.rules array with an item in the 'value` array that contains a decimal.
Piece of cake, right?
Here's as far as I've gotten.
myCollection.find({"filters.rules": })
but I'm not sure where to go from here.
UPDATE: After running the regex provided by #R2D2, I found that it also brings up records with a valid date string , e.g.
"rules" : [
{
"id" : "dueDate",
"operator" : "between",
"value" : [
"2018-09-10T19:04:00.000Z",
null
]
},
so what I need to do is filter out cases where the period has a double 0 on either side (i.e. 00.00). If I read the regex correctly, this part
[^\.]
is excluding characters, so I would want something like
[^00\.00]
but running this query
db.collection.find( {
"filters.rules.value": { $regex: /\.[^00\.00]*/ }
} )
still returns the same records, even though it works as expected in a regex tester. What am I missing?
To find all documents containing at least one value string with (.) , try:
db.collection.find( {
"filters.rules.value": { $regex: /\.[^\.]*/ }
} )
Or you can filter only the fields that need fix via aggregation as follow:
[direct: mongos]> db.tes.aggregate([ {$unwind:"$filters.rules"}, {$unwind:"$filters.rules.value"}, {$match:{ "filters.rules.value": {$regex: /\.[^\.]*/ } }} ,{$project:{_id:1,oldValue:"$filters.rules.value"}} ])
[
{ _id: ObjectId("6202f3357a02e8740039f343"), oldValue: 'now-3.5y/d' }
]
[direct: mongos]>
Later to update those values:
db.collection.update({
"filters.rules.value": "now-3.5y/d"
},
{
$set: {
"filters.rules.$[x].value.$": "now-3,5y/d-CORRECTED"
}
},
{
arrayFilters: [
{
"x.value": "now-3.5y/d"
}
]
})
playground

Terraform - transform list of lists to into a new list of lists

In Terraform, I need to transform my input data structure from e.g.:
vip_lists = [
["1.0.1.1", "1.0.1.2", "1.0.1.3", "1.0.1.4"]
["1.0.2.1", "1.0.2.2", "1.0.2.3", "1.0.2.4"]
["1.0.0.1", "1.0.0.2", "1.0.0.3", "1.0.0.4"]
]
to produce an output like this:
vip_sets = [
["1.0.1.1", "1.0.2.1", "1.0.0.1"]
["1.0.1.2", "1.0.2.2", "1.0.0.2"]
["1.0.1.3", "1.0.2.3", "1.0.0.3"]
["1.0.1.4", "1.0.2.4", "1.0.0.4"]
]
So essentially, i need to take my input list of lists and create an output which is again a list of lists but whose 0th list is a list of the 0th elements from each of the lists in the input...then the same again for the 1st and so on.
I can't know in advance how many lists will be in the input or how long they will be, but we can assume the lists will all be the same length if that helps.
I've tried pretty much everything I can think of and searched the web but since with no luck. All suggestions would be very welcome!
I once wrote version of this for lists of lists that are not the same length for one of our modules on github.com/mineiros-io where we used such transformations to create 2 dimensional sets of resources using count. (Those are not in use atm as we transformed them to maps for use with ressource level for_each).
locals {
matrix = [
["1.0.1.1", "1.0.1.4"],
["1.0.2.1", "1.0.2.2", "1.0.2.3", "1.0.2.4"],
["1.0.0.1", "1.0.0.3", "1.0.0.4"]
]
row_lengths = [
for row in local.matrix : length(row)
]
max_row_length = max(0, local.row_lengths...)
output = [
for i in range(0, local.max_row_length) : [
for j, _ in local.matrix : try(local.matrix[j][i], null)
]
]
output_compact = [
for i in range(0, local.max_row_length) : compact([
for j, _ in local.matrix : try(local.matrix[j][i], null)
])
]
}
output "matrix" {
value = local.output
}
output "compact" {
value = local.output_compact
}
which can handle dynamic list sizes and output them compact or filled with null values:
Outputs:
compact = [
[ "1.0.1.1", "1.0.2.1", "1.0.0.1" ],
[ "1.0.1.4", "1.0.2.2", "1.0.0.3" ],
[ "1.0.2.3", "1.0.0.4" ],
[ "1.0.2.4" ],
]
matrix = [
[ "1.0.1.1", "1.0.2.1", "1.0.0.1" ],
[ "1.0.1.4", "1.0.2.2", "1.0.0.3" ],
[ null, "1.0.2.3", "1.0.0.4" ],
[ null, "1.0.2.4", null ],
]
I know an answer is already accepted, but maybe some one can still make use of this dynamic version.
This is sort of horrible, but it works (Although I haven't tested what it'd do if vip_lists was empty. Probably crash, as I'm indexing to vip_lists[0] without checking):
locals {
vip_lists = [
["1.0.1.1", "1.0.1.2", "1.0.1.3", "1.0.1.4"],
["1.0.2.1", "1.0.2.2", "1.0.2.3", "1.0.2.4"],
["1.0.0.1", "1.0.0.2", "1.0.0.3", "1.0.0.4"]
]
vip_sets = [for i in range(0, length(local.vip_lists[0])): [for j in range(0, length(local.vip_lists)): local.vip_lists[j][i]]]
}
output "vip_sets" {
value = local.vip_sets
}
$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
vip_sets = [
[
"1.0.1.1",
"1.0.2.1",
"1.0.0.1",
],
[
"1.0.1.2",
"1.0.2.2",
"1.0.0.2",
],
[
"1.0.1.3",
"1.0.2.3",
"1.0.0.3",
],
[
"1.0.1.4",
"1.0.2.4",
"1.0.0.4",
],
]

Implement auto-complete feature using MongoDB search

I have a MongoDB collection of documents of the form
{
"id": 42,
"title": "candy can",
"description": "canada candy canteen",
"brand": "cannister candid",
"manufacturer": "candle canvas"
}
I need to implement auto-complete feature based on the input search term by matching in the fields except id. For example, if the input term is can, then I should return all matching words in the document as
{ hints: ["candy", "can", "canada", "canteen", ...]
I looked at this question but it didn't help. I also tried searching how to do regex search in multiple fields and extract matching tokens, or extracting matching tokens in a MongoDB text search but couldn't find any help.
tl;dr
There is no easy solution for what you want, since normal queries can't modify the fields they return. There is a solution (using the below mapReduce inline instead of doing an output to a collection), but except for very small databases, it is not possible to do this in realtime.
The problem
As written, a normal query can't really modify the fields it returns. But there are other problems. If you want to do a regex search in halfway decent time, you would have to index all fields, which would need a disproportional amount of RAM for that feature. If you wouldn't index all fields, a regex search would cause a collection scan, which means that every document would have to be loaded from disk, which would take too much time for autocompletion to be convenient. Furthermore, multiple simultaneous users requesting autocompletion would create considerable load on the backend.
The solution
The problem is quite similar to one I have already answered: We need to extract every word out of multiple fields, remove the stop words and save the remaining words together with a link to the respective document(s) the word was found in a collection. Now, for getting an autocompletion list, we simply query the indexed word list.
Step 1: Use a map/reduce job to extract the words
db.yourCollection.mapReduce(
// Map function
function() {
// We need to save this in a local var as per scoping problems
var document = this;
// You need to expand this according to your needs
var stopwords = ["the","this","and","or"];
for(var prop in document) {
// We are only interested in strings and explicitly not in _id
if(prop === "_id" || typeof document[prop] !== 'string') {
continue
}
(document[prop]).split(" ").forEach(
function(word){
// You might want to adjust this to your needs
var cleaned = word.replace(/[;,.]/g,"")
if(
// We neither want stopwords...
stopwords.indexOf(cleaned) > -1 ||
// ...nor string which would evaluate to numbers
!(isNaN(parseInt(cleaned))) ||
!(isNaN(parseFloat(cleaned)))
) {
return
}
emit(cleaned,document._id)
}
)
}
},
// Reduce function
function(k,v){
// Kind of ugly, but works.
// Improvements more than welcome!
var values = { 'documents': []};
v.forEach(
function(vs){
if(values.documents.indexOf(vs)>-1){
return
}
values.documents.push(vs)
}
)
return values
},
{
// We need this for two reasons...
finalize:
function(key,reducedValue){
// First, we ensure that each resulting document
// has the documents field in order to unify access
var finalValue = {documents:[]}
// Second, we ensure that each document is unique in said field
if(reducedValue.documents) {
// We filter the existing documents array
finalValue.documents = reducedValue.documents.filter(
function(item,pos,self){
// The default return value
var loc = -1;
for(var i=0;i<self.length;i++){
// We have to do it this way since indexOf only works with primitives
if(self[i].valueOf() === item.valueOf()){
// We have found the value of the current item...
loc = i;
//... so we are done for now
break
}
}
// If the location we found equals the position of item, they are equal
// If it isn't equal, we have a duplicate
return loc === pos;
}
);
} else {
finalValue.documents.push(reducedValue)
}
// We have sanitized our data, now we can return it
return finalValue
},
// Our result are written to a collection called "words"
out: "words"
}
)
Running this mapReduce against your example would result in db.words look like this:
{ "_id" : "can", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
{ "_id" : "canada", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
{ "_id" : "candid", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
{ "_id" : "candle", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
{ "_id" : "candy", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
{ "_id" : "cannister", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
{ "_id" : "canteen", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
{ "_id" : "canvas", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
Note that the individual words are the _id of the documents. The _id field is indexed automatically by MongoDB. Since indices are tried to be kept in RAM, we can do a few tricks to both speed up autocompletion and reduce the load put to the server.
Step 2: Query for autocompletion
For autocompletion, we only need the words, without the links to the documents.
Since the words are indexed, we use a covered query – a query answered only from the index, which usually resides in RAM.
To stick with your example, we would use the following query to get the candidates for autocompletion:
db.words.find({_id:/^can/},{_id:1})
which gives us the result
{ "_id" : "can" }
{ "_id" : "canada" }
{ "_id" : "candid" }
{ "_id" : "candle" }
{ "_id" : "candy" }
{ "_id" : "cannister" }
{ "_id" : "canteen" }
{ "_id" : "canvas" }
Using the .explain() method, we can verify that this query uses only the index.
{
"cursor" : "BtreeCursor _id_",
"isMultiKey" : false,
"n" : 8,
"nscannedObjects" : 0,
"nscanned" : 8,
"nscannedObjectsAllPlans" : 0,
"nscannedAllPlans" : 8,
"scanAndOrder" : false,
"indexOnly" : true,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 0,
"indexBounds" : {
"_id" : [
[
"can",
"cao"
],
[
/^can/,
/^can/
]
]
},
"server" : "32a63f87666f:27017",
"filterSet" : false
}
Note the indexOnly:true field.
Step 3: Query the actual document
Albeit we will have to do two queries to get the actual document, since we speed up the overall process, the user experience should be well enough.
Step 3.1: Get the document of the words collection
When the user selects a choice of the autocompletion, we have to query the complete document of words in order to find the documents where the word chosen for autocompletion originated from.
db.words.find({_id:"canteen"})
which would result in a document like this:
{ "_id" : "canteen", "value" : { "documents" : [ ObjectId("553e435f20e6afc4b8aa0efb") ] } }
Step 3.2: Get the actual document
With that document, we can now either show a page with search results or, like in this case, redirect to the actual document which you can get by:
db.yourCollection.find({_id:ObjectId("553e435f20e6afc4b8aa0efb")})
Notes
While this approach may seem complicated at first (well, the mapReduce is a bit), it is actual pretty easy conceptually. Basically, you are trading real time results (which you won't have anyway unless you spend a lot of RAM) for speed. Imho, that's a good deal. In order to make the rather costly mapReduce phase more efficient, implementing Incremental mapReduce could be an approach – improving my admittedly hacked mapReduce might well be another.
Last but not least, this way is a rather ugly hack altogether. You might want to dig into elasticsearch or lucene. Those products imho are much, much more suited for what you want.
Thanks to #Markus solution, I came up with something similar with aggregations instead. Knowing that map-reduce are flagged as deprecated for later versions.
const { MongoDBNamespace, Collection } = require('mongodb')
//.replace(/(\b(\w{1,3})\b(\W|$))/g,'').split(/\s+/).join(' ')
const routine = `function (text) {
const stopwords = ['the', 'this', 'and', 'or', 'id']
text = text.replace(new RegExp('\\b(' + stopwords.join('|') + ')\\b', 'g'), '')
text = text.replace(/[;,.]/g, ' ').trim()
return text.toLowerCase()
}`
// If the pipeline includes the $out operator, aggregate() returns an empty cursor.
const agg = [
{
$match: {
a: true,
d: false,
},
},
{
$project: {
title: 1,
desc: 1,
},
},
{
$replaceWith: {
_id: '$_id',
text: {
$concat: ['$title', ' ', '$desc'],
},
},
},
{
$addFields: {
cleaned: {
$function: {
body: routine,
args: ['$text'],
lang: 'js',
},
},
},
},
{
$replaceWith: {
_id: '$_id',
text: {
$trim: {
input: '$cleaned',
},
},
},
},
{
$project: {
words: {
$split: ['$text', ' '],
},
qt: {
$const: 1,
},
},
},
{
$unwind: {
path: '$words',
includeArrayIndex: 'id',
preserveNullAndEmptyArrays: true,
},
},
{
$group: {
_id: '$words',
docs: {
$addToSet: '$_id',
},
weight: {
$sum: '$qt',
},
},
},
{
$sort: {
weight: -1,
},
},
{
$limit: 100,
},
{
$out: {
db: 'listings_db',
coll: 'words',
},
},
]
// Closure for db instance only
/**
*
* #param { MongoDBNamespace } db
*/
module.exports = function (db) {
/** #type { Collection } */
let collection
/**
* Runs the aggregation pipeline
* #return {Promise}
*/
this.refreshKeywords = async function () {
collection = db.collection('listing')
// .toArray() to trigger the aggregation
// it returns an empty curson so it's fine
return await collection.aggregate(agg).toArray()
}
}
Please check for very minimal changes for your convenience.

Filter duplicates in MongoDB C++

I am looking to find all duplicates in my collection by flagging duplicates based on the date. The following was my attempt but I am not sure how to use cmdResult within update. Any clues?
//filter duplicates
bson::bo cmdResult;
bool ok = c.runCommand(dbcol, BSON("distinct" << "date"), cmdResult);
c.update(dbcol,Query("date"<<cmdResult<<NOT<<"_id"), BSON("$set"<<BSON("noise"<<"true")), false, true);
The "distinct" command will return you a list of all unique "date" values there are in the collection. But what you need is a list of "date" values that occur more than once.
You can get this list using the aggregate command, by grouping by "date" and counting the entries, then matching for counts > 1:
aggregate([
{ $group: { "_id": "$name", count: {$sum:1} } },
{ $match: { $gt: [ count, 1 ] } }
])
You would then update your collection (multi:true) by querying for "date" IN that list, setting the "noise" field:
update( {"name": {$in: [<list>]} },{$set: {"noise": true} }, true, false )
For help on aggregation, see http://docs.mongodb.org/manual/reference/aggregation/