C++ - Efficient way to group double vectors following a certain criteria - c++

Question

I have a list of objects saved in a CSV-like file using the following scheme:
[value11],...,[value1n],[label1]
[value21],...,[value2n],[label2]
...
[valuen1],...,[valuenn],[labeln]
(each line is a single object, i.e. a vector of doubles and the respective label).
I would like to collect them in groups with a certain custom criteria (i.e. same values at n-th and (n+1)-th position of all objects of that group). And i need to do that in the most efficient way, since the text file contains hundreds of thounsands of objects. I'm using the C++ programming language.
To do so, firstly I load all the CSV lines in a simple custom container (with getObject, getLabel and import methods). Then i use the following code to read them and make groups. "verifyGroupRequirements" is a function which returns true if the group conditions are satisfied, false otherwise.
for (size_t i = 0; i < ObjectsList.getSize(); ++i) {
MyObject currentObj;
currentObj.attributes = ObjectsList.getObject(i);
currentObj.label = ObjectsList.getLabel(i);
if (i == 0) {
// Sequence initialization with the first object
ObjectsGroup currentGroup = ObjectsGroup();
currentGroup.objectsList.push_back(currentObj);
tmpGroupList.push_back(currentGroup);
} else {
// if it is not the first pattern, then we check sequence conditions
list<ObjectsGroup>::iterator it5;
for (it5 = tmpGroupList.begin(); it5 != tmpGroupList.end(); ++it5) {
bool AddObjectToGroupRequirements =
verifyGroupRequirements(it5->objectsList.back(), currentObj) &
( (it5->objectsList.size() < maxNumberOfObjectsPerGroup) |
(maxNumberOfObjectsPerGroup == 0) );
if (AddObjectToGroupRequirements) {
// Object added to the group
it5->objectsList.push_back(currentObj);
break;
} else {
// If we can't find a group which satisfy those conditions and we
// arrived at the end of the list of groups, then we create a new
// group with that object.
size_t gg = std::distance(it5, tmpGroupList.end());
if (gg == 1) {
ObjectsGroup tmp1 = ObjectsGroup();
tmp1.objectsList.push_back(currentObj);
tmpGroupList.push_back(tmp1);
break;
}
}
}
}
if (maxNumberOfObjectsPerGroup > 0) {
// With a for loop we can take all the elements of
// tmpGroupList which have reached the maximum size
list<ObjectsGroup>::iterator it2;
for (it2 = tmpGroupList.begin(); it2 != tmpGroupList.end(); ++it2) {
if (it2->objectsList.size() == maxNumberOfObjectsPerGroup)
finalGroupList.push_back(*it2);
}
// Since tmpGroupList is a list we can use remove_if to remove them
tmpGroupList.remove_if(rmCondition);
}
}
if (maxNumberOfObjectsPerGroup == 0)
finalGroupList = vector<ObjectsGroup> (tmpGroupList.begin(), tmpGroupList.end());
else {
list<ObjectsGroup>::iterator it6;
for (it6 = tmpGroupList.begin(); it6 != tmpGroupList.end(); ++it6)
finalGroupList.push_back(*it6);
}
Where tmpGroupList is a list<MyObject>, finalGroupList is a vector<MyObject> and rmCondition is a boolean function that returns true if the size of a ObjectsGroup is bigger than a fixed value. MyObject and ObjectsGroup are two simple data structures, written in the following way:
// Data structure of the single object
class MyObject {
public:
MyObject(
unsigned short int &spaceToReserve,
double &defaultContent,
string &lab) {
attributes = vector<double>(spaceToReserve, defaultContent);
label = lab;
}
vector<double> attributes;
string label;
};
// Data structure of a group of object
class ObjectsGroup {
public:
list<MyObject> objectsList;
double health;
};
This code seems to work, but it is really slow. Since, as i said before, i have to apply it on a large set of objects, is there a way to improve it and make it faster? Thanks.
[EDIT] What I'm trying to achieve is to make groups of objects where each object is a vector<double> (got from a CSV file). So what I'm asking here is, is there a more efficient way to collect those kind of objects in groups than what is exposed in the code example above?
[EDIT2] I need to make groups using all of those vectors.

Answer 1

So, I'm reading your question...
... I would like to collect them in groups with a certain custom
criteria (i.e. same values at n-th and (n+1)-th position of all
objects of that group) ...
Ok, I read this part, and kept on reading...
... And i need to do that in the most efficient way, since the text file
contains hundreds of thounsands of objects...
I'm still with you, makes perfect sense.
... To do so, firstly I load all the CSV lines ...
{thud} {crash} {loud explosive noises}
Ok, I stopped reading right there, and didn't pay much attention to the rest of the question, including the large code sample. This is because we have a basic problem right from the start:
1) You say that your intention is, typically, to read only a small
portion of this huge CSV file, and...
2) ... to do that you load the entire CSV file, into a fairly sophisticated data structure.
These two statements are at odds with each. You're reading a huge number of values from a file. You are creating an object for each value. Based on the premise of your question, you're going to have a large number of these objects. But then, when all is said and done, you're only going to look at a small number of them, and throw the rest away?
You are doing a lot of work, presumably using up a lot of memory, and CPU cycles, loading a huge data set, only to ignore most of it. And you are wondering why you're having performance issues? Seems pretty cut and dry to me.
What would be an alternative way of doing this? Well, let's turn this whole problem inside out, and approach it piecemeal. Let's read a CSV file, one line at a time, parse the values in the CSV-formatted file, and pass the resulting strings to a lambda.
Something like this:
template<typename Callback> void parse_csv_lines(std::ifstream &i,
Callback &&callback)
{
std::string line;
while (1)
{
line.clear();
std::getline(i, line);
// Deal with missing newline on the last line...
if (i.eof() && line.empty())
break;
std::vector<std::string> words;
// At this point, you'll take this "line", and split it apart, at
// the commas, into the individual words. Parsing a CSV-
// formatted file. Not very exciting, you're doing this
// already, the algorithm is boring to implement, you know
// how to do it, so let's say you replace this entire comment
// with your boiler-plate CSV parsing logic from your existing
// code
callback(words);
}
}
Ok, now we've done that task of parsing the CSV file. Now, let's say we want to do the task you've set out in the beginning of your question, grab every nth and n+1th position. So...
void do_something_with_n_and_nplus1_words(size_t n)
{
std::ifstream input_file("input_file.csv");
// Insert code to check if input_file.is_open(), and if not, do
// whatever
parse_csv_lines(input_file,
[n]
(const auto &words)
{
// So now, grab words[n] and words[n+1]
// (after checking, of course, for a malformed
// CSV file with fewer than "n+2" values)
// and do whatever you want with them.
});
}
That's it. Now, you end up simply reading the CSV file, and doing the absolute minimum amount of work required to extract the nth and the n+1th values from each CSV file. It's going to be fairly difficult to come up with an approach that does less work (except, of course, micro-optimizations related to CSV parsing, and word buffers; or perhaps foregoing the overhead of std::ifstream, but rather mmap-ing the entire file, and then parsing it out by scanning its mmap-ed contents, something like that), I'd think.
For other similar one-off tasks, requiring only a small number of values from the CSV files, just write an appropriate lambda to fetch them out.
Perhaps, you need to retrieve two or more subsets of values from the large CSV file, and you want to read the CSV file once, maybe? Well, hard to give the best general approach. Each one of these situations will require individual analysis, to pick the best approach.