How to speed up loading text file to multi vector - c++

Question

I have to load large files (several GB) with data and I want to load them to two dimensional vector. Code below does the job, but it is insanely slow. To be more specific, the goal is to get all lines where values in 2nd column are equal to index(_lh,_sh). And then exclude the lines where 4th column value is same as line+1 and line-1.
Now, I'am new to c++ and I usualy code in Python (have working code for this problem already). But I need it to be as fast as posible so I tried to rewrite my python code to C++. But it rus slower than Python now (and only getting the data to vector is implemented)... so before I proceed, I want to improve that.
From what I have found in similar questions, the problem would be dynamic vectors, .push_back() and getline().
I am rather confused about maping and chunk loading mentioned in similar questions so I am not able to change the code acording to these.
Could you help me to optimize this code?
Thank you.
#include <iostream>
#include <sstream>
#include <fstream>
#include <array>
#include <string>
#include <vector>
using namespace std;
int pixel(int radek, int sloupec, int rozmer = 256) {
int index = (radek - 1) * rozmer + sloupec;
int index_lh = (index - rozmer - 1);
int index_sh = (index - rozmer);
int index_ph = (index - rozmer + 1);
int index_l = (index - 1);
int index_p = (index + 1);
int index_ld = (index + rozmer - 1);
int index_sd = (index + rozmer);
int index_pd = (index + rozmer + 1);
array<int, 9> index_all = { {index, index_lh, index_sh, index_ph, index_l, index_p, index_ld, index_sd, index_pd } };
vector<vector<string>> Data;
vector<string> Line;
string line;
for (int m = 2; m < 3; m++) {
string url = ("e:/TPX3 - kalibrace - 170420/ToT_ToA_calib_Zn_" + to_string(m) + string(".t3pa"));
cout << url << endl;
ifstream infile(url);
if (!infile)
{
cout << "Error opening output file" << endl;
system("pause");
return -1;
}
while (getline(infile, line))
{
Line.push_back(line);
istringstream txtStream(line);
string txtElement;
vector<string> Element;
while (getline(txtStream, txtElement, '\t')){
Element.push_back(txtElement);
}
Data.push_back(Element);
}
}
cout << Data[1][0] << ' ' << Data[1][1] << ' ' << Data[1][2] << endl;
return 0;
}
int main()
{
int x = pixel(120, 120);
cout << x << endl;
system("pause");
return 0;
}

Answer 1

Vectors can get slow if their underlying buffer gets reallocated often. A vector is required to be implemented on a buffer of continuous memory, and every time the buffer limit is exceeded, it will have to allocate a new and larger buffer, and then copy the content from the old buffer to the new buffer. If you have an idea of how big buffers you require (you don't need to be excact), you can help the program to allocate a buffer of appropriate size by using e.g. Data.reserve(n) (where n is approximately the number of elements you think you need). This does note change the "size" of the vector, just the size of the underlying buffer. As a concluding remark, I have to say I haven't really ever benchmarked this, so this may or may not improve the performance of your program.
EDIT: Though, I deem it a bit more likely that the performance is a bit bottled by the line Data.push_back(Element); which makes a copy of the Element-vector. If you're using C++11, I believe it's possible to work around this by doing something like Data.emplace_back(std::move(Element)); in which case you can't alter Element afterwards (it's content is moved). You would also need to include memory for std::move.

Answer 2

In the while loop, you could try changing the lines from
while (getline(infile, line))
{
Line.push_back(line);
istringstream txtStream(line);
string txtElement;
vector<string> Element;
while (getline(txtStream, txtElement, '\t')){
Element.push_back(txtElement);
}
Data.push_back(Element);
}
to:
while (getline(infile, line))
{
Line.push_back(line);
istringstream txtStream(line);
string txtElement;
//vector<string> Element; [-]
Data.emplace_back(); // [+]
while (getline(txtStream, txtElement, '\t')) {
//Element.push_back(txtElement); [-]
Data.back().push_back(txtElement); // [+]
}
//Data.push_back(Element); [-]
}
That way, the vectors in Data don't need to get moved or copied there -- they are already constructed, albeit empty. The vectors in Data are default-constructed with .emplace_back(). We get the last element in Data with the .back() function, and push our values as usual with .push_back(). Hopefully this helps :)

Answer 3

You can try using old C file reading API (FILE*, fopen(), etc.) or setting a bigger buffer for std::istringstream as follows
constexp std::size_t dimBuff { 10240 } // 10K, by example
char myBuff[dimBuff];
// ...
istringstream txtStream(line);
txtStream.rdbuf()->pubsetbuf(myBuff, dimBuff);
Another thing that you can try is using std::deques instead of std::vectors (but I've no idea if this is useful).
As suggested by muos, you can use move semantics; you can use emplace_back() also.
So I suggest to try with
Element.push_back(std::move(txtElement));
Data.push_back(std::move(Element));
or
Element.emplace_back(std::move(txtElement));
Data.emplace_back(std::move(Element));
You can also swith the following lines (there isn't a move constructor from a string for std::istringstream, if I'm not wrong)
Line.push_back(line);
istringstream txtStream(line);
adding move semantics (and emplace_back())
istringstream txtStream(line);
Line.emplace_back(std::move(line));
p.s.: obviously reserve() is usefull

Answer 4

You can also use reserve(int) on the vectors so they are created closer to the target size.
That too can avoid a lot of vector hopping around the heap, as the vector will only be recreated of it passes the target size.
You can call reserve again if vector passes the size you previously reserved:
vector<int> vec;
vec.reserve(10);
for (int i=0;i < 1000; i++)
{
if ( vec.size() == vec.capacity() )
{
vec.reserve(vec.size()+10);
}
vec.push_back(i);
}