Processing a large image text file - c++

Question

I'm using C++ and have a 1234 by 1234 text file with values 0 to 255. I have been trying to speed up my code because its used in real time with the user. Right now it takes .5 seconds to run with .4 seconds devoted to reading the text file to a vector<vector<int>>. I am using getline then istringstream. Below is the code I'm currently using. There is some stuff in there where I get rid of the first and last 50 columns as well as take the first chunk of rows into one vector and the second chunk into another vector because that's how I need it for processing purposes.
void readInRawData(string fileName, int start, int split, int finish, vector< vector <int> > &rawArrayTop, vector< vector <int> > &rawArrayBottom)
{
string line;
vector<int> rawRow;
int counter=0;
int value=0;
int numberOfColumns=0, numberOfRows=0;
ifstream rawImage;
rawImage.open(fileName.c_str()); //open file using fileName
if (rawImage.is_open()&&!is_empty(rawImage))
{
int length=0;
getline(rawImage,line);
istringstream ss(line);
while(ss>>value)//clump into values between spaces
{
length++;
}
while(getline(rawImage, line))//get row
{
if(counter<start)
{
}
else
{
break;
}
counter++;
}
while(getline(rawImage, line))//get row
{
if(counter<split)
{
rawRow.clear();
istringstream ss(line);
for(int i=0;i<50;i++)
{
ss>>value;
}
for(int i=0; i<length-100; i++)
{
ss>>value;
rawRow.push_back(value);
}
rawArrayTop.push_back(rawRow);
}
else
{
break;
}
counter++;
}
while(getline(rawImage, line))//get row
{
if(counter<finish)
{
rawRow.clear();
istringstream ss(line);
for(int i=0;i<50;i++)
{
ss>>value;
}
for(int i=0; i<length-100; i++)
{
ss>>value;
rawRow.push_back(value);
}
rawArrayBottom.push_back(rawRow);
}
else
{
break;
}
counter++;
}
rawImage.close();
}
//if it can't be opened throw error
else
{
throw rawArrayTop;
}
}

Answer 1

To get a real increase in performance, you'll have to rewrite totally.
while((ch = fgetc(fp)) != EOF)
{
if(isdigit(ch))
{
sample = sample * 10 + ch - '0';
onsample = 1;
}
else
{
if(onsample)
{
*out++ = sample;
sample = 0;
onsample = 0;
}
}
}
Set up out with malloc(width * height). Now it should zip through the file almost as fast as it can read it.

Answer 2

I will not give you the code, but I will suggest how to proceed here:
parsing text takes a long time. If real-time is important, pre-process the file to a binary format, since it can be loaded directly with read/write functions. You will need to create a stream in binary mode from the binary file and use istream::read.
try to avoid vector<vector<int>> unless you use scoped allocators, which I assume you are not using. This is bad for the cache. It is a much better fit to use a vector with n * m reserved space.
If you need bidimensional accesss, you can just code your functions for that:
using Matrix = vector<int>;
int & idx(Matrix, size_t row, size_t col);
Matrix mat(m * n);
idx(mat, 2, 3) = 17;
Another concern is that you must load into the Matrix. If you want to avoid redundant initialization and at the same time prereserve memory before loading the data, that is not possible with the stl vector, but you can use Boost.Container, which has an overload for reserve with default_init_t. That will not trigger initialization of elements in the vector.
if the values are between 0 and 255 use char, not int. You will fit more data at once in cache.