10 fold size increase when reading file into struct - c++

Question

I am trying to read a csv file into a struct containing a vector of vector of strings. The file contains ~2 million lines and size on disk is ~350 mb. When I read the file into struct top shows me that the on reading the full file, the program is now using almost 3.5GB of my memory. I have used vector reserve to try to limit vector capacity increase on push_back.
#include<iomanip>
#include<stdio.h>
#include<stdlib.h>
#include<iostream>
#include<fstream>
#include<string.h>
#include<sstream>
#include<math.h>
#include<vector>
#include<algorithm>
#include<array>
#include<ctime>
#include<boost/algorithm/string.hpp>
using namespace std;
struct datStr{
vector<string> colNames;
vector<vector<string>> data;
};
datStr readBoost(string fileName)
{
datStr ds;
ifstream inFile;
inFile.open(fileName);
string line;
getline(inFile, line);
vector<string> colNames;
stringstream ss(line);
string item;
int i = 0;
vector<int> colTypeInt;
while(getline(ss, item, ','))
{
item.erase( remove( item.begin(), item.end(), ' ' ), item.end() );
colNames.push_back(item);
vector<string> colVec;
ds.data.push_back(colVec);
ds.data[i].reserve(3000000);
i++;
}
int itr = 0;
while(getline(inFile, line))
{
vector<string> rowStr;
boost::split(rowStr, line, boost::is_any_of(","));
for(int ktr = 0; ktr < rowStr.size(); ktr++)
{
rowStr[ktr].erase( remove( rowStr[ktr].begin(), rowStr[ktr].end(), ' ' ), rowStr[ktr].end() );
ds.data[ktr].push_back(rowStr[ktr]);
}
itr++;
}
int main()
{
datStr ds = readBoost("file.csv");
while(true)
{
}
}
PS: The last while is just so I can monitor the memory usage on completion of the program.
Is this something expected when using vectors or am I missing something here?
Another interesting fact. I started adding up size and capacity for each string in the read loop. Surprisingly it just adds up to 1/10 of what I am shown in top on ubuntu? Could it be that top is misreporting or my compiler is allocating too much space?

Answer 1

I tested your code with an input file that has 1886850 lines of text, with a size of 105M.
With your code, the memory consumption was about 2.5G.
Then, I started modifying how data is stored.
First test:
Change datStr to:
struct datStr{
vector<string> colNames;
vector<string> lines;
};
This reduced the memory consumption to 206M. That's more than 10 fold reduction in size. It's clear that the penalty for using
vector<vector<string>> data;
is rather stiff.
Second test:
Change datStr to:
struct datStr{
vector<string> colNames;
vector<string> lines;
vector<vector<string::size_type>> indices;
};
with indices keeping track of where the tokens in lines start. You can extract the tokens from each line by using lines and indices.
With this change, the memory consumption went to 543MB but sill is five times smaller than the original.
Third test
Change dataStr to:
struct datStr{
vector<string> colNames;
vector<string> lines;
vector<vector<unsigned int>> indices;
};
With this change, the memory consumption came down to 455MB. This should work if you don't expect your lines to be longer or equal to UINT_MAX.
Fourth Test
Change dataStr to:
struct datStr{
vector<string> colNames;
vector<string> lines;
vector<vector<unsigned short>> indices;
};
With this change, the memory consumption came down to 278MB. This should work if you don't expect your lines to be longer or equal to USHRT_MAX. For this case, the overhead of indices is really small, only 72MB.
Here's the modified code I used for my tests.
#include<iomanip>
#include<stdio.h>
#include<stdlib.h>
#include<iostream>
#include<fstream>
#include<string.h>
#include<sstream>
#include<math.h>
#include<vector>
#include<algorithm>
#include<array>
#include<ctime>
// #include<boost/algorithm/string.hpp>
using namespace std;
struct datStr{
vector<string> colNames;
vector<string> lines;
vector<vector<unsigned short>> data;
};
void split(vector<unsigned short>& rowStr, string const& line)
{
string::size_type begin = 0;
string::size_type end = line.size();
string::size_type iter = begin;
while ( iter != end)
{
++iter;
if ( line[iter] == ',' )
{
rowStr.push_back(static_cast<unsigned short>(begin));
++iter;
begin = iter;
}
}
if (begin != end )
{
rowStr.push_back(static_cast<unsigned short>(begin));
}
}
datStr readBoost(string fileName)
{
datStr ds;
ifstream inFile;
inFile.open(fileName);
string line;
getline(inFile, line);
vector<string> colNames;
stringstream ss(line);
string item;
int i = 0;
vector<int> colTypeInt;
while(getline(ss, item, ','))
{
item.erase( remove( item.begin(), item.end(), ' ' ), item.end() );
ds.colNames.push_back(item);
}
int itr = 0;
while(getline(inFile, line))
{
ds.lines.push_back(line);
vector<unsigned short> rowStr;
split(rowStr, line);
ds.data.push_back(rowStr);
}
}
int main(int argc, char** argv)
{
datStr ds = readBoost(argv[1]);
while(true)
{
}
}

Answer 2

Your vector<vector<string>> suffers from the costs of indirection (pointer members to dynamically allocated memory), housekeeping (members supporting size()/end()/capacity()), and the housekeeping and rounding-up internal to the dynamic memory allocation functions... if you look at the first graph titled Real memory consumption for different string lengths here it suggests total overheads around 40-45 bytes per string for a 32 bit app built with G++ 4.6.2, though an implementation could potentially get this as low as 4 bytes for strings of up to ~4 characters. Then there's waste for vector overheads....
You can address the issue in any of several ways, depending on your input data and efficiency needs:
store vector<std::pair<string, Column_Index>>, where Column_Index is a class you write that records the offsets in the string where each field appear
store vector<std::string> where column values are padded to known maximum widths, which will help most if the lengths are small, fixed and/or similar (e.g. date/times, small monetary amounts, ages)
memory map the file, then store offsets (but unquoting/unescaping is an issue - you could do that in-place, e.g. abc""def or abc\"def (whichever you support) -> abc"deff)
with the last two approaches, you can potentially overwrite the character after each field with a NUL if that's useful to you, so you can treat the fields as "C"-style ASCIIZ NUL-terminated strings
if some/all fields contain values like values like say 1.23456789012345678... where the textual representation may be longer than a binary inbuilt type (float, double, int64_t), doing a conversion before storage makes sense
similarly, if there's a set of repeating values - like a field of what are logically enumeration identifiers, you can encoding them as integers, or if the values are repetitive but not known until runtime, you can create a bi-directional mapping from incrementing indices to values

Answer 3

A couple things come to mind:
You say your file has about 2 million lines, but you reserve space for 3 million strings for each column. Even if you only have one column, that's a lot of wasted space. If you have a bunch of columns, that's a ton of wasted space. It might be informative to see how much space difference it makes if you don't reserve at all.
string has a small* but nonzero amount of overhead that you're paying for every single field in your 2-million line file. If you really need to hold all the data in memory at once and it's causing problems to do so, this may actually be a case where you're better off just using char* instead of string. But I'd only resort to this if adjusting reserve doesn't help.
* The overhead due to metadata is small, but if the strings are allocating extra capacity for their internal buffers, that could really add up. See this recent question.
Update: The problem with your update is that you are storing pointers to temporary std::string objects in datStr. By the time you get around to printing, those strings have been destroyed and your pointers are wild.
If you want a simple, safe way to store your strings in datStr that doesn't allocate more space than it needs, you could use something like this:
class TrivialReadOnlyString
{
private:
char* m_buffer;
public:
TrivialReadOnlyString(const std::string& src)
{
InitFrom(src.c_str());
}
TrivialReadOnlyString(const TrivialReadOnlyString& src)
{
InitFrom(src.m_buffer);
}
~TrivialReadOnlyString()
{
delete[] m_buffer;
}
const char* Get()
{
return m_buffer;
}
private:
void InitFrom(const char* src)
{
// Can switch to the safe(r) versions of these functions
// if you're using vc++ and it complains.
size_t length = strlen(src);
m_buffer = new char[length + 1];
strcpy(m_buffer, src);
}
};
There are a lot of further enhancements that could be made to this class, but I think it is sufficient for your program's needs as shown. This will fragment memory more than Blastfurnace's idea of storing the whole file in a single buffer. however.
If there is a lot of repetition in your data, you might also consider "folding" the repeats into a single object to avoid redundantly storing the same strings in memory over and over (flyweight pattern).

Answer 4

Indulge me while I take a very different approach in answering your question. Others have already answered your direct question quite well, so let me provide another perspective entirely.
Do you realize that you could store that data in memory with a single allocation, plus one pointer for each line, or perhaps one pointer per cell?
On a 32 bit machine, that's 350MB + 8MB (or 8MB * number columns).
Did you know that it's easy to parallelize CSV parsing?
The problem you have is layers and layers of bloat. ifstream, stringstream, vector<vector<string>>, and boost::split are wonderful if you don't care about size or speed. All of that can be done more directly and at lower cost.
In situations like this, where size and speed do matter, you should consider doing things the manual way. Read the file using an API from your OS. Read it into a single memory location, and modify the memory in place by replacing commas or EOLs with '\0'. Store pointers to those C strings in your datStr and you're done.
You can write similar solutions for variants of the problem. If the file is too large for memory, you can process it in pieces. If you need to convert data to other formats like floating point, that's easy to do. If you'd like to parallelize parsing, that's far easier without the extra layers between you and your data.
Every programmer should be able to choose to use convenience layers or to use simpler methods. If you lack that choice, you won't be able to solve some problems.