Segfault on reading large file with ifstream on 64 bit Debian - c++

Question

I am trying to read a large file (~5GB) using ifstream in C++.
Since I'm on a 64bit OS, I thought this shouldn't be a problem.
Still, I get a segfault. Everything runs fine with smaller files,
so I'm pretty sure that is where the problem is.
I'm using g++ (4.4.5-8) and libstdc++6 (4.4.5-8).
Thanks.
The code looks like this:
void load (const std::string &path, int _dim, int skip = 0, int gap = 0) {
std::ifstream is(path.c_str(), std::ios::binary);
BOOST_VERIFY(is);
is.seekg(0, std::ios::end);
size_t size = is.tellg();
size -= skip;
long int line = sizeof(float) * _dim + gap;
BOOST_VERIFY(size % line == 0);
long int _N = size / line;
reset(_dim, _N);
is.seekg(skip, std::ios::beg);
char *off = dims;
for (long int i = 0; i < N; ++i) {
is.read(off, sizeof(T) * dim);
is.seekg(gap, std::ios::cur);
off += stride;
}
BOOST_VERIFY(is);
}
The segfault is in the is.read line for i=187664.
T is float and I'm reading dim=1000 floats at a time.
When the segfault occures, i * stride is way smaller than size, so I'm not running past the end of the file.
dims is allocated here
void reset (int _dim, int _N)
{
BOOST_ASSERT((ALIGN % sizeof(T)) == 0);
dim = _dim;
N = _N;
stride = dim * sizeof(T) + ALIGN - 1;
stride = stride / ALIGN * ALIGN;
if (dims != NULL) delete[] dims;
dims = (char *)memalign(ALIGN, N * stride);
std::fill(dims, dims + N * stride, 0);
}

Answer 1

I don't know if this is the bug, but this code looks very C like and plenty of opportunity to leak. Any way try changing
void reset (int _dim, int _N)
to
void reset (size_t dim, size_t _N)
//I would avoid using leading underscores that is usually used to identify elements of the standard library.
When you are dealing with the size or index of something in memory ALWAYS use size_t, it is guaranteed to be able to hold the maximum size of an object including arrays.

Answer 2

I think you have to use _ftelli64 etc... to have the right size of your file, and to use long long (or _int64) variables to manage it. But it's C library. I don't find how to use ifstream with so big file (actualy > 2Go). Did you find the way ?
PS : In your case size_t is fine, but I'm not sure that's OK with 32-bit software. I'm sure it's OK with 64-bit.
int main()
{
string name="tstFile.bin";
FILE *inFile,*inFile2;
fopen_s(&inFile,name.c_str(),"rb");
if (!inFile)
{
cout<<"\r\n***error -> File not found\r\n";
return 0;
}
_fseeki64 (inFile,0L,SEEK_END);
long long fileLength = _ftelli64(inFile);
_fseeki64 (inFile,0L,SEEK_SET);
cout<<"file lg : "<<fileLength<<endl;
return 1;
}