How to get line in a file partially by C++ - c++

Question

I want to read data in an input file partially. For example, input file is 1GB, I want to read only 100MB each time, then store in a vector. How can I continue reading the next line after the first loop? As you can see in my code below, after the first loop of i, maybe the vector v stored 1000 lines from the input file. I'm not sure if the next loop of i, the command while(std::getline(infile, line)) will continue to read from line 1001 from the input file or not? If not, how can I modify my code to get lines from the input in several groups (1~1000), (1001~2000), (2001~3000)... then store in vector v?
#define FILESIZE 1000000000 // size of the file on disk
#define TOTAL_MEM 100000 // max items the memory buffer can hold
void ExternalSort(std::string infilepath, std::string outfilepath)
{
std::vector<std::string> v;
int runs_count;
std::ifstream infile;
if(!infile.is_open())
{
std::cout << "Unable to open file\n";
}
infile.open(infilepath, std::ifstream::in);
if(FILESIZE % TOTAL_MEM > 0)
runs_count = FILESIZE/TOTAL_MEM + 1;
else
runs_count = FILESIZE/TOTAL_MEM;
// Iterate through the elements in the file
for(i = 0; i < runs_count; i++)
{
// Step 1: Read M-element chunk at a time from the file
for (j = 0; j < (TOTAL_MEM < FILESIZE ? TOTAL_MEM : FILESIZE); j++)
{
while(std::getline(infile, line))
{
// If line is empty, ignore it
if(line.empty())
continue;
new_line = line + "\n";
// Line contains string of length > 0 then save it in vector
if(new_line.size() > 0)
v.push_back(new_line);
}
}
// Step 2: Sort M elements
sort(v.begin(), v.end()); //sort(v.begin(), v.end(), compare);
// Step 3: Create temporary files and write sorted data into those files.
std::ofstream tf;
tf.open(tfile + ToString(i) + ".txt", std::ofstream::out | std::ofstream::app);
std::ostream_iterator<std::string> output_iterator(tf, "\n");
std::copy(v.begin(), v.end(), output_iterator);
v.clear();
//for(std::vector<std::string>::iterator it = v.begin(); it != v.end(); ++it)
// tf << *it << "\n";
tf.close();
}
infile.close();

Answer 1

I didn’t have the patience to check the whole code. It was easier to write a splitter from scratch. Here are some observations, anyhow:
std::ifstream infile;
if (!infile.is_open())
{
std::cout << "Unable to open file\n";
}
infile.open(infilepath, std::ifstream::in);
You will always get the message since you check before opening the file. One correct way to open a file is:
std::ifstream infile(infilepath);
if (!infile)
throw "could not open the input file";
if (infile.peek() == std::ifstream::traits_type::eof())
This will be true, for instance, even for nonexistent files. The algorithm should work for empty files, too.
if(FILESIZE % TOTAL_MEM > 0)
runs_count = FILESIZE/TOTAL_MEM + 1;
else
runs_count = FILESIZE/TOTAL_MEM;
Why do you need the number of resulting files before generate them? You will never be able to calculate it correctly since it depends on how long lines are (you cannot read half of line just to fit it into TOTAL_MEM). You should read from input file at most TOTAL_MEM bytes (but a line, at least), sort & save and then continue from where you left (see the loop in execute, below).
How can I continue reading the next line after the first loop?
If you do not close the input stream, the next read will continue from exactly where you left.
A solution:
#include <iostream>
#include <fstream>
#include <string>
#include <algorithm>
#include <vector>
#include <iterator>
std::vector<std::string> split_file(const char* fn, std::size_t mem); // see the implementation below
int main()
{
const std::size_t max_mem = 8;
auto r = split_file("input.txt", max_mem);
std::cout << "generated files:" << std::endl;
for (const auto& fn : r)
std::cout << fn << std::endl;
}
class split_file_t
{
public:
split_file_t(std::istream& is, std::size_t mem) :is_{ is }, mem_{ mem }
{
// nop
}
std::vector<std::string> execute()
{
while (make_file())
;
return std::move(ofiles_);
}
protected:
std::istream& is_;
std::size_t mem_;
std::vector<std::string> ofiles_;
static std::string make_temp_file()
{
std::string fn(512, 0);
tmpnam_s(&fn.front(), fn.size()); // this might be system dependent
std::ofstream os(fn);
os.close();
return fn;
}
bool make_file()
{
using namespace std;
// read lines
vector<string> lines;
{
streamsize max_gpos = is_.tellg() + streamsize(mem_);
string line;
while (is_.tellg() < max_gpos && getline(is_, line))
lines.push_back(line);
}
//
if (lines.empty())
return false;
// sort lines
sort(lines.begin(), lines.end());
// save lines
{
string ofile = make_temp_file();
ofstream os{ ofile };
if (!os)
throw "could not open output file";
copy(lines.begin(), lines.end(), ostream_iterator<string>(os, "\n"));
ofiles_.push_back(ofile);
}
//
return bool(is_);
}
};
std::vector<std::string> split_file(const char* fn, std::size_t mem)
{
using namespace std;
ifstream is{ fn };
if (!is)
return vector<string>();
return split_file_t{ is, mem }.execute();
}