How to read only some previously know lines using ifstream (C++) - c++

Question

By preprocessing on the file i found some line for further processing, know i want to read that lines. is there any faster solution than reading lines one by one using ifstream::getline(...) ?
For example i know that i want only lines of product 4 (0-4-8-12-16-...) or special line numbers stored in a vector...
Now I'm doing this :
string line;
int counter = 0;
while( getline(ifstr,line) ){
if(counter%4 =0){
// some code working with line
}
}
but i want something like this (if faster)
while(getline(ifstr,line)){
// some code working with line
while(++counter%4 !=0){ // or checking on index vector
skipline(ifstr)
}
}
Let me mention again that i have some line index (sorted but not this regular) but i use this example of product4 for simplicity.
Edit: and i want to jump to line at the begining, for example i know that i need to read from line number 2000, how to skip 1999 lines quickly ?
Thanks all

Answer 1

Because #caps said this left him with the feeling there's nothing in the standard library to help with this kind of task, I felt compelled to demonstrate otherwise :)
Live On Coliru
template <typename It, typename Out, typename Filter = std::vector<int> >
Out retrieve_lines(It begin, It const end, Filter lines, Out out, char const* delim = "\\n") {
if (lines.empty())
return out;
// make sure input is orderly
assert(std::is_sorted(lines.begin(), lines.end()));
assert(lines.front() >= 0);
std::regex re(delim);
std::regex_token_iterator<It> line(begin, end, re, -1), eof;
// make lines into incremental offsets
std::adjacent_difference(lines.begin(), lines.end(), lines.begin());
// iterate advancing by each offset requested
auto advanced = [&line, eof](size_t n) { while (line!=eof && n--) ++line; return line; };
for (auto offset = lines.begin(); offset != lines.end() && advanced(*offset) != eof; ++offset) {
*out++ = *line;
}
return out;
}
This is noticably more generic. The trade off (for now) is that the tokenizing iterator requires a random access iterator. I find this a good trade-off because "random access" on files really asks for memory mapped files anyways
Live Demo 1: from string to vector<string>
Live On Coliru
int main() {
std::vector<std::string> output_lines;
std::string is(" a b c d e\nf g hijklmnop\nqrstuvw\nxyz");
retrieve_lines(is.begin(), is.end(), {0,3,999}, back_inserter(output_lines));
// for debug purposes
for (auto& line : output_lines)
std::cout << line << "\n";
}
Prints
a b c d e
xyz
Live Demo 2: From file to cout
Live On Coliru
#include <boost/iostreams/device/mapped_file.hpp>
int main() {
boost::iostreams::mapped_file_source is("/etc/dictionaries-common/words");
retrieve_lines(is.begin(), is.end(), {13,784, 9996}, std::ostream_iterator<std::string>(std::cout, "\n"));
}
Prints e.g.
ASL's
Apennines
Mercer's
The use of boost::iostreams::mapped_file_source can easily be replaced with straight up ::mmap but I found it uglier in the presentation sample.

Answer 2

Store std::fstream::streampos instances corresponding to line beginnings of your file into a std::vector and then you can access a specific line using the index of this vector. A possible implementation follows,
class file_reader {
public:
// load file streampos offsets during construction
explicit file_reader(const std::string& filename)
: fs(filename) { cache_file_streampos(); }
std::size_t lines() const noexcept { return line_streampos_vec.size(); }
// get a std::string representation of specific line in file
std::string read_line(std::size_t n) {
if (n >= line_streampos_vec.size() - 1)
throw std::out_of_range("out of bounds");
navigate_to_line(n);
std::string rtn_str;
std::getline(fs, rtn_str);
return rtn_str;
}
private:
std::fstream fs;
std::vector<std::fstream::streampos> line_streampos_vec;
const std::size_t max_line_length = // some sensible value
// store file streampos instances in vector
void cache_file_streampos() {
std::string s;
s.reserve(max_line_length);
while (std::getline(fs, s))
line_streampos_vec.push_back(fs.tellg());
}
// go to specific line in file stream
void navigate_to_line(std::size_t n) {
fs.clear();
fs.seekg(line_streampos_vec[n]);
}
};
Then you can read a specific line of your file via,
file_reader fr("filename.ext");
for (int i = 0; i < fr.lines(); ++i) {
if (!(i % 4))
std::string line_contents = fr.read_line(i); // then do something with the string
}

Answer 3

ArchbishopOfBanterbury's answer is nice, and I would agree with him that you will get cleaner code and better efficiency by just storing the character positions of the beginning of each line when you do your preprocessing.
But, supposing that is not possible (perhaps the preprocessing is handled by some other API, or is from user input), there is a solution that should do the minimal amount of work necessary to read in only the specified lines.
The fundamental problem is that, given a file with variable line lengths, you cannot know where each line begins and ends, since a line is defined as a sequence of characters that end in '\n'. So, you must parse every character to check and see if it is '\n' or not, and if so, advance your line counter and read in the line if the line counter matches one of your desired inputs.
auto retrieve_lines(std::ifstream& file_to_read, std::vector<int> line_numbers_to_read) -> std::vector<std::string>
{
auto begin = std::istreambuf_iterator<char>(file_to_read);
auto end = std::istreambuf_iterator<char>();
auto current_line = 0;
auto next_line_num = std::begin(line_numbers_to_read);
auto output_lines = std::vector<std::string>();
output_lines.reserve(line_numbers_to_read.size()); //this may be a silly "optimization," since all the strings are still separate unreserved buffers
//we can bail if we've reached the end of the lines we want to read, even if there are lines remaining in the stream
//we *must* bail if we've reached the end of the stream, even if there are supposedly lines left to read; that input must have been incorrect
while(begin != end && next_line_num != std::end(line_numbers_to_read))
{
if(current_line == *next_line_num)
{
auto matching_line = std::string();
if(*begin != '\n')
{
//potential optimization: reserve matching_line to something that you expect will fit most/all of your input lines
while(begin != end && *begin != '\n')
{
matching_line.push_back(*begin++);
}
}
output_lines.emplace_back(matching_line);
++next_line_num;
}
else
{
//skip this "line" by finding the next '\n'
while(begin != end && *begin != '\n')
{
++begin;
}
}
//either code path in the previous if/else leaves us staring at the '\n' at the end of a line,
//which is not the right state for the next iteration of the loop.
//So skip this '\n' to get to the beginning of the next line
if (begin != end && *begin == '\n')
{
++begin;
}
++current_line;
}
return output_lines;
}
Here it is live on Coliru, along with the input I tested it with. As you can see, it correctly handles empty lines as well as correctly handling being told to grab more lines than are in the file.