Elegantly (iterating) parsing data in chunks? - c++

Question

We have some "iterable" collection of data, for instance: std::vector<Foo> bar.
We wish to process Foo elements from bar until some condition is met in which point we "yield" (think Python) the result of all these processed Foos and wait until the next chunk of parsed information is requested.
So far what we're doing is this:
ParsedChunk foobar( std::vector<Foo> bar, size_t* start_from) {
size_t& i = *start_from;
ParsedChunk result_so_far;
for (;i < bar.size(); i++) {
process_some_foo_and_update_chunk(result_so_far, bar[i]);
if (check_condition(? ? ?) {
return result_so_far;
}
}
}
Any suggestions for doing this better?

Answer 1

As I already pointed out in my comment, this is IMO a very good case for an custom iterator:
The iterator scans through your range, as long as some predicate holds, and
when the predicate isn't true for some element, calls a function with the sub range of elements where the predicate held (plus the one that didn't satisfy the predicate). The result of that function call is then the value you get when you dereference the iterator.
template<typename Fn, typename Predicate, typename Iterator>
struct collector {
using value_type = typename std::result_of<Fn(Iterator, Iterator)>::type;
using pointer = value_type const *;
using reference = value_type const &;
using difference_type = std::ptrdiff_t;
using iterator_category = std::forward_iterator_tag;
value_type cache;
Fn fn;
Predicate predicate;
Iterator pos, from, stop;
collector(Fn&& fn, Predicate&& p, Iterator&& s, Iterator&& e)
: fn(std::forward<Fn>(fn)),
predicate(std::forward<Predicate>(p)),
pos(std::forward<Iterator>(s)),
from(pos),
stop(std::forward<Iterator>(e))
{
next_range();
}
collector & operator++(void) {
next_range();
return *this;
}
reference operator*(void) const {
return cache;
}
void next_range(void) {
from = pos;
if (pos == stop) return;
for (; pos != stop; ++pos) {
if (not predicate(*pos)) {
++pos;
break;
}
}
cache = fn(from, pos);
}
collector end_of_range(void) const {
auto copy = collector{*this};
copy.pos = copy.stop;
copy.from = copy.stop;
return copy;
}
bool operator==(collector const & rhs) const {
return (from == rhs.from) and (pos == rhs.pos) and (stop == rhs.stop);
}
bool operator!=(collector const & rhs) const {
return (from != rhs.from) or (pos != rhs.pos) or (stop != rhs.stop);
}
};
template<typename Fn, typename Predicate, typename Iterator>
auto make_collector_range(Fn&& fn, Predicate&& p, Iterator&& s, Iterator&& e) {
using I = typename std::decay<Iterator>::type;
using P = typename std::decay<Predicate>::type;
using F = typename std::decay<Fn>::type;
using C = collector<F,P,I>;
auto start = C{
std::forward<Fn>(fn), std::forward<Predicate>(p),
std::forward<Iterator>(s), std::forward<Iterator>(e)};
auto stop = start.end_of_range();
return make_pair(std::move(start), std::move(stop));
}
An example usage, calculating the sum of the numbers till 50, but not in one step, but in steps of 15 numbers each:
int main(int, char**) {
vector<int> numbers = vector<int>(50);
generate(begin(numbers), end(numbers),
[i = 0] (void) mutable{
return ++i;
});
copy(begin(numbers), end(numbers), ostream_iterator<int>{cout, " "});
cout << endl;
auto collected = make_collector_range(
[](auto const & from, auto const & to) {
return accumulate(from, to, 0);
},
[](auto const & num) {
return not ((num % 3 == 0) and (num % 5 == 0));
},
begin(numbers), end(numbers));
copy(collected.first, collected.second, ostream_iterator<int>{cout, " "});
cout << endl;
bool passed = accumulate(collected.first, collected.second, 0) == (50*51)/2;
cout << "test " << (passed ? "passed" : "failed") << endl;
return 0;
}
(Live demo here)
(Note: This example uses a fixed "step" width, and predicate and function are unrelated to each other and don't maintain state, but none of this is required by the iterator.)
I hope the intention of the code is clear, if not I can try to provide a more detailed explanation about its workings.