Understanding composition of lazy range-based functions - c++

TL;DR
What's wrong with the last commented block of lines below?
// headers and definitions are in the down the question
int main() {
std::vector<int> v{10,20,30};
using type_of_temp = std::vector<std::pair<std::vector<int>,int>>;
// seems to work, I think it does work
auto temp = copy_range<type_of_temp>(v | indexed(0)
| transformed(complex_keep_index));
auto w = temp | transformed(distribute);
print(w);
// shows undefined behavior
//auto z = v | indexed(0)
// | transformed(complex_keep_index)
// | transformed(distribute);
//print(z);
}
Or, in other words, what makes piping v | indexed(0) into transformed(complex_keep_index) well defined, but piping v | indexed(0) | transformed(complex_keep_index) into transformed(distribute) undefined behavior?
Extended version
I have a container of things,
std::vector<int> v{10,20,30};
and I have a function which generates another container from each of those things,
// this is in general a computation of type
// T -> std::vector<U>
constexpr auto complex_comput = [](auto const& x){
return std::vector{x,x+1,x+2}; // in general the number of elements changes
};
so if I was to apply the complex_comput to v, I'd get,
{{10, 11, 12}, {20, 21, 22}, {30, 31, 32}}
and if I was to also concatenate the results, I'd finally get this:
{10, 11, 12, 20, 21, 22, 30, 31, 32}
However, I want to keep track of the index where each number came from, in a way that the result would encode something like this:
0 10
0 11
0 12
1 20
1 21
1 22
2 30
2 31
2 32
To accomplish this, I (eventually) came up with this solution, where I attempted to make use of ranges from Boost. Specifically I do the following:
use boost::adaptors::indexed to attach the index to each element of v
transform each resulting "pair" in a std::pair storing the index and result of the application of complex_comput to the value,
and finally transforming each std::pair<st::vector<int>,int> in a std::vector<std::pair<int,int>>.
However, I had to give up on the range between 2 and 3, using a helper "true" std::vector in between the two transformations.
#include <boost/range/adaptor/indexed.hpp>
#include <boost/range/adaptor/transformed.hpp>
#include <boost/range/iterator_range_core.hpp>
#include <iostream>
#include <utility>
#include <vector>
using boost::adaptors::indexed;
using boost::adaptors::transformed;
using boost::copy_range;
constexpr auto complex_comput = [](auto const& x){
// this is in general a computation of type
// T -> std::vector<U>
return std::vector{x,x+1,x+2};
};
constexpr auto complex_keep_index = [](auto const& x){
return std::make_pair(complex_comput(x.value()), x.index());
};
constexpr auto distribute = [](auto const& pair){
return pair.first | transformed([n = pair.second](auto x){
return std::make_pair(x, n);
});
};
template<typename T>
void print(T const& w) {
for (auto const& elem : w) {
for (auto i : elem) {
std::cout << i.second << ':' << i.first << ' ';
}
std::cout << std::endl;
}
}
int main() {
std::vector<int> v{10,20,30};
using type_of_temp = std::vector<std::pair<std::vector<int>,int>>;
auto temp = copy_range<type_of_temp>(v | indexed(0)
| transformed(complex_keep_index));
auto w = temp | transformed(distribute);
print(w);
//auto z = v | indexed(0)
// | transformed(complex_keep_index)
// | transformed(distribute);
//print(z);
}
Indeed, decommenting the lines defining and using z gives you a code that compiles but generates rubbish results, i.e. undefined behavior. Note that applying copy_range<type_of_temp> to the first, working, range is necessary, otherwise the resulting code is essetially the same as the one on the right of auto z =.
Why do I have to do so? What are the details that makes the oneliner not work?
I partly understand the reason, and I'll list my understanding/thoughts in the following, but I'm asking this question to get a thorough explanation of all the details of this.
I understand that the undefined behavior I observe stems from z being a range whose defining a view on some temporary which has been destroyed;
given the working version of the code, it is apparent that temporary is v | indexed(0) | transformed(complex_keep_index);
however, isn't v | indexed(0) itself a temporary that is fed to transformed(complex_keep_index)?
Probably one important detail is that the expression v | indexed(0) is no more than a lazy range, which evaluates nothing, but just sets things up such that when one iterates on the range the computations is done; after all I can easily do v | indexed(0) | indexed(0) | indexed(0), which is well defined;
and also the whole v | indexed(0) | transformed(complex_keep_index) is well defined, otherwise the code above using w would probably misbehave (I know that UB doesn't mean that the result has to show something is wrong, and things could just look ok on this hardware, in this moment, and break tomorrow).
So there's something inherently wrong is passing an rvalue to transformed(distribute);
but what's wrong in doing so lies in distribute, not in transformed, because for instance changing distribute to [](auto x){ return x; } seems to be well defined.
So what's wrong with distribute? Here's the code
constexpr auto distribute = [](auto const& pair){
return pair.first | transformed([n = pair.second](auto x){
return std::make_pair(x, n);
});
};
What's the problem with it? The returned range (output of this transformed) will hold some iterators/pointers/references to pair.first which is part of goes out of scope when distribute returns, but pair is a reference to something in the caller, which keeps living, right?
However I know that even though a const reference (e.g. pair) can keep a temporary (e.g. the elements of v | indexed(0) | transformed(complex_keep_index)) alive, that doesn't mean that the temporary stays alive when that reference goes out of scope just because it is in turn referenced by something else (references/pointers/iterators in the output of transformed([n = …](…){ … })) which doesn't go out of scope.
I think/hope that probably the answer is already in what I've written above, however I need some help to streamline all of that so that I can understand it once and for all.

Related

In C++ and range-v3, how to convert a string of space-separated numbers to a vector of integers?

Using C++ and range-v3 library, what's the optimal approach to converting a string with space-separated numbers to a vector of integers?
I tried the following code:
#include <iostream>
#include <range/v3/all.hpp>
using namespace std::literals;
int main() {
auto r = "1 1 2 3 5 8 13"sv
| ranges::views::split(" "sv)
| ranges::views::transform([](auto &&i){ return std::stoi(std::string{i}); })
| ranges::to<std::vector<int>>();
for (auto i: r)
std::cout << "Value: " << i << std::endl;
}
It doesn't compile however. In clang, the error is as follows:
repro-range.cpp:10:60: error: no matching constructor for initialization of 'std::string' (aka 'basic_string<char>')
| ranges::view::transform([](auto &&i){ return std::stoi(std::string{i}); })
^ ~~~
It seems that the type of i is ranges::detail::split_outer_iterator and it's not convertible to string. Actually, I don't understand how to use i, can't dereference it, can't convert it to anything useful... replacing string_views by strings also doesn't improve the situation.
What's weird, the code below works fine:
auto r = "1 1 2 3 5 8 13"sv
| ranges::views::split(" "sv)
| ranges::to<std::vector<std::string>>();
which suggest me the problem is netiher split nor to, but the transform itself.
How to make the first piece code working?
If you have a string containing space separated numbers you can first create an std::istringstream over the string and then use ranges::istream to parse the numbers (assuming ints here):
auto s = "1 1 2 3 5 8 13";
auto ss = std::istringstream{s};
auto r = ranges::istream<int>(ss)
| ranges::to<std::vector<int>>();
Here's a demo.
Digging deeper, I found out that i in my example isn't an iterator nor a wrapper over string_view (like I expected) but a range of characters (a special type with begin and end iterators).
Meaning, my code works if I first convert i to a string the range way:
auto r = "1 1 2 3 5 8 13"sv
| ranges::views::split(" "sv)
| ranges::views::transform([](auto &&i){
return std::stoi(i | ranges::to<std::string>());
})
| ranges::to<std::vector<int>>();
Although I'll be delighted if somebody posts a nicer (at least less verbose) way to do that.

Lifetime of the returned range-v3 object in C++

I want to make a function that works like np.arange(). With range-v3, the code is
auto arange(double start, double end, double step){
assert(step != 0);
const auto element_count = static_cast<int>((end - start) / step) + 1;
return ranges::views::iota(0, element_count) | ranges::views::transform([&](auto i){ return start + step * i; });
}
and to use it,
auto range = arange(1, 5, 0.5);
for (double x : range){
std::cout << x << ' '; // expect 1 1.5 2 2.5 3 3.5 4 4.5 5
}
However, the result told me a dummy value. I think the lifetime of returned range object is expired, and I found that by making them to vector can pass the result well. (And it will cause overhead for constructing vector.)
Is there any way to return range itself without expired lifetime ?
You fell victim to Undefined Behaviour due to capturing of local variables via [&].
If you capture by value [start, step](auto i){ return start + step * i; }, the code will work correctly.
Note that views are always non-owning, can be copied around and are generally O(1) in their storage. Since iota is a generating view and stores its full state inside itself, the code is safe.

Broadcasting Row and Column Vector in Eigen C++

I have following Python Code written in NumPy:
> r = 3
> y, x = numpy.ogrid[-r : r + 1, -r : r + 1]
> mask = numpy.sqrt(x**2 + y**2)
> mask
array([[4.24264, 3.60555, 3.16228, 3.00000, 3.16228, 3.60555, 4.24264],
[3.60555, 2.82843, 2.23607, 2.00000, 2.23607, 2.82843, 3.60555],
[3.16228, 2.23607, 1.41421, 1.00000, 1.41421, 2.23607, 3.16228],
[3.00000, 2.00000, 1.00000, 0.00000, 1.00000, 2.00000, 3.00000],
[3.16228, 2.23607, 1.41421, 1.00000, 1.41421, 2.23607, 3.16228],
[3.60555, 2.82843, 2.23607, 2.00000, 2.23607, 2.82843, 3.60555],
[4.24264, 3.60555, 3.16228, 3.00000, 3.16228, 3.60555, 4.24264]])
Now, I am making the mask in Eigen where I need to broadcast row and column vector. Unfortunately, it is not allowed so I made the following workaround:
int len = 1 + 2 * r;
MatrixXf mask = MatrixXf::Zero(len, len);
ArrayXf squared_yx = ArrayXf::LinSpaced(len, -r, r).square();
mask = (mask.array().colwise() + squared_yx) +
(mask.array().rowwise() + squared_yx.transpose());
mask = mask.cwiseSqrt();
cout << "mask" << endl << mask << endl;
4.24264 3.60555 3.16228 3 3.16228 3.60555 4.24264
3.60555 2.82843 2.23607 2 2.23607 2.82843 3.60555
3.16228 2.23607 1.41421 1 1.41421 2.23607 3.16228
3 2 1 0 1 2 3
3.16228 2.23607 1.41421 1 1.41421 2.23607 3.16228
3.60555 2.82843 2.23607 2 2.23607 2.82843 3.60555
4.24264 3.60555 3.16228 3 3.16228 3.60555 4.24264
It works. But I wonder if there is another and shorter way to do it. Therefore my question is how to broadcast Row and Column Vector in Eigen C++?
System Info
Tool
Version
Eigen
3.3.7
GCC
9.4.0
Ubuntu
20.04.4 LTS
I think the easiest approach (as in: most readable), is replicate.
int r = 3;
int len = 1 + 2 * r;
const auto& squared_yx = Eigen::ArrayXf::LinSpaced(len, -r, r).square();
const auto& bcast = squared_yx.replicate(1, len);
Eigen::MatrixXf mask = (bcast + bcast.transpose()).sqrt();
Note that what you do is numerically unstable (for large r) and the hypot function exists to work around these issues. So even your python code could be better:
r = 3
y, x = numpy.ogrid[-r : r + 1, -r : r + 1]
mask = numpy.hypot(x, y)
To achieve the same in Eigen, do something like this:
const auto& yx = Eigen::ArrayXf::LinSpaced(len, -r, r);
const auto& bcast = yx.replicate(1, len);
Eigen::MatrixXf mask = bcast.binaryExpr(bcast.transpose(),
[](float x, float y) noexcept -> float {
return std::hypot(x, y);
});
Eigen's documentation on binaryExpr is currently broken, so this is hard to find.
To be fair, you will probably never run into stability issues in this particular case because you will run out of memory first. However, it'd still like to point this out because seeing a naive sqrt(x**2 + y**2) is always a bit of a red flag. Also, in Python hypot might still worth it from a performance point because it reduces the number of temporary memory allocations and function calls.
BinaryExpr
The documentation on binaryExpr is missing, I assume because the parser has trouble with Eigen's C++ code. In any case, one can find it indirectly as CwiseBinaryOp and similarly CwiseUnaryOp, CwiseNullaryOp and CwiseTernaryOp.
The use looks a bit weird but is pretty simple. It takes a functor (either a struct with operator(), a function pointer, or a lambda) and applies this element-wise.
The unary operation makes this pretty clear. If Eigen::Array.sin() didn't exist, you could write this:
array.unaryExpr([](double x) -> double { return std::sin(x); }) to achieve exactly the same effect.
The binary and ternary versions take one or two more Eigen expressions as the second and third argument to the function. That's what I did above. The nullary version is explained in the documentation in its own chapter.
Use of auto
Eigen is correct to warn about auto but only in that you have to know what you do. It is important to realize that auto on an Eigen expression just keeps the expression around. It does not evaluate it into a vector or matrix.
This is fine and very useful if you want to compose a complex expression that would be hard to read when put in a single statement. In my code above, there are no temporary memory allocations and no floating point computations take place until the final expression is assigned to the matrix.
As long as the programmer knows that these are expressions and not final matrices, everything is fine.
I think the main take-away is that use of auto with Eigen should be limited to short-lived (as in: inside a single function) scalar expressions. Any coding style that uses auto for everything will quickly break or be hard to read with Eigen. But it can be used safely and make the code more readable in the process without sacrificing performance in the same way as evaluating into matrices would.
As for why I chose const auto& instead of auto or const auto: Mostly force of habit that is unrelated to the task at hand. I mostly do it for instances like this:
const Eigen::Vector& Foo::get_bar();
void quz(Foo& foo)
{
const auto& bar = foo.get_bar();
}
Here, bar will remain a reference whereas auto would create a copy. If the return value is changed, everything stays valid.
Eigen::Vector Foo::get_bar();
void quz(Foo& foo)
{
const auto& bar = foo.get_bar();
}
Now a copy is created anyway. But everything continues to work because assigning the return value to a const-reference extends the lifetime of the object. So this may look like a dangling pointer, but it is not.

How to remove the Nth element in a range?

I can write:
my_range | ranges::views::remove(3)
using the ranges-v3 library, to remove the element(s) equal to 3 from the range my_range. This can also be done in C++20 with
my_range | std::views::filter([](auto const& val){ return val != 3; })
But - how can I remove the element at position 3 in my_range, keeping the elements at positions 0, 1, 2, 4, 5 etc.?
Here's one way to do it:
#include <iostream>
#include <ranges>
#include <range/v3/view/take.hpp>
#include <range/v3/view/drop.hpp>
#include <range/v3/view/concat.hpp>
int main() {
const auto my_range = { 10, 20, 30, 40, 50, 60, 70 };
auto index_to_drop = 3; // so drop the 40
auto earlier = my_range | ranges::views::take(index_to_drop - 1);
auto later = my_range | ranges::views::drop(index_to_drop);
auto both = ranges::views::concat(earlier, later);
for (auto const & num : both) { std::cout << num << ' '; }
}
This will produce:
10 20 30 50 60 70
... without the 40.
See it working on Godbolt. Compilation time is extremely poor though. Also, concat() is not part of C++20. Maybe in C++23?
The most straightforward way I can think of in range-v3 would be:
auto remove_at_index(size_t idx) {
namespace rv = ranges::views;
return rv::enumerate
| rv::filter([=](auto&& pair){ return pair.first != idx; })
| rv::values;
}
To be used like:
my_range | remove_at_index(3);
enumerate (and its more general cousin zip) is not in C++20, but will hopefully be in C++23.

Running std::normal_distribution with user-defined random generator

I am about to generate an array of normally distributed pseudo-random numbers. As I know the std library offers the following code for that:
std::random_device rd;
std::mt19937 gen(rd());
std::normal_distribution<> d(mean,std);
...
double number = d(gen);
The problem is that I want to use a Sobol' quasi-random sequence instead of Mersenne
Twister pseudo-random generator. So, my question is:
Is it possible to run the std::normal_distribution with a user-defined random generator (with a Sobol' quasi-random sequence generator in my case)?
More details: I have a class called RandomGenerators, which is used to generate a Sobol' quasi-random numbers:
RandomGenerator randgen;
double number = randgen.sobol(0,1);
Yes, it is possible. Just make it comply to the requirements of a uniform random number generator (§26.5.1.3 paragraphs 2 and 3):
2 A class G satisfies the requirements of a uniform random number
generator if the expressions shown in Table 116 are valid and have the
indicated semantics, and if G also satisfies all other requirements
of this section. In that Table and throughout this section:
a) T is the type named by G’s associatedresult_type`, and
b) g is a value of G.
Table 116 — Uniform random number generator requirements
Expression | Return type | Pre/post-condition | Complexity
----------------------------------------------------------------------
G::result_type | T | T is an unsigned integer | compile-time
| | type (§3.9.1). |
----------------------------------------------------------------------
g() | T | Returns a value in the | amortized constant
| | closed interval |
| | [G::min(), G::max()]. |
----------------------------------------------------------------------
G::min() | T | Denotes the least value | compile-time
| | potentially returned by |
| | operator(). |
----------------------------------------------------------------------
G::max() | T | Denotes the greatest value | compile-time
| | potentially returned by |
| | operator(). |
3 The following relation shall hold: G::min() < G::max().
A word of caution here - I came across a big gotcha when I implemented this. It seems that if the return types of max()/min()/operator() are not 64 bit then the distribution will resample. My (unsigned) 32 bit Sobol implementation was getting sampled twice per deviate thus destroying the properties of the numbers. This code reproduces:
#include <random>
#include <limits>
#include <iostream>
#include <cstdint>
typedef uint32_t rng_int_t;
int requested = 0;
int sampled = 0;
struct Quasi
{
rng_int_t operator()()
{
++sampled;
return 0;
}
rng_int_t min() const
{
return 0;
}
rng_int_t max() const
{
return std::numeric_limits<rng_int_t>::max();
}
};
int main()
{
std::uniform_real_distribution<double> dist(0.0,1.0);
Quasi q;
double total = 0.0;
for (size_t i = 0; i < 10; ++i)
{
dist(q);
++requested;
}
std::cout << "requested: " << requested << std::endl;
std::cout << "sampled: " << sampled << std::endl;
}
Output (using g++ 5.4):
requested: 10
sampled: 20
and even when compiled with -m32. If you change rng_int_t to 64bit the problem goes away. My workaround is to stick the 32 bit value into the most significant bits of the return value, e.g
return uint64_t(val) << 32;
You can now generate Sobol sequences directly with Boost. See boost/random/sobol.hpp.