Why does std::reduce need commutativity? - c++

Question

https://en.cppreference.com/w/cpp/algorithm/reduce
It says that the behavior of an operation is not defined if the operation is not commutative, but why? We just divide the array into blocks and then merge the result. Is it only necessary to have associativity?

Answer 1

std::reduce requires both associativity and commutativity. Associativity is clearly needed for a parallel algorithm, since you want to perform the calculation on separate chunks and then combine them.
As for commutativity: According to a reddit post by MSVC STL developer Billy O'Neal, this is required in order to allow vectorization to SIMD instructions:
Commutativity is also necessary to enable vectorization, since the code you want for reduce to come out as something like:
vecRegister = load_contiguous(first);
while (a vector register sized chunk is left) {
first += packSize;
vecRegister = add_packed(load_contiguous(first), vecRegister);
}
// combine vecRegister's packed components
etc., which given ints and SSE registers and a * b * c * d * e * f * g * h gives something like (a * e) * (b * f) * (c * g) * (d * h).
Most other languages aren't doing explicit things to make vectorizing their reduction possible. And nothing says we can't add a noncommutative_reduce or something like that in the future if someone comes up with a compelling use case.

Answer 2

The behavior is actually non-deterministic if the operation between the operands is not commutative. "non-deterministic" is not the same as "undefined". Floating point math is not commutative, for example. This is one reason why a call to std::reduce may not be deterministic, because the binary function is applied in an unspecified order.
Refer to this note in the standard:
Note: The difference between reduce and accumulate is that reduce applies binary_op in an
unspecified order, which yields a nondeterministic result for non-associative or non-commutative
binary_op such as floating-point addition. —end note ]

Answer 3

The standard defines the generalized sum as follows: numeric.defns
Define GENERALIZED_­NONCOMMUTATIVE_­SUM(op, a1, ..., aN) as follows:
a1 when N is 1, otherwise
op(GENERALIZED_­NONCOMMUTATIVE_­SUM(op, a1, ..., aK),
op(GENERALIZED_­NONCOMMUTATIVE_­SUM(op, aM, ..., aN)) for any K where 1
Define GENERALIZED_­SUM(op, a1, ..., aN) as GENERALIZED_­NONCOMMUTATIVE_­SUM(op, b1, ..., bN), where b1, ..., bN may be any permutation of a1, ..., aN.
So, the order of summation as well as the order of operands is unspecified. So if the binary operation is not commutative or not associative, the result is unspecified.
That is also explicitly stated here.
Regarding why: It gives the library vendors more freedom, so they may or may not implement it better. As an example where the implementation can benefit from commutativity. Consider the sum a+b+c+d+e, we first calculate a+b and c+d in parallel. Now a+b returns before c+d does (as it can happen, because it is done in parallel). Instead of waiting for the return value of c+d we now can directly compute (a+b)+e and then add this result to the result of c+d. So in the end, we computed ((a+b)+e)+(c+d), which is a rearrangement of a+b+c+d+e.

Answer 4

Why does std::reduce need commutativity?
For speed.
If the operator is commutative, then you can rearrange the order of operations without affecting the results.
And if you can rearrange the order of operations, you can have different threads, or processes, or hardware accelerators or what-not working independently on some of the operations to be performed, and not care about the order in which they complete their partial sums, nor their internal ordering of operations, and then finally add up the partial sums whichever way is convenient.