Create MPI reduction operator from c++ lambda function - c++

Question

I would like to write a function that wraps MPI_Allreduce, and which accepts any binary operator (as std::reduce) to be used as reduction operator by MPI. Especially, the user of such a function could use a lambda.
The following simple sample code illustrates that:
#include <mpi.h>
#include <iostream>
#include <functional>
template<typename BinaryOp>
void reduce(double *data, int len, BinaryOp op) {
auto lambda=[op](void *a, void *b, int *len, MPI_Datatype *){
double *aa=static_cast<double *>(a);
double *bb=static_cast<double *>(bb);
for (int i=0; i<*len; ++i) {
bb[i]=op(aa[i], bb[i]);
}
};
// MPI_User_function is a typedef to: void (MPI_User_function) ( void * a, void * b, int * len, MPI_Datatype * )
MPI_User_function *opPtr=/* black magic code that get the function pointer from the lambda */;
MPI_Op mpiOp;
MPI_Op_create(*opPtr, 1, &mpiOp);
MPI_Allreduce(MPI_IN_PLACE, data, len, MPI_DOUBLE, mpiOp, MPI_COMM_WORLD);
MPI_Op_free(&mpiOp);
}
int main() {
MPI_Init(nullptr, nullptr);
double data[4]={1.,2.,3.,4.};
reduce(data, 4, [](double a, double b){return a+b;});
int pRank;
MPI_Comm_rank(MPI_COMM_WORLD, &pRank);
if (pRank==0) {
for (int i=0; i<4; ++i) {
std::cout << data[i] << " ";
}
std::cout << std::endl;
}
MPI_Finalize();
return 1;
}
The missing part is the code that get a function pointer from the lambda in the reduce function. From several related questions, this problem of getting a function pointer from a capturing lambda seems to be tricky but possible to solve. But I failed to have something working on this simple code (I tried some tricks with std::function, std::bind, storage of the lambda in a static variable)... So a little help would be great!
EDIT: Following #noma answer, I tried the following simplified code without MPI in goldbolt
#include <iostream>
#include <functional>
typedef double MPI_Datatype;
template<typename BinaryOp, BinaryOp op> // older standards
void non_lambda(void *a, void *b, int *len, MPI_Datatype *)
{}
template<typename BinaryOp>
void reduce(double *data, int len, BinaryOp op) {
typedef void (MPI_User_function) ( void * a, void * b, int * len, MPI_Datatype * );
MPI_User_function *opPtr = &non_lambda<decltype(+op), +op>; // older standards;
}
int main() {
double data[4]={1.,2.,3.,4.};
reduce(data, 4, [](double a, double b){return a+b;});
return 1;
}
It compile on some compilers. Here are the results:
icc >= 19.0.1 (with -std=c++17) : OK
clang++ >= 5.0.0 (with --std=c++17): OK
clang++ 10.0.0 (with --std=c++14): NOK
g++ 9.3 (with --std=c++17): NOK
icc >= 19.0.0 (with -std=c++17) : NOK
The error message with icc 19.0.0 with -std=c++17 (or icc 19.0.1 with -std=c++14) is interesting:
<source>(15): error: expression must have a constant value
MPI_User_function *opPtr = &non_lambda<decltype(+op), +op>; // older standards;
^
detected during instantiation of "void reduce(double *, int, BinaryOp) [with BinaryOp=lambda [](double, double)->double]" at line 21
And indeed, I don't really understand the passing of the 'op' variable which is a runtime argument of the function reduce as the second template parameter of the non_lambda function... Is it an obscure c++17 functionality that only some of the compilers support?

Answer 1

I think the lambda approach is not possible here as it is a capturing lambda, see
https://stackoverflow.com/a/28746827/7678171
We can use a function template with the BinaryOp as a template value parameter instead of a Lambda here. This assumes that the BinaryOp is either a function pointer, or a capture-less lambda, that can be converted into one. Instead of the lambda inside your reduce we introduce:
template<auto op> // this is C++17, so use --std=c++17
// template<typename BinaryOp, BinaryOp op> // older standards
void non_lambda(void *a, void *b, int *len, MPI_Datatype *)
{
double *aa=static_cast<double *>(a);
double *bb=static_cast<double *>(bb);
for (int i=0; i<*len; ++i) {
bb[i]=op(aa[i], bb[i]);
}
}
The Black Magic line then is:
/* black magic code that get the function pointer from the lambda */
MPI_User_function *opPtr = &non_lambda<+op>; // NOTE: the + implies the lamda to function pointer conversion here
// MPI_User_function *opPtr = &non_lambda<decltype(+op), +op>; // older standards;
Hope this helps.
NOTE: I got this compiled using Clang 6.0, but g++ 7.5 failed (possible compiler bug?):
error: no matches converting function ‘non_lambda’ to type ‘void (*)(void*, void*, int*, struct ompi_datatype_t**)’
MPI_User_function *opPtr = &non_lambda<+op>;
^~~~~
note: candidate is: template<auto op> void non_lambda(void*, void*, int*, ompi_datatype_t**)
void non_lambda(void *a, void *b, int *len, MPI_Datatype *)
Maybe newer g++ versions work.