MPI Gatherv with submatrices - c++

Question

I'm having trouble with getting MPI_Gatherv to work how I intend, and was wondering those of you who are more experienced can see what I'm doing wrong.
I have a large matrix (TEST) of [N, M]. Each process does some work on a subset [nrows, M] (WORK_MATRIX) and then every process gathers these submatrices (along the row dimension) into the full matrix.
It seems like it doesn't gather any of the data, and I'm struggling to figure out why!
Here I'm using Eigen to wrap these (contiguous) matrices.
Output:
mpirun -np 5 ./pseudo.x
1 1 1 1 1
0 1 2 3 4
TEST: 5 10
0 0 2 0 0 0 0 0 0 0
1 1 2 0 0 0 0 0 0 0
2 2 2 0 0 0 0 0 0 0
3 2 2 0 0 0 0 0 0 0
4 2 0 0 0 0 0 0 0 0
I've created a simple version of the code below:
mpiicc -I/path/to/Eigen -o pseudo.x pseudo.cpp
#include <mpi.h>
#include <Eigen/Dense>
#include <iostream>
using namespace Eigen;
using namespace std;
int main(int argc, char ** argv) {
int RSIZE = 5;
int CSIZE = 10;
int rank;
int num_tasks;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &num_tasks);
MatrixXd TEST_MATRIX = MatrixXd::Zero(RSIZE, CSIZE);
VectorXi recv = VectorXi::Zero(num_tasks);
VectorXi displs = VectorXi::Zero(num_tasks);
int nrows = (RSIZE + rank) / num_tasks;
MPI_Allgather(&nrows, 1, MPI_INT, recv.data(), 1, MPI_INT, MPI_COMM_WORLD);
int start = 0;
for (int i = 0; i < rank; i++)
start += recv[i];
MPI_Allgather(&start, 1, MPI_INT, displs.data(), 1, MPI_INT, MPI_COMM_WORLD);
if (rank == 0) {
cout << recv.transpose() << endl;
cout << displs.transpose() << endl;
}
MatrixXd WORK_MATRIX = MatrixXd::Zero(nrows, CSIZE);
for (int row = 0; row < nrows; row++)
for (int col = 0; col < CSIZE; col++)
WORK_MATRIX(row, col) += rank;
MPI_Datatype rowsized, row;
int sizes[2] = { RSIZE, CSIZE };
int subsizes[2] = { nrows, CSIZE };
int starts[2] = { 0, 0 };
MPI_Type_create_subarray(2, sizes, subsizes, starts, MPI_ORDER_C, MPI_DOUBLE, &rowsized);
MPI_Type_create_resized(rowsized, 0, sizeof(double), &row);
MPI_Type_commit(&row);
MPI_Allgatherv(WORK_MATRIX.data(), recv[rank], row, TEST_MATRIX.data(), recv.data(), displs.data(), row, MPI_COMM_WORLD);
if (rank == 0) {
cout << "TEST: " << TEST_MATRIX.rows() << " " << TEST_MATRIX.cols() << endl;
for (int i = 0; i < TEST_MATRIX.rows(); i++) {
for (int j = 0; j < TEST_MATRIX.cols(); j++) {
cout << TEST_MATRIX(i, j) << " ";
}
cout << endl;
}
}
}

Answer 1

in C, 2D matrixes are stored by rows, and I doubt the Eigen changes that.
that means you do not need to resize your datatypes, and the displacement should be adjusted
start += recv[i] * CSIZE;
As a matter of taste, you do not need two MPI_Allgather() at all since nrows and start can be computed locally.
I'd rather suggest you simply create a derived datatype for one row with MPI_Type_contiguous() (and this type should not be resized), since MPI_Type_create_subarray() is really an overkill here.