MPI - sending parts of image to different processes - c++

Question

I'm writing a program in which process 0 sends parts of image to other processes which transform (long operation) this part and send back to the rank 0. I have a problem with one thing. To reproduce my issue I wrote a simple example. An image with size 512x512px is split on 4 parts (vertical stripes) by process 0. Next other processes save this part on disk. The problem is that each process saves the same part. I discovered that the image is split on parts correctly but problem is probably with sending data. What's wrong in my code?
Run:
mpirun -np 5 ./example
Main:
int main(int argc, char **argv) {
int size, rank;
MPI_Request send_request, rec_request;
MPI_Status status;
ostringstream s;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (rank == 0) {
Mat mat = imread("/home/user/original.jpg", CV_LOAD_IMAGE_COLOR);
if (!mat.data) exit(-1);
int idx = 1;
for (int c = 0; c < 512; c += 128) {
Mat slice = mat(Rect(c, 0, 128, 512)).clone();
MPI_Isend(slice.data, 128 * 512 * 3, MPI_BYTE, idx, 0, MPI_COMM_WORLD, &send_request);
idx++;
}
}
if (rank != 0) {
Mat test = Mat(512, 128, CV_8UC3);
MPI_Irecv(test.data, 128 * 512 * 3, MPI_BYTE, 0, 0, MPI_COMM_WORLD, &rec_request);
MPI_Wait(&rec_request, &status);
s << "/home/user/p" << rank << ".jpg";
imwrite(s.str(), test);
}
MPI_Finalize();
return 0;
}

Answer 1

If you insist on using non-blocking operations, then the proper way to issue multiple of them at the same time is:
MPI_Request *send_reqs = new MPI_Request[4];
int idx = 1;
for (int c = 0; c < 512; c += 128) {
Mat slice = mat(Rect(c, 0, 128, 512)).clone();
MPI_Isend(slice.data, 128 * 512 * 3, MPI_BYTE, idx, 0, MPI_COMM_WORLD, &send_reqs[idx-1]);
idx++;
}
MPI_Waitall(4, send_reqs, MPI_STATUSES_IGNORE);
delete [] send_reqs;
Another (and IMHO better) option would be to utilise MPI_Scatterv to scatter the original data buffer. Thus you could even save cloning parts of the image matrix.
if (rank == 0) {
Mat mat = imread("/home/user/original.jpg", CV_LOAD_IMAGE_COLOR);
if (!mat.data) exit(-1);
int *send_counts = new int[size];
int *displacements = new int[size];
// The following calculations assume row-major storage
for (int i = 0; i < size; i++) {
send_counts[i] = displacements[i] = 0;
}
int idx = 1;
for (int c = 0; c < 512; c += 128) {
displacements[idx] = displacements[idx-1] + send_counts[idx-1];
send_counts[idx] = 128 * 512 * 3;
idx++;
}
MPI_Scatterv(mat.data, send_counts, displacements, MPI_BYTE,
NULL, 0, MPI_BYTE, 0, MPI_COMM_WORLD);
delete [] send_counts;
delete [] displacements;
}
if (1 <= rank && rank <= 4) {
Mat test = Mat(512, 128, CV_8UC3);
MPI_Scatterv(NULL, NULL, NULL, MPI_BYTE,
test.data, 128 * 512 * 3, MPI_BYTE, 0, MPI_COMM_WORLD);
s << "/home/user/p" << rank << ".jpg";
imwrite(s.str(), test);
}
Note how the arguments to MPI_Scatterv are prepared. Since you are scattering to 4 MPI processes only, setting certain elements of send_counts[] to zero allows the program to function correctly with more than 5 MPI processes. Also, the root rank in your original code doesn't send to itself, therefore send_counts[0] must be zero.

Answer 2

The problem is that you are not waiting till the send operation completes before the matrix Mat is destructed. Use MPI_Send instead of MPI_Isend.
If you really want to use non blocking communication, you have to keep track of all MPI_Request objects and of all Mat images until the send is complete.