MPI receiving in multiple parts - c++

Question

Usually when I want to send a buffer to next processor and receive another one from previous one I use the following:
MPI_Irecv(rcv_buff,rcv_size,
MPI_DOUBLE,rcv_p,0,world,
&request);
MPI_Send(snd_buff,snd_size,
MPI_DOUBLE,snd_p,0,world);
MPI_Wait(&request,&status);
Suppose that I want to put the first rcv_size0 elements of rcv_buff in array0 and the rest (rcv_size1 elements) in array1, where:
rcv_size1=rcv_size-rcv_size0;
normally what I do is that I first create a dummy array like rcv_buff here and then start copying the values to array0 and array1. My question is that is there any way in MPI to receive the sent bytes in two or more sequences? for example directly receive the first size0 elements in array0 and the rest in array1?

Answer 1

You can do this - receiving into two buffers - by creating a type specific to that pair of buffers:
#include <stdio.h>
#include <mpi.h>
#include <stdlib.h>
int recv_split(const int total, const int src, const int tag,
double *buffA, const int sizeA, double *buffB) {
if (total <= 0) return -1;
if (sizeA > total) return -1;
if (buffA == NULL) return -2;
if (buffB == NULL) return -2;
const int sizeB = total - sizeA;
int blocksizes[2] = {sizeA, sizeB};
MPI_Datatype types[2] = {MPI_DOUBLE, MPI_DOUBLE};
MPI_Aint displacements[2], addrA, addrB;
MPI_Datatype splitbuffer;
MPI_Status status;
displacements[0] = 0;
MPI_Get_address(buffA, &addrA);
MPI_Get_address(buffB, &addrB);
displacements[1] = addrB - addrA;
MPI_Type_create_struct(2, blocksizes, displacements, types, &splitbuffer);
MPI_Type_commit(&splitbuffer);
MPI_Recv(buffA, 1, splitbuffer, src, tag, MPI_COMM_WORLD, &status);
MPI_Type_free(&splitbuffer);
return 0;
}
int main(int argc, char **argv) {
int rank, size;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
const int sendSize = 15;
const int tag = 1;
if (rank == 0 && size >= 2) {
double sendbuff[sendSize];
for (int i=0; i<sendSize; i++)
sendbuff[i] = 1.*i;
MPI_Send(sendbuff, sendSize, MPI_DOUBLE, 1, tag, MPI_COMM_WORLD);
}
if (rank == 1) {
const int buffLen = 12;
const int recvIntoA = 10;
double buffA[buffLen];
double buffB[buffLen];
for (int i=0; i<buffLen; i++) {
buffA[i] = buffB[i] = -1.;
}
recv_split(sendSize, 0, tag, buffA, recvIntoA, buffB);
printf("---Buffer A--\n");
for (int i=0; i<buffLen; i++)
printf("%5.1lf ", buffA[i]);
printf("\n---Buffer B--\n");
for (int i=0; i<buffLen; i++)
printf("%5.1lf ", buffB[i]);
printf("\n");
}
MPI_Finalize();
return 0;
}
compiling and running gives
$ mpicc -o recvsplit recvsplit.c -std=c99
$ mpirun -np 2 ./recvsplit
---Buffer A--
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 -1.0 -1.0
---Buffer B--
10.0 11.0 12.0 13.0 14.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0 -1.0
Just note that this type will only work for this pair of buffers; different pairs will generally have different relative displacements. You can also of course always receive into one large staging buffer and manually unpack into different buffers, using your own code or MPI_Unpack.

Answer 2

There's nothing that I know of directly in MPI that would allow you to do that, though there's probably some nasty pointer magic you could use to make it work. In general, it would be much cleaner to do it as two sends if you wanted to do things that way.
Another thing that's not a direct answer to your question, but you might not have known about is that your three line command above can be combined into one using MPI_SENDRECV. Try this line out:
MPI_Sendrecv(snd_buff, snd_size, MPI_DOUBLE, snd_p, 0,
rcv_buff, rcv_size, MPI_DOUBLE, rcv_p, 0,
world, &status);