A not safe MPI non-blocking communication example? - c++

Question

I am implementing MPI non-blocking communication inside my program. I see on MPI_Isend man_page, it says:
A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not modify any part of the send buffer after a nonblocking send operation is called, until the send completes.
My code works like this:
// send messages
if(s > 0){
MPI_Requests s_requests[s];
MPI_Status s_status[s];
for(int i = 0; i < s; ++i){
// some code to form the message to send
std::vector<doubel> send_info;
// non-blocking send
MPI_Isend(&send_info[0], ..., s_requests[i]);
}
MPI_Waitall(s, s_requests, s_status);
}
// recv info
if(n > 0){ // s and n will match
for(int i = 0; i < n; ++i){
MPI_Status status;
// allocate the space to recv info
std::vector<double> recv_info;
MPI_Recv(&recv_info[0], ..., status)
}
}
My question is: am I modify the send buffers since they are in the inner curly brackets (the send_info vector get killed after the loop finishes)? Therefore, this is not a safe communication mode? Although my program works fine now, I still being suspected. Thank you for your reply.

Answer 1

There are two points I want to emphasize in this example.
The first one is the problem I questioned: send buffer gets modified before MPI_Waitall. The reason is what Gilles said. And the solution could be allocated a big buffer before the for loop, and use MPI_Waitall after the loop is finished or put MPI_Wait inside the loop. But the latter one is equivalent to use MPI_Send in the sense of performance.
However, I found if you simply transfer to blocking send and receive, a communication scheme like this could cause deadlock. It is similar to the classic deadlock:
if (rank == 0) {
MPI_Send(..., 1, tag, MPI_COMM_WORLD);
MPI_Recv(..., 1, tag, MPI_COMM_WORLD, &status);
} else if (rank == 1) {
MPI_Send(..., 0, tag, MPI_COMM_WORLD);
MPI_Recv(..., 0, tag, MPI_COMM_WORLD, &status);
}
And the explaination could be found here.
My program could cause a similar situation: all the processors called MPI_Send then it is a deadlock.
So my solution is to use a large buffer and stick to non-blocking communication scheme.
#include <vector>
#include <unordered_map>
// send messages
if(s > 0){
MPI_Requests s_requests[s];
MPI_Status s_status[s];
std::unordered_map<int, std::vector<double>> send_info;
for(int i = 0; i < s; ++i){
// some code to form the message to send
send_info[i] = std::vector<double> ();
// non-blocking send
MPI_Isend(&send_info[i][0], ..., s_requests[i]);
}
MPI_Waitall(s, s_requests, s_status);
}
// recv info
if(n > 0){ // s and n will match
for(int i = 0; i < n; ++i){
MPI_Status status;
// allocate the space to recv info
std::vector<double> recv_info;
MPI_Recv(&recv_info[0], ..., status)
}
}