is it possible to call the MPI_send and MPI_recv commands inside a subroutine and not the main program? I have written a minimal program for what I am trying to do. It is compiling fine but it is not working. The program just hangs in the "sendrecv" subroutine. Any ideas how can I do it?
main.f
program main
implicit none
include 'mpif.h'
integer me, np, ierror
call MPI_init( ierror )
call MPI_comm_rank( mpi_comm_world, me, ierror )
call MPI_comm_size( mpi_comm_world, np, ierror )
call sendrecv(me, np)
call mpi_finalize( ierror )
stop
end
sendrecv.f
subroutine sendrecv(me, np)
include 'mpif.h'
integer np, me, sender, tag
integer, dimension(mpi_status_size) :: status
integer, dimension(1) :: recv, send
if (me.eq.0) then
do sender = 1, np-1
call mpi_recv(recv, 1, mpi_integer, sender, tag,
& mpi_comm_world, status, ierror)
end do
end if
if ((me.ge.1).and.(me.lt.np)) then
send(1) = me*12
call mpi_send(send, 1, mpi_integer, 0, tag,
& mpi_comm_world, ierror)
end if
return
end
Related
I am trying to write an MPI code to process a large 2D matrix. I'm basically dividing the matrix into chunks and giving those chunks to individual processes. I see that the processes complete their task, send the processed array back to the Master. However, after calling MPI_Finalize(), a random process gives me a seg fault. I have checked and debugged the addressing (like whether a process is accessing some invalid memory) but haven't found any issues in that. I have attached my code below :-
#include "mpi/mpi.h"
#include <iostream>
using namespace std;
#define n 6
#define iter 1
#define MASTER 0
#define FROM_MASTER 1
#define FROM_SLAVE 2
int main(int argc, char* argv[]) {
int num_tasks, num_workers, task;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &num_tasks);
MPI_Comm_rank(MPI_COMM_WORLD, &task);
num_workers = num_tasks - 1;
MPI_Status status;
double *mat;
int rows_per_task = (n - 2) / num_workers;
int index = 0;
int size;
if(task == MASTER) {
// Allocate n*n matrix
mat = (double *)malloc(sizeof(double) * n * n);
// Initialize the matrix
for(int i = 1; i <= num_workers; ++i) {
// Accommodate for extra rows per task
size = (i <= (n % num_workers)) ? rows_per_task + 1 : rows_per_task;
// Send rows to the Slave processes
MPI_Send(&index, 1, MPI_INT, i, FROM_MASTER, MPI_COMM_WORLD); // Start index
MPI_Send(&size, 1, MPI_INT, i, FROM_MASTER, MPI_COMM_WORLD); // Size of the array chunk
MPI_Send(&mat[index], n * size, MPI_DOUBLE, i, FROM_MASTER, MPI_COMM_WORLD); // Array
index += n * size;
}
for(int i = 1; i <= num_workers; ++i) {
// Get array size, start index, actual array from all the processes
MPI_Recv(&size, 1, MPI_INT, i, FROM_SLAVE, MPI_COMM_WORLD, &status);
MPI_Recv(&index, 1, MPI_INT, i, FROM_SLAVE, MPI_COMM_WORLD, &status);
MPI_Recv(&mat[index], n * size, MPI_DOUBLE, i, FROM_SLAVE, MPI_COMM_WORLD, &status);
}
printf("MASTER DONE!\n");
}
if(task > 0) {
// Get index, size, rows
MPI_Recv(&index, 1, MPI_INT, MASTER, FROM_MASTER, MPI_COMM_WORLD, &status);
MPI_Recv(&size, 1, MPI_INT, MASTER, FROM_MASTER, MPI_COMM_WORLD, &status);
MPI_Recv(&mat[index], n * size, MPI_DOUBLE, MASTER, FROM_MASTER, MPI_COMM_WORLD, &status);
// Repeat
for(int it = 0; it < iter; ++it) {
for(int i = 0; i < size; ++i) {
for(int j = 0; j < n; ++j) {
int idx = index + n * i + j; // 2D -> 1D index transformation
// Do something with the array element mat[idx]
}
}
}
MPI_Send(&size, 1, MPI_INT, MASTER, FROM_SLAVE, MPI_COMM_WORLD);
MPI_Send(&index, 1, MPI_INT, MASTER, FROM_SLAVE, MPI_COMM_WORLD);
MPI_Send(&mat[index], n * size, MPI_DOUBLE, MASTER, FROM_SLAVE, MPI_COMM_WORLD);
printf("SLAVE %d DONE\n", task);
}
printf("TASK %d calling finalize\n", task);
MPI_Finalize();
printf("TASK %d called finalize\n", task);
return 0;
}
In this example, n is set to 6. If my number of processes = 3 (divisible by 6), equal sharing of rows happens and the program works. If I change n to 7, then I start getting an error :-
SLAVE 2 DONE
TASK 2 calling finalize
SLAVE 1 DONE
TASK 1 calling finalize
SLAVE 3 DONE
TASK 3 calling finalize
MASTER DONE!
TASK 0 calling finalize
[<node_name>:70041] *** Process received signal ***
[<node_name>:70041] Signal: Segmentation fault (11)
[<node_name>:70041] Signal code: (128)
[<node_name>:70041] Failing at address: (nil)
[<node_name>:70041] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f24107b6090]
[<node_name>:70041] [ 1] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x20)[0x7f241080d6f0]
[<node_name>:70041] [ 2] /lib/x86_64-linux-gnu/libopen-pal.so.40(+0x409e2)[0x7f24106269e2]
[<node_name>:70041] [ 3] /lib/x86_64-linux-gnu/libmpi.so.40(ompi_datatype_finalize+0x79)[0x7f2410bc2eb9]
[<node_name>:70041] [ 4] /lib/x86_64-linux-gnu/libmpi.so.40(ompi_mpi_finalize+0x773)[0x7f2410bb12e3]
[<node_name>:70041] [ 5] ./a.out(+0xd2cd)[0x5642eb9e52cd]
[<node_name>:70041] [ 6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f2410797083]
[<node_name>:70041] [ 7] ./a.out(+0xc5ce)[0x5642eb9e45ce]
[<node_name>:70041] *** End of error message ***
TASK 1 called finalize
TASK 0 called finalize
TASK 3 called finalize
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node <node_name> exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
From the output, I see that the processes finish their tasks and every task (including master) calls finalize. Then, a seg fault happens for task 2 while the other 3 tasks finish. Although this is happening when I change the value of n such that n % number_of_slave_tasks != 0, I am positive nothing is wrong with that particular logic and I am accessing valid memory locations.
Help me out with this please!
I try to get a MPI-CUDA program working with MVAPICH CUDA8. I did run the program successfully with openMPI before but I want to test if I get better performance with MVAPICH. Unfortunately the program gets stuck in MPI_Isend if a CUDA kernel is running at the same time when using MVAPICH.
I downloaded MVAPICH2-2.2 and built it from the source with the configuration flags
--enable-cuda --disable-mcast
to enable MPI calls on cuda memory. mcast was disabled because I could not compile it without the flag.
I used the following flags before running the application:
export MV2_USE_CUDA=1
export MV2_GPUDIRECT_GDRCOPY_LIB=/path/to/gdrcopy/
export MV2_USE_GPUDIRECT=1
MPI_Isend/recv work fine when at the same time no CUDA kernel is running. But in my program it is important that MPI sends and receives data from and to GPU memory, while a kernel is running.
I came up with two possible reasons for that behavior. First, MVAPICH tries to run his own CUDA kernel for some reason to send data from GPU memory and this kernel does not get scheduled because the GPU is already fully utilized. Second possibility: MVAPICH uses cudaMemcpy somewhere (not the async version), which blocks until the kernel finishes execution.
Could someone confirm one of my assumptions? And is there a flag in MVAPICH that solves this problem that I am not aware of?
EDIT:
Here a "simpel" code that illustrates my problem. When executing the code with openMPI, it executes and terminates correctly. With mvapich2 it deadlocks at the marked MPI_Send function.
#include <stdio.h>
#include <stdlib.h>
#include <cuda.h>
#include <mpi.h>
__global__ void kernel(double * buffer, int rank)
{
volatile double *buf = buffer;
if(rank == 0){
while(buf[0] != 3){}
} else {
while(buf[0] != 2){}
}
}
int main(int argc, char **argv)
{
double host_buffer[1];
MPI_Init(&argc, &argv);
int world_size, world_rank;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
printf("Im rank %d\n", world_rank);
cudaSetDevice(world_rank);
double * dev_buffer;
cudaError_t err = cudaMalloc(&dev_buffer, sizeof(double));
if(world_rank == 0){
host_buffer[0] = 1;
cudaError_t err = cudaMemcpy(dev_buffer, host_buffer, sizeof(double), cudaMemcpyHostToDevice);
MPI_Send(dev_buffer, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD);
printf("[%d]First send does not deadlock\n", world_rank);
}else {
MPI_Recv(dev_buffer, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("[%d]Received first message\n", world_rank);
}
cudaStream_t stream, kernel_stream;
cudaStreamCreate(&stream);
cudaStreamCreate(&kernel_stream);
printf("[%d]launching kernel\n", world_rank);
kernel<<<208, 128, 0, kernel_stream>>>(dev_buffer, world_rank);
if(world_rank == 0){
//rank 0
host_buffer[0] = 2;
cudaMemcpyAsync(
dev_buffer, host_buffer, sizeof(double),
cudaMemcpyHostToDevice,
stream
);
cudaStreamSynchronize(stream);
printf("[%d]Send message\n", world_rank);
MPI_Send(dev_buffer, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD); //mvapich2 deadlocks here
printf("[%d]Message sent\n", world_rank);
printf("[%d]Receive message\n", world_rank);
MPI_Recv(dev_buffer, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("[%d]Message received\n", world_rank);
cudaStreamSynchronize(kernel_stream);
printf("[%d]kernel finished\n", world_rank);
} else {
//rank 1
printf("[%d]Receive message\n", world_rank);
MPI_Recv(dev_buffer, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("[%d]Message received\n", world_rank);
cudaStreamSynchronize(kernel_stream);
printf("[%d]kernel finished\n", world_rank);
host_buffer[0] = 3;
cudaMemcpyAsync(
dev_buffer, host_buffer, sizeof(double),
cudaMemcpyHostToDevice,
stream
);
cudaStreamSynchronize(stream);
printf("[%d]Send message\n", world_rank);
MPI_Send(dev_buffer, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD);
printf("[%d]Message sent\n", world_rank);
}
printf("[%d]Stopped execution\n", world_rank);
MPI_Finalize();
}
I got back to this problem and used gdb to debug the code.
Apparently, the problem is the eager protocol of MVAPICH2 implemented in src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c. The eager protocol uses a cuda_memcpy without async, which blocks until the kernel execution finishes.
The program posted in the question runs fine by passing MV2_IBA_EAGER_THRESHOLD 1 to mpirun. This prevents MPI to use the eager protocol and uses the rendez-vous protocol instead.
Patching the MVAPICH2 source code does solve the problem as well. I changed the synchronous cudaMemcpys to cudaMemcpyAsync in the files
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c
src/mpid/ch3/channels/mrail/src/gen2/ibv_recv.c
src/mpid/ch3/src/ch3u_request.c
The change in the third file is only needed for MPI_Isend/MPI_Irecv. Other MPI functions might need some additional code changes.
I'm having trouble with my MPI_Isend and MPI_Irecv blocks of code. I need to send a number Cin to the next process up the line, then the current process can go about it's business.
The receiving process needs to receive before it can go further in it's calculations, but when I don't have MPI_Wait it nevers gets the data, and when I do it just hangs forever. What am I doing wrong?
Note: I only set Cin to 3 in order to see when the message doesn't go through. Currently it just hangs.
void ComputeS5C()
{
MPI_Request send_request, recv_request;
MPI_Status status;
int Cin[1] = {3};
if(my_rank == 0){Cin[0] = 0;}
else {
MPI_Irecv(Cin, 1, MPI_INT, my_rank - 1, 0, MPI_COMM_WORLD, &recv_request);
MPI_Wait(&recv_request, &status);
fprintf(stderr, "RANK:%d Message Received from rank%d: Cin=%d\n", my_rank, my_rank-1, Cin[0]);
}
int k;
for(k = 0; k < Size_5; k++)
{
int s5clast;
if(k==0)
{
s5clast = Cin[0];
}
else
{
s5clast = s5c[k-1];
}
s5c[k] = s5g[k] | (s5p[k]&s5clast);
}
//if not highest rank, pass the carryin upstream
if(my_rank < world_size - 1){
MPI_Isend(&s5c[k], 1, MPI_INT, my_rank+1, 1, MPI_COMM_WORLD, &send_request);
fprintf(stderr, "RANK:%d Message sent to rank%d: Cin=%d\n", my_rank, my_rank+1, s5c[k]);
}
MPI_Wait(&send_request, &status);
}
The error in your code has to do with the missmatch of tags. Messages are sent using a tag = 1 and received using tag = 0. Sends and receives are not matching explaining why all processes are stuck waiting that sent messages get consumed. Change the tags so that they match.
A note, when using MPI_Irecv you always need an MPI_Wait to be sure to know when it is safe to consume received data. I think in your example use of MPI_Recv is more approriate.
It seems that you communicate one rank after the other sequentially. Quite large overhead.
I bring my code below but when I run it, I encounter with this error:
aborting job:
Fatal error in MPI_Waitall: Invalid MPI_Request, error stack:
MPI_Wait(171): MPI_Waitall(count=4, req_array=0x000001400E0DE0,status_array = 0x0000000000012FD8C) failed
MPI_Waitall(96). : Invalid MPI_Request
MPI_Waitall(96). : Invalid MPI_Request
When I’m using a blocking send and receive, my code giving me a correct answer but when I’m using a non blocking send and receive I getting an error.
This is my code :
integer reqs(4) ! required variable for non-blocking calls
integer stats(MPI_status_SIZE,4) ! required variable for WAITALL routine
call MPI_INIT( ierr )
call MPI_COMM_RANK( MPI_COMM_WORLD, taskid, ierr )
call MPI_COMM_SIZE( MPI_COMM_WORLD, numtasks, ierr )
! Send data to the left neighbor
if ((taskid > 0) ) then
call MPI_ISEND(phi0(1,1),N_z,MPI_DOUBLE_PRECISION,taskid-1,11,MPI_COMM_WORLD,&
reqs(1),ierr)
end if
! Send data to the right neighbor
if (taskid < numtasks-1) then
call MPI_ISEND(phi0(1,cols),N_z,MPI_DOUBLE_PRECISION,taskid+1,10,MPI_COMM_WORLD,&
reqs(2),ierr)
end if
! Receives data from the neighbor on the left
if (taskid > 0) then
call MPI_IRECV(phi0(1,0),N_z,MPI_DOUBLE_PRECISION,taskid-1,10,MPI_COMM_WORLD,&
reqs(3),ierr)
end if
! From the right side of the neighbors to get the data
if (taskid < numtasks-1) then
call MPI_IRECV(phi0(1,cols+1),N_z,MPI_DOUBLE_PRECISION,taskid+1,11,MPI_COMM_WORLD,&
reqs(4),ierr)
end if
call MPI_WAITALL(4, reqs, status_array, ierr)
i have two questions ; the first one is :
i'm gonna use msmpi and i meant by "only mpi" that we mustn't use sockets, my application is about a scalable distributed data structure; initially, we have a server contain a file which has a variable size (the size could be increased by insertions and decreased by deletion) and when the size of the file exceed certain limit the file will be splitted, the half remain in the first server and the second half will be moved to a new server and so on... and the client need to be always informed by the address of the data he want to retrieve so he should have an image of the split operation of the file. finally, i hope i make it clearer.
and the second one is:
i've tried to compile simple client/server application(the code source is bellow) with msmpi or mpich2 and it doesn't work and gives me the error message "fatal error in mpi_open_port() and other errors of stack", so i installed open mpi on ubunto 11.10, and tried to run the same example it worked with server side and it gave me a port name but on the client side it gave me the error message:
[user-Compaq-610:03833] [[39604,1],0] ORTE_ERROR_LOG: Not found in file ../../../../../../ompi/mca/dpm/orte/dpm_orte.c at line 155
[user-Compaq-610:3833] *** An error occurred in MPI_Comm_connect
[user-Compaq-610:3833] *** on communicator MPI_COMM_WORLD
[user-Compaq-610:3833] *** MPI_ERR_INTERN: internal error
[user-Compaq-610:3833] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 3833 on
node toufik-Compaq-610 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
so i'm confused what the problem is, and i spent a while trying to fix it,
i'd be greatfull if any body could help me with it, and thank u in advance.
the source code is here:
/* the server side */
#include <stdio.h>
#include <mpi.h>
main(int argc, char **argv)
{
int my_id;
char port_name[MPI_MAX_PORT_NAME];
MPI_Comm newcomm;
int passed_num;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
passed_num = 111;
if (my_id == 0)
{
MPI_Open_port(MPI_INFO_NULL, port_name);
printf("%s\n\n", port_name); fflush(stdout);
} /* endif */
MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &newcomm);
if (my_id == 0)
{
MPI_Send(&passed_num, 1, MPI_INT, 0, 0, newcomm);
printf("after sending passed_num %d\n", passed_num); fflush(stdout);
MPI_Close_port(port_name);
} /* endif */
MPI_Finalize();
exit(0);
} /* end main() */
and at the client side:
#include <stdio.h>
#include <mpi.h>
int main(int argc, char **argv)
{
int passed_num;
int my_id;
MPI_Comm newcomm;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
MPI_Comm_connect(argv[1], MPI_INFO_NULL, 0, MPI_COMM_WORLD, &newcomm);
if (my_id == 0)
{
MPI_Status status;
MPI_Recv(&passed_num, 1, MPI_INT, 0, 0, newcomm, &status);
printf("after receiving passed_num %d\n", passed_num); fflush(stdout);
} /* endif */
MPI_Finalize();
return 0;
//exit(0);
} /* end main() */
How exactly do you run the application? It seems that provided client and server codes are the same.
Usually the code is the same for all MPI processes and program decides what to execute basing on rank as in this snippet if (my_id == 0) { ... }. The application is executed with mpiexec. For example mpiexec -n 2 ./application would run two MPI processes with ranks 1 and 2 in one MPI_COMM_WORLD communicator. Where exactly the prococesses would be executed (on the same node or on different ones) depends on configuration.
Nevertheless, you should create port with MPI_Open_port and the pass it to MPI_Comm_connect. Here is an example on how to use these functions: MPI_Comm_connect
Moreover, for MPI_Recv there must be corresponding MPI_Send. Otherwise receiving process would wait forever.