i have two questions ; the first one is :
i'm gonna use msmpi and i meant by "only mpi" that we mustn't use sockets, my application is about a scalable distributed data structure; initially, we have a server contain a file which has a variable size (the size could be increased by insertions and decreased by deletion) and when the size of the file exceed certain limit the file will be splitted, the half remain in the first server and the second half will be moved to a new server and so on... and the client need to be always informed by the address of the data he want to retrieve so he should have an image of the split operation of the file. finally, i hope i make it clearer.
and the second one is:
i've tried to compile simple client/server application(the code source is bellow) with msmpi or mpich2 and it doesn't work and gives me the error message "fatal error in mpi_open_port() and other errors of stack", so i installed open mpi on ubunto 11.10, and tried to run the same example it worked with server side and it gave me a port name but on the client side it gave me the error message:
[user-Compaq-610:03833] [[39604,1],0] ORTE_ERROR_LOG: Not found in file ../../../../../../ompi/mca/dpm/orte/dpm_orte.c at line 155
[user-Compaq-610:3833] *** An error occurred in MPI_Comm_connect
[user-Compaq-610:3833] *** on communicator MPI_COMM_WORLD
[user-Compaq-610:3833] *** MPI_ERR_INTERN: internal error
[user-Compaq-610:3833] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 3833 on
node toufik-Compaq-610 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
so i'm confused what the problem is, and i spent a while trying to fix it,
i'd be greatfull if any body could help me with it, and thank u in advance.
the source code is here:
/* the server side */
#include <stdio.h>
#include <mpi.h>
main(int argc, char **argv)
{
int my_id;
char port_name[MPI_MAX_PORT_NAME];
MPI_Comm newcomm;
int passed_num;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
passed_num = 111;
if (my_id == 0)
{
MPI_Open_port(MPI_INFO_NULL, port_name);
printf("%s\n\n", port_name); fflush(stdout);
} /* endif */
MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &newcomm);
if (my_id == 0)
{
MPI_Send(&passed_num, 1, MPI_INT, 0, 0, newcomm);
printf("after sending passed_num %d\n", passed_num); fflush(stdout);
MPI_Close_port(port_name);
} /* endif */
MPI_Finalize();
exit(0);
} /* end main() */
and at the client side:
#include <stdio.h>
#include <mpi.h>
int main(int argc, char **argv)
{
int passed_num;
int my_id;
MPI_Comm newcomm;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
MPI_Comm_connect(argv[1], MPI_INFO_NULL, 0, MPI_COMM_WORLD, &newcomm);
if (my_id == 0)
{
MPI_Status status;
MPI_Recv(&passed_num, 1, MPI_INT, 0, 0, newcomm, &status);
printf("after receiving passed_num %d\n", passed_num); fflush(stdout);
} /* endif */
MPI_Finalize();
return 0;
//exit(0);
} /* end main() */
How exactly do you run the application? It seems that provided client and server codes are the same.
Usually the code is the same for all MPI processes and program decides what to execute basing on rank as in this snippet if (my_id == 0) { ... }. The application is executed with mpiexec. For example mpiexec -n 2 ./application would run two MPI processes with ranks 1 and 2 in one MPI_COMM_WORLD communicator. Where exactly the prococesses would be executed (on the same node or on different ones) depends on configuration.
Nevertheless, you should create port with MPI_Open_port and the pass it to MPI_Comm_connect. Here is an example on how to use these functions: MPI_Comm_connect
Moreover, for MPI_Recv there must be corresponding MPI_Send. Otherwise receiving process would wait forever.
Related
I am new to MPI programming and I write two simple programs to check if I can do the cross nodes communication. I first run the following program by using my own hostfile
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
// Print off a hello world message
printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name, world_rank, world_size);
// Finalize the MPI environment.
MPI_Finalize();
}
And then I got the follow results. It seems that everything works will and I can do the communication cross nodes. However, when I tried to do the Allreduce operation, I got the error message.
Hello world from processor lifs1.math.ust.hk, rank 4 out of 8 processors
Hello world from processor lifs1.math.ust.hk, rank 6 out of 8 processors
Hello world from processor lifs1.math.ust.hk, rank 5 out of 8 processors
Hello world from processor lifs1.math.ust.hk, rank 7 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 0 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 1 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 2 out of 8 processors
Hello world from processor lifs2.math.ust.hk, rank 3 out of 8 processors
[lifs1.math.ust.hk:29387] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[lifs1.math.ust.hk:29387] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
The allreduce programming is
int main(int argc, char** argv) {
MPI_Init(&argc, &argv);
int total_process;
int id;
MPI_Comm_size(MPI_COMM_WORLD, &total_process);
MPI_Comm_rank(MPI_COMM_WORLD, &id);
if (id == 0) {
std::cout << "========== Testing MPI Across Nodes ==========" << std::endl;
}
int N = 1000;
float* vec = new float[N];
for (int i = 0; i < N; ++i) {
vec[i] = 12.345;
}
float* global_sum = new float[N];
MPI_Allreduce(vec, global_sum, N, MPI_FLOAT, MPI_SUM, MPI_COMM_WORLD);
std::cout << "Result " << global_sum[0] << std::endl;
delete [] vec;
delete [] global_sum;
MPI_Finalize();
return 0;
}
I think 1000 single-precision floating point should not take too much time to finish the communication. The error message is
[lifs1.math.ust.hk:30146] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[lifs1.math.ust.hk:30146] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[lifs2][[13004,1],0][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs2][[13004,1],2][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs2][[13004,1],1][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs2][[13004,1],3][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.207 failed: Connection timed out (110)
[lifs1][[13004,1],4][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.208 failed: Connection timed out (110)
[lifs1][[13004,1],5][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] [lifs1][[13004,1],6][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.208 failed: Connection timed out (110)
[lifs1][[13004,1],7][btl_tcp_endpoint.c:803:mca_btl_tcp_endpoint_complete_connect] connect() to 143.89.16.208 failed: Connection timed out (110)
connect() to 143.89.16.208 failed: Connection timed out (110)
My questions are
Why can't I do the allreduce operation cross nodes?
My program is very simple, but I still get this message 7 more processes have sent help message help-mpi-btl-base.txt. What's this?
I try to get a MPI-CUDA program working with MVAPICH CUDA8. I did run the program successfully with openMPI before but I want to test if I get better performance with MVAPICH. Unfortunately the program gets stuck in MPI_Isend if a CUDA kernel is running at the same time when using MVAPICH.
I downloaded MVAPICH2-2.2 and built it from the source with the configuration flags
--enable-cuda --disable-mcast
to enable MPI calls on cuda memory. mcast was disabled because I could not compile it without the flag.
I used the following flags before running the application:
export MV2_USE_CUDA=1
export MV2_GPUDIRECT_GDRCOPY_LIB=/path/to/gdrcopy/
export MV2_USE_GPUDIRECT=1
MPI_Isend/recv work fine when at the same time no CUDA kernel is running. But in my program it is important that MPI sends and receives data from and to GPU memory, while a kernel is running.
I came up with two possible reasons for that behavior. First, MVAPICH tries to run his own CUDA kernel for some reason to send data from GPU memory and this kernel does not get scheduled because the GPU is already fully utilized. Second possibility: MVAPICH uses cudaMemcpy somewhere (not the async version), which blocks until the kernel finishes execution.
Could someone confirm one of my assumptions? And is there a flag in MVAPICH that solves this problem that I am not aware of?
EDIT:
Here a "simpel" code that illustrates my problem. When executing the code with openMPI, it executes and terminates correctly. With mvapich2 it deadlocks at the marked MPI_Send function.
#include <stdio.h>
#include <stdlib.h>
#include <cuda.h>
#include <mpi.h>
__global__ void kernel(double * buffer, int rank)
{
volatile double *buf = buffer;
if(rank == 0){
while(buf[0] != 3){}
} else {
while(buf[0] != 2){}
}
}
int main(int argc, char **argv)
{
double host_buffer[1];
MPI_Init(&argc, &argv);
int world_size, world_rank;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
printf("Im rank %d\n", world_rank);
cudaSetDevice(world_rank);
double * dev_buffer;
cudaError_t err = cudaMalloc(&dev_buffer, sizeof(double));
if(world_rank == 0){
host_buffer[0] = 1;
cudaError_t err = cudaMemcpy(dev_buffer, host_buffer, sizeof(double), cudaMemcpyHostToDevice);
MPI_Send(dev_buffer, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD);
printf("[%d]First send does not deadlock\n", world_rank);
}else {
MPI_Recv(dev_buffer, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("[%d]Received first message\n", world_rank);
}
cudaStream_t stream, kernel_stream;
cudaStreamCreate(&stream);
cudaStreamCreate(&kernel_stream);
printf("[%d]launching kernel\n", world_rank);
kernel<<<208, 128, 0, kernel_stream>>>(dev_buffer, world_rank);
if(world_rank == 0){
//rank 0
host_buffer[0] = 2;
cudaMemcpyAsync(
dev_buffer, host_buffer, sizeof(double),
cudaMemcpyHostToDevice,
stream
);
cudaStreamSynchronize(stream);
printf("[%d]Send message\n", world_rank);
MPI_Send(dev_buffer, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD); //mvapich2 deadlocks here
printf("[%d]Message sent\n", world_rank);
printf("[%d]Receive message\n", world_rank);
MPI_Recv(dev_buffer, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("[%d]Message received\n", world_rank);
cudaStreamSynchronize(kernel_stream);
printf("[%d]kernel finished\n", world_rank);
} else {
//rank 1
printf("[%d]Receive message\n", world_rank);
MPI_Recv(dev_buffer, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("[%d]Message received\n", world_rank);
cudaStreamSynchronize(kernel_stream);
printf("[%d]kernel finished\n", world_rank);
host_buffer[0] = 3;
cudaMemcpyAsync(
dev_buffer, host_buffer, sizeof(double),
cudaMemcpyHostToDevice,
stream
);
cudaStreamSynchronize(stream);
printf("[%d]Send message\n", world_rank);
MPI_Send(dev_buffer, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD);
printf("[%d]Message sent\n", world_rank);
}
printf("[%d]Stopped execution\n", world_rank);
MPI_Finalize();
}
I got back to this problem and used gdb to debug the code.
Apparently, the problem is the eager protocol of MVAPICH2 implemented in src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c. The eager protocol uses a cuda_memcpy without async, which blocks until the kernel execution finishes.
The program posted in the question runs fine by passing MV2_IBA_EAGER_THRESHOLD 1 to mpirun. This prevents MPI to use the eager protocol and uses the rendez-vous protocol instead.
Patching the MVAPICH2 source code does solve the problem as well. I changed the synchronous cudaMemcpys to cudaMemcpyAsync in the files
src/mpid/ch3/channels/mrail/src/gen2/ibv_send.c
src/mpid/ch3/channels/mrail/src/gen2/ibv_recv.c
src/mpid/ch3/src/ch3u_request.c
The change in the third file is only needed for MPI_Isend/MPI_Irecv. Other MPI functions might need some additional code changes.
I have install MPI and found simple example of thread that I'm don't fully understand. Can I get a simple example of MPI thread that prints something. Thanks
MPI_Init(&argc, &argv);
int errs = 0;
int provided, flag, claimed;
MPI_Init_thread(0, 0, MPI_THREAD_MULTIPLE, &provided); //what does it MPI_THREAD_MULTIPLE mean
MPI_Is_thread_main(&flag); //why there is flag here
if (!flag) {
errs++; //why there is counter for errs
printf("..."); fflush(stdout); // doesn't print anything
}
MPI_Finalize();
//return errs; I get error here
I have a serial C++ program that I wish to parallelize. I know the basics of MPI, MPI_Send, MPI_Recv, etc. Basically, I have a data generation algorithm that runs significantly faster than the data processing algorithm. Currently they run in series, but I was thinking that running the data generation in the root process, having the data processing done on the slave processes, and sending a message from the root to a slave containing the data to be processed. This way, each slave processes a data set and then waits for its next data set.
The problem is that, once the root process is done generating data, the program hangs because the slaves are waiting for more.
This is an example of the problem:
#include "mpi.h"
#include <cassert>
#include <cstdio>
class Generator {
public:
Generator(int min, int max) : value(min - 1), max(max) {}
bool NextValue() {
++value;
return value < max;
}
int Value() { return value; }
private:
int value, max;
Generator() {}
Generator(const Generator &other) {}
Generator &operator=(const Generator &other) { return *this; }
};
long fibonnaci(int n) {
assert(n > 0);
if (n == 1 || n == 2) return 1;
return fibonnaci(n-1) + fibonnaci(n-2);
}
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
int rank, num_procs;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
if (rank == 0) {
Generator generator(1, 2 * num_procs);
int proc = 1;
while (generator.NextValue()) {
int value = generator.Value();
MPI_Send(&value, 1, MPI_INT, proc, 73, MPI_COMM_WORLD);
printf("** Sent %d to process %d.\n", value, proc);
proc = proc % (num_procs - 1) + 1;
}
} else {
while (true) {
int value;
MPI_Status status;
MPI_Recv(&value, 1, MPI_INT, 0, 73, MPI_COMM_WORLD, &status);
printf("** Received %d from process %d.\n", value, status.MPI_SOURCE);
printf("Process %d computed %d.\n", rank, fibonnaci(2 * (value + 10)));
}
}
MPI_Finalize();
return 0;
}
Obviously not everything above is "good practice", but it is sufficient to get the point across.
If I remove the while(true) from the slave processes, then the program exits when each of the slaves have exited. I would like the program to exit only after the root process has done its job AND all of the slaves have processed everything that has been sent.
If I knew how many data sets would be generated, I could have that many process running and everything would exit nicely, but that isn't the case here.
Any suggestions? Is there anything in the API that will do this? Could this be solved better with a better topology? Would MPI_Isend or MPI_IRecv do this better? I am fairly new to MPI so bear with me.
Thanks
The usual practice is to send to all worker processes an empty message with a special tag that signals them to exit the infinite processing loop. Let's say this tag is 42. You would do something like that in the worker loop:
while (true) {
int value;
MPI_Status status;
MPI_Recv(&value, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
if (status.MPI_TAG == 42) {
printf("Process %d exiting work loop.\n", rank);
break;
}
printf("** Received %d from process %d.\n", value, status.MPI_SOURCE);
printf("Process %d computed %d.\n", rank, fibonnaci(2 * (value + 10)));
}
The manager process would do something like this after the generator loop:
for (int i = 1; i < num_procs; i++)
MPI_Send(&i, 0, MPI_INT, i, 42, MPI_COMM_WORLD);
Regarding your next question. Using MPI_Isend() in the master process would deserialise the execution and increase the performance. The truth however is that you are sending very small messages and those are typically internally buffered (WARNING - implementation dependent!) so your MPI_Send() is actually non-blocking and you already have non-serial execution. MPI_Isend() returns an MPI_Request handle that you need to take care of later. You could either wait for it to finish with MPI_Wait() or MPI_Waitall() but you could also just call MPI_Request_free() on it and it will be automatically freed when the operation is over. This is usually done when you'd like to send many messages asynchronously and would not care on when the sends will be completed, but it's a bad practice nevertheless since having a large number of outstanding requests can consume lots of precious memory. As for the worker processes - they need the data in order to proceed with the computation so using MPI_Irecv() is not necessary.
Welcome to the wonderful world of MPI programming!
I had a problem with a program that uses MPI and I have just fixed it, however, I don't seem to understand what was wrong in the first place. I'm quite green with programming relates stuff, so please be forgiving.
The program is:
#include <iostream>
#include <cstdlib>
#include <mpi.h>
#define RNumber 3
using namespace std;
int main() {
/*Initiliaze MPI*/
int my_rank; //My process rank
int comm_sz; //Number of processes
MPI_Comm GathComm; //Communicator for MPI_Gather
MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
/*Initialize an array for results*/
long rawT[RNumber];
long * Times = NULL; //Results from threads
if (my_rank == 0) Times = (long*) malloc(comm_sz*RNumber*sizeof(long));
/*Fill rawT with results at threads*/
for (int i = 0; i < RNumber; i++) {
rawT[i] = i;
}
if (my_rank == 0) {
/*Main thread recieves data from other threads*/
MPI_Gather(rawT, RNumber, MPI_LONG, Times, RNumber, MPI_LONG, 0, GathComm);
}
else {
/*Other threads send calculation results to main thread*/
MPI_Gather(rawT, RNumber, MPI_LONG, Times, RNumber, MPI_LONG, 0, GathComm);
}
/*Finalize MPI*/
MPI_Finalize();
return 0;
};
On execution the program returns the following message:
Fatal error in PMPI_Gather: Invalid communicator, error stack:
PMPI_Gather(863): MPI_Gather(sbuf=0xbf824b70, scount=3, MPI_LONG,
rbuf=0x98c55d8, rcount=3, MPI_LONG, root=0, comm=0xe61030) failed
PMPI_Gather(757): Invalid communicator Fatal error in PMPI_Gather:
Invalid communicator, error stack: PMPI_Gather(863):
MPI_Gather(sbuf=0xbf938960, scount=3, MPI_LONG, rbuf=(nil), rcount=3,
MPI_LONG, root=0, comm=0xa6e030) failed PMPI_Gather(757): Invalid
communicator
After I remove GathComm altogether and substitute it with MPI_COMM_WORLD default communicator everything works fine.
Could anyone be so kind to explain what was I doing wrong and how did this adjustment made everything work?
That's because GathComm has not been assigned a valid communicator. "MPI_Comm GathComm;" only declares the variable to hold a communicator but doesn't create one.
You can use the default communicator (MPI_COMM_WORLD) if you simply want to include all procs in the operation.
Custom communicators are useful when you want to organised your procs in separate groups or when using virtual communication topologies.
To find out more, check out this article which describes Groups, Communicator and Topologies.