I had a problem with a program that uses MPI and I have just fixed it, however, I don't seem to understand what was wrong in the first place. I'm quite green with programming relates stuff, so please be forgiving.
The program is:
#include <iostream>
#include <cstdlib>
#include <mpi.h>
#define RNumber 3
using namespace std;
int main() {
/*Initiliaze MPI*/
int my_rank; //My process rank
int comm_sz; //Number of processes
MPI_Comm GathComm; //Communicator for MPI_Gather
MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
/*Initialize an array for results*/
long rawT[RNumber];
long * Times = NULL; //Results from threads
if (my_rank == 0) Times = (long*) malloc(comm_sz*RNumber*sizeof(long));
/*Fill rawT with results at threads*/
for (int i = 0; i < RNumber; i++) {
rawT[i] = i;
}
if (my_rank == 0) {
/*Main thread recieves data from other threads*/
MPI_Gather(rawT, RNumber, MPI_LONG, Times, RNumber, MPI_LONG, 0, GathComm);
}
else {
/*Other threads send calculation results to main thread*/
MPI_Gather(rawT, RNumber, MPI_LONG, Times, RNumber, MPI_LONG, 0, GathComm);
}
/*Finalize MPI*/
MPI_Finalize();
return 0;
};
On execution the program returns the following message:
Fatal error in PMPI_Gather: Invalid communicator, error stack:
PMPI_Gather(863): MPI_Gather(sbuf=0xbf824b70, scount=3, MPI_LONG,
rbuf=0x98c55d8, rcount=3, MPI_LONG, root=0, comm=0xe61030) failed
PMPI_Gather(757): Invalid communicator Fatal error in PMPI_Gather:
Invalid communicator, error stack: PMPI_Gather(863):
MPI_Gather(sbuf=0xbf938960, scount=3, MPI_LONG, rbuf=(nil), rcount=3,
MPI_LONG, root=0, comm=0xa6e030) failed PMPI_Gather(757): Invalid
communicator
After I remove GathComm altogether and substitute it with MPI_COMM_WORLD default communicator everything works fine.
Could anyone be so kind to explain what was I doing wrong and how did this adjustment made everything work?
That's because GathComm has not been assigned a valid communicator. "MPI_Comm GathComm;" only declares the variable to hold a communicator but doesn't create one.
You can use the default communicator (MPI_COMM_WORLD) if you simply want to include all procs in the operation.
Custom communicators are useful when you want to organised your procs in separate groups or when using virtual communication topologies.
To find out more, check out this article which describes Groups, Communicator and Topologies.
Related
i've some problems with this piece of code.
NEIGHBOORDHOOD_FLAG = 1234;
int number_of_nodes=90;
MPI_Request neig_request[2*number_of_nodes];
MPI_Status neig_status[2*number_of_nodes];
int neig_counter=0;
int is_neighbour_send[number_of_nodes];
int is_neighbour_receive[number_of_nodes];
for(int v=2;v<number_of_nodes+2;++v){
is_neighbour_send[v-2] = (util.check_targets(neighbours,v)==true)?1:0;
this->communicate_neighborhood(&(is_neighbour_send[v-2]),v,&(neig_request[neig_counter]));
++neig_counter;
}
//ricevo la comunicazione dai nodi per sapere di chi sono vicino
for(int v=2;v<number_of_nodes+2;++v){
this->receive_neighborhood(&(is_neighbour_receive[v-2]),v,&(neig_request[neig_counter]));
++neig_counter;
}
MPI_Waitall(neig_counter,neig_request,neig_status);
The two methods above are essentially two wrapped versions of the MPI send and receive:
void NodeAgent::communicate_neighborhood(int *neigh,int dest_node,MPI_Request *req){
MPI_Class::send_message(neigh,MPI_INT,1,dest_node,NEIGHBOORDHOOD_FLAG,req);
}
void NodeAgent::receive_neighborhood(int *is_neighbour,int node_mitt,MPI_Request *req){
MPI_Class::receive_message(is_neighbour,MPI_INT,1,node_mitt,NEIGHBOORDHOOD_FLAG,req);
}
with send and receive static methods:
void MPI_Class::send_message(void *content,MPI_Datatype datatype,int length,int dest,int tag,MPI_Request *request){
int np = MPI_Class::count();
MPI_Isend(content, length, datatype, (dest % np), tag, MPI_COMM_WORLD, request);
}
void MPI_Class::receive_message(void *buffer,MPI_Datatype datatype,int length,int dest,int tag,MPI_Request *request){
int np = MPI_Class::count();
MPI_Irecv(buffer,length,datatype,(dest % np),tag,MPI_COMM_WORLD,request);
}
The number of nodes is the number of processes - 2 (there are two processes that i use for others operations) and i've used the " % " in send and receive to ensure that the sender and the recipient are always in the range that i want.
The interesting thing is that this code executes properly also with large number of process (100). The problem arises when i launch the program on two machines or more...
It gives me: Fatal error in PMPI_Waitall: See the MPI_ERROR field in MPI_Status for the error code
I tried to print the MPI_ERROR of each status, but are all zeroes...
Can someone help me?
many thanks in advance
Edit, here the MCVE:
#include <mpi.h>
#include <stdlib.h>
int main(int argc,char **argv){
MPI_Init(&argc,&argv);
int my_id;
MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Status * statuses = (MPI_Status *)malloc(2*world_size * sizeof(MPI_Status));
MPI_Request * requestes = (MPI_Request *)malloc(2*world_size * sizeof(MPI_Request));
int counter =0;
int *tosend = (int *) malloc((world_size - 2) * sizeof(int));
int *toreceive = (int *) malloc((world_size - 2) * sizeof(int));
for(int i=0;i<world_size-2;++i){
tosend[i]=0;
}
if(my_id>1){
for(int i=2;i<world_size;++i){
MPI_Isend(&tosend[i-2],1,MPI_INT,i,0,MPI_COMM_WORLD,&requestes[counter]);
++counter;
}
}
if(my_id>1){
for(int i=2;i<world_size;++i){
MPI_Irecv(&(toreceive[i-2]),1,MPI_INT,i,0,MPI_COMM_WORLD,&(requestes[counter]));
++counter;
}
}
MPI_Waitall(counter,requestes,statuses);
free(statuses);
free(requestes);
free(tosend);
free(toreceive);
MPI_Finalize();
}
Also this code executes correctly on a single machine. The problem arises when i launch it on 2 machines mutual authenticated (ssh). I just discovered that after the MPI_Waitall fatal error, if i try to connect via ssh from a machine to other, it ask me the password....
Why? At this point i think the problem is the communication via ssh and not the code..
Edit2: solved! it was a ssh problem that i solved in some way (i don't know how honestly, i redid the authentication procedures ten times as long as it worked....
I have install MPI and found simple example of thread that I'm don't fully understand. Can I get a simple example of MPI thread that prints something. Thanks
MPI_Init(&argc, &argv);
int errs = 0;
int provided, flag, claimed;
MPI_Init_thread(0, 0, MPI_THREAD_MULTIPLE, &provided); //what does it MPI_THREAD_MULTIPLE mean
MPI_Is_thread_main(&flag); //why there is flag here
if (!flag) {
errs++; //why there is counter for errs
printf("..."); fflush(stdout); // doesn't print anything
}
MPI_Finalize();
//return errs; I get error here
Why I get the following error for the following code with mpirun -np 2 ./out command? I called make_layout() after resizing the std::vector so normally I should not get this error. It works if I do not resize. What is the reason?
main.cpp:
#include <iostream>
#include <vector>
#include "mpi.h"
MPI_Datatype MPI_CHILD;
struct Child
{
std::vector<int> age;
void make_layout();
};
void Child::make_layout()
{
int nblock = 1;
int age_size = age.size();
int block_count[nblock] = {age_size};
MPI_Datatype block_type[nblock] = {MPI_INT};
MPI_Aint offset[nblock] = {0};
MPI_Type_struct(nblock, block_count, offset, block_type, &MPI_CHILD);
MPI_Type_commit(&MPI_CHILD);
}
int main()
{
int rank, size;
MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
Child kid;
kid.age.resize(5);
kid.make_layout();
int datasize;
MPI_Type_size(MPI_CHILD, &datasize);
std::cout << datasize << std::endl; // output: 20 (5x4 seems OK).
if (rank == 0)
{
MPI_Send(&kid, 1, MPI_CHILD, 1, 0, MPI_COMM_WORLD);
}
if (rank == 1)
{
MPI_Recv(&kid, 1, MPI_CHILD, 0, 0, MPI_COMM_WORLD, NULL);
}
MPI_Finalize();
return 0;
}
Error message:
*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: 0x14ae7b8
[ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7fe1ad91c3d0]
[ 1] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x22)[0x7fe1ad5c5a92]
[ 2] ./out[0x400de4]
[ 3] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fe1ad562830]
[ 4] ./out[0x400ec9]
*** End of error message ***
Here is an example with several std::vector members that uses MPI datatypes with absolute addresses:
struct Child
{
int foo;
std::vector<float> bar;
std::vector<int> baz;
Child() : dtype(MPI_DATATYPE_NULL) {}
~Child() { if (dtype != MPI_DATATYPE_NULL) MPI_Type_free(dtype); }
const MPI_Datatype mpi_dtype();
void invalidate_dtype();
private:
MPI_Datatype dtype;
void make_dtype();
};
const MPI_Datatype Child::mpi_dtype()
{
if (dtype == MPI_DATATYPE_NULL)
make_dtype();
return dtype;
}
void Child::invalidate_dtype()
{
if (dtype != MPI_DATATYPE_NULL)
MPI_Datatype_free(&dtype);
}
void Child::make_dtype()
{
const int nblock = 3;
int block_count[nblock] = {1, bar.size(), baz.size()};
MPI_Datatype block_type[nblock] = {MPI_INT, MPI_FLOAT, MPI_INT};
MPI_Aint offset[nblock];
MPI_Get_address(&foo, &offset[0]);
MPI_Get_address(&bar[0], &offset[1]);
MPI_Get_address(&baz[0], &offset[2]);
MPI_Type_struct(nblock, block_count, offset, block_type, &dtype);
MPI_Type_commit(&dtype);
}
Sample use of that class:
Child kid;
kid.foo = 5;
kid.bar.resize(5);
kid.baz.resize(10);
if (rank == 0)
{
MPI_Send(MPI_BOTTOM, 1, kid.mpi_dtype(), 1, 0, MPI_COMM_WORLD);
}
if (rank == 1)
{
MPI_Recv(MPI_BOTTOM, 1, kid.mpi_dtype(), 0, 0, MPI_COMM_WORLD, NULL);
}
Notice the use of MPI_BOTTOM as the buffer address. MPI_BOTTOM specifies the bottom of the address space, which is 0 on architectures with flat address space. Since the offsets passed to MPI_Type_create_struct are the absolute addresses of the structure members, when those are added to 0, the result is again the absolute address of each structure member. Child::mpi_dtype() returns a lazily constructed MPI datatype specific to that instance.
Since resize() reallocates memory, which could result in the data being moved to a different location in memory, the invalidate_dtype() method should be used to force the recreation of the MPI datatype after resize() or any other operation that might trigger memory reallocation:
// ...
kid.bar.resize(100);
kid.invalidate_dtype();
// MPI_Send / MPI_Recv
Please excuse any sloppy C++ code above.
The problem here is that you're telling MPI to send a block of integers from &kid, but that's not where your data is. &kid points to an std::vector object, which has an internal pointer to your block of integers allocated somewhere on the heap.
Replace &kid with kid.age.data() and it should work. The reason it "works" when you don't resize is that the vectors will be of 0 size, so MPI will try to send an empty message and no actual memory access takes place.
Be careful, you faced several problems.
First std::vector stores object in heap, so data is not really stored inside your struct.
Second you are not able to send STL containers even between dynamic libraries, also for app instances this is also true. Because they may be compiled with different versions of STL and work on different architectures differently.
Here is good answer about this part of question: https://stackoverflow.com/a/22797419/440168
I have a serial C++ program that I wish to parallelize. I know the basics of MPI, MPI_Send, MPI_Recv, etc. Basically, I have a data generation algorithm that runs significantly faster than the data processing algorithm. Currently they run in series, but I was thinking that running the data generation in the root process, having the data processing done on the slave processes, and sending a message from the root to a slave containing the data to be processed. This way, each slave processes a data set and then waits for its next data set.
The problem is that, once the root process is done generating data, the program hangs because the slaves are waiting for more.
This is an example of the problem:
#include "mpi.h"
#include <cassert>
#include <cstdio>
class Generator {
public:
Generator(int min, int max) : value(min - 1), max(max) {}
bool NextValue() {
++value;
return value < max;
}
int Value() { return value; }
private:
int value, max;
Generator() {}
Generator(const Generator &other) {}
Generator &operator=(const Generator &other) { return *this; }
};
long fibonnaci(int n) {
assert(n > 0);
if (n == 1 || n == 2) return 1;
return fibonnaci(n-1) + fibonnaci(n-2);
}
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
int rank, num_procs;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &num_procs);
if (rank == 0) {
Generator generator(1, 2 * num_procs);
int proc = 1;
while (generator.NextValue()) {
int value = generator.Value();
MPI_Send(&value, 1, MPI_INT, proc, 73, MPI_COMM_WORLD);
printf("** Sent %d to process %d.\n", value, proc);
proc = proc % (num_procs - 1) + 1;
}
} else {
while (true) {
int value;
MPI_Status status;
MPI_Recv(&value, 1, MPI_INT, 0, 73, MPI_COMM_WORLD, &status);
printf("** Received %d from process %d.\n", value, status.MPI_SOURCE);
printf("Process %d computed %d.\n", rank, fibonnaci(2 * (value + 10)));
}
}
MPI_Finalize();
return 0;
}
Obviously not everything above is "good practice", but it is sufficient to get the point across.
If I remove the while(true) from the slave processes, then the program exits when each of the slaves have exited. I would like the program to exit only after the root process has done its job AND all of the slaves have processed everything that has been sent.
If I knew how many data sets would be generated, I could have that many process running and everything would exit nicely, but that isn't the case here.
Any suggestions? Is there anything in the API that will do this? Could this be solved better with a better topology? Would MPI_Isend or MPI_IRecv do this better? I am fairly new to MPI so bear with me.
Thanks
The usual practice is to send to all worker processes an empty message with a special tag that signals them to exit the infinite processing loop. Let's say this tag is 42. You would do something like that in the worker loop:
while (true) {
int value;
MPI_Status status;
MPI_Recv(&value, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
if (status.MPI_TAG == 42) {
printf("Process %d exiting work loop.\n", rank);
break;
}
printf("** Received %d from process %d.\n", value, status.MPI_SOURCE);
printf("Process %d computed %d.\n", rank, fibonnaci(2 * (value + 10)));
}
The manager process would do something like this after the generator loop:
for (int i = 1; i < num_procs; i++)
MPI_Send(&i, 0, MPI_INT, i, 42, MPI_COMM_WORLD);
Regarding your next question. Using MPI_Isend() in the master process would deserialise the execution and increase the performance. The truth however is that you are sending very small messages and those are typically internally buffered (WARNING - implementation dependent!) so your MPI_Send() is actually non-blocking and you already have non-serial execution. MPI_Isend() returns an MPI_Request handle that you need to take care of later. You could either wait for it to finish with MPI_Wait() or MPI_Waitall() but you could also just call MPI_Request_free() on it and it will be automatically freed when the operation is over. This is usually done when you'd like to send many messages asynchronously and would not care on when the sends will be completed, but it's a bad practice nevertheless since having a large number of outstanding requests can consume lots of precious memory. As for the worker processes - they need the data in order to proceed with the computation so using MPI_Irecv() is not necessary.
Welcome to the wonderful world of MPI programming!
i have two questions ; the first one is :
i'm gonna use msmpi and i meant by "only mpi" that we mustn't use sockets, my application is about a scalable distributed data structure; initially, we have a server contain a file which has a variable size (the size could be increased by insertions and decreased by deletion) and when the size of the file exceed certain limit the file will be splitted, the half remain in the first server and the second half will be moved to a new server and so on... and the client need to be always informed by the address of the data he want to retrieve so he should have an image of the split operation of the file. finally, i hope i make it clearer.
and the second one is:
i've tried to compile simple client/server application(the code source is bellow) with msmpi or mpich2 and it doesn't work and gives me the error message "fatal error in mpi_open_port() and other errors of stack", so i installed open mpi on ubunto 11.10, and tried to run the same example it worked with server side and it gave me a port name but on the client side it gave me the error message:
[user-Compaq-610:03833] [[39604,1],0] ORTE_ERROR_LOG: Not found in file ../../../../../../ompi/mca/dpm/orte/dpm_orte.c at line 155
[user-Compaq-610:3833] *** An error occurred in MPI_Comm_connect
[user-Compaq-610:3833] *** on communicator MPI_COMM_WORLD
[user-Compaq-610:3833] *** MPI_ERR_INTERN: internal error
[user-Compaq-610:3833] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 3833 on
node toufik-Compaq-610 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
so i'm confused what the problem is, and i spent a while trying to fix it,
i'd be greatfull if any body could help me with it, and thank u in advance.
the source code is here:
/* the server side */
#include <stdio.h>
#include <mpi.h>
main(int argc, char **argv)
{
int my_id;
char port_name[MPI_MAX_PORT_NAME];
MPI_Comm newcomm;
int passed_num;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
passed_num = 111;
if (my_id == 0)
{
MPI_Open_port(MPI_INFO_NULL, port_name);
printf("%s\n\n", port_name); fflush(stdout);
} /* endif */
MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &newcomm);
if (my_id == 0)
{
MPI_Send(&passed_num, 1, MPI_INT, 0, 0, newcomm);
printf("after sending passed_num %d\n", passed_num); fflush(stdout);
MPI_Close_port(port_name);
} /* endif */
MPI_Finalize();
exit(0);
} /* end main() */
and at the client side:
#include <stdio.h>
#include <mpi.h>
int main(int argc, char **argv)
{
int passed_num;
int my_id;
MPI_Comm newcomm;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
MPI_Comm_connect(argv[1], MPI_INFO_NULL, 0, MPI_COMM_WORLD, &newcomm);
if (my_id == 0)
{
MPI_Status status;
MPI_Recv(&passed_num, 1, MPI_INT, 0, 0, newcomm, &status);
printf("after receiving passed_num %d\n", passed_num); fflush(stdout);
} /* endif */
MPI_Finalize();
return 0;
//exit(0);
} /* end main() */
How exactly do you run the application? It seems that provided client and server codes are the same.
Usually the code is the same for all MPI processes and program decides what to execute basing on rank as in this snippet if (my_id == 0) { ... }. The application is executed with mpiexec. For example mpiexec -n 2 ./application would run two MPI processes with ranks 1 and 2 in one MPI_COMM_WORLD communicator. Where exactly the prococesses would be executed (on the same node or on different ones) depends on configuration.
Nevertheless, you should create port with MPI_Open_port and the pass it to MPI_Comm_connect. Here is an example on how to use these functions: MPI_Comm_connect
Moreover, for MPI_Recv there must be corresponding MPI_Send. Otherwise receiving process would wait forever.