openmpi/c++: defining a mpi data type for class with members of variable length (pointers pointing to malloced memory)? - c++

Question

i am currently learning to use openmpi, my aim is to parallelize a simple program whose code i will post bellow.
The program is for testing my concept of paralleling a much bigger program, i hope to learn all i need to know for my actual problem if i succeed with this.
Basically it is a definition of a simple c++ class for lists. A list consists of two arrays, one integer and one double. Entries with the same indicies belong together, in a way that the integer entry is some kind of list entry identifier (maybe an object ID) and the double entry is some kind of quantifier (maybe the weight if an object).
The basic purpose of the program is to add lists together (this is the task i want to parallelize). Adding works as follows: For each entry in one list it is checked if there is the same integer entry in the the other list, if so then the double entry gets added to the double entry in the other list, if there is no such entry in the other list then both the integer and the double entries gets added to the end of the list.
Basically each summand in this list addition represents a storage and each entry is a type of object with a given amount (int is the type and double is the amount), so adding two lists means putting the stuff from the second storage to the first.
The order of the list entries is irrelevant, this means that the addition of lists is not only associative but commutative too!
My plan is to add a very large number of such lists (a few billions) so parallelizing could be to let each thread add a subset of lists first and when this is finished distribute all such sublists (one for each thread) to all of the threads.
My current understanding of openmpi is that only the last step (distributing of finished sublists) needs any special non standard stuff. Basically i need a AllReduce but with a custom data type and a custom operaton.
The first problem i have is understanding how to create a fitting MPI data type. I came to the conclusion that i probably need MPI_Type_create_struct to create a struct type.
I found this site with a nice example: http://mpi.deino.net/mpi_functions/MPI_Type_create_struct.html
from which i learned a lot but the problem is, that in this case there are fixed member arrays. In my case i have lists with arbitrary sized member variables or better with pointers pointing to memory blocks of arbitrary size. So doing it like in the example would lead to creating a new MPI datatype for each list size (using fixed sized lists could help but only in this minimalistic case, but i want to learn how to do it with arbitrary sized lists are preparation for my actual problem).
So my question is: how to create a data type for this special case? What is the best way?
I even thought to maybe write some non mpi code to serialize my class/object, (which would be a lot of work for my real problem but in this example it should be easy) to a single block of bits. Then i could simply use a MPI function to distribute those blocks to all threads and then i just have to translate it back to the actual object, and then i could let each thread simply add the "number-of-threads" lists together to have the same full reduced list on all threads (because the operation is commutative it is not important if the order is the same on each thread in the end).
The problem is that i do not know which MPI function to use to distribute a such memory blocks to each thread so that in the end each thread has an array of "number-of-threads" such blocks (similar like AllReduce but with blocks).
But thats just another idea, i would like to hear from you whats the best way.
Thank you, here is my fully working example program (ignore the MPI parts thats just preparation, you can simply compile with: g++)
As you can see, i needed to create custom copy constructors because standard of the pointer members. I hope thats not a problem for MPI?
#include <iostream>
#include <cstdlib>
#if (CFG_MPI > 0)
#include <mpi.h>
#else
#define MPI_Barrier(xxx) // dummy code if not parallel
#endif
class list {
private:
int *ilist;
double *dlist;
int n;
public:
list(int n, int *il, double *dl) {
int i;
if (n>0) {
this->ilist = (int*)malloc(n*sizeof(int));
this->dlist = (double*)malloc(n*sizeof(double));
if (!ilist || !dlist) std::cout << "ERROR: malloc in constructor failed!" << std::endl;
} else {
this->ilist = NULL;
this->dlist = NULL;
}
for (i=0; i<n; i++) {
this->ilist[i] = il[i];
this->dlist[i] = dl[i];
}
this->n = n;
}
~list() {
free(ilist);
free(dlist);
ilist = NULL;
dlist = NULL;
this->n=0;
}
list(const list& cp) {
int i;
this->n = cp.n;
this->ilist = NULL;
this->dlist = NULL;
if (this->n > 0) {
this->ilist = (int*)malloc(this->n*sizeof(int));
this->dlist = (double*)malloc(this->n*sizeof(double));
if (!ilist || !dlist) std::cout << "ERROR: malloc in copy constructor failed!" << std::endl;
}
for (i=0; i<this->n; i++) {
this->ilist[i] = cp.ilist[i];
this->dlist[i] = cp.dlist[i];
}
}
list& operator=(const list& cp) {
if(this == &cp) return *this;
this->~list();
int i;
this->n = cp.n;
if (this->n > 0) {
this->ilist = (int*)malloc(this->n*sizeof(int));
this->dlist = (double*)malloc(this->n*sizeof(double));
if (!ilist || !dlist) std::cout << "ERROR: malloc in copy constructor failed!" << std::endl;
} else {
this->ilist = NULL;
this->dlist = NULL;
}
for (i=0; i<this->n; i++) {
this->ilist[i] = cp.ilist[i];
this->dlist[i] = cp.dlist[i];
}
return *this;
}
void print() {
int i;
for (i=0; i<this->n; i++)
std::cout << i << " : " << "[" << this->ilist[i] << " - " << (double)dlist[i] << "]" << std::endl;
}
list& operator+=(const list& cp) {
int i,j;
if(this == &cp) {
for (i=0; i<this->n; i++)
this->dlist[i] *= 2;
return *this;
}
double *dl;
int *il;
il = (int *) realloc(this->ilist, (this->n+cp.n)*sizeof(int));
dl = (double *) realloc(this->dlist, (this->n+cp.n)*sizeof(double));
if (!il || !dl)
std::cout << "ERROR: 1st realloc in operator += failed!" << std::endl;
else {
this->ilist = il;
this->dlist = dl;
il = NULL;
dl = NULL;
}
for (i=0; i<cp.n; i++) {
for (j=0; j<this->n; j++) {
if (this->ilist[j] == cp.ilist[i]) {
this->dlist[j] += cp.dlist[i];
break;
}
} if (j == this->n) {// no matching entry found in this
this->ilist[this->n] = cp.ilist[i];
this->dlist[this->n] = cp.dlist[i];
this->n++;
}
}
il = (int *) realloc(this->ilist, (this->n)*sizeof(int));
dl = (double *) realloc(this->dlist, (this->n)*sizeof(double));
if (!il || !dl)
std::cout << "ERROR: 2nd realloc in operator += failed!" << std::endl;
else {
this->ilist = il;
this->dlist = dl;
}
return *this;
}
};
int main(int argc, char **argv) {
int npe, myid;
#if (CFG_MPI > 0)
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD,&npe);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
#else
npe=1;
myid=0;
#endif
if (!myid) // reduce output
std::cout << "NPE = " << npe << " MYID = " << myid << std::endl;
int ilist[5] = {14,17,4,29,0};
double dlist[5] = {0.0, 170.0, 0.0, 0.0, 24.523};
int ilist2[6] = {14,117,14,129,0, 34};
double dlist2[6] = {0.5, 170.5, 0.5, 0.5, 24.0, 1.2};
list tlist(5, ilist, dlist);
list tlist2(6, ilist2, dlist2);
if (!myid) {
tlist.print();
tlist2.print();
}
tlist +=tlist2;
if (myid) tlist.print();
#if (CFG_MPI > 0)
MPI_Finalize();
#endif
return 0;
}