Memory with Vectors and Pointers in MPI+OpenMP App? - c++

Question

I have been reading about vectors with respect to memory allocation and trying to work through some hybrid parallelized code that seems to be heavily chewing through memory unexpectedly. The program used to use only OpenMP so was limited in processing the data on a per node basis, but code was recently added to utilize MPI as well. The data itself is of a fixed size for the current problem (15GB) and is distributed equally amongst all of the MPI processes.
The program overall is using vectors almost everywhere, but prior to MPI code, was able to process much much larger amounts of data without running out of mem.
The current data it is trying to process is only 15gb in size, but any attempts to run are wrought with bad_alloc errors -- even when requesting nodes with >64GB mem.
I am shortening down the code to explain the critical points and adding some inline comments. It is mixed MPI+openmp.
Here are my questions:
If I am running out of memory on a per node basis, does that definitively mean that its the OpenMP threads that are causing the problem?
Could it be that increasing these vectors on-the-fly is causing the memory issue? It seems like the vectors are only used for storing data and then for the allocation of other memory -- could I get away with using a std::list instead?
The vectors are never used after this step, but that mem remains allocated. I could certainly free the memory associated with them, but these vectors were not previously a problem when they ran only on a single node over a series of ~50 hours on the full set of data (not a smaller chunk). Is there anything about the MPI code below that could be causing it to ask for too much memory in some way? Previously, the mallocs for the pointers were not in the code.
svec<int> ind_leftFW;
svec<int> ind_rightFW;
svec<int> ind_leftRC;
svec<int> ind_rightRC;
svec<int> ind_toasted;
// For additional examples, I will be using only leftFront and rightFront but all of these have similar behavior
/* Searching through previously created [outside of function] vectors for things + some math */
#pragma omp critical
{
// first use of declared vectors
ind_leftFront.push_back(i);
ind_rightFront.push_back(c);
}
/* Other push_backs into the other vector structures */
int* all_indexFront_sizes;
int* all_IndexFront_left;
int* all_IndexFront_right;
int* leftFront;
int* rightFront;
int numFront;
numFront = (int)ind_leftFront.size();
leftFront = (int*)malloc(sizeof(int)*numFront);
rightFront = (int*)malloc(sizeof(int)*numFront);
for(int count=0; count<numFront; count++) {
leftFront[count] = ind_leftFront[count];
rightFront[count] = ind_rightFront[count];
}
all_indexFront_sizes = (int*)malloc(sizeof(int)*numranks);
MPI_Allgather(&numFront,1,MPI_INT,all_indexFront_sizes,1,MPI_INT,MPI_COMM_WORLD);
int* Front_displs;
Front_displs = (int*)malloc(sizeof(int)*numranks);
Front_displs[0] = 0;
for(int count=1;count<numranks;count++)
Front_displs[count] = Front_displs[count-1] + all_indexFront_sizes[count-1];
all_IndexFront_left = (int*)malloc(sizeof(int)*total_indexFront);
all_IndexFront_right = (int*)malloc(sizeof(int)*total_indexFront);
MPI_Allgatherv(leftFront, numFront,MPI_INT,all_IndexFront_left ,all_indexFront_sizes, FW_displs, MPI_INT,MPI_COMM_WORLD);
MPI_Allgatherv(rightFront,numFront,MPI_INT,all_IndexFront_right,all_indexFront_sizes, FW_displs, MPI_INT,MPI_COMM_WORLD);
free(Front_displs);
free(all_indexFront_sizes);
free(all_IndexFront_left);
free(all_IndexFront_right);
free(leftFront);
free(rightFront);