C++ openmp much slower than serial implementation - c++

Question

I am doing a thermodynamic simulation on a double dimension array. The array is 1024x1024. The while loop iterates through a specified amount of times or until goodTempChange is false. goodTempChange is set true or false based on the change in temperature of a block being greater than a defined EPSILON value. If every block in the array is below that value, then the plate is in stasis. The program works, I have no problems with the code, my problem is that the serial code is absolutely blowing the openmp code out of the water. I don't know why. I have tried removing everything except the average calculation which is just the average of the 4 blocks up, down, left, right around your desired square and still it is getting destroyed by the serial code. I've never done openmp before and I looked up some things online to do what I have. I have the variables within critical regions in the most efficient way I could see possible, I have no race conditions. I really don't see what is wrong. Any help would be greatly appreciated. Thanks.
while(iterationCounter < DESIRED_ITERATIONS && goodTempChange) {
goodTempChange = false;
if((iterationCounter % 1000 == 0) && (iterationCounter != 0)) {
cout << "Iteration Count Highest Change Center Plate Temperature" << endl;
cout << "-----------------------------------------------------------" << endl;
cout << iterationCounter << " "
<< highestChange << " " << newTemperature[MID][MID] << endl;
cout << endl;
}
highestChange = 0;
if(iterationCounter != 0)
memcpy(oldTemperature, newTemperature, sizeof(oldTemperature));
for(int i = 1; i < MAX-1; i++) {
#pragma omp parallel for schedule(static)
for(int j = 1; j < MAX-1; j++) {
bool tempGoodChange = false;
double tempHighestChange = 0;
newTemperature[i][j] = (oldTemperature[i-1][j] + oldTemperature[i+1][j] +
oldTemperature[i][j-1] + oldTemperature[i][j+1]) / 4;
if((iterationCounter + 1) % 1000 == 0) {
if(abs(oldTemperature[i][j] - newTemperature[i][j]) > highestChange)
tempHighestChange = abs(oldTemperature[i][j] - newTemperature[i][j]);
if(tempHighestChange > highestChange) {
#pragma omp critical
{
if(tempHighestChange > highestChange)
highestChange = tempHighestChange;
}
}
}
if(abs(oldTemperature[i][j] - newTemperature[i][j]) > EPSILON
&& !tempGoodChange)
tempGoodChange = true;
if(tempGoodChange && !goodTempChange) {
#pragma omp critical
{
if(tempGoodChange && !goodTempChane)
goodTempChange = true;
}
}
}
}
iterationCounter++;
}

Answer 1

Trying to get rid of those critical sections may help. For example:
#pragma omp critical
{
if(tempHighestChange > highestChange)
{
highestChange = tempHighestChange;
}
}
Here, you can store the highestChange computed by each thread in a local variable and, when the parallel section finishes, get the maximum of the highestChange's you have.

Answer 2

Here is my attempt (not tested).
double**newTemperature;
double**oldTemperature;
while(iterationCounter < DESIRED_ITERATIONS && goodTempChange) {
if((iterationCounter % 1000 == 0) && (iterationCounter != 0))
std::cout
<< "Iteration Count Highest Change Center Plate Temperature\n"
<< "---------------------------------------------------------------\n"
<< iterationCounter << " "
<< highestChange << " "
<< newTemperature[MID][MID] << '\n' << std::endl;
goodTempChange = false;
highestChange = 0;
// swap pointers to arrays (but not the arrays themselves!)
std::swap(newTemperature,oldTemperature);
if(iterationCounter != 0)
std::swap(newTemperature,oldTemperature);
bool CheckTempChange = (iterationCounter + 1) % 1000 == 0;
#pragma omp parallel
{
bool localGoodChange = false;
double localHighestChange = 0;
#pragma omp for
for(int i = 1; i < MAX-1; i++) {
//
// note that putting a second
// #pragma omp for
// here has (usually) zero effect. this is called nested parallelism and
// usually not implemented, thus the new nested team of threads has only
// one thread.
//
for(int j = 1; j < MAX-1; j++) {
newTemperature[i][j] = 0.25 * // multiply is faster than divide
(oldTemperature[i-1][j] + oldTemperature[i+1][j] +
oldTemperature[i][j-1] + oldTemperature[i][j+1]);
if(CheckTempChange)
localHighestChange =
std::max(localHighestChange,
std::abs(oldTemperature[i][j] - newTemperature[i][j]));
localGoodChange = localGoodChange ||
std::abs(oldTemperature[i][j] - newTemperature[i][j]) > EPSILON;
// shouldn't this be < EPSILON? in the previous line?
}
}
//
// note that we have moved the critical sections out of the loops to
// avoid any potential issues with contentions (on the mutex used to
// implement the critical section). Also note that I named the sections,
// allowing simultaneous update of goodTempChange and highestChange
//
if(!goodTempChange && localGoodChange)
#pragma omp critical(TempChangeGood)
goodTempChange = true;
if(CheckTempChange && localHighestChange > highestChange)
#pragma omp critical(TempChangeHighest)
highestChange = std::max(highestChange,localHighestChange);
}
iterationCounter++;
}
There are several changes to your original:
The outer instead of the inner of the nested for loops is performed in parallel. This should make a significant difference.
added in edit: It appears from the comments that you don't understand the significance of this, so let me explain. In your original code, the outer loop (over i) was done only by the master thread. For every i, a team of threads was created to perform the inner loop over j in parallel. This creates an synchronisation overhead (with significant imbalance) at every i! If one instead parallelises the outer loop over i, this overhead is encountered only once and each thread will run the entire inner loop over j for its share of i. Thus, always to parallelise the outermost loop possible is a basic wisdom for multi-threaded coding.
The double for loop is inside a parallel region to minimise the critical region calls to one per thread per while loop. You may also consider to put the whole while loop inside a parallel region.
I also swap between two arrays (similar as suggested in other answers) to avoid to memcpy, but this shouldn't really be performance critical.
added in edit: std::swap(newTemperature,oldTemperature) only swaps the pointer values and not the memory pointed to, of course, that's the point.
Finally, don't forget that the proof of the pudding is in the eating: just try what difference it makes to have the #pragma omp for in front of the inner or the outer loop. Always do experiments like this before asking on SO -- otherwise you can be rightously accused of not having done sufficient research.

Answer 3

I assume that you are concerned with the time taken by the entire code inside the while loop, not just by the time taken by the loop beginning for(int i = 1; i < MAX-1; i++).
This operation
if(iterationCounter != 0)
{
memcpy(oldTemperature, newTemperature, sizeof(oldTemperature));
}
is unnecessary and, for large arrays, may be enough to kill performance. Instead of maintaining 2 arrays, old and new, maintain one 3D array with two planes. Create two integer variables, let's call them old and new, and set them to 0 and 1 initially. Replace
newTemperature[i][j] = ((oldTemperature[i-1][j] + oldTemperature[i+1][j] + oldTemperature[i][j-1] + oldTemperature[i][j+1]) / 4);
by
temperature[new][i][j] =
(temperature[old][i-1][j] +
temperature[old][i+1][j] +
temperature[old][i][j-1] +
temperature[old][i][j+1])/4;
and, at the end of the update swap the values of old and new so that the updates go the other way round. I'll leave it to you to determine whether old/new should be the first index into your array or the last. This approach eliminates the need to move (large amounts of) data around in memory.
Another possible cause of serious slowdown, or failure to accelerate, is covered in this SO question and answer. Whenever I see arrays with sizes of 2^n I suspect cache issues.