Parallelize inner loops - c++

Question

I'm pretty new to OpenMP, so I'm fine if I have this wrong. Also I wasn't successful in finding information about this but I'm sure I missed something obvious.
I have some nested loops, I would like to parallelize a certain way.
This is a sequential version. Notice f(i) is a larger integer between 100 and 100,000 roughly.
for (int a = 0; a < 10; a++)
{
for (int b = 0; b < 10; b++)
{
for (int c = 0; c < f(a); c++)
{
for (int d = 0; d < f(b); d++)
{
if (comp(c, d))
{
result[a][b]++;
}
}
}
}
}
Naively, I came up with this method of parallelizing the code.
#pragma omp parallel
{
// Create a result_local array to avoid critical sections in the loop
#pragma omp for collapse(2) schedule(guided) nowait
for (int a = 0; a < 10; a++)
{
for (int b = 0; b < 10; b++)
{
for (int c = 0; c < f(a); c++)
{
for (int d = 0; d < f(b); d++)
{
if (comp(c, d))
{
result_local[a][b]++;
}
}
}
}
}
// Add the result_local to result
}
This is the part where I'm not so sure. If my understanding is correct, OpenMP will not parallelize the c and d loops meaning each thread will execute a c loop in its entirety. Given f(i) can return relatively low numbers like 100 or relatively high numbers like 100,000, this means some of the threads might get stuck with a lot more work than other threads which is not ideal.
So then the question is how can I parallelize the inner loops to share the work better. I can't change collapse(2) to collapse(4) because the c and d loops iterate up to a number that is a function of the a and b variables.
I saw something in my research that maybe is helpful.
#pragma omp parallel
{
// Create a result_local array to avoid critical sections in the loop
for (int a = 0; a < 10; a++)
{
for (int b = 0; b < 10; b++)
{
#pragma omp parallel for collapse(2) schedule(guided)
for (int c = 0; c < f(a); c++)
{
for (int d = 0; d < f(b); d++)
{
if (comp(c, d))
{
result_local[a][b]++;
}
}
}
}
}
// Add the result_local to result
}
Admittedly, I don't know enough to know if this helpful at all. What I saw indicates this might be parallelizing the c and d loops but leaving the a and b loops serial?
Any help is appreciated.

Answer 1

omp will not parallelize the c and d loops meaning each thread will execute a c loop in its entirety.
This is correct.
some of the threads might get stuck with a lot more work than other threads
You are right: the work imbalance between thread is a performance issue in the first code. A schedule(dynamic) help a bit to fix this, but there is not much more you can do on this version.
I don't know enough to know if this helpful at all. What I saw indicates this might be parallelizing the c and d loops but leaving the a and b loops serial?
Technically, the a and b loops are executed in parallel too (since they are in a parallel section, but all the threads will completely execute all the iterations in lockstep (because the omp parallel for contains an implicit synchronization). You should not use a second omp parallel: regarding the runtime, this can created new threads 100 times, and even when no new threads are created, this result in an inefficient code (for example because of a bad default thread pinning). Moreover, schedule(guided) is not needed here and should be less efficient than a schedule(static). Thus, use omp for collapse(2) schedule(static).
how can I parallelize the inner loops to share the work better.
The last code is not soo bad in term of work balancing although it introduces some unwanted overheads:
The implicit synchronization of the omp for can be skipped using nowait since all threads are working on thread-private data.
The access to result_local[a][b] can be replaced by a fast thread-private variable access.
The conditional increment can be replaced by a branch-less boolean increment.
f(a) and f(b) can be per-computed although optimizing compilers should already do this.
When f(a) * f(b) is very small, this could be better not to execute the loop in parallel (because of the expensive cost to communicate between cores). However this is highly dependent of whether cond is expensive or not.
When f(a) is big, there is no need to use a costly collapse(2) as there will be enough work for all threads (collapse(2) usually slow down the execution since compilers often generate a slow modulus instruction to find the value of the loop iterators at runtime).
Here is the resulting code tacking into account most fixes:
#pragma omp parallel
{
// Create a result_local array to avoid critical sections in the loop
// Arbritrary threshold (this may not be optimal)
const int threshold = 4 * omp_get_num_threads();
for (int a = 0; a < 10; a++)
{
const int c_lim = f(a);
for (int b = 0; b < 10; b++)
{
const int d_lim = f(b);
int64_t local_sum = 0;
if(c_lim < threshold)
{
#pragma omp for collapse(2) schedule(static) nowait
for (int c = 0; c < c_lim; c++)
for (int d = 0; d < d_lim; d++)
local_sum += comp(c, d);
}
else
{
#pragma omp for schedule(static) nowait
for (int c = 0; c < c_lim; c++)
for (int d = 0; d < d_lim; d++)
local_sum += comp(c, d);
}
result_local[a][b] += local_sum;
}
}
// Add the result_local to result
}
Another more efficient strategy is to redesign the sequential algorithm to significantly reduce the amount of work.
Redesigning of the algorithm
One can note that comp(c, d) is recomputed with the same value several times (up to 100 times) and the same for result_local[a][b]++ or even f(b) (up to 1,000,000 times). In such cases, the generic solution is to memoize the results (see here for more information) to avoid expensive parts of the algorithm to be recomputed over and over.
Note that you cannot pre-compute all the needed comp(a, b) values: this solution would be too expensive in terms of memory usage (up to 10 Gio needed). Thus, the trick is to split the 2D space in tiles. Here is how the algorithm works:
compute all the f(a) and f(b) sequentially (100 values);
split the iteration space in tiles of reasonable size (eg. 100x100) and pre-compute all the required tiles that should be completely computed (possibly in parallel, although this is tedious);
compute the sum of all comp(a, b) for each tile (i.e. for a in [a_tile_begin;a_tile_end[ and b in [b_tile_begin;b_tile_end[) in parallel (each thread should work on several tiles) and write the sums in a shared array.
compute the final result using the tile sums (partial tiles are computed on the fly in this last step) in parallel.
This algorithm is definitively much more complex, but it should be up to 100 time faster than the above one since most operations are computed only once.

Answer 2

For your attempt of parallelizing the inner loops to have a chance to work, you need to do something about the data race to result_local:
If you have enough memory for every thread to have it's own private version of result_local, you might be able to specify reduction(+: result_local[:10][:10]) in the pragma, but I haven't used it with multidimensional arrays yet. You might have to use a linear array and "lexic indexing" (idx = a * 10 + b). If result_local is dynamically allocated (on the heap), this might be the better way of dealing with it anyway (better than some std::vector<std::vector<int>>, due to cache locality).
If comp is computationally intensive enough you might be better off by putting #pragma omp atomic update in front of result_local[a][b]++. This takes less memory. In your example with a * b == 100 memory is probably not an issue.
As branching inside the innermost loop can be bad for performance, you might want to try out if result_local[a][b] += comp(c, d); gives better performance, as addition is quite cheap.