triangular matrix conversion and auto parallelization - c++

Question

I'm playing a bit with auto parallelization in ICC (11.1; old, but can't do anything about it) and I'm wondering why the compiler can't parallelize the inner loop for a simple gaussian elimination:
void makeTriangular(float **matrix, float *vector, int n) {
for (int pivot = 0; pivot < n - 1; pivot++) {
// swap row so that the row with the largest value is
// at pivot position for numerical stability
int swapPos = findPivot(matrix, pivot, n);
std::swap(matrix[pivot], matrix[swapPos]);
std::swap(vector[pivot], vector[swapPos]);
float pivotVal = matrix[pivot][pivot];
for (int row = pivot + 1; row < n; row++) { // line 72; should be parallelized
float tmp = matrix[row][pivot] / pivotVal;
for (int col = pivot + 1; col < n; col++) { // line 74
matrix[row][col] -= matrix[pivot][col] * tmp;
}
vector[row] -= vector[pivot] * tmp;
}
}
}
We're only writing to the arrays dependent on the private row (and col) variable and row is guaranteed to be larger than pivot, so it should be obvious to the compiler that we aren't overwriting anything.
I'm compiling with -O3 -fno-alias -parallel -par-report3 and get lots of dependencies ala: assumed FLOW dependence between matrix line 75 and matrix line 73. or assumed ANTI dependence between matrix line 73 and matrix line 75. and the same for line 75 alone. What problem does the compiler have? Obviously I could tell it exactly what to do with some pragmas, but I want to understand what the compiler can get alone.

Answer 1

Basically the compiler can't figure out that there's no dependency due to the name matrix and the name vector being both read from and written too (even though with different regions). You might be able to get around this in the following fashion (though slightly dirty):
void makeTriangular(float **matrix, float *vector, int n)
{
for (int pivot = 0; pivot < n - 1; pivot++)
{
// swap row so that the row with the largest value is
// at pivot position for numerical stability
int swapPos = findPivot(matrix, pivot, n);
std::swap(matrix[pivot], matrix[swapPos]);
std::swap(vector[pivot], vector[swapPos]);
float pivotVal = matrix[pivot][pivot];
float **matrixForWriting = matrix; // COPY THE POINTER
float *vectorForWriting = vector; // COPY THE POINTER
// (then parallelize this next for loop as you were)
for (int row = pivot + 1; row < n; row++) {
float tmp = matrix[row][pivot] / pivotVal;
for (int col = pivot + 1; col < n; col++) {
// WRITE TO THE matrixForWriting VERSION
matrixForWriting[row][col] = matrix[row][col] - matrix[pivot][col] * tmp;
}
// WRITE TO THE vectorForWriting VERSION
vectorForWriting[row] = vector[row] - vector[pivot] * tmp;
}
}
}
Bottom line is just give the ones you're writing to a temporarily different name to trick the compiler. I know that it's a little dirty and I wouldn't recommend this kind of programming in general. But if you're sure that you have no data dependency, it's perfectly fine.
In fact, I'd put some comments around it that are very clear to future people who see this code that this was a workaround, and why you did it.
Edit: I think the answer was basically touched on by #FPK and an answer was posted by #Evgeny Kluev. However, in #Evgeny Kluev's answer he suggests making this an input parameter and that might parallelize but won't give the correct value since the entries in matrix won't be updated. I think the code I posted above will give the correct answer too.

Answer 2

The same auto-parallelization problem is on icc 12.1. So I used this newer version for experiments.
Adding an output matrix to your function's parameter list and changing body of the third loop to this
out[row][col] = matrix[row][col] - matrix[pivot][col] * tmp;
fixed the "FLOW dependence" problem. Which means, "-fno-alias" affects only function parameters, while contents of the single parameter remain under suspicion of being aliased. I don't know why this option does not affect everything. Since different parts of your matrix do not really alias each other, you can just leave this additional parameter to the function and pass the same matrix through this parameter.
Interestingly, while complaining about 'matrix', compiler say nothing about 'vector', which really has aliasing problems: this line vector[row] -= vector[pivot] * tmp; may lead to false aliasing (writing to vector[row] in one thread may touch the cache line, storing vector[pivot], used by every thread).
"FLOW dependence" is not the only problem in this code. After it was fixed, compiler still refuses to parallelize second and third loops because of "insufficient computational work". So I tried to give it some extra work:
float tmp = matrix[row][pivot] * pivotVal;
...
out[row][col] = matrix[row][col] - matrix[pivot][col] *tmp /pivotVal /pivotVal;
And after all this, the second loop was at last parallelized, though I'm not sure if it gained any speed improvement.
Update: I found a better alternative to giving computer "some extra work". Option -par-threshold50 does the trick.

Answer 3

I have no access to an icc to test my idea but I suspect the compiler fears aliasing: matrix is defined as float**: an array of pointers pointing to arrays of floats. All those pointers could point to the same float array so parallizing this would be very dangerous. This would make no sense, but the compiler cannot know.