openMP Alternating direction implicit method - c++

Question

Hi I am trying to parralelize the calculation of this code with openMP. It is calculation of hydrodynamics vorticity with finite difference implicite method. I am using the Alternating direction implicit method to do so.
I would like to speed up its execution. (Here Nx=Ny=100)
The probleme is that using openMp this way slow down the code instead of speeding it up. I have try to specify the shared variable but this not helping much.
Any idea?
All the best
void ADI(double vort[][Ny], double psi[][Ny], double n[][Ny],
double cls[][Ny],double AAx[], double BBx[], double CCx[], double DDx[],
double AAy[], double BBy[], double CCy[], double DDy[],
double cx[][Ny], double cy[][Ny], double epsx[][Ny], double epsy[][Ny],
double vortx[], double vorty[Ny-2], double dx, double Dxs, double coefMass,
double coefMasCls)
{
////////////calcul sur y////////////
//calcul coef ADI
int i=0, j=0;
#pragma omp parallel for private(Dxs,i) shared(psi,vort)
for (i=0; i<Nx; i++) //Boundary condition sur x
{
vort[i][0]=(psi[i][0]-psi[i][1])*2/Dxs;
vort[i][Ny-1] = (psi[i][Ny-1]-psi[i][Ny-2])*2/Dxs;
}
#pragma omp parallel for private(Dxs,j) shared(psi,vort)
for (j=0; j<Ny; j++) //Boundary condition
{
vort[0][j] = (psi[0][j]-psi[1][j])*2/Dxs;
vort[Nx-1][j] = (psi[Nx-1][j]-psi[Nx-2][j])*2/Dxs;
}
for (j=1; j<Ny-1; j++) //interior points
{
#pragma omp parallel for private(coefMasCls,coefMasCls,i) shared(psi,vort,n,cls)
for (i=1; i<Nx-1; i++) //interior points
{
vort[i][j] = vort[i][j] - coefMass * (n[i+1][j]-n[i-1][j])- coefMasCls * (cls[i+1][j]-cls[i-1][j]);;
}
//i=0;
//vort[i][j] = vort[i][j] + coefMass*(n[1][j]-n[1][j]);
//i=Nx-1;
//vort[i][j] = vort[i][j] + coefMass*(n[Nx-2][j]-n[Nx-2][j]);
}
for (i=1; i<Nx-1; i++) //interior points
{
for (j=1; j<Ny-1; j++) //interior points
{
AAy[j] = -.5 * ( .5 * (1 + epsy[i][j]) * cy[i][j-1] + dx);
BBy[j] = 1 + dx + .5 * epsy[i][j] * cy[i][j];
CCy[j] = .5 * ( .5 * ( 1 - epsy[i][j] ) * cy[i][j+1] - dx);
DDy[j] = .5 * (.5 * ( 1 + epsx[i][j] ) * cx[i-1][j] + dx ) * vort[i-1][j]
+ ( 1 - dx - .5 * epsx[i][j] * cx[i][j] ) * vort[i][j]
+ .5 * (- .5 * ( 1 - epsx[i][j] ) * cx[i+1][j] + dx ) * vort[i+1][j];
vorty[j] = vort[i][j];
}
DDy[1]=DDy[1] - AAy[1] * vort[i][0]; //the AA[0] are not taken into account in the tridiag methode. Include it in the second hand
DDy[Ny-2]=DDy[Ny-2] - CCy[Ny-2]* vort[i][Ny-1]; //moving boundary condition
//DDy[Ny-3]= DDy[Ny-3]; //vorticity nul on the free slip boundary condition
tridiag(AAy, BBy, CCy, DDy, vorty, Ny-1); //ne calcul pas le point en 0 et en Ny-1
for (j=1; j<Ny-1; j++)
{
vort[i][j]=vorty[j];
}
}
////////////calcul sur x //////////
//calcul coef ADI
for (j=1; j<Ny-1; j++)
{
for (i=1; i<Nx-1; i++)
{
AAx[i] = -.5* ( .5 * ( 1 + epsx[i][j] ) * cx[i-1][j] + dx );
BBx[i] = 1 + dx + .5 * epsx[i][j] * cx[i][j];
CCx[i] = .5 * ( .5 * ( 1 - epsx[i][j] ) * cx[i+1][j] - dx) ;
DDx[i]= .5 * ( .5 * ( 1 + epsy[i][j] ) * cy[i][j-1] + dx ) * vort[i][j-1]
+ ( 1 - dx - .5 * epsy[i][j] * cy[i][j] ) * vort[i][j]
+ .5 * (-.5 * ( 1 - epsy[i][j] ) * cy[i][j+1] + dx ) * vort[i][j+1];
vortx[i]=vort[i][j];
}
DDx[1] = DDx[1] - AAx[1]* vort[0][j];
DDx[Nx-2] = DDx[Nx-2] - CCx[Nx-2] * vort[Nx-1][j];
tridiag(AAx, BBx, CCx, DDx, vortx, Nx-1); //ne calcul pas le point en 0 et en Nx-1
for (i=1; i<Nx-1; i++)
{
vort[i][j]=vortx[i];
}
}
}

Answer 1

The first thing to do is indeed to isolate which loop parallelizations have the most bad impact, but the last loop there looks very much like you would be experiencing cache thrashing. Simplifying the structure a bit:
double vort[Nx][Ny];
// ...
for (int j=1; j<Ny-1; ++j) {
#pragma omp parallel for
for (int i=1; i<Nx-1; ++i) {
vort[i][j] -= f(i, j);
}
}
Any given thread is going to read and update in turn the values in vort at offsets j+k*Ny, j+(k+1)*Ny, j+(k+2)*Ny etc. depending on how the for loop is chunked across the threads. Each of these accesses is going to pull in a cache-line's worth of data to update 8 bytes. And when the outer loop starts again, chances are none of the data you just accessed is still going to be in cache.
All things being equal, if you can arrange your array accesses so that you're moving in the direction of the smallest stride (for C arrays, that's the last index), your cache behaviour will be much better. For dimension size 100, the arrays are likely not so big that this makes a huge difference. For e.g. Nx, Ny = 1000, it will likely be devastating to access the array the 'wrong way'.
This would give poorer performance in serial code, but I think adding threads to it just makes it that much worse.
That all said, the amount of computation done in each of these inner loops is quite small; there's a good chance you're going to be constrained by memory bandwidth regardless.
Addendum
Just to be explicit, the 'right' loop access would look like:
for (int i=1; i<Nx-1; ++i) {
for (int j=1; j<Ny-1; ++j) {
vort[i][j] -= f(i, j);
}
}
And to parallelize it, you can allow the compiler to better chunk the data across threads by using the collapse directive:
#pragma omp parallel for collapse(2)
for (int i=1; i<Nx-1; ++i) {
for (int j=1; j<Ny-1; ++j) {
vort[i][j] -= f(i, j);
}
}
Lastly, in order to avoid false sharing (threads treading on each other's cache lines), it's good to make sure that two adjacent rows of the array don't share data in the same cache line. One could make sure that each row is aligned on to a multiple of the cache-line size in memory, or more simply just add padding to the end of each row.
double vort[Nx][Ny+8]; // 8 doubles ~ 64 bytes
(Assuming a cache-line of 64 bytes, this should suffice.)