PETSC DMDA vec values are assigned to wrang place - c++

Question

I recently started learning PETSc and encountered a problem while trying to accomplish some simple task. What is wrong with this code:
static char help[] = "Test 2d DMDAs Vecs.\n\n";
#include <petscdm.h>
#include <petscdmda.h>
#include <petscsys.h>
PetscReal process_value(int rank, int i) {
return i*PetscPowScalar(10,rank*2);
}
int main(int argc,char **argv) {
PetscErrorCode ierr;
PetscMPIInt rank;
PetscInt M = -5,N = -3;
DM da;
Vec local,global;
ierr = PetscInitialize(&argc,&argv,(char*)0,help);CHKERRQ(ierr);
ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
ierr = DMDACreate2d(PETSC_COMM_WORLD , DM_BOUNDARY_NONE , DM_BOUNDARY_NONE
, DMDA_STENCIL_BOX , M , N , PETSC_DECIDE, PETSC_DECIDE
, 1 , 1 , NULL , NULL , &da); CHKERRQ(ierr);
ierr = DMCreateGlobalVector(da,&global);CHKERRQ(ierr);
ierr = DMCreateLocalVector(da,&local);CHKERRQ(ierr);
{
PetscInt v,i, j, xm, ym, xs, ys;
PetscScalar **array;
ierr = DMDAGetCorners(da, &xs, &ys, 0, &xm, &ym, 0); CHKERRQ(ierr);
PetscSynchronizedPrintf(PETSC_COMM_WORLD,"%d:xs=%d\txm=%d\tys=%d\tym=%d\n",rank,xs,xm,ys,ym);
PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT);
ierr = DMDAVecGetArray(da, global, &array); CHKERRQ(ierr);
v=0;
for (j = ys; j < ys + ym; j++) {
for (i = xs; i < xs + xm; i++) {
array[j][i] = process_value(rank,v+=1);
}
}
ierr = DMDAVecRestoreArray(da, global, &array); CHKERRQ(ierr);
}
ierr = VecView(global,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);
ierr = VecDestroy(&local);CHKERRQ(ierr);
ierr = VecDestroy(&global);CHKERRQ(ierr);
ierr = DMDestroy(&da);CHKERRQ(ierr);
ierr = PetscFinalize();
return 0;
}
It fills small array with quantities labeled by process rank.
After successful compilation and linking it gives the following result:
> mpiexec -n 2 ./problem
0:xs=0 xm=3 ys=0 ym=3
1:xs=3 xm=2 ys=0 ym=3
Vec Object: 2 MPI processes
type: mpi
Vec Object:Vec_0x84000004_0 2 MPI processes
type: mpi
Process [0]
1.
2.
3.
100.
200.
4.
5.
6.
300.
Process [1]
400.
7.
8.
9.
500.
600.
>
VecView shows that processes have written to places that belong to other process. Where is an error? DMDAVecGetArray/DMDAVecRestoreArray give wrong array or VecView is not suitable to view Vec obtained from DM object ?

Answer 1

DMDAVecGetArray() and DMDAVecRestoreArray() work perfectly fine. Indeed, you are dealing with a feature of VecView() described in the mailing list of petsc a few years ago.
As is, VecView() prints the vector from the DMDA using the natural ordering.
proc0 proc1
1 2 | 3 4
5 6 | 7 8
___________
9 10 | 11 12
13 14 | 15 16
proc2 proc3
The difference between the natural ordering and Petsc's ordering is underlined in the documentation of Petsc, in section 2.5 about structured grids. Here is what Petsc's ordering looks like:
proc0 proc1
1 2 | 5 6
3 4 | 7 8
___________
9 10 | 13 14
11 12 | 15 16
proc2 proc3
As signaled in the thread VecView doesn't work properly with DA global vectors, it is still possible to print the vector using Petsc's ordering by using:
PetscViewerPushFormat(PETSC_VIEWER_STDOUT_WORLD,PETSC_VIEWER_NATIVE);
VecView(global,PETSC_VIEWER_STDOUT_WORLD);
PetscViewerPopFormat(PETSC_VIEWER_STDOUT_WORLD);
Let's take a closer look to the source of Petsc to see how it works. The VecView() operation for DMDA vector is overloaded as the DMCreateGlobalVector_DA() function is called (see dadist.c): the new method is VecView_MPI_DA() in gr2.c. Unsurprisingly, it calls a function DMDACreateNaturalVector() and later prints the natural vector using the native VecView(). If the format PETSC_VIEWER_NATIVE is used, the vector interface calls the operation *vec->ops->viewnative, which likely points to the native VecView() function VecView_MPI_ASCII() in pdvec.c. This explains the strange ( but very practical! ) behavior of VecView for DMDA vectors.
If you wish to keep the natural ordering, you can revome the meaningless Process[0]...Process[3] that get printed by using:
PetscViewerPushFormat(PETSC_VIEWER_STDOUT_WORLD, PETSC_VIEWER_ASCII_COMMON);