fftw_mpi_plan_dft_3d fails with Segmentation fault - fortran

Question

The following 3D complex FFT Fortran MPI program runs when the number of processes P = 1, 2, 4, 8, and 16. But, it fails when P = 12 with the following error at the place where function fftw_mpi_plan_dft_3d is called.
"Program received signal SIGSEGV: Segmentation fault - invalid memory reference."
Why does this program fail with P = 12?
PROGRAM test
USE, INTRINSIC :: ISO_C_BINDING
IMPLICIT NONE
INCLUDE 'mpif.h'
INCLUDE 'fftw3-mpi.f03'
INTEGER :: ier
integer(C_INTPTR_T), parameter :: L = 256
integer(C_INTPTR_T), parameter :: N = 256
integer(C_INTPTR_T), parameter :: M = 48
type(C_PTR) :: plan, cdata
complex(C_DOUBLE_COMPLEX), pointer :: data(:,:,:)
integer(C_INTPTR_T) :: i, j, alloc_local, local_M, local_j_offset
CALL MPI_INIT(ier)
CALL fftw_mpi_init
alloc_local = fftw_mpi_local_size_3d(M, N, L, MPI_COMM_WORLD, local_M, local_j_offset)
cdata = fftw_alloc_complex(alloc_local)
CALL C_F_POINTER(cdata, data, [L, N, local_M])
plan = fftw_mpi_plan_dft_3d(M, N, L, data, data, MPI_COMM_WORLD, FFTW_FORWARD, FFTW_ESTIMATE)
CALL fftw_destroy_plan(plan)
CALL fftw_free(cdata)
CALL MPI_FINALIZE(ier)
STOP
END PROGRAM test

Answer 1

I see the same problem with fftw_mpi_plan_dft_c2r on a Cray XE6 using cray-mpich 7.1.3. It doesn't matter if fftw is compiled with gcc, Cray cc, or Intel icc. The segmentation fault only occurs for certain combinations of array dimensions and number of processors. One combination that results in a segfault is array size 1920,3840,3840 on 128 processes. As with the original post, I used FFTW_ESTIMATE and so no transform is done by the planner. Stack trace follows.
#0 0x00000000008f26f0 in ?? ()
#1 0x00000000004179bb in fftw_plan_destroy_internal (ego=0x8e2af0) at plan.c:49
#2 0x000000000040d755 in mkplan (ego_=0x8def60, p_=0x8f1440, plnr=0x8d6110)
at transpose-pairwise.c:467
#3 0x0000000000418a07 in invoke_solver (ego=0x8d6110, p=0x8f1440, s=0x8def60,
nflags=0x7fffffff4fc0) at planner.c:486
#4 0x0000000000418b9c in search0 (ego=0x8d6110, p=0x8f1440, slvndx=0x7fffffff4fcc,
flagsp=0x7fffffff4fc0) at planner.c:529
#5 0x0000000000418db8 in search (ego=0x8d6110, p=0x8f1440, slvndx=0x7fffffff4fcc,
flagsp=0x7fffffff4fc0) at planner.c:600
#6 0x00000000004191d8 in mkplan (ego=0x8d6110, p=0x8f1440) at planner.c:711
#7 0x0000000000419d35 in fftw_mkplan_d (ego=0x8d6110, p=0x8f1440) at planner.c:970
#8 0x0000000000411d47 in mkplan (ego_=0x8df140, p_=0x8df930, plnr=0x8d6110)
at dft-rank1-bigvec.c:165
#9 0x0000000000418a07 in invoke_solver (ego=0x8d6110, p=0x8df930, s=0x8df140,
nflags=0x7fffffff5290) at planner.c:486
#10 0x0000000000418b9c in search0 (ego=0x8d6110, p=0x8df930, slvndx=0x7fffffff529c,
flagsp=0x7fffffff5290) at planner.c:529
#11 0x0000000000418db8 in search (ego=0x8d6110, p=0x8df930, slvndx=0x7fffffff529c,
flagsp=0x7fffffff5290) at planner.c:600
#12 0x00000000004191d8 in mkplan (ego=0x8d6110, p=0x8df930) at planner.c:711
#13 0x0000000000419d35 in fftw_mkplan_d (ego=0x8d6110, p=0x8df930) at planner.c:970
#14 0x00000000004157b1 in mkplan (ego_=0x8df4a0, p_=0x8df650, plnr=0x8d6110)
at rdft2-rank-geq2.c:179
#15 0x0000000000418a07 in invoke_solver (ego=0x8d6110, p=0x8df650, s=0x8df4a0,
nflags=0x7fffffff5550) at planner.c:486
#16 0x0000000000418b9c in search0 (ego=0x8d6110, p=0x8df650, slvndx=0x7fffffff555c,
flagsp=0x7fffffff5550) at planner.c:529
#17 0x0000000000418db8 in search (ego=0x8d6110, p=0x8df650, slvndx=0x7fffffff555c,
flagsp=0x7fffffff5550) at planner.c:600
#18 0x00000000004191d8 in mkplan (ego=0x8d6110, p=0x8df650) at placner.c:711
#19 0x000000000041d1e2 in mkplan0 (plnr=0x8d6110, flags=64, prb=0x8df650, hash_info=0,
wisdom_state=WISDOM_NORMAL) at apiplan.c:34
#20 0x000000000041d229 in mkplan (plnr=0x8d6110, flags=64, prb=0x8df650, hash_info=0)
at apiplan.c:48
#21 0x000000000041d438 in fftw_mkapiplan (sign=0, flags=0, prb=0x8df650)
at apiplan.c:111
#22 0x000000000040a385 in plan_guru_rdft2 (rnk=3, dims0=0x8df540, howmany=1,
r=0x2aab74c21010, c=0x2aab73000010, comm=1140850688, kind=HC2R00, flags=64)
at api.c:784
#23 0x000000000040a542 in fftw_mpi_plan_many_dft_c2r (rnk=3, n=0x7fffffff5830,
howmany=1, iblock=0, oblock=0, in=0x2aab73000010, out=0x2aab74c21010,
comm=1140850688, flags=64) at api.c:831