Seg fault after MPI_Finalize() while using OpenMPI in C++ - c++

Question

I am trying to write an MPI code to process a large 2D matrix. I'm basically dividing the matrix into chunks and giving those chunks to individual processes. I see that the processes complete their task, send the processed array back to the Master. However, after calling MPI_Finalize(), a random process gives me a seg fault. I have checked and debugged the addressing (like whether a process is accessing some invalid memory) but haven't found any issues in that. I have attached my code below :-
#include "mpi/mpi.h"
#include <iostream>
using namespace std;
#define n 6
#define iter 1
#define MASTER 0
#define FROM_MASTER 1
#define FROM_SLAVE 2
int main(int argc, char* argv[]) {
int num_tasks, num_workers, task;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &num_tasks);
MPI_Comm_rank(MPI_COMM_WORLD, &task);
num_workers = num_tasks - 1;
MPI_Status status;
double *mat;
int rows_per_task = (n - 2) / num_workers;
int index = 0;
int size;
if(task == MASTER) {
// Allocate n*n matrix
mat = (double *)malloc(sizeof(double) * n * n);
// Initialize the matrix
for(int i = 1; i <= num_workers; ++i) {
// Accommodate for extra rows per task
size = (i <= (n % num_workers)) ? rows_per_task + 1 : rows_per_task;
// Send rows to the Slave processes
MPI_Send(&index, 1, MPI_INT, i, FROM_MASTER, MPI_COMM_WORLD); // Start index
MPI_Send(&size, 1, MPI_INT, i, FROM_MASTER, MPI_COMM_WORLD); // Size of the array chunk
MPI_Send(&mat[index], n * size, MPI_DOUBLE, i, FROM_MASTER, MPI_COMM_WORLD); // Array
index += n * size;
}
for(int i = 1; i <= num_workers; ++i) {
// Get array size, start index, actual array from all the processes
MPI_Recv(&size, 1, MPI_INT, i, FROM_SLAVE, MPI_COMM_WORLD, &status);
MPI_Recv(&index, 1, MPI_INT, i, FROM_SLAVE, MPI_COMM_WORLD, &status);
MPI_Recv(&mat[index], n * size, MPI_DOUBLE, i, FROM_SLAVE, MPI_COMM_WORLD, &status);
}
printf("MASTER DONE!\n");
}
if(task > 0) {
// Get index, size, rows
MPI_Recv(&index, 1, MPI_INT, MASTER, FROM_MASTER, MPI_COMM_WORLD, &status);
MPI_Recv(&size, 1, MPI_INT, MASTER, FROM_MASTER, MPI_COMM_WORLD, &status);
MPI_Recv(&mat[index], n * size, MPI_DOUBLE, MASTER, FROM_MASTER, MPI_COMM_WORLD, &status);
// Repeat
for(int it = 0; it < iter; ++it) {
for(int i = 0; i < size; ++i) {
for(int j = 0; j < n; ++j) {
int idx = index + n * i + j; // 2D -> 1D index transformation
// Do something with the array element mat[idx]
}
}
}
MPI_Send(&size, 1, MPI_INT, MASTER, FROM_SLAVE, MPI_COMM_WORLD);
MPI_Send(&index, 1, MPI_INT, MASTER, FROM_SLAVE, MPI_COMM_WORLD);
MPI_Send(&mat[index], n * size, MPI_DOUBLE, MASTER, FROM_SLAVE, MPI_COMM_WORLD);
printf("SLAVE %d DONE\n", task);
}
printf("TASK %d calling finalize\n", task);
MPI_Finalize();
printf("TASK %d called finalize\n", task);
return 0;
}
In this example, n is set to 6. If my number of processes = 3 (divisible by 6), equal sharing of rows happens and the program works. If I change n to 7, then I start getting an error :-
SLAVE 2 DONE
TASK 2 calling finalize
SLAVE 1 DONE
TASK 1 calling finalize
SLAVE 3 DONE
TASK 3 calling finalize
MASTER DONE!
TASK 0 calling finalize
[<node_name>:70041] *** Process received signal ***
[<node_name>:70041] Signal: Segmentation fault (11)
[<node_name>:70041] Signal code: (128)
[<node_name>:70041] Failing at address: (nil)
[<node_name>:70041] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f24107b6090]
[<node_name>:70041] [ 1] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x20)[0x7f241080d6f0]
[<node_name>:70041] [ 2] /lib/x86_64-linux-gnu/libopen-pal.so.40(+0x409e2)[0x7f24106269e2]
[<node_name>:70041] [ 3] /lib/x86_64-linux-gnu/libmpi.so.40(ompi_datatype_finalize+0x79)[0x7f2410bc2eb9]
[<node_name>:70041] [ 4] /lib/x86_64-linux-gnu/libmpi.so.40(ompi_mpi_finalize+0x773)[0x7f2410bb12e3]
[<node_name>:70041] [ 5] ./a.out(+0xd2cd)[0x5642eb9e52cd]
[<node_name>:70041] [ 6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f2410797083]
[<node_name>:70041] [ 7] ./a.out(+0xc5ce)[0x5642eb9e45ce]
[<node_name>:70041] *** End of error message ***
TASK 1 called finalize
TASK 0 called finalize
TASK 3 called finalize
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node <node_name> exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
From the output, I see that the processes finish their tasks and every task (including master) calls finalize. Then, a seg fault happens for task 2 while the other 3 tasks finish. Although this is happening when I change the value of n such that n % number_of_slave_tasks != 0, I am positive nothing is wrong with that particular logic and I am accessing valid memory locations.
Help me out with this please!