solving 2d heat equation using mpi - c++

Question

Basically, I am trying to solve the 2d heat equation using OpenMPI.
We start off with a matrix(which represents the heat distribution of a 2-dimensional object) with the heat at the bottom(i.e, all the bottom values are 100), time passes and all the heat flows from the hotter surface to the colder surface and we are simulating this through each iteration.
We start off
int max_iterations = 15000;
double epsilon = 1.0e-3;
int N = 20;
int M = 20;
// Initialize MPI
MPI_Init(&argc, &argv);
int pid, np;
MPI_Comm_rank(MPI_COMM_WORLD, &pid);
MPI_Comm_size(MPI_COMM_WORLD, &np);
// Calculate the number of rows per process
int rows_per_process = M / np;
rows_per_process += 2; // boundary rows
then we declare and initialize the local matrix by dividing the main matrix horizontally in to np parts where np is number of processes
Matrix U(rows_per_process, N);
Matrix W(rows_per_process, N);
// Init & Boundary
for (i = 0; i < rows_per_process; ++i) {
for (j = 0; j < N; ++j) {
U(i, j) = 0.0;
}
}
for (j = 0; j < N; ++j) {
U(rows_per_process - 1, j) = 100.0;
}
after that, we distribute the boundaries of local matrices to the other processes (this is the part, I think doesn't work as expected)
// Check if this is not process 0
if (pid != 0) {
MPI_Send(&U(1, 0), N, MPI_DOUBLE, pid - 1, 0, MPI_COMM_WORLD);
MPI_Recv(&U(0, 0), N, MPI_DOUBLE, pid - 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
// Check if this is not the last process
if (pid != np - 1) {
MPI_Send(&U(rows_per_process-2, 0), N, MPI_DOUBLE, pid + 1, 0, MPI_COMM_WORLD);
MPI_Recv(&U(rows_per_process-1, 0), N, MPI_DOUBLE, pid + 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
after this main loop begins
iteration_count = 0;
do
{
iteration_count++;
diffnorm = 0.0;
for (i = 1; i < rows_per_process - 1; ++i)
{
for (j = 1; j < N - 1; ++j)
{
W(i, j) = (U(i, j + 1) + U(i, j - 1) + U(i + 1, j) + U(i - 1, j)) * 0.25;
diffnorm += (W(i, j) - U(i, j)) * (W(i, j) - U(i, j));
}
}
for (i = 1; i < rows_per_process - 1; ++i)
for (j = 1; j < N - 1; ++j)
U(i, j) = W(i, j);
// Check if this is not process 0
if (pid != 0) {
MPI_Send(&U(1, 0), N, MPI_DOUBLE, pid - 1, 0, MPI_COMM_WORLD);
MPI_Recv(&U(0, 0), N, MPI_DOUBLE, pid - 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
// Check if this is not the last process
if (pid != np - 1) {
MPI_Send(&U(rows_per_process-2, 0), N, MPI_DOUBLE, pid + 1, 0, MPI_COMM_WORLD);
MPI_Recv(&U(rows_per_process-1, 0), N, MPI_DOUBLE, pid + 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
diffnorm = sqrt(diffnorm);
MPI_Allreduce(&diffnorm, &diffnorm, 1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD);
} while (epsilon <= diffnorm && iteration_count < max_iterations);
after this we gather the processed data from the children (this part is also in doubt)
Matrix U_MPI(M, N);
int nElm = N*(rows_per_process -2), nth = (pid*(rows_per_process - 2));
MPI_Gather(&U(1, 0), nElm, MPI_DOUBLE,
&U_MPI(nth, 0), nElm, MPI_DOUBLE, 0, MPI_COMM_WORLD);
OUTPUT
0 0.93421 1.67984 2.09131 2.09131 1.67984 0.93421 0
0 2.05719 3.69419 4.59451 4.59451 3.69419 2.05719 0
0 3.60071 6.44586 7.99882 7.99882 6.44586 3.60071 0
0 5.90026 10.4906 12.9572 12.9572 10.4906 5.90026 0
0 9.51028 16.6601 20.3833 20.3833 16.6601 9.51028 0
0 15.4813 26.2571 31.5339 31.5339 26.2571 15.4813 0
0 26.1585 41.3538 47.9623 47.9623 41.3538 26.1585 0
Expected output
0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
0.000000 2.285033 4.078976 5.051108 5.051108 4.078976 2.285033 0.000000
0.000000 5.061334 8.980086 11.074751 11.074751 8.980086 5.061334 0.000000
0.000000 8.980539 15.705866 19.193785 19.193785 15.705866 8.980539 0.000000
0.000000 15.155358 25.669779 30.801644 30.801644 25.669779 15.155358 0.000000
0.000000 25.971518 41.016975 47.542273 47.542273 41.016975 25.971518 0.000000
0.000000 47.714062 64.884910 70.808927 70.808927 64.884910 47.714062 0.000000
100.000000 100.000000 100.000000 100.000000 100.000000 100.000000 100.000000 100.000000
Matrix
struct Mat {
int height;
int width;
std::vector<double> data;
Mat();
Mat(int height, int width)
: height(height), width(width), data(height*width)
{ }
double& operator()(int h, int w) {
int id = h*width + w;
if(id >= (height * width)) {
throw std::runtime_error("Invalid acess");
}
return data[h*width + w];
}
};

Answer 1

This code:
for (j = 0; j < N; ++j) {
U(rows_per_process - 1, j) = 100.0;
}
Appears to be initializing the boundary condition into the lowest ghost-zone row on each process, not the lowest row of the domain. Is that what you intended?
Also here:
int nElm = N*(rows_per_process -2), nth = (pid*(rows_per_process - 2));
MPI_Gather(&U(1, 0), nElm, MPI_DOUBLE,
&U_MPI(nth, 0), nElm, MPI_DOUBLE, 0, MPI_COMM_WORLD);
The &U_MPI(nth, 0) expression looks like an out-of-bounds access.
There may be other problems...