MPI_Scatter issue, can't scatter and gather a picture matrix - c++

Question

.Hello, i have a problem with my c++ code. I'm trying to make a parallel implementation from my sequential code for sobel operator using OpenCV.
My actual idea is scatter a picture using 2d buffer, make the sobel operation to averaged_rows*cols. and then make the gathering When i have sent the averaged_rows and every rank receive it, i try to use MPI_Scatter and this execution error appears:
sent to 1
sent to 2
sent to 3
recieved by 1
recieved by 2
recieved by 3
[roronoasins-GL552VW:3245] *** An error occurred in MPI_Scatter
[roronoasins-GL552VW:3245] *** reported by process [1759117313,1]
[roronoasins-GL552VW:3245] *** on communicator MPI_COMM_WORLD
[roronoasins-GL552VW:3245] *** MPI_ERR_TRUNCATE: message truncated
[roronoasins-GL552VW:3245] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[roronoasins-GL552VW:3245] *** and potentially your MPI job)
[roronoasins-GL552VW:03239] 2 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[roronoasins-GL552VW:03239] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
What I actually do is scatter pic buffer, bcast the picture to the rest of ranks and the gathering.
MPI_Scatter(pic, cols*rows_av, MPI_INT, picAux, cols*rows_av, MPI_INT, 0, MPI_COMM_WORLD);
MPI_Bcast (pic3, cols*rows, MPI_INT, 0, MPI_COMM_WORLD);
int ip_gx, ip_gy, sum;
for(int y = ip*pic_struct[2]; y < (ip+1)*pic_struct[2] -1; y++){
for(int x = 1; x < pic_struct[1]- 1; x++){
int gx = x_gradient(pic3, x, y);
int gy = y_gradient(pic3, x, y);
int sum = abs(gx) + abs(gy);
sum = sum > 255 ? 255:sum;
sum = sum < 0 ? 0 : sum;
picAux[y][x] = sum;
}
}
MPI_Gather(picAux, cols*rows_av, MPI_INT, pic, cols*rows_av, MPI_INT, 0, MPI_COMM_WORLD);
I'd like to know what is happening with the Scatter function, i thought that i could scatter single picture pieces to the rest ranks to calculate sobel, maybe i'm wrong.
My code is here if u want to check it. Thanks for your time.
// > compile with mpic++ mpi_sobel.cpp -o mpi_sobel `pkg-config --libs opencv` -fopenmp -lstdc++
#include <iostream>
#include <cmath>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <omp.h>
#include <mpi.h>
using namespace std;
using namespace cv;
Mat src, dst;
/*
Computes the x component of the gradient vector
at a given point in a image.
returns gradient in the x direction
| 1 0 -1 |
Gx = | 2 0 -2 |
| 1 0 -1 |
*/
int x_gradient(int** image, int x, int y)
{
return image[y-1][x-1] +
2*image[y][x-1] +
image[y+1][x-1] -
image[y-1][x+1] -
2*image[y][x+1] -
image[y+1][x+1];
}
/*
Computes the y component of the gradient vector
at a given point in a image
returns gradient in the y direction
| 1 2 1 |
Gy = | 0 0 0 |
|-1 -2 -1 |
*/
int y_gradient(int** image, int x, int y)
{
return image[y+1][x-1] +
2*image[y+1][x] +
image[y+1][x+1] -
image[y-1][x-1] -
2*image[y-1][x] -
image[y-1][x+1];
}
int main(int argc, char** argv)
{
string picture;
if (argc == 2) {
picture = argv[1];
src = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE);
}
else {
picture = "input/logan.jpg";
src = imread(picture.c_str(), CV_LOAD_IMAGE_GRAYSCALE);
}
if( !src.data )
{ return -1; }
dst.create(src.rows, src.cols, src.type());
int rows_av, rows_extra;
Size s = src.size();
int rows = s.height;
int cols = s.width;
int pic[rows][cols];int picAux[rows][cols];
int ** pic3;
pic3 = new int*[rows];
for(int y = 0; y < rows; y++)
pic3[y] = new int[cols];
int pic_struct[3], pic_struct_recv[3];
int np, ip;
double start_time = omp_get_wtime();
if (MPI_Init(&argc, &argv) != MPI_SUCCESS){
exit(1);
}
MPI_Comm_size(MPI_COMM_WORLD, &np);
MPI_Comm_rank(MPI_COMM_WORLD, &ip);
MPI_Status status;
if(ip==0)
{
for(int y = 0; y < rows ; y++)
for(int x = 0; x < cols; x++)
{
pic3[y][x] = src.at<uchar>(y,x);
pic[y][x] = 0;
picAux[y][x] = 0;
}
src.release();
rows_av = rows/np;
//cols_av = cols/np;
pic_struct[0] = rows;
pic_struct[1] = cols;
pic_struct[2] = rows_av;
//pic_struct[3] = cols_av:
for(int i=1; i < np; i++)
{
//rows = (i <= rows_extra) ? rows_av+1 : rows_av;
pic_struct[0] = rows;
MPI_Send(&pic_struct, sizeof(pic_struct), MPI_BYTE, i, 0, MPI_COMM_WORLD);
cout << "sent to " << i << endl;
}
}else{//ip
MPI_Recv(&pic_struct, sizeof(pic_struct), MPI_BYTE, 0, 0, MPI_COMM_WORLD, &status);
cout << "recieved by " << ip << endl;
}
MPI_Scatter(pic, cols*rows_av, MPI_INT, picAux, cols*rows_av, MPI_INT, 0, MPI_COMM_WORLD);
MPI_Bcast (pic3, cols*rows, MPI_INT, 0, MPI_COMM_WORLD);
cout << "bcast" << endl;
//MPI_Barrier(MPI_COMM_WORLD);
int ip_gx, ip_gy, sum;
for(int y = ip*pic_struct[2]; y < (ip+1)*pic_struct[2] -1; y++){
for(int x = 1; x < pic_struct[1]- 1; x++){
ip_gx = x_gradient(src, x, y);
ip_gy = y_gradient(src, x, y);
sum = abs(ip_gx) + abs(ip_gy);
sum = sum > 255 ? 255:sum;
sum = sum < 0 ? 0 : sum;
picAux[y][x] = sum;
}
}
MPI_Gather(picAux, cols*rows_av, MPI_INT, pic, cols*rows_av, MPI_INT, 0, MPI_COMM_WORLD);
cout << "gather" << endl;
MPI_Finalize();
if(!ip)
{
double time = omp_get_wtime() - start_time;
for( int i = 0 ; i < rows ; i++ )
{
delete [] pic3[i] ;
delete [] pic3 ;
}
cout << "Number of processes: " << np << endl;
cout << "Rows, Cols: " << rows << " " << cols << endl;
cout << "Rows, Cols(Division): " << rows_av << ", " << cols << endl << endl;
cout << "Processing time: " << time << endl;
for(int i=0; i < 6 ; i++) picture.erase(picture.begin());
for(int i=0; i < 4 ; i++) picture.pop_back();
picture.insert(0,"output/");
picture += "-sobel.jpg";
for(int y = 0; y < rows; y++)
for(int x = 0; x < cols; x++)
dst.at<uchar>(y,x) = pic[y][x];
if(imwrite(picture.c_str(), dst)) cout << "Picture correctly saved as " << picture << endl;
else cout << "\nError has occurred being saved." << endl;
}
return 0;
}
Update: I forgot rows_av in ranks != 0 and pic3 sending is fixed. I've packed src in contiguous buffer and it is right in each rank.
updated code here: https://pastebin.com/jPV9mGFW
I have noticed that into the 3/4 dark there is noise, with this new issue i dont know if gathering is the problem now or i am doing the operations with number_process*rows/total_processes wrong.
MPI_Scatter(pic, cols*rows_av, MPI_UNSIGNED_CHAR, picAux, cols*rows_av, MPI_UNSIGNED_CHAR, 0, MPI_COMM_WORLD);
int ip_gx, ip_gy, sum;
for(int y = ip*rows_av+1; y < (ip+1)*rows_av-1; y++){
for(int x = 1; x < cols ; x++){
ip_gx = x_gradient(src, x, y);
ip_gy = y_gradient(src, x, y);
sum = abs(ip_gx) + abs(ip_gy);
sum = sum > 255 ? 255:sum;
sum = sum < 0 ? 0 : sum;
picAux[y][x] = sum;
//picAux[y*rows_av+x] = sum;
}
}
MPI_Gather(picAux, cols*rows_av, MPI_UNSIGNED_CHAR, pic, cols*rows_av, MPI_UNSIGNED_CHAR, 0, MPI_COMM_WORLD);
Loop updated and image is full calculated now but i cant use images bigger than 2048x1536.
for(int y = 1; y < rows_av-1; y++){
for(int x = 1; x < cols ; x++){
ip_gx = x_gradient(src, x, ip*rows_av+y);
ip_gy = y_gradient(src, x, ip*rows_av+y);
sum = abs(ip_gx) + abs(ip_gy);
sum = sum > 255 ? 255:sum;
sum = sum < 0 ? 0 : sum;
picAux[y*cols+x] = sum;
}
}
How could i send larger images than 2048x1536?
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 0 on node roronoasins-GL552VW exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Images size issue was the stack limited size. with ulimit -s unlimited works fine but im now working to improve memory efficiency. Last code will be updated in the pastebin code above.