Error writing to files PBS MPI - c++

Question

I have a big trouble while writing some data to files using MPI on a cluster with PBS. Here is the example of simple problem-emulating programm.
#include <mpi.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <iostream>
#include <fstream>
#include <string>
#include <sstream>
#include <cstdlib>
#include <unistd.h>
int main(int argc, char* argv[]){
int rank;
int size;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
// Define hostname
char hostname[128];
gethostname(hostname, 128);
// check and create dump directory
struct stat buf;
int rc;
char *dir="Res";
rc = stat( dir, &buf );
if( rc ) // no dir, create
{ if( rank == 0 )
{
rc = mkdir( dir, 0771);
if( rc )
{std::ostringstream oss;
oss << "Can't create dump directory \""
<< dir
<< "\"";
}
}
else {
sleep (2);
}
}
else if( !S_ISDIR( buf.st_mode ) )
{std::ostringstream oss;
oss << "Path \""
<< dir
<< "\" is not directory for dump";
}
MPI_Barrier(MPI_COMM_WORLD);
// Every process defines name of file for output (res_0, res_1, res_2.....)
std::ostringstream filename;
filename << dir << "/res_"<< rank;
// Open file
std::ofstream file(filename.str().c_str());
// Output to file . Output seems like "I am 0 from 24. hostname"
file << "I am " << rank << " from " << size << ". " << hostname << std::endl;
file.close();
MPI_Finalize();
return 0;
}
I compile it with openmpi_intel-1.4.2, using comand
mpicxx -Wall test.cc -o test
Then I queue this program with script:
#!/bin/bash
#PBS -N test
#PBS -l select=8:ncpus=6:mpiprocs=6
#PBS -l walltime=00:01:30
#PBS -m n
#PBS -e stderr.txt
#PBS -o stdout.txt
cd $PBS_O_WORKDIR
echo "I run on node: `uname -n`"
echo "My working directory is: $PBS_O_WORKDIR"
echo "Assigned to me nodes are:"
cat $PBS_NODEFILE
mpirun -hostfile $PBS_NODEFILE ./test
I expected this result:
1. New directory "Res" to be created
2. 8*6 different files (res_0, res_1, res_2, ...) to be written to the Res dir
But only res_* file from the first node are written (res_{0..5}) while the rest are not.
What is the problem?
Thank you!

Answer 1

OK, let's assume you run on a file system coherently mounted across all your compute nodes. This is the case, right?
So then the main issue I see with your code snippet is that all processes do state the directory at the same time and then try to create it if it doesn't exist. I'm not sure what truly happens but I'm sure this isn't the smartest idea ever.
Since in essence what you want is a serial sanity check of the directory and/or it's creation if needed, why not just letting MPI process of rank 0 doing it?
That would give you something like this:
if ( rank == 0 ) { // Only master manages the directory creation
int rc = stat( dir, &buf );
... // sanity check goes here and directory creation as well
// calling MPI_Abort() in case of failure seems also a good idea
}
// all other processes wait here
MPI_Barrier( MPI_COMM_WORLD );
// now we know the directory exists and is accessible
// let's do our stuff
Could this work for you?