OpenMP share file handler - c++

Question

I've got a loop, which I parallelize using OpenMP. In this loop, I read a triangle from a file, and perform some operations on this data. These operations are independent from each triangle to another, so I thought this would be easy to parallelize, as long as I kept the actual reading of files in a critical section.
Order in which triangles are read is not important
Some triangles are read and get discarded pretty quickly, some need some more algorithmic work (bbox construction, ...)
I'm doing binary I/O
Using C++ ifstream *tri_data*
I'm testing this on an SSD
ReadTriangle calls file.read() and reads 12 floats from an ifstream.
#pragma omp parallel for shared (tri_data)
for(int i = 0; i < ntriangles ; i++) {
vec3 v0,v1,v2,normal;
#pragma omp critical
{
readTriangle(tri_data,v0,v1,v2,normal);
}
(working with the triangle here)
}
Now, the behaviour I'm observing is that with OpenMP enabled, the whole process is slower.
I've added some timers to my code to track time spent in the I/O method, and time spent in the loop itself.
Without OpenMP:
Total IO IN time : 41.836 s.
Total algorithm time : 15.495 s.
With OpenMP:
Total IO IN time : 48.959 s.
Total algorithm time : 44.61 s.
My guess is, since the reading is in a critical section, the threads are just waiting for eachother to finish using the file handler, resulting in a longer waiting time.
Any pointers on how to resolve this? My program would really benefit from the possibility to process read triangles with multiple processes. I've tried toying with thread scheduling and related stuff, but that doesn't seem to help a lot in this instance.
Since I'm working on an out-of-core algorithm, introducing a buffer to hold a multitude of triangles is not really an option.

Answer 1

So, the solution I would propose is based on a master/slave strategy, where:
the master (thread 0) performs all the I/O
the slaves do some work on the retrieved data
The pseudo-code would read something like the following:
#include<omp.h>
vector<vec3> v0;
vector<vec3> v1;
vector<vec3> v2;
vector<vec3> normal;
vector<int> tdone;
int nthreads;
int triangles_read = 0;
/* ... */
#pragma omp parallel shared(tri_data)
{
int id = omp_get_thread_num();
/*
* Initialize all the buffers in the master thread.
* Notice that the size in memory is similar to your example.
*/
#pragma omp single
{
nthreads = omp_get_num_threads();
v0.resize(nthreads);
v1.resize(nthreads);
v2.resize(nthreads);
normal.resize(nthreads);
tdone.resize(nthreads,1);
}
if ( id == 0 ) { // Producer thread
int next = 1;
while( triangles_read != ntriangles ) {
if ( tdone[next] ) { // If the next thread is free
readTriangle(tri_data,v0[next],v1[next],v2[next],normal[next]); // Read data and fill the correct buffer
triangles_read++;
tdone[next] = 0; // Set a flag for thread next to start working
#pragma omp flush (tdone[next],triangles_read) // Flush it
}
next = next%(nthreads - 1) + 1; // Set next
} // while
} else { // Consumer threads
while( true ) { // Wait for work
if( tdone[id] == 0) {
/* ... do work here on v0[id], v1[id], v2[id], normal[id] ... */
tdone[id] == 1;
#pragma omp flush (tdone[id]) // Flush it
}
if( tdone[id] == 1 && triangles_read == ntriangles) break; // Work finished for all
}
}
#pragma omp barrier
}
I am not sure if this is still valuable to you but that was a nice teaser anyhow!