Boost interprocess_condition multiple threads calling wait() fails - c++

Question

Running into a very strange issue with 2+ threads waiting on an interprocess_condition variable.
Boost 1.60.0
With 1 thread calling wait() and a 2nd calling notify_all(), everything works as expected.
When there are 2+ calling wait(), I get an assertion failure on do_wait() and the process exits.
Test.cpp:
#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/sync/interprocess_mutex.hpp>
#include <boost/interprocess/sync/interprocess_condition.hpp>
#include <iostream>
using namespace boost::interprocess;
struct Data {
interprocess_mutex mux_;
interprocess_condition cond_;
};
int main(int argc, char *argv[]) {
if (argc > 1 && atoi(argv[1]) == 0) {
struct shm_remove {
shm_remove() { shared_memory_object::remove("MySharedMemory"); }
~shm_remove() { shared_memory_object::remove("MySharedMemory"); }
} remover;
managed_shared_memory seg(create_only, "MySharedMemory", 65536);
Data *const d = seg.construct<Data>(unique_instance)();
scoped_lock<interprocess_mutex> lock(d->mux_);
std::cout << "Waiting" << std::endl;
d->cond_.wait(lock);
} else if (argc > 1 && atoi(argv[1]) == 1) {
managed_shared_memory seg(open_only, "MySharedMemory");
std::pair<Data *, std::size_t> res = seg.find<Data>(unique_instance);
scoped_lock<interprocess_mutex> lock(res.first->mux_);
std::cout << "Waiting" << std::endl;
res.first->cond_.wait(lock);
} else {
managed_shared_memory seg(open_only, "MySharedMemory");
std::pair<Data *, std::size_t> res = seg.find<Data>(unique_instance);
scoped_lock<interprocess_mutex> lock(res.first->mux_);
std::cout << "Notifying" << std::endl;
res.first->cond_.notify_all();
}
}
Compiled as:
$ clang++ -I/usr/local/include test.cpp
Running with 1 wait() and 1 notify():
$ ./a.out 0&
[8] 25889
Waiting
$ ./a.out 2&
[9] 25901
Notifying
[8]- Done ./a.out 0
[9]+ Done ./a.out 2
Running with 2 waits:
$ ./a.out 0&
[8] 25986
Waiting
$ ./a.out 1&
[9] 25998
Waiting
Assertion failed: (res == 0), function do_wait, file /usr/local/include/boost/interprocess/sync/posix/condition.hpp, line 175.
Tested on OSX El Capitan
$ uname -a
Darwin LUS-JOHUGHES2 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64
I also tried the above example on an Ubuntu Trusty machine and all examples work as expected, leading me to believe there is an issue with the OSX implementation. I have not tried it on Windows.

Answer 1

Did some digging and found a definite answer to the problem.
The boost assertion error above is failing when the second process calls do_wait(), which calls pthread_wait() which returns immediately with EINVAL (instead of a successful 0).
In OSX's pthread implementation, the condition variable stores a raw pointer to the mutex variable. The first call to pthread_wait() in the first process sets this pointer. The second call to pthread_wait() checks this stored mutex pointer against the mutex pointer passed into pthread_wait(). {Can be found in the source here: https://opensource.apple.com/source/libpthread/libpthread-137.1.1/src/pthread_cond.c}
Since the two processes have mapped the shared mutex and condition variables in different address spaces, the second call to pthread_wait() will never work, since it compares raw pointers.
Thus the 2 options to make this work are as follows:
Use Fixed Address Mapping: http://www.boost.org/doc/libs/1_60_0/doc/html/interprocess/sharedmemorybetweenprocesses.html#interprocess.sharedmemorybetweenprocesses.mapped_region.mapped_region_fixed_address_mapping which guarantees that the mapped regions will be at the same address, and thus raw pointers will work, or
Instead of exec()'ing new processes, use fork() which means that the child process will have a copy of the original segment manager, mapped to the same address, and thus raw pointers will work.
I didn't dig much into the glibc pthreads code to see what they do differently than Apple, so I'm not sure why the original example works on Linux but not OSX.
I think the Boost docs would definitely benefit with a paragraph discussing this pitfall.

Answer 2

This is a bug both on Darwin's C library and Boost.Interprocess. First, that C library claims it's posix-compliant and supports process shared memory condition variables, which is false (as it uses a raw pointer to store the address of the mutex). Second, Boost.Interprocess should detect this platform as a buggy one, should disable the use of pthreads and fallback to emulation.
In boost/interprocess/detail/workaround.hpp, you will find a comment saying:
//Mac Os X < Lion (10.7) might define _POSIX_THREAD_PROCESS_SHARED but there is no real support.
Some old reports claimed that newer macos versions really support process shared condition variables, but this claim is false, so the __APPLE__ section should be just:
#define BOOST_INTERPROCESS_BUGGY_POSIX_PROCESS_SHARED