Lock a mutex from mutex array with atomic index - c++

Question

I'm trying to write a buffer which can push data to the buffers, checks if full, swaps the buffer if necessary. Another thread can get a buffer for file output.
I've successfully implemented the buffer but I wanted to add a ForceSwapBuffer method that would force an incomplete buffer to be swapped and return the data from the incomplete buffer. In order to do this I check if the read and write buffer are the same (there is no use in trying to force swap a buffer to write to a file while there are still other full buffers that could be written).
I want this method to be able to run side by side with the GetBuffer method (not really necessary but I wanted to try it and stumbled upon this problem).
The GetBuffer would block and when ForceSwapBuffer is finished it would still block until the new buffer is completely full, because in the ForceSwapBuffer I change the atomic _read_buffer_index. I wonder if this will always work? Will the blocking lock of GetBuffer detect the change of the atomic read_buffer_index and change the mutex it is trying to lock or would it check at the start of the lock what mutex it has to lock and keep trying to lock the same mutex even when the index changes?
/* selection of member data */
unsigned int _size, _count;
std::atomic<unsigned int> _write_buffer_index, _read_buffer_index;
unsigned int _index;
std::unique_ptr< std::unique_ptr<T[]>[] > _buffers;
std::unique_ptr< std::mutex[] > _mutexes;
std::recursive_mutex _force_swap_buffer;
/* selection of implementation of member functions */
template<typename T> // included to show the use of the recursive_mutex
void Buffer<T>::Push(T *data, unsigned int length) {
std::lock_guard<std::recursive_mutex> lock(_force_swap_buffer);
if (_index + length <= _size) {
memcpy(&_buffers[_write_buffer_index][_index], data, length*sizeof(T));
_index += length;
} else {
memcpy(&_buffers[_write_buffer_index][_index], data, (_size - _index)*sizeof(T));
unsigned int t_index = _index;
SwapBuffer();
Push(&data[_size - t_index], length - (_size - t_index));
}
}
template<typename T>
std::unique_ptr<T[]> Buffer<T>::GetBuffer() {
std::lock_guard<std::mutex> lock(_mutexes[_read_buffer_index]); // where the magic should happen
std::unique_ptr<T[]> result(new T[_size]);
memcpy(result.get(), _buffers[_read_buffer_index].get(), _size*sizeof(T));
_read_buffer_index = (_read_buffer_index + 1) % _count;
return std::move(result);
}
template<typename T>
std::unique_ptr<T[]> Buffer<T>::ForceSwapBuffer() {
std::lock_guard<std::recursive_mutex> lock(_force_swap_buffer); // lock that forbids pushing and force swapping at the same time
if (_write_buffer_index != _read_buffer_index)
return nullptr;
std::unique_ptr<T[]> result(new T[_index]);
memcpy(result.get(), _buffers[_read_buffer_index].get(), _index*sizeof(T));
unsigned int next = (_write_buffer_index + 1) % _count;
_mutexes[next].lock();
_read_buffer_index = next; // changing the read_index while the other thread it blocked, the new mutex is already locked so the other thread should remain locked
_mutexes[_write_buffer_index].unlock();
_write_buffer_index = next;
_index = 0;
return result;
}

Answer 1

There are some problems with your code. First, be careful when modifying atomic variables. Only a small set of operations is really atomic (see http://en.cppreference.com/w/cpp/atomic/atomic), and combinations of atomic operations are not atomic. Consider:
_read_buffer_index = (_read_buffer_index + 1) % _count;
What happens here is that you have an atomic read of the variable, an increment, a modulo operation, and an atomic store. However, the whole statement itself is not atomic! If _count is a power of 2, you can just use the ++-operator. If it is not, you have to read _read_buffer_index into a temporary variable, perform the above calculations, and then use a compare_exchange function to store the new value if the variable was not changed in the mean time. Obviously the latter has to be done in a loop until it succeeds. You also have to worry about the possibility that one thread increments the variable _count times between the read and compare_exchange of a second thread, in which case the second thread erroneously thinks the variable was not changed.
The second problem is cache-line bouncing. If you have multiple mutexes on the same cache line, then if two or more threads try to access them simultaneously, the performance will be very bad. What the size of a cache-line is depends on your platform.
The main problem is that while ForceSwapBuffer() and Push() both lock the _force_swap_buffer mutex, GetBuffer() does not. GetBuffer() however does change _read_buffer_index. So in ForceSwapBuffer():
std::lock_guard<std::recursive_mutex> lock(_force_swap_buffer);
if (_write_buffer_index != _read_buffer_index)
return nullptr;
// another thread can call GetBuffer() here and change _read_buffer_index
// rest of the code here
The assumption that _write_buffer_index == _read_buffer_index after the if-statement is actually invalid.