OpenGL multithreading - c++

Question

Ok I am trying to switch my Game Engine to multithreading. I have done the research on how to make it work to use OpenGL in multithreaded application. I have no problem with rendering or switching contexts. Let my piece of code explain the problem :) :
for (it = (*mIt).Renderables.begin(); it != (*mIt).Renderables.end(); it++)
{
//Set State Modeling matrix
CORE_RENDERER->state.ModelMatrix = (*it).matrix;
CORE_RENDERER->state.ActiveSubmesh = (*it).submesh;
//Internal Model Uniforms
THREAD_POOL->service.post([&]
{
for (unsigned int i = 0; i < CORE_RENDERER->state.ActiveShaderProgram->InternalModelUniforms.size(); i++)
{
CORE_RENDERER->state.ActiveShaderProgram->InternalModelUniforms[i]->Set( CORE_RENDERER->state.ModelMatrix);
}
CORE_RENDERER->state.ActiveSubmesh->_Render();
});
//Sleep(10);
}
I'll quickly explain what are the elements in the code to make my problem more clear. Renderables is a simple std::vector of elements with _Render() function which works perfectly. CORE_RENDERER->state is a struct holding information about current render state such as current material properties as well as current submesh ModelMatrix. So Matrix and Submesh are stored to state struct (I KNOW THIS IS SLOW, I'll probably change that in time :) ) The next piece of code is sent to THREAD_POOL->service which is actually boost::asio::io_service and has only one thread so it acts like a queue of rendering commands. The idea is that the main thread provides information about what to render and do frustum culling and other tests while an auxilary thread does actual rendering. This works fine, except there is a slight problem:
The code that is sent to thread pool starts to execute, but before all InternalModelUniforms are set and submesh is rendered the next iteration of Renderables is executed and both ModelMatrix and ActiveSubmesh are changed. The program doesn't crash but both informations change and some meshes are rendered some matrices are right others not which results in flickering image. Objects apear on frame and the next frame they are gone. The problem is only fixed if I enable that Sleep(10) function which makes sure that the code is executed before next iteration which obviously kills the idea of gaining preformance. What is the best possible solution for this? How can I send commands to the queue each with unique built in data? Maybe I need to implement my own queue for commands and a single thread without io_service?
I will continue my research as I know there is a way. The idea is right cause I get preformance boost as not a single if/else statement is processed by the rendering thread :) Any help or tips will really help!
Thanks!
Update:
After struggling for few nights I have created a very primitive model of communication between main thread and an Aux Thread. I created a class that represents a base command to be executed by aux thread:
class _ThreadCommand
{
public:
_ThreadCommand() {}
~_ThreadCommand() {}
virtual void _Execute() = 0;
virtual _ThreadCommand* Clone() = 0;
};
These commands that are childs of this class have _Execute() function to do whatever operation needs to be done. The main thread upon rendering fills a boost::ptr_vector of these commands While aux thread keeps on checking if there are any commands to process. When commands are found it copies entire vector to it's own vector inside _AuxThread and clears the original one. Commands are then executed by calling _Execute functions on each:
void _AuxThread()
{
//List of Thread commands
boost::ptr_vector<_ThreadCommand> _cmd;
//Infinitive loop
while(CORE_ENGINE->isRunning())
{
boost::lock_guard<boost::mutex> _lock(_auxMutex);
if (CORE_ENGINE->_ThreadCommands.size() > 0)
{
boost::lock_guard<boost::mutex> _auxLock(_cmdMutex);
for (unsigned int i = 0; i < CORE_ENGINE->_ThreadCommands.size(); i++)
{
_cmd.push_back(CORE_ENGINE->_ThreadCommands[i].Clone());
}
//Clear commands
CORE_ENGINE->_ThreadCommands.clear();
//Execute Commands
for (unsigned int i = 0; i < _cmd.size(); i++)
{
//Execute
_cmd[i]._Execute();
}
//Empty _cmd
_cmd.clear();
}
}
//Notify main thread that we have finished
CORE_ENGINE->_ShutdownCondition->notify_one();
}
I know that this is a really bad way to do it. Preformance is quite slower which I'm quite sure is because of all the copying and mutex locks. But at least the renderer works. You can get the idea of what I want to achieve but as I said I am very new to multithreading. What is the best solution for this scenario? Should I return back to ThreadPool system with asio::io_service? How can I feed commands to AuxThread with all values that must be sent to renderer to preform tasks in correct way?

Answer 1

First, a warning. Your "slight problem" is not slight at all. It is race condition, which is undefined behavior in C++, which, in turn, implies that anything could happen, including:
Everything renders fine
Image flickers
Nothing renders at all
It crashes on the last Saturday of every month. Or working fine on your computer and crashing on everyone's else.
Seriously, do not ever rely on UB, especially when writing library/framework/game engine.
Now about your question.
Lets leave aside any practical benefits of your approach and fix it first.
Actually, OpenGL implementation uses something very similar under the hood. Commands are executed asynchronously by the driver thread. I recommend you to read about their implementation to get some ideas on how to improve your design.
What you need to do, is to somehow "capture" the state at the time you post a rendering command. Simplest possible thing - copy the CORE_RENDERER->state into closure and use this copy to do the rendering. If state is large enough, it can be costly, though.
Alternative solution (and OpenGL goes that way) is to make every change in the state a command also, so
CORE_RENDERER->state.ModelMatrix = (*it).matrix;
CORE_RENDERER->state.ActiveSubmesh = (*it).submesh;
translates into
Matrix matrix = (*it).matrix;
Submesh submesh = (*it).submesh;
THREAD_POOL->service.post([&,matrix,submesh]{
CORE_RENDERER->state.ModelMatrix = matrix;
CORE_RENDERER->state.ActiveSubmesh = submesh;
});
Notice, however, that now you can't simply read CORE_RENDERER->state.ModelMatrix from your main thread, as it is changing in a different thread. You must first ensure that command queue is empty.