C++ How to Design Command Queue - c++

Question

I have 8 threads providing commands that need to be executed on the GPU, and I want the GPU execution to happen on the same thread via a buffer filled by the 8 other threads. I thought it would be fairly simple but am having issues designing it correctly.
struct CommandStruct {
CComPtr<IDirect3DPixelShader9> Shader;
const char* Param[9];
bool IsCompleted;
};
concurrent_queue<CommandStruct*> Shader::cmdBuffer;
std::mutex Shader::startLock;
std::thread* Shader::WorkerThread = NULL;
void Shader::AddCommandToQueue(CommandStruct* cmd) {
// Add command to queue.
cmdBuffer.push(cmd);
if (WorkerThread == NULL) {
startLock.lock();
if (WorkerThread == NULL) {
WorkerThread = new std::thread(StartWorkerThread, env);
}
startLock.unlock();
}
}
StartWorkerThread is a function that takes all elements within cmdBuffer and execute them one by one until the buffer is empty and stops after being idle for a few seconds.
The problem I'm having is how to make the caller wait until execution is completed? At first I tried SetEvent, but performance is VERY poor. Then I tried looping with Sleep(10), but then it wastes the whole CPU.
Ideally AddCommandToQueue would return once the command is completed, but then I have the same problem about how to wait for completion.
What's the right way of implementing such a queue? It needs to be fast because it contains many instructions for the GPU. I tried with an event, I tried with a IsCompleted flag on the structure, but the design isn't yet working.

Answer 1

I realize I'm a few years late, but I thought I'd provide some thoughts. Since you're looking at a command system, I would take the time to make sure your commands are type-erased or abstract. Here is an example system that may be useful to you. You might also take a look at his blogs on external and internal references:
https://blog.molecular-matters.com/2014/12/16/stateless-layered-multi-threaded-rendering-part-3-api-design-details/
The reason I'd keep your command system type-erased or abstract, is to maintain some level of a layered architecture. Here is a typical diagram of how a realtime game engine is architected: http://hightalestudios.com/2017/03/game-engine-architecture-2nd-edition-overview-ch-1-part-2/
One thing to note in the above diagram is that an entire layer is dedicated to platform independence. Above that level, I would avoid using constructs such as com pointers, since they are largely platform-dependent. Instead, think about creating a resource management system using a pool or arena allocator where you can store, cache, and reuse resources, vertex buffers, and such. You will set yourself up for success if you take this step.
This also sets yourself up for success when creating your command system. At a high level, think of your commands as small structs that hold handles to data held by the data manager. Again, by doing it this way, you won't need to load/reload VAOs, VBOs, IBOs, shaders, materials, etc. You'll get O(1) access to the corresponding data in the data manager if you setup your handle system correctly, and your commands will be small, which allows you to sort them. Sorting will become important when it comes time to render draw calls, since you'll want to draw items that are far away before drawing items that are closer.
If you follow the above advice, the content of your command should be nothing a small amount of data (handles/pointers only) and dispatch function. So here is a concrete example of what some of the commands in my homemade GUI/game engine look like:
struct make_viewport
{
int width = platform::default_viewport_width;
int height = platform::default_viewport_height;
static auto dispatch(const void* data) {
auto command = static_cast<const make_viewport*>(data);
auto handle = platform::make_viewport(command->width, command->height);
resource::manager::store(handle);
};
};
In the above example, the screen width and height are small enough that it would not make sense to use a handle and store the sizes in the resource manager. Also notice that the command knows nothing about the platform specifics at this level. There is a custom linker-based interface that is implemented for various operating systems, cpus, gpus, etc. and compiled on the host machine. There are other ways to do this, but this was the solution I thought of first. Here is a library I found for abstracting away several platform-specific things you may be interested in: https://github.com/ThePhD/infoware
So what exactly does this dispatch function do? Most command systems use some form of dispatch mechanism to execute commands lazily. A few common ways I've seen these implemented is as follows:
auto execute() noexcept -> void
{
// do stuff with command data
}
auto dispatch() noexcept -> void
{
// do stuff with command data
}
auto operator()() noexcept -> void
{
// do stuff with command data
}
The signature of your dispatch function is not important, but it is important to keep the signature consistent between each of your commands so that you can create polymorphic arrays of your commands. You will need to be able to sort and execute this polymorphic array, and its contents cannot be determined at compile time, so you won't be able to use something like std::tuple (you can use std::variant, but your data locality won't be as great as it could be).
If I had to make a suggestion, I would tend toward overloading operator(). Why? If you notice, this would make any command a valid functor. This allows for deep integration with modern C++ techniques such as lambdas and standard algorithms. In other words, rewriting the make_viewport command from above could be as simple as:
auto make_viewport = [width = platform::default_viewport_width, height = platform::default_viewport_height](const void* data)
{
auto command = static_cast<const make_viewport*>(data);
auto handle = platform::make_viewport(command->width, command->height);
resource::manager::store(handle);
};
Of course, you'd need to find a way to deal with polymorphism/sorting/executing, but if you find a way to do this, the lambda approach could be pretty interesting.
If you're interested in ways to handle type erasure, here's how I did it:
struct command_packet
{
void* command = nullptr;
dispatch_type dispatch = nullptr;
};
template<typename command_t> [[nodiscard]]
constexpr auto make_packet(command_t&& command) noexcept
{
return command_packet
{
.command = &command,
.dispatch = command.dispatch
};
}
You can also use a virtual base class to do this, but I wanted the ability to have POD commands so I could reason about the size of the commands. I then create an array of commands to execute every frame. This array should ideally have thread-local storage, which makes it difficult to fill from multiple threads. But the good news, is that you can create multiple command queues depending on your needs. For instance, draw calls will always need to be sorted by depth and therefore will need to be in the same queue. But if you're clever, you can choose to sort this queue using a radix sort. When implementing your radix sort, you can sort data into buckets from multiple threads since radix sort does not require any comparison operations. If your commands are small enough and numerous enough, you'll drastically out-perform std::sort in graphics applications.
And finally, if you need to perform culling, merging, and redundancy optimizations, you can add steps to the pipeline to do this. I have not yet implemented these optimizations on my system, but it should not be difficult to imagine that commands can be removed/added/modified/merged/chained with a little extra logic, making the aforementioned optimizations possible.
In summary, a command system that is likely to fit your needs would have the following attributes:
Commands are POD or their size can be reasoned about
Commands are polymorphic
Commands do not contain platform-specific code
Command system is coupled with a resource manager
Commands are designed to be aligned with cache lines (in other words, static_cast(sizeof(command)) / sizeof(void*) is a whole number, and your allocator provides the appropriate padding/alignment to avoid cache misses)
Your command pipeline would look something like this:
Load data (preferrably asynchronously, since handles can be returned immediately without waiting for resources to load) into the resource manager and obtain a handle to that data
create commands using the aforementioned handles using a task/job system
shove each command into your command queue (if you use a radix sort, you can directly stuff them into the first bucket for improved efficiency).
At the end of the frame, sort the data (preferrably using a radix sort).
Remove any redundant commands if so-desired
sequentially submit commands from your queue on a single thread. I'm sure you're already aware based on the question, but graphics APIs are typically stateful and require you to configure the state before each render call (which makes it impossible to perform in parallel... not to mention any kind of hardware-based limitations on parallelism that might exist on your platform). The thing to take note here is that if you setup your sort criteria correctly, you should only need to swap out shaders/textures/materials/vertex buffers/etc when they have changed. This further improves efficiency by getting rid of redundant api calls and copies.
Clear your queue. If using std::vector<T, Allocator>, for instance, a call to std::vector<T, Allocator>::clear() would be sufficient. The idea is to not deallocate memory, but to move the current index back to the beginning of the container. You do not even need to delete the commands if they are POD, you can simply overwrite it on the next frame if you're careful about how you set this up.
And finally, I just want to leave you with the note that what you're doing can be as difficult as you want it to be. Start small and simply get your system working. You can slowly add some of the features I've talked about as you get each piece working and have verified that it works. graphics commands are what I would call a critical bottleneck in most systems that use them, so do yourself a favor and set yourself up for success by creating an architecture that can be optimized if you find a reason to do so. Many command systems do not allow for sorting, for instance, which is a huge opportunity missed. Even if you don't implement sorting immediately, if you somewhat follow the cook book, you can implement sorting at a later time without too much pain. What you're doing almost requires a data-oriented approach to be efficient, so think about laying out your data in a manner that offers O(1) access, O(1) storage, doesn't result in redundancies, and is aligned to the cache lines on your platform. You will save yourself a world of hurt by spending some time on this. And perhaps you may find some additional useful tips in this book: https://www.amazon.com/Engine-Architecture-Third-Jason-Gregory/dp/1138035459/ref=pd_lpo_14_t_0/137-2195898-2699658?_encoding=UTF8&pd_rd_i=1138035459&pd_rd_r=100afaaf-5542-476a-89eb-87ae8bed3571&pd_rd_w=AaXTZ&pd_rd_wg=E6BRu&pf_rd_p=7b36d496-f366-4631-94d3-61b87b52511b&pf_rd_r=KQ0CFH5Y2JCA4817ATNM&psc=1&refRID=KQ0CFH5Y2JCA4817ATNM