Multi-Rendering API Engine shader file management - c++

Question

I am developing a 3D engine that is designed to support the implementation of any given graphics API. I would like your feedback on how I'm planning to manage the shader files:
I thought about creating a struct that contains 3 string variables, the directory and the file name (both vertex and fragment), something like this:
class ShaderFile : public SerializableAsset
{
std::string nameID; //Identifier
std::string directory;
std::string vertexName;
std::string fragmentName;
};
The user would be able to set these variables in the editor. Then, my engine would load the shader files in like:
void createShader(RenderAPI api)
{
ShaderFile shaderFile //Get it from some place
std::string vertexPath = shaderFile.directory + shader.vertexName + api.name + api.extension;
std::string fragmentPath = shaderFile.directory + shader.fragmentName + api.name + api.extension;
//Create shader...
}
Which would create something like: Project/Assets/Shaders/standardVulkan.spv.
Am I thinking in the right direction or is this a completely idiotic approach? Any feedback

Answer 1

It's an interesting idea and we've actually done exactly this, but we discovered some things along the way that are not easy to deal with:
If you take a deeper look at Shader API's, although they are close to offering the same capabilities on paper, they often do not support features in the same way and have to be managed differently. By extension, so do the shaders. The driver implementation is key here, and sometimes differs considerably when it comes to managing internal state (synchronization and buffer handling).
Flexibility
You'll find that OpenGL is flexible in the way it handles attributes and uniforms, where DirectX is more focussed on minimizing uploads to the hardware by binding them in blocks according to your renderpass configurations, usually on a per-object/per-frame/per-pass basis etc.. While you can also do this by creating tiny blocks, this obviously would give different performance.
Obviously, there are multiple ways to do binds, or even buffer objects, or shaders, even in a single API. Also, getting shader variable names and bind points is not that flexible to query in DirectX and some of the parameters needs to be set from code. In Vulkan, binding shader attributes and uniforms is even more generalized: you can completely configure the bind points as you wish.
Versioning
Another topic is everything that has to do with GLSL/HLSL shading versioning: You may need to write different shaders for different hardware capabilities that support lower shader models. If you're going to write unique shaders and are not going for the uber-shader approach (and also to a large extend IF you use this approach) this can get complicated if it ties too tightly into your design, and given the number of permutations might be unrealistic.
Extensions
OpenGL and Vulkan extensions can be 'queried' from within the shader, while other API's such as DirectX require setting this from the code side. Still, within the same compute_capability, you can have extensions that only work on NVidia, or are ARB approved but not CORE, etc. This is really quite messy and in most cases application specific.
Deprecation
Considerable parts of API's are getting deprecated all the time. This can be problematic if your engine expects those features to remain in place, especially if you like to deal with multiple API's that support that feature.
Compilation & Caching
Most API's by now support some form of offline compilation that can be loaded later. Compilation takes a considerable amount of time so caching that makes sense. Since the hardware shader code is compiled uniquely for the hardware that you have, you need to do this exercise for each platform that the code should run on, either the first time the app needs the shader, or in some other clever way in your production pipeline. Your filename would in that case be replaced by a hash so that the shader can be retrieved from the cache. But this means the cache needs a timestamp so it can detect new versions of the source shader, because if the shader source should change, then the cache entry needs to be rebuilt. etc.. :)
Long story short
If you aim for maximum flexibility in whatever API, you'd end up adding a useless layer in your engine that in the best case simply duplicates the underlying calls. If you aim for a generalized API, you'll quickly get trapped in the version story that is totally not synchronized between the different API's in terms of extensions, deprecation and driver implementation support.