MVP matrix for DirectX11 using GLM - c++

Question

First for the part that does work using XMMATH where data.model is a XMMatrix:
static auto model_matrix = DirectX::XMMatrixIdentity();
static auto pos = DirectX::XMVectorSet(0.0f, 0.0f, -10.0f, 0.0f);
static auto focus = DirectX::XMVectorSet(0.0f, 0.0f, 0.0f, 0.0f);
static auto up = DirectX::XMVectorSet(0.0f, 1.0f, 0.0f, 0.0f);
static auto view_matrix = DirectX::XMMatrixLookAtLH(pos, focus, up);
static auto proj_matrix = DirectX::XMMatrixPerspectiveFovLH(glm::radians(45.0f), 16.0f / 9.0f, 0.1f, 10000.0f);
building the mvp:
data.model = model_matrix * view_matrix * proj_matrix;
data.model = DirectX::XMMatrixTranspose(data.model);
when I hand the data.model over to my HLSL shader, everything works fine and and I can change the pos vector to look at my cube from different angles. HLSL vertex shader:
cbuffer myCbuffer : register(b0) {
float4x4 mat;
}
float4 main(float3 pos : POSITION) : SV_POSITION
{
return mul(float4(pos, 1), mat);
}
Now when I try to make something similar using GLM (I changed the type of data.model to be a glm::mat4 ):
auto gl_m = glm::mat4(1.0f);
static auto gl_pos = glm::vec3(0.0f, 0.0f, -10.0f);
static auto gl_focus = glm::vec3(0.0f, 0.0f, 0.0f);
static auto gl_up = glm::vec3(0.0f, 1.0f, 0.0f);
auto gl_v = glm::lookAtLH(gl_pos, gl_focus, gl_up);
auto gl_p = glm::perspectiveFovLH_ZO(glm::radians(45.0f), 1280.0f, 720.0f, 0.1f, 10000.0f);
building the MVP:
data.model = gl_m * gl_v * gl_p;
Now when I pass this to data.model the cube does get rendered but the whole screen is filled black. (my cube is black and the clear color is light-blue, so I think it's rendering the cube but really close or inside it).
I don't know where to look on how to fix this, the projection matrix should be in the correct clipping space since I'm using perspectiveFovLH_ZO, the ZO fixes the clipping space to [0..1]. It could be how the HLSL shader float4x4 deals with the glm::mat4, but both are column major I believe so no need to transpose.
It might have something to do with the rasterizer culling settings and the FrontCounterClockwise setting, but I'm fairly new to DirectX and don't know what it does exactly.
D3D11_RASTERIZER_DESC raster_desc = {0};
raster_desc.FillMode = D3D11_FILL_MODE::D3D11_FILL_SOLID;
raster_desc.CullMode = D3D11_CULL_MODE::D3D11_CULL_NONE;
raster_desc.FrontCounterClockwise = false;
d3device->CreateRasterizerState(&raster_desc, rasterize_state.GetAddressOf());
Any help is appreciated, let me know if I forgot anything.

Answer 1

Managed to fix it (seems like a bandaid fix but I'll work with it for now).
After I build the mvp I added:
gl_mvp[3][3] = -10.0f;
gl_mvp = glm::transpose(gl_mvp);
the -10 is the z coordinate of the camera position (the same one you pass as the first argument to glm::lookAtLH). I thought HLSL/DirectX matched the column-major GLM matrices but apparently not, it needed the extra transpose call. I'm not sure why that is and what the bottom left element of the MVP is that it has to match the positional z, maybe someone with a better understanding of the math behind it can clarify.

Answer 2

Removed gl_mvp[3][3] = -10.0f; since it's a bad fix, instead I got this now:
changed both the lookAtLH and PerspectiveFovLH_ZO to their RH variants.
Also changed the order of building the MVP from M * V * P to P * V * M.
New code that seems to work well (even flying around using my Camera class):
auto gl_m = glm::mat4(1.0f);
auto gl_v = glm::lookAtRH(position, position + dir, {0, 1, 0});
auto gl_p = glm::perspectiveFovRH_ZO(glm::radians(FOV), 1280.0f, 720.0f, 0.1f, 10000.0f);
glm::mat4 gl_mvp = gl_p * gl_v * gl_m;
return glm::transpose(gl_mvp);
It's a bit different from the previous code cause this is inside my Camera class, so position, dir and FOV are variables I keep track off but you get the idea. Passed this return result to my HLSL shader and all seems well so far.