Rotate a plane defined by its normal and its distance - opengl

Question

I've been trying to work around rotating a plane in 3D space, but I keep hitting dead ends. The following is the situation:
I have a physics engine where I simulate a moving sphere inside a cube. To make things simpler, I have only drawn the top and bottom plane and moved the sphere vertically. I have defined my two planes as follows:
CollisionPlane* p = new CollisionPlane(glm::vec3(0.0, 1.0, 0.0), -5.0);
CollisionPlane* p2 = new CollisionPlane(glm::vec3(0.0, -1.0, 0.0), -5.0);
Where the vec3 defines the normal of the plane, and the second parameter defines the distance of the plane from the normal. The reason I defined their distance as -5 is because I have scaled the the model that represents my two planes by 10 on all axis, so now the distance from the origin is 5 to top and bottom, if that makes any sense.
To give you some reference, I am creating my two planes as two line loops, and I have a model which models those two line loop, like the following:
top plane:
std::shared_ptr<Mesh> f1 = std::make_shared<Mesh>(GL_LINE_LOOP);
std::vector<Vertex> verts = { Vertex(glm::vec3(0.5, 0.5, 0.5)), Vertex(glm::vec3(0.5, 0.5, -0.5)), Vertex(glm::vec3(-0.5, 0.5, -0.5)), Vertex(glm::vec3(-0.5, 0.5, 0.5)) };
f1->BufferVertices(verts);
bottom plane:
std::shared_ptr<Mesh> f2 = std::make_shared<Mesh>(GL_LINE_LOOP);
std::vector<Vertex> verts2 = { Vertex(glm::vec3(0.5, -0.5, 0.5)), Vertex(glm::vec3(0.5, -0.5, -0.5)), Vertex(glm::vec3(-0.5, -0.5, -0.5)), Vertex(glm::vec3(-0.5, -0.5, 0.5)) };
f2->BufferVertices(verts2);
std::shared_ptr<Model> faceModel = std::make_shared<Model>(std::vector<std::shared_ptr<Mesh>> {f1, f2 });
And like I said I scale the model by 10.
Now I have a sphere that moves up and down, and collides with each face, and the collision response is implemented as well.
The problem I am facing is when I try to rotate my planes. It seems to work fine when I rotate around the Z-axis, but when I rotate around the X axis it doesn't seem to work. The following shows the result of rotating around Z:
However If I try to rotate around X, the ball penetrates the bottom plane, as if the collisionplane has moved down:
The following is the code I've tried to rotate the normals and the planes:
for (int i = 0; i < m_entities.size(); ++i)
{
glm::mat3 normalMatrix = glm::mat3_cast(glm::angleAxis(glm::radians(6.0f), glm::vec3(0.0, 0.0, 1.0)));
CollisionPlane* p = (CollisionPlane*)m_entities[i]->GetCollisionVolume();
glm::vec3 normalDivLength = p->GetNormal() / glm::length(p->GetNormal());
glm::vec3 pointOnPlane = normalDivLength * p->GetDistance();
glm::vec3 newNormal = normalMatrix * normalDivLength;
glm::vec3 newPointOnPlane = newNormal * (normalMatrix * (pointOnPlane - glm::vec3(0.0)) + glm::vec3(0.0));
p->SetNormal(newNormal);
float newDistance = newPointOnPlane.x + newPointOnPlane.y + newPointOnPlane.z;
p->SetDistance(newDistance);
}
I've done the same thing for rotating around X, except changed the glm::vec3(0.0, 0.0, 1.0) to glm::vec3(1.0, 0.0, 0.0)
m_entites are basically my physics entities that hold the different collision shapes (spheres planes etc). I based my code on the answer here Rotating plane with normal and distance
I can't seem to figure at all why it works when I rotate around Z, but not when I rotate around X. Am I missing something crucial?