Using glut to prevent the mouse from leaving the window - c++

Question

I'm using glut for a game right now and I'm trying to keep the mouse inside the window. This isn't a first person shooter so locking it in the center is no good. I already know about glutWarpPointer(int, int); and I've trying things that work (kinda).
I've tried having the mouse warp back to the nearest edge of the window when it leaves, this works, but for a split second you see the mouse outside of the window and teleport back in. I don't want that, I want it to seem like the mouse just hits the edge of the window and stops going any further in that direction, while keeping movement in any other available direction. Like you would expect it to work.

Answer 1

This is not exactly an answer to your question, but it is an answer to your problem!
Almost every game has its own cursors. They would hide the mouse, and draw the cursor manually where the mouse should be positioned.
If you get your own cursor image and do as I said, you can simply draw the curser at the edge of the screen, even though the mouse position reads out of boundaries. Then you can warp the mouse back in.

Answer 2

Tried to search and figure this out and couldn't find an answer, so I implemented it myself. Here is what worked for my first person camera case:
callback from glutPassiveMotion
CODE SAMPLE
void Game::passiveMouseMotion(int x, int y)
{
//of my code for doing the cam, yours is may be different, this is based on the example from https://learnopengl.com/Getting-started/Camera
if (firstMouse) {
lastX = x;
lastY = y;
firstMouse = false;
}
float xoffset = x - lastX;
float yoffset = lastY - y; // reversed since y-coordinates go from bottom to top
lastX = x;
lastY = y;
camera->ProcessMouseMovement(xoffset, yoffset);
glutPostRedisplay();
//this is the main thing that keeps it from leaving the screen
if ( x < 100 || x > win_w - 100 ) { //you can use values other than 100 for the screen edges if you like, kind of seems to depend on your mouse sensitivity for what ends up working best
lastX = win_w/2; //centers the last known position, this way there isn't an odd jump with your cam as it resets
lastY = win_h/2;
glutWarpPointer(win_w/2, win_h/2); //centers the cursor
} else if (y < 100 || y > win_h - 100) {
lastX = win_w/2;
lastY = win_h/2;
glutWarpPointer(win_w/2, win_h/2);
}
}
Hope this helps!