opengl - how to implement "portal rendering" - opengl

I've been trying to implement something like the game Antichamber (to be more precisly this trick shown below) for the past week:
Here's a video of what I'm hoping to achieve (even though it was done with the Unreal Engine 4; I'm not using that): https://www.youtube.com/watch?v=Of3JcoWrMZs
I looked up the best way to do this and I found out about the stencil buffer. Between this article and this code (the "drawPortals()" function) I found online I managed to almost implement it.
It works nicely with one portal to another room (not a crossable portal, meaning you can't walk through it and be teleported in the other room). In my example I'm drawing a portal to a simple squared room with a sphere in it; behind the portal there is another sphere, that I used to check whether the depth buffer was working correctly and drawing it behind the portal:
The problems arise when I add another portal that is near this one. In that case I manage to display the other portal correctly (the lighting is off, but the sphere on the right is of a different color to show that it's another sphere):
But if I turn the camera so that the first portal has to be drawn over the second one, then the depth of the first one becomes wrong, and the second portal gets drawn over the first one, like this:
while it should be something like this:
So, this is the problem. I'm probably doing something wrong with the depth buffer, but I can't find what.
My code for the rendering part is pretty much this:
glClear(GL_DEPTH_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
// First portal
glPushMatrix();
// Disable writing to the color and depht buffer; disable depth testing
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glDisable(GL_DEPTH_TEST);
// Make sure that the stencil always fails
glStencilFunc(GL_NEVER, 1, 0xFF);
// On fail, put 1 on the buffer
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
// Enable writing to the stencil buffer
glStencilMask(0xFF);
// Clean the buffer
glClear(GL_STENCIL_BUFFER_BIT);
// Finally draw the portal's frame, so that it will have only 1s in the stencil buffer; the frame is basically the square you can see in the pictures
portalFrameObj1.Draw();
/* Now I compute the position of the camera so that it will be positioned at the portal's room; the computation is correct, so I'm skipping it */
// I'm going to render the portal's room from the new perspective, so I'm going to need the depth and color buffers again
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glClear(GL_DEPTH_BUFFER_BIT);
// Disable writing to the stencil buffer and enable drawing only where the stencil values are 1s (so only on the portal frame previously rendered)
glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 1, 0xFF);
// Draw the room from this perspective
portalRoomObj1.Draw();
glPopMatrix();
// Now the second portal; the procedure is the same, so I'm skipping the comments
glPushMatrix();
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE);
glDepthMask(GL_FALSE);
glDisable(GL_DEPTH_TEST);
glStencilFunc(GL_NEVER, 1, 0xFF);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
glStencilMask(0xFF);
glClear(GL_STENCIL_BUFFER_BIT);
portalFrameObj2.Draw();
/* New camera perspective computation */
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glClear(GL_DEPTH_BUFFER_BIT);
glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 1, 0xFF);
portalRoomObj2.Draw();
glPopMatrix();
// Finally, I have to draw the portals' frames once again but this time on the depth buffer, so that they won't get drawn over; first off, disable the stencil buffer
glDisable(GL_STENCIL_TEST);
// Disable the color buffer
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glClear(GL_DEPTH_BUFFER_BIT);
// Draw portals' frames
portalFrameObj1.Draw();
portalFrameObj2.Draw();
// Enable the color buffer again
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
/* Here I draw the rest of the scene */
UPDATE
I managed to find out what the problem is, but I'm still not able to solve it. It isn't related to the depth buffer actually, but to the stencil.
Basically, the way I'm drawing the first portal is this:
1) Fill the portal frame's bits in the stencil buffer with 1s; outside the portal there are only 0s
2) Draw the portal room where the stencil has 1s (so that it gets drawn onto the frame's portal
And I repeat this for the second portal.
For the first portal, I get at step 1 something like this then (pardon the stupid drawings, I'm lazy):
Then after step 2:
Then I start with the second portal:
But now, between step 1 and 2, I tell the stencil to draw only where the bits are 1s; since the buffer is now cleared, I lost track of the 1s of the first portal, so if part of the second portal's frame is behind the previous frame, the 1s of the second frames will still be drawn with the second portal's room, regardless of the depth buffer. Kind of like in this image:
I don't know if I managed to explain it right...

It's been a while now. Since there aren't going to be anwsers (probably) I'm writing about the workaround I'm using now.
I tried solving the problem using different stencil functions and not empting the stencil buffer when I render a new portal; the idea was to exploit the fact that I knew by looking at the stencil buffer where the previous portals where being rendered.
In the end, I couldn't manage to do it; in the examples I posted in my original question, one of the 2 portals always obscured parts of the other.
So, I decided to do something simplier: I check if a portal is visible, and then I draw it exactly like in the code I posted in the question (so emptying the stencil buffer each time).
In pseudo code, I do this:
for(var i = 0; i < PORTALS_NUMBER; i++)
{
// I get the normal to the portal's frame and its position
var normal = mPortalFramesNormals[i];
var framePos = mPortalFrames[i].GetPosition();
// I compute the scalar product between the normal and the direction vector between the camera's position and the frame's position
var dotProduct = normal * (currentCameraPosition - framePos);
// If the dot product is 0 or positive, the portal is visible
if(dotProduct >= 0)
{
// I render the portal
DrawPortal(mPortalFrames[i], mPortalRooms[i]);
}
}
glDisable(GL_STENCIL_TEST);
// Now I draw the portals' frames in the depth buffer, so they don't get overwritten by other objects in the scene
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glClear(GL_DEPTH_BUFFER_BIT);
for(var i = 0; i < PORTALS_NUMBER; i++)
{
mPortalFrames[i].Draw();
}
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
In my case, where I need to display 4 portals in total positioned as a cube (so I know that at most 2 portals are directly visible at a time) and only one face of the frame displays the portal while the other one donesn't, it works perfectly.
I know that there is probably a more elegant way to do this by using only the stencil buffer, but right now this works for me.
If anyone knows a better way to do this, I'm still interested in knowing!
EDIT: I've found out about the other versions of the stencil functions (i.e., glStencilFuncSeparate, glStencilOpSeparate, glStencilMaskSeparate) that allow to do different things with back and front faces. That seems like what I would need to solve the problems I described, but I won't be able to try this anytime soon. I'm just pointing it out in case someone who has my same problem wanders here.

I'm a few years late, but this question still shows up when searching for this problem, and I've had to fix it as well, so I figured I'll drop my solution for anyone else who comes by.
1. Don't clear the depth buffer (except between frames)
The main issue with your solution is that you're periodically clearing the entire depth buffer. And every time you do, you're getting rid of any information you might have to figure out which portals to draw and which are obscured. The only depth data you actually need to clobber is on the pixels where your stencils are set up; you can keep the rest.
Just before you draw the objects "through" the portal, set up your depth buffer like so:
// first, make sure you're writing to the depth buffer
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_TRUE);
glStencilMask(0x00);
// you can have the stencil enabled for this step, if you did anything fancy with it earlier (like depth testing)
glStencilFunc(GL_EQUAL, 1, 0xFF);
// the depth test has to be enabled to write to the depth mask. But don't worry; it'll always pass.
glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_ALWAYS);
// set the depth range so we only draw on the far plane, leaving a "hole" for later
glDepthRange(1, 1);
// now draw the portal object again
portalFrameObj1.Draw();
// and reset what you changed so the rest of the code works
glDepthFunc(GL_LESS);
glDepthRange(0, 1);
glColorMask(GL_TRUE,GL_TRUE,GL_TRUE,GL_TRUE);
Now, when you draw the objects "through" the portal, they'll show up where they're needed, but the rest of the screen will still have the depth information it used to! Everybody wins!
Of course, don't forget to overwrite the depth information with a third portalFrameObj1.Draw() like you've been doing. That'll help with the next part:
2. Check if your portal is obscured before setting up the stencil
At the start of the code, when you set up your stencil, you disable depth testing. You don't need to!
glStencilOp has three arguments:
sfail applies when the stencil test fails.
dpfail applies when the stencil test succeeds, but the depth test fails.
dppass applies when both the stencil and depth test succeed (or the stencil succeeds and the depth test is disabled or has no data).
Leave depth testing on. Set glStencilFunc(GL_ALWAYS, 0, 0xFF); rather than GL_NEVER, so the stencil test succeeds, and use dppass to set your stencil instead of sfail: glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
Now, you could also play around with glStencilFuncSeparate to optimize things a little, but you don't need it to get portals to obscure each other.

Related

How do values get drawn to the stencil buffer in this example?

I have a serious problem understanding how stencil buffering in OpenGL works. I followed the examples of a tutorial and the code works of course, but I really don't get at what points I'm writing to the stencil buffer. To me it seems as I'm setting up the stencil tests, masks and everything, but then I just call glDrawArrays(...)
and write the triangles to the output without applying the stencil buffer at all.
I also don't understand why I clear the buffer using glClear(GL_STENCIL_BUFFER_BIT)
before I even draw the triangles. I would be very thankful if somebody could enlighten me.
// Draws the initial cube facing upwards
glDrawArrays(GL_TRIANGLES, 0, 36);
//enable stencil testing
glEnable(GL_STENCIL_TEST);
// Draws the floor between the upward cube and downward cube
glStencilFunc(GL_ALWAYS, 1, 0xFF); /*sets the test function, mask and ref value*/
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); /*defines what happens on restults*/
glStencilMask(0xFF); /*Mask that is applied on the bits before they are written to the stencil buffer*/
glDepthMask(GL_FALSE);
glClear(GL_STENCIL_BUFFER_BIT); /*Why do I clear the buffer here?!?*/
glDrawArrays(GL_TRIANGLES, 36, 6);
// Draws the downward cube (reflection)
glStencilFunc(GL_EQUAL, 1, 0xFF);
glDepthMask(GL_TRUE);
glStencilMask(0x00);
model = glm::scale(
glm::translate(model, glm::vec3(0, 0, -1)),
glm::vec3(1, 1, -1)
);
glUniformMatrix4fv(uniModel, 1, GL_FALSE, glm::value_ptr(model));
glUniform3f(uniColor, 0.3f, 0.3f, 0.3f);
glDrawArrays(GL_TRIANGLES, 0, 36);
glUniform3f(uniColor, 1.0f, 1.0f, 1.0f);
glDisable(GL_STENCIL_TEST);
The third value passed to glStencilOp (GL_REPLACE) tells OpenGL to set the stencil buffer to the ref value previously specified in glStencilFunc whenever both the stencil test and the depth test are passed. In this case: 1.
So then the call to glDrawArrays updates the stencil buffer to 1 everyplace where it draws successfully (i.e., passes both tests).
(And the reason you clear it first, is to get rid of any 1's that happened to be there when you started: you don't want them acting as false positives where you didn't draw triangles).
See https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/glStencilOp.xml for details.
Okay I think I finally get it. And just to provide the answer to anybody else who might have the same thinking barrier I had here is what was my problem.
In my head I thought I have to write the values somehow to the stencil buffer first and afterwards I will write them to the display. Which is wrong as #NicolBolas kindly stated.
The command glDrawArrays (while stencilTesting is active) writes to the display as well as to the stencil buffer. By that for everything following we have the stencil of the in my example floor in the buffer and can use it to evaluate whether or not a pixel will be drawn. By that we can effectively make the second cube be drawn only on the Floor and not outside of it. Due to setting glStencilMask to all zero before we draw the cube we make sure it doesn't affect the stencil buffer even though the stencil test is applied on the action. If anybody else stumbles upon this I hope this might help. And if this is complete nonsense, please anybody leave a comment with the mistake in my thinking.

glDisable(GL_DEPTH_TEST) makes nothing but Sky-Sphere Render

Edit: Rendering the skybox before all other objects in the scene fixed this problem.
I've seen the question here but adding
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
doesn't appear to help.
I'm trying to render a spherical Skybox for my scene and for some reason when I Disable depth testing before doing so, the Skybox is the only thing rendered.
[Render other objects..]
// Disable depth test & mask, faceculling
glDisable(GL_DEPTH_TEST); // Adding this makes everything else invisible
glDepthMask(GL_FALSE);
glCullFace(GL_FRONT);
[Render texture onto inside of sphere..]
// Re-enable faceculling, & depth
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glCullFace(GL_BACK);
Any idea why this might be happening?
I wasn't sure of what code to include to keep this clear, so don't hesitate to ask for more.
Just to let you know, this:
glDisable(GL_DEPTH_TEST);
Disables both depth test and writing, so you don't need both that and set the depth mask to GL_FALSE.
When you clear the depth buffer each frame by default it should clear it the maximum value, probably 1.0. By default the depth function is GL_LESS meaning any depth value coming out of the fragment shader less than the one in the depth buffer passes and is written to the framebuffer.
It seems to me that what you're doing is clearing the depth buffer to 1.0, disabling depth testing and writing, drawing your objects, then enabling depth testing and writing and drawing your skybox. The problem with this is that the drawing of your objects doesn't write anything to the depth buffer, and so when it comes time to draw your skybox (with depth testing enabled) all the pixel depth values in the buffer are 1.0 (because you never wrote anything to it), and because the depth function is GL_LESS every pixel you draw of your skybox passes the depth test and is written to the framebuffer.
If there is a special need to have your objects always drawn in front of the skybox, for example the skybox follows the camera position around, then:
1) Disable the depth writing.
2) Draw the skybox.
3) Enable the depth writing.
4) Draw your objects.
Well, yes. That's what the depth test is for. Without it, there's nothing to indicate to OpenGL that the skybox shouldn't be rendered on top of everything else.
If you don't want this to happen, don't disable the depth test… or draw the skybox before everything else, instead of afterwards.

Stencil buffers seem to not work properly

I am writing an SDL2/modern OpenGL application that uses stencil buffers. I have written the following code in my renderer:
glEnable(GL_STENCIL_FUNC);
glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_NEVER, 1, 0xFF); //Always fail the stencil test
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); //Set the pixels which failed to 1
glStencilMask(0xFF);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
//Drawing small rectangle here
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 0xFF); //Only pass the stencil test where the pixel is 1 in the stencil buffer
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //Dont change the stencil buffer any further
//Drawing big rectangle here
glDisable(GL_STENCIL_FUNC);
The goal of the code above is to only draw the part of the big rectangle that fits in the small rectangle. Unfortunately, when I run the code the opposite happens, it renders the big rectangle with a hole in it the size of the small rectangle.
I have tried many more stencil functions, but they all result in the same, and this seems like it should work. So, does anybody have any ideas or can tell me where I am going wrong?
I apperantly don't have the reputation to embed pictures in my post but:
Intended result:
http://i.imgur.com/RpbHzCV.jpg
Actual result:
http://i.imgur.com/Z5qDqHk.jpg
The call glEnable(GL_STENCIL_FUNC); is just wrong, the correct enum is GL_STENCIL_TEST. So your code doesn't use the stencil buffer at all.
I can only guess why you get the result you got, then: Your code might draw the first rectangle into the depth buffer, so when you draw the second one, the fragments in that area might fail the depth test. So even when you correctly enable the stencil test, you still have to take care about the depht buffer here.

How to render a mesh behind another mesh, like a mask?

I would like it so that when mesh A (the character), is behind mesh B (a wall), it is still rendered but with a solid gray color.
I'm beginning opengles 2.0 and I'm still unsure as to go about this. From what I understand the depth buffer allows meshes to fight out who will be seen in the fragments they encompass, also there are various blend functions that could possibly involved in this, finally the stencil buffer looks like it would also have this desirable functionality.
So is there a way to output different colors through the shader based on a failed depth test? Is there a way to do this through blending? Or must I use the stencil buffer some how?
And what is this technique called for future reference? I've seen it used in a lot of video games.
This can be done using the stencil buffer. The stencil buffer gives each pixel some additional bits which can be used as a bitmask or a counter. In your case you'd configure the stencil test unit to set a specific bitmask when the depth test for the character fails (because it's obstructed by the well). Then you switch the stencil test mode operation to pass the stencil test for this specific bitmask, and render a full viewport, solid quad in the desired color, with depth testing and depth writes disabled.
Code
I strongly recommend you dive deep into the documentation for the stencil test unit. It's a very powerful mechanism, often overlooked. Your particular problem would be solved by the following. I stuggest you take this example code, read it in parallel to the stencil test functions references glStencilFunc, glStencilOp.
You must add a stencil buffer to your frame buffer's pixel format – how you do that is platform dependent. For example, if you're using GLUT, then you'd add |GLUT_STENCIL to the format bitmask of glutInitDisplayMode; on iOS you'd set a property on your GLKView; etc. Once you've added a stencil buffer, you should clear it along with your other render buffers by adding |GL_STENCIL_BUFFER_BIT to the initial glClear call of each drawing.
GLint const silouhette_stencil_mask = 0x1;
void display()
{
/* ... */
glEnable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);
glDepthFunc(GL_LESS);
glDisable(GL_STENCIL_TEST);
/* The following two are not necessary according to specification.
* But drivers can be buggy and this makes sure we don't run into
* issues caused by not wanting to change the stencil buffer, but
* it happening anyway due to a buggy driver.
*/
glStencilFunc(GL_NEVER, 0, 0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
draw_the_wall();
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_REPLACE, GL_KEEP);
draw_the_character();
glStencilFunc(GL_EQUAL, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glDisable(GL_DEPTH_TEST);
glDepthMask(GL_FALSE);
draw_full_viewport_solid_color();
/* ... */
}

Opengl Stencil Buffer set when not transparent

I was trying to configure my stencil buffer so that, when enabled, it would set when the pixel drawn is not transparent (thus creating a map of pixels that light can collide with). What I've done is:
glClearStencil(0); //clear stencil
glStencilFunc(GL_EQUAL, 0xFF, 0x000000FF); //only where alpha (mask : 0x000000FF) is 0xFF (opaque)
glStencilOp(GL_INCR, GL_KEEP, GL_KEEP); //increment if passes (if it is opaque)
render(); //withing this method I sometimes disable the whole thing to draw the floor, for example
Then, i use the following code to test:
/* TURN OFF STENCIL */
glEnable(GL_STENCIL_TEST); //re-enable
glStencilFunc(GL_EQUAL, 0, 1); //if the test is equal to 1
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //do not change stencil buffer
ImageInfo.drawColorSquare(0, 0, Configurations.SCREEN_WIDTH, Configurations.GAME_HEIGHT, Color.BLUE); //drwa blue square
glDisable(GL_STENCIL_TEST); //disable
However, there are two problems:
It doesn't seem to be ignoring transparent pixels, as it should;
If a region overlaps with another, then it reverses - for example, it sets to one, then another region is drawn in the same area, and it reset it to 0 again.
I don't know why that is happening. Probably something wrong with my mask, I guess - I wasn't absolutely sure how many pixels OpneGL used in the Color Buffer.
Also, GL_INCR should add up to the max, and not go back, according to the documentation. Since my stencil buffer size is one bit, it should set to one, try to increase again, fail, and keep on one (instead of reseting).
The Stencil Test is independent of what happens in the color buffer. Setting glStencilFunc
( http://www.opengl.org/sdk/docs/man/xhtml/glStencilFunc.xml )
you can specify how the Stencil Test interacts with what is already stored in the Stencil Buffer.
Setting glStencilOp ( http://www.opengl.org/sdk/docs/man/xhtml/glStencilOp.xml ) gives you the possibility of using the result of the depth test to perform the Stencil Test.
A good tutorial that explain the Stencil Test and a very instructive algorithm based upon it can be found here http://ogldev.atspace.co.uk/www/tutorial37/tutorial37.html
The stencil buffer is usually an 8-bit buffer that reserves a small portion of the memory normally used for the depth buffer and is used for advanced rejection of fragments; things like masking to arbitrary shapes rather than using rectangular scissor boxes. It has nothing to do with your color buffer, and to make sure that fragments that have a specific alpha value do not affect the pixels on screen you would use something called an alpha test.
In core OpenGL 3, the fixed-function alpha test is no longer supported, so you would have to implement it in a fragment shader and then discard if it failed to meet your condition.