Related
Well, I’m trying to draw tiles into a stencil buffer, but while drawing obviously happens something that I don’t understand and during the drawing disappearing (not drawn) part of the tiles.
So, how I draw:
// Enable blending.
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
// Enable testing.
glEnable(GL_STENCIL_TEST);
// Disable depth test.
glDisable(GL_DEPTH_TEST);
// Set stencil buff to 0.
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
// Here I got all visible tiles by a camera.
auto const visible_tiles = camera.visible_tiles();
// Draw tiles into a stencil.
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilMask(0xFF);
// Don't output the color.
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
// ...
// Here I had a loop where I go through each tile from all which was visible for the camera.
auto tile_id = 1;
visible_tiles.for_each_tile([this, &tile_id](auto const& tile_position) {
// Output tile ID into stencil buffer. I assume there will never be more than
// 255 tiles.
glStencilFunc(GL_ALWAYS, tile_id, 0xFF);
mesh_->draw(); // << Here I draw.
++tile_id;
});
// Disable testing.
glDisable(GL_STENCIL_TEST);
However, how my problem looks like?
]1
If I disable stencil testing at all, everything is OK.
And we draw it correctly.
UPD: With the help of the debug, I managed to narrow the circle of suspects )))
And I realized that the problem was somewhere in glEnable(GL_STENCIL_TEST), namely when I repeatedly call my method “draw(…)”.
So, I just did the following:
static bool flag = false;
if (!flag) {
glEnable(GL_STENCIL_TEST);
flag = true;
}
So, now I call glEnable(GL_STENCIL_TEST) only once, at first call of draw, and didn’t call glDisable(GL_STENCIL_TEST);
And it looks as if everything works correctly, at least now I didn’t see any defects.
But why this works?
I don't understand why those two codes doesn't yield the same result. I have enabled depth testing and stencil testing and my main loop looks like this :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
Obviously, my cube renders in this case.
However, if I use this code, nothing renders at all :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_LEQUAL, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
With only one object in my scene, both code should behave the same way. Yet, it doesn't.
I thought it had something to do with the default value in the stencil buffer. However, since I'm clearing the stencil buffer before doing this, the default value should be 0.
By the way, with this code, the cube renders as well :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_LEQUAL, 0, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
I don't understand what is happening.
glStencilFunc(GL_LEQUAL, 1, 0xFF) means if 1 <= 0, because the stencil buffer contains 0, since you cleared it. The reference value is 1, because you pass this value to the function. This always fails.
See the Khronos reference page (glStencilFunc):
GL_LEQUAL Passes if ( ref & mask ) <= ( stencil & mask ).
I want to setup a really simple two-pass effect. The first pass draws a texture object to a texture. The second pass creates a full screen quad in the geometry shader and textures it with the texture written in pass one.
The texture and framebuffer is set up in the following way:
gl.glGenFramebuffers(1, frameBufferHandle, 0);
gl.glBindFramebuffer(GL3.GL_FRAMEBUFFER, frameBufferHandle[0]);
texture = new Texture(gl, new TextureData(gl.getGLProfile(), GL3.GL_RGB, viewportWidth, viewportHeight,
0, GL3.GL_RGB, GL3.GL_UNSIGNED_BYTE, false, false, false, null, null));
texture.setTexParameteri(gl, GL3.GL_TEXTURE_MAG_FILTER, GL3.GL_LINEAR);
texture.setTexParameteri(gl, GL3.GL_TEXTURE_MIN_FILTER, GL3.GL_LINEAR);
gl.glFramebufferTexture(GL3.GL_FRAMEBUFFER, GL3.GL_COLOR_ATTACHMENT0, texture.getTextureObject(), 0);
int drawBuffers[] = {GL3.GL_COLOR_ATTACHMENT0};
gl.glDrawBuffers(1, drawBuffers, 0);
if (gl.glCheckFramebufferStatus(GL3.GL_FRAMEBUFFER) != GL3.GL_FRAMEBUFFER_COMPLETE)
throw new Exception("error while creating framebuffer");
The render function looks like:
// 1st pass
gl.glBindFramebuffer(GL3.GL_FRAMEBUFFER, frameBufferHandle[0]);
gl.glClearColor(0.2f, 0.2f, 0.2f, 1.0f);
gl.glClear(GL3.GL_STENCIL_BUFFER_BIT | GL3.GL_COLOR_BUFFER_BIT | GL3.GL_DEPTH_BUFFER_BIT);
texturePass.apply();
texturePass.updatePerObject(world);
texturePass.updateTexture(object.getDiffuseMap());
object.draw(gl);
// 2nd pass
gl.glBindFramebuffer(GL3.GL_FRAMEBUFFER, 0);
gl.glClearColor(0.2f, 0.2f, 0.2f, 1.0f);
gl.glClear(GL3.GL_STENCIL_BUFFER_BIT | GL3.GL_COLOR_BUFFER_BIT | GL3.GL_DEPTH_BUFFER_BIT);
fullscreenQuadPass.apply();
fullscreenQuadPass.updateTexture(texture)
;
gl.glDrawArrays(GL3.GL_POINTS, 0, 1);
The picture below shows the result of applying this effect:
As you hopefully can see, one can see through the golem and see his right hand. It seems like there is some kind of depth-test or transparency error.
Everything looks fine if I comment the 2nd pass out and replace
gl.glBindFramebuffer(GL3.GL_FRAMEBUFFER, frameBufferHandle[0]);
by
gl.glBindFramebuffer(GL3.GL_FRAMEBUFFER, 0);
Does anyone have an idea, what goes on here?
EDIT: In fact, I'm actually missing a depth buffer for the 2nd pass. Thus, I've updated my initialization sequence to
// Create framebuffer
gl.glGenFramebuffers(1, frameBufferHandle, 0);
gl.glBindFramebuffer(GL4.GL_FRAMEBUFFER, frameBufferHandle[0]);
// Set up color texture
colorTexture = new Texture(gl, new TextureData(gl.getGLProfile(),
GL4.GL_RGBA, width, height, 0, GL4.GL_RGBA, GL4.GL_UNSIGNED_BYTE,
false, false, false, null, null));
gl.glFramebufferTexture(GL4.GL_FRAMEBUFFER, GL4.GL_COLOR_ATTACHMENT0,
colorTexture.getTextureObject(), 0);
// Create and set up depth renderbuffer
gl.glGenRenderbuffers(GL4.GL_RENDERBUFFER, depthRenderBufferHandle, 0);
gl.glBindRenderbuffer(GL4.GL_RENDERBUFFER, depthRenderBufferHandle[0]);
gl.glRenderbufferStorage(GL4.GL_RENDERBUFFER, GL4.GL_DEPTH_COMPONENT,
width, height);
gl.glFramebufferRenderbuffer(GL4.GL_FRAMEBUFFER, GL4.GL_DEPTH_ATTACHMENT,
GL4.GL_RENDERBUFFER, depthRenderBufferHandle[0]);
int drawBuffers[] = {GL4.GL_COLOR_ATTACHMENT0};
gl.glDrawBuffers(1, drawBuffers, 0);
However, now my system crashes with a "fatal error" by the Java Runtime Environment. If I comment the newly added lines out, everything "works fine". What's the point now?
EDIT2: I've no idea why I've written
gl.glGenRenderbuffers(GL4.GL_RENDERBUFFER, depthRenderBufferHandle, 0);
Of course, it should be
gl.glGenRenderbuffers(1, depthRenderBufferHandle, 0);
That solved my problem.
Your Framebuffer Object currently lacks a depth attachment.
Here is some C pseudo-code that will address your problem:
GLuint depth_rbo;
glGenRenderbuffers (1, &depth_rbo);
glBindRenderbuffer (GL_RENDERBUFFER, depth_rbo);
glRenderbufferStorage (GL_RENDERBUFFER, GL_DEPTH_COMPONENT, width, height);
glFramebufferRenderbuffer (GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, depth_rbo);
In fact, it also lacks a stencil attachment, so I am not sure why you are clearing the stencil buffer?
If you have stencil operations to perform you will need to allocate storage for it as well. Moreover, if you need both depth and stencil in an FBO, you must use a packed Depth-Stencil format (GL_DEPTH_STENCIL_ATTACHMENT).
I'm trying to do this example.
I couldn't understand how it works still and I'm having difficulties in visualizing how the code would look like.
I stumbled upon this post which was the closest I think I could find.
From the redbook link, how can I do a "two-pass algorithm" to get the desired output? How do I use GL_INVERT and check if the pixel is covered even number of times?
I somehow managed to understand how the stencil works for the concave polygon but my problem now is the polygon is being rendered incorrectly.
http://img.photobucket.com/albums/v442/ardo/ScreenShot2013-12-30at24155PM.png
The location of where I am drawing this is in a framebuffer which is set up like this:
//multisample
glGenRenderbuffersEXT(1, &colorBuffer);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, colorBuffer);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, multisampling, GL_RGBA,800, 600);
//multi sample depth
glGenRenderbuffersEXT(1, &depthBuffer);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, depthBuffer);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, multisampling, GL_DEPTH24_STENCIL8, 800, 600);
//multisamplefbo
glGenFramebuffersEXT(1, &mFBO);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_RENDERBUFFER_EXT, colorBuffer);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER_EXT, depthBuffer);
Then my drawing code is:
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_INVERT, GL_INVERT);
glBegin(GL_TRIANGLE_FAN);
glVertex3f(1.396900,3.130690,0);
glVertex3f(2.034830,2.466900,0);
glVertex3f(2.486338,2.441036,0);
glVertex3f(2.802204,2.437803,0);
glVertex3f(2.910181,2.447098,0);
glVertex3f(2.957240,2.466900,0);
glVertex3f(3.019335,2.552570,0);
glVertex3f(3.130733,2.673799,0);
glVertex3f(3.284830,2.828970,0);
glVertex3f(3.474490,3.087590,0);
glVertex3f(4.319320,2.596210,0);
glVertex3f(4.508980,2.147930,0);
glVertex3f(4.714798,1.866680,0);
glVertex3f(4.907215,1.620586,0);
glVertex3f(5.101651,1.397930,0);
glVertex3f(5.186114,1.317364,0);
glVertex3f(5.254869,1.269024,0);
glVertex3f(5.302510,1.261700,0);
glVertex3f(5.323633,1.304180,0);
glVertex3f(5.312832,1.405254,0);
glVertex3f(5.264703,1.573711,0);
glVertex3f(5.173841,1.818340,0);
glVertex3f(5.034840,2.147930,0);
glVertex3f(4.875695,2.492816,0);
glVertex3f(4.727900,2.780675,0);
glVertex3f(4.591307,3.016304,0);
glVertex3f(4.465765,3.204503,0);
glVertex3f(4.351127,3.350070,0);
glVertex3f(4.247241,3.457803,0);
glVertex3f(4.153958,3.532502,0);
glVertex3f(4.071129,3.578965,0);
glVertex3f(3.998605,3.601990,0);
glVertex3f(3.936235,3.606376,0);
glVertex3f(3.841363,3.578427,0);
glVertex3f(3.766900,3.510000,0);
glVertex3f(2.974490,3.458280,0);
glVertex3f(2.672770,2.958280,0);
glVertex3f(2.379670,3.070350,0);
glVertex3f(1.853810,3.303110,0);
glVertex3f(1.396910,3.303110,0);
glVertex3f(1.396910,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glEnd();
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glColor3f(1,0,1);
glBegin(GL_TRIANGLE_FAN);
glVertex3f(1.396900,3.130690,0);
glVertex3f(2.034830,2.466900,0);
glVertex3f(2.486338,2.441036,0);
glVertex3f(2.802204,2.437803,0);
glVertex3f(2.910181,2.447098,0);
glVertex3f(2.957240,2.466900,0);
glVertex3f(3.019335,2.552570,0);
glVertex3f(3.130733,2.673799,0);
glVertex3f(3.284830,2.828970,0);
glVertex3f(3.474490,3.087590,0);
glVertex3f(4.319320,2.596210,0);
glVertex3f(4.508980,2.147930,0);
glVertex3f(4.714798,1.866680,0);
glVertex3f(4.907215,1.620586,0);
glVertex3f(5.101651,1.397930,0);
glVertex3f(5.186114,1.317364,0);
glVertex3f(5.254869,1.269024,0);
glVertex3f(5.302510,1.261700,0);
glVertex3f(5.323633,1.304180,0);
glVertex3f(5.312832,1.405254,0);
glVertex3f(5.264703,1.573711,0);
glVertex3f(5.173841,1.818340,0);
glVertex3f(5.034840,2.147930,0);
glVertex3f(4.875695,2.492816,0);
glVertex3f(4.727900,2.780675,0);
glVertex3f(4.591307,3.016304,0);
glVertex3f(4.465765,3.204503,0);
glVertex3f(4.351127,3.350070,0);
glVertex3f(4.247241,3.457803,0);
glVertex3f(4.153958,3.532502,0);
glVertex3f(4.071129,3.578965,0);
glVertex3f(3.998605,3.601990,0);
glVertex3f(3.936235,3.606376,0);
glVertex3f(3.841363,3.578427,0);
glVertex3f(3.766900,3.510000,0);
glVertex3f(2.974490,3.458280,0);
glVertex3f(2.672770,2.958280,0);
glVertex3f(2.379670,3.070350,0);
glVertex3f(1.853810,3.303110,0);
glVertex3f(1.396910,3.303110,0);
glVertex3f(1.396910,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glEnd();
glDisable(GL_STENCIL_TEST);
The way I have my multisampling set up is here (This is before the drawing is written)
[self checkError];
glEnable(GL_DEPTH_TEST);
[self checkError];
glDepthMask(GL_TRUE);
[self checkError];
glDepthFunc(GL_LESS);
[self checkError];
glClearDepth(10000.0);
[self checkError];
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
[self checkError];
if(multisampling != 0){
//Set multisampled framebuffer
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
//Antialiasing functions
glEnable (GL_POLYGON_SMOOTH);
glEnable (GL_BLEND);
glBlendFunc (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glHint (GL_POLYGON_SMOOTH_HINT, GL_NICEST);
glLineWidth (1.5);
}
glDisable(GL_CULL_FACE);
And my blit call:
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, mFBO);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, currentFramebuffer);
glBlitFramebuffer(0, 0, 800, 600, 0, 0, 800, 600, GL_COLOR_BUFFER_BIT, GL_NEAREST);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glReadBuffer(GL_COLOR_ATTACHMENT0_EXT);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, currentFramebuffer);
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, currentFramebuffer);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, currentFramebuffer);
What don't you get? The example starts out with a stencil buffer cleared to 0x00 and each time you draw a fragment (whether it passes or fails a depth/stencil test) it performs a bitwise inversion of the stencil buffer (e.g. ~0x00 --> 0xff). If you do this an odd number of times, the stencil buffer will be non-zero, but if you have an even number of fragments it will be zero.
Some things that might be confusing are the use of 1 as the bitmask for the stencil test and the stencil op in the second pass. This effectively limits the test to a single bit. In other words, the bit flipping I mentioned earlier will only happen for bit 1. Thus, you can actually simplify your test for even/odd to a test of a single bit... if the stencil buffer stores a value of 1 then you drew an odd number of fragments. If it stores a value of 0 then you drew an even number.
The second pass in your example actually does the exact stencil test I described. It tests the stencil buffer for 1 and fails the stencil test if the first bit != 1.
// The stencil op below is a convoluted way of clearing the stencil buffer
glStencilOp (GL_ZERO, GL_ZERO, GL_ZERO);
glStencilFunc (GL_EQUAL, 1, 1); // Test: (Stencil & 1) == 1
I am trying to figure out the best way to mask of sections of a texture when they ar drawn. My issue comes in the fact that I seem to have run our of alpha masks!
We are using openGL to draw a custom built 2D game engine. The game is built up off of sprites and simple block textures.
My desired outcome is like this:
A character sprite is drawn in place (using it's alpha color to not just be a box)
An item is drawn into the players hand (also using it's alpha color to draw into the scene without being a box)
The item should appear behind the characters arm/hand, but above the rest of the body.
For the moment the only way I can figure out how to accomplish this, is by drawing them in order (Body, Item, Arm) but I would like to avoid this to make art assets a bit easier to deal with. My idea solution would be to draw the character, then draw the item with an alpha mask that blocks out areas of the texture that should be "under" the arm.
Other solutions that I have seen are like this, where the glBlendFuncSeparate() function is used. I am trying to avoid bringing in extensions, as my current version of OpenGL doesn't support it. Not to say that I am opposed to the idea, but it seems a bit of a handle to brig it in just to draw an alpha mask?
I fully admit that this is a learning process for me, and I am using it as an excuse to really see how OpenGL handles. Any suggestions as to where I should head to get this to draw correctly? Is there a way for OpenGL in the fixed pipeline to take a texture, apply an alpha mask on top of it, and THEN draw it into the buffer? Should I give in and separate my character into several parts of its model?
[UPDATE: 8/12/12]
Tried to add the code suggested by Tim, but I seem to be having an issue. When I enable the stencil buffer, everything just gets blocked out, NOT just what I wanted. Here is my test example code.
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// Disable writing to any of the color fields
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
glStencilFunc(GL_ALWAYS, 0,0);
// Draw our blocking poly
glBegin(GL_POLYGON);
glVertex2f( 50, 50 );
glVertex2f( 50, 50+128 );
glVertex2f( 50+128, 50+128 );
glEnd();
glStencilFunc(GL_GREATER, 0, -1);
glEnable(GL_STENCIL_TEST);
// Re enable drawing of colors
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
// Enable use of textures
glEnable(GL_TEXTURE_2D);
// Bind desired texture for drawing
glBindTexture(GL_TEXTURE_2D,(&texture)[0]);
// Draw the box with colors
glBegin(GL_QUADS);
glTexCoord2d( 0, 0 ); glVertex2f( 50, 50 );
glTexCoord2d( 0, 1 ); glVertex2f( 50, 50+128 );
glTexCoord2d( 1, 1 ); glVertex2f( 50+128, 50+128 );
glTexCoord2d( 1, 0 ); glVertex2f( 50+128, 50 );
glEnd();
// Swap buffers and display!
SDL_GL_SwapBuffers();
Just to be clear, here is my init code as well to set this system up.
When the code is run with stencil disabled, I get this:
When I use glEnable(GL_STENCIL_TEST), I get this:
I've tried playing around with various options, but I cannot see a clear reason why my stencil buffer is blocking everything.
[Update#2 8/12/12]
We got some working code, Thanks tim! Here is what I ended up running to work correctly.
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// Disable writing to any of the color fields
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilOp(GL_INCR, GL_INCR, GL_INCR);
glEnable(GL_STENCIL_TEST);
// Draw our blocking poly
glBegin(GL_POLYGON);
glVertex2f( 50, 50 );
glVertex2f( 50, 50+128 );
glVertex2f( 50+128, 50+128 );
glEnd();
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
// Re enable drawing of colors
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
// Enable use of textures
glEnable(GL_TEXTURE_2D);
// Bind desired texture for drawing
glBindTexture(GL_TEXTURE_2D,(&texture)[0]);
// Draw the box with colors
glBegin(GL_QUADS);
glTexCoord2d( 0, 0 ); glVertex2f( 50, 50 );
glTexCoord2d( 0, 1 ); glVertex2f( 50, 50+128 );
glTexCoord2d( 1, 1 ); glVertex2f( 50+128, 50+128 );
glTexCoord2d( 1, 0 ); glVertex2f( 50+128, 50 );
glEnd();
glDisable(GL_STENCIL_TEST);
// Swap buffers and display!
SDL_GL_SwapBuffers();
Here's my idea for the situation where you have one texture and one alpha mask:
Draw the character onto the scene like normal.
Lock the RGB color channels so that it cannot be changed with glColorMask
Setup the stencil buffer with glStencilOp(GL_KEEP, GL_KEEP, GL_INCR); glStencilFunc(GL_ALWAYS, 0,0);
Draw the alpha mask with alpha testing enabled. This will increment the stencil buffer anywhere the alpha test passes (you may have to flip this based on your mask polarity)
At this point, you have a character texture in the framebuffer, and a mask outline in the stencil buffer.
Reenable the color channels with glColorMask
Setup the stencil buffer for the weapon with glStencilFunc(GL_GREATER, 0, -1); This will only draw the weapon texels where the stencil buffer is greater than zero, and reject pixels where the stencil is not updated.
Draw the weapon texture as normal.
Tim was pretty clear in his comment, but I want to present you the solution I find the most intuitive. It's 3D, so hold on... ;)
Basically, you can just use the Z coordinate of your images to create virtual "layers". It then doesnt' matter, in which order you draw them. Just alphatest every image individually, and draw it on correct Z value. If it still isn't enough, you could use separate texture containing "depth" of every pixel, and then use the 2nd texture to perform some sort of depth-testing.
Be sure to call glEnable(GL_DEPTH_TEST); if you want to use this approach.
As I see it, the problem is that you have one texture, but part of it represents the arm and part of it the rest of the character. The issue is that you want to draw the weapon over the character, but draw the arm over both.
This means, while drawing two objects, you want to put them into three different "layers". This fundamentally doesn't make sense, so you're kind of stuck.
Here's an idea though: use a fragment program (i.e., a shader).
I suggest you overload the character's texture's alpha channel to encode both transparency and layer. For example, let's use 0=transparent body, 64=opaque body, 128=transparent arm, 255=opaque arm.
From here, you draw your objects, but conditionally set the depth of your objects into three layers. Basically, you write a fragment program that draws your character into two different layers, the character gets pushed backward while the arm gets pulled forward. When the weapon is drawn, it is drawn without a shader, but it's tested against the characters' pixels' depths. It works something like this (untested, obviously).
Define a shader my_shader, which contains a fragment program:
uniform sampler2D character_texture;
void main(void) {
vec4 sample = texture2D(character_texture,gl_TexCoord[0].st);
int type; //Figure out what type of character texel we're looking at
if (fabs(sample.a-0.00)<0.01) type = 0; //transparent body
else if (fabs(sample.a-0.25)<0.01) type = 1; //opaque body
else if (fabs(sample.a-0.50)<0.01) type = 2; //transparent arm
else if (fabs(sample.a-1.00)<0.01) type = 3; //opaque arm
//Don't draw transparent pixels.
if (type==0 || type==2) discard;
gl_FragColor = vec4(sample.rgb,1.0);
//Normally, you (can) write "gl_FragDepth = gl_FragCoord.z". This
//is how OpenGL will draw your weapon. However, for the character,
//we alter that so that the arm is closer and the body is farther.
//Move body farther
if (type==1) gl_FragDepth = gl_FragCoord.z * 1.1;
//Move arm closer
else if (type==3) gl_FragDepth = gl_FragCoord.z * 0.9;
}
Here's some pseudocode for your draw function:
//...
//Algorithm to draw your character
glUseProgram(my_shader);
glBindTexture(GL_TEXTURE_2D,character.texture.texture_gl_id);
glUniform1i(glGetUniformLocation(my_shader,"character_texture"),1);
character.draw();
glUseProgram(0);
//Draw your weapon
glEnable(GL_DEPTH_TEST);
character.weapon.draw();
glDisable(GL_DEPTH_TEST);
//...