So, I'm making a 2D game and I have some textures. I will like some of them to drop a shadow, there is something like drop-shadow in css for SDL2?
Render the texture first, then render a slightly larger semi-transparent gray square slightly behind it. If you want rounded corners, use a shader that increases alpha as you get further from the corners.
Since noone mentionned it yet, here it is:
int SDL_SetTextureColorMod(SDL_Texture* texture,
Uint8 r,
Uint8 g,
Uint8 b)
https://wiki.libsdl.org/SDL_SetTextureColorMod
This function multiplies a texture color channels when copying it to the SDL_Renderer*. Using it with 0 as r, g and b arguments would make your texture pitch black but not affect the alpha, so the texture would keep its shape (like in the case of a transparent PNG). You just have to copy that shadow before (= behind) your texture, with a slight offset. You can also change the overall transparency of the shadow, with SDL_SetTextureAlphaMod(SDL_Texture* texture, Uint8 a)
Just don't forget to set the values back to 255 when you're done.
Code example:
SDL_SetRenderDrawColor(renderer, 0, 0, 0, 255);
SDL_RenderClear(renderer);
// [...] draw background, etc... here
SDL_Rect characterRect;
// [...] fill the rect however you want
SDL_Rect shadowRect(characterRect);
shadowRect.x += 30; // offsets the shadow
shadowRect.y += 30;
SDL_SetTextureColorMod(characterTexture, 0, 0, 0); // makes the texture black
SDL_SetTextureAlphaMod(characterTexture, 191); // makes the texture 3/4 of it's original opacity
SDL_RenderCopy(renderer, characterTexture, NULL, &shadowRect); // draws the shadow
SDL_SetTextureColorMod(characterTexture, 255, 255, 255); // sets the values back to normal
SDL_SetTextureAlphaMod(characterTexture, 255);
SDL_RenderCopy(renderer, characterTexture, NULL, &characterRect); // draws the character
// [...] draw UI, additionnal elements, etc... here
SDL_RenderPresent(renderer);
I am using an FBO (or "Render Texture") which has an alpha channel (32bpp ARGB) and clear that with a color that is not fully opaque, for example (R=1, G=0, B=0, A=0) (i.e. completely transparent). Then I am rendering a translucent object, for example a rectangle with color (R=1, G=1, B=1, A=0.5), on top of that. (All values normalized from 0 to 1)
According to common sense, as well as imaging software such as GIMP and Photoshop, as well as several articles on Porter-Duff compositing, I would expect to get a texture that is
fully transparent outside of the rectangle
white (1.0, 1.0, 1.0) with 50 % opacity inside the rectangle.
Like so (you won't see this on the SO website):
Instead, the background color RGB values, which are (1.0, 0.0, 0.0) are weighted overall with (1 - SourceAlpha) instead of (DestAlpha * (1 - SourceAlpha)). The actual result is this:
I have verified this behavior using OpenGL directly, using SDL's wrapper API, and using SFML's wrapper API. With SDL and SFML I have also saved the results as an image (with alpha channel) instead of merely rendering to the screen to be sure that it's not a problem with the final rendering step.
What do I need to do to produce the expected SourceOver result, either with SDL, SFML, or using OpenGL directly?
Some sources:
W3 article on compositing, specifies co = αs x Cs + αb x Cb x (1 – αs), weight of Cb should be 0 if αb is 0, no matter what.
English Wiki shows destination ("B") being weighted according to αb (as well as αs, indirectly).
German Wiki shows 50% transparency examples, clearly the transparent background's original RGB values do not interfere with either the green or the magenta source, also shows that the intersection is clearly asymmetric in favor of the element that is "on top".
There are also several questions on SO that seemingly deal with this at first glance, but I could not find anything that talks abut this specific issue. People suggest different OpenGL blending functions, but the general consensus seems to be glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA), which is what both SDL and SFML use by default. I have also tried different combinations with no success.
Another suggested thing is premultiplying the color with the destination alpha, since OpenGL can only have 1 factor, but it needs 2 factors for correct SourceOver. However, I cannot make sense of that at all. If I'm premultiplying (1, 0, 0) with the destination alpha value of, say, (0.1), I get (0.1, 0, 0) (as suggested here for example). Now I can tell OpenGL the factor GL_ONE_MINUS_SRC_ALPHA for this (and source with just GL_SRC_ALPHA), but then I'm effectively blending with black, which is incorrect. Though I am not a specialist on the topic, I put a fair amount of effort into trying to understand (and at least got to the point where I managed to program a working pure software implementation of every compositing mode). My understanding is that applying an alpha value of 0.1 "via premultiplication" to (1.0, 0.0, 0.0) is not at all the same as treating the alpha value correctly as the fourth color component.
Here is a minimal and complete example using SDL. Requires SDL2 itself to compile, optionally SDL2_image if you want to save as PNG.
// Define to save the result image as PNG (requires SDL2_image), undefine to instead display it in a window
#define SAVE_IMAGE_AS_PNG
#include <SDL.h>
#include <stdio.h>
#ifdef SAVE_IMAGE_AS_PNG
#include <SDL_image.h>
#endif
int main(int argc, char **argv)
{
if (SDL_Init(SDL_INIT_VIDEO) != 0)
{
printf("init failed %s\n", SDL_GetError());
return 1;
}
#ifdef SAVE_IMAGE_AS_PNG
if (IMG_Init(IMG_INIT_PNG) == 0)
{
printf("IMG init failed %s\n", IMG_GetError());
return 1;
}
#endif
SDL_Window *window = SDL_CreateWindow("test", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, 800, 600, SDL_WINDOW_OPENGL | SDL_WINDOW_SHOWN);
if (window == NULL)
{
printf("window failed %s\n", SDL_GetError());
return 1;
}
SDL_Renderer *renderer = SDL_CreateRenderer(window, 1, SDL_RENDERER_ACCELERATED | SDL_RENDERER_TARGETTEXTURE);
if (renderer == NULL)
{
printf("renderer failed %s\n", SDL_GetError());
return 1;
}
// This is the texture that we render on
SDL_Texture *render_texture = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_TARGET, 300, 200);
if (render_texture == NULL)
{
printf("rendertexture failed %s\n", SDL_GetError());
return 1;
}
SDL_SetTextureBlendMode(render_texture, SDL_BLENDMODE_BLEND);
SDL_SetRenderDrawBlendMode(renderer, SDL_BLENDMODE_BLEND);
printf("init ok\n");
#ifdef SAVE_IMAGE_AS_PNG
uint8_t *pixels = new uint8_t[300 * 200 * 4];
#endif
while (1)
{
SDL_Event event;
while (SDL_PollEvent(&event))
{
if (event.type == SDL_QUIT)
{
return 0;
}
}
SDL_Rect rect;
rect.x = 1;
rect.y = 0;
rect.w = 150;
rect.h = 120;
SDL_SetRenderTarget(renderer, render_texture);
SDL_SetRenderDrawColor(renderer, 255, 0, 0, 0);
SDL_RenderClear(renderer);
SDL_SetRenderDrawColor(renderer, 255, 255, 255, 127);
SDL_RenderFillRect(renderer, &rect);
#ifdef SAVE_IMAGE_AS_PNG
SDL_RenderReadPixels(renderer, NULL, SDL_PIXELFORMAT_ARGB8888, pixels, 4 * 300);
// Hopefully the masks are fine for your system. Might need to randomly change those ff parts around.
SDL_Surface *tmp_surface = SDL_CreateRGBSurfaceFrom(pixels, 300, 200, 32, 4 * 300, 0xff0000, 0xff00, 0xff, 0xff000000);
if (tmp_surface == NULL)
{
printf("surface error %s\n", SDL_GetError());
return 1;
}
if (IMG_SavePNG(tmp_surface, "t:\\sdltest.png") != 0)
{
printf("save image error %s\n", IMG_GetError());
return 1;
}
printf("image saved successfully\n");
return 0;
#endif
SDL_SetRenderTarget(renderer, NULL);
SDL_SetRenderDrawColor(renderer, 255, 255, 255, 255);
SDL_RenderClear(renderer);
SDL_RenderCopy(renderer, render_texture, NULL, NULL);
SDL_RenderPresent(renderer);
SDL_Delay(10);
}
}
Thanks to #HolyBlackCat and #Rabbid76 I was able to shed some light on this entire thing. I hope this can help out other people who want to know how about correct alpha blending and the details behind premultiplied alpha.
The basic problem is that correct "Source Over" alpha blending in actually not possible with OpenGL's built-in blend functionality (that is glEnable(GL_BLEND), glBlendFunc[Separate](...), glBlendEquation[Separate](...)) (this is the same for D3D by the way). The reason is the following:
When calculating the result color and alpha values of the blending operation (according to correct Source Over), one would have to use these functions:
Each RGB color values (normalized from 0 to 1):
RGB_f = ( alpha_s x RGB_s + alpha_d x RGB_d x (1 - alpha_s) ) / alpha_f
The alpha value (normalized from 0 to 1):
alpha_f = alpha_s + alpha_d x (1 - alpha_s)
Where
sub f is the result color/alpha,
sub s is the source (what is on top) color/alpha,
d is the destionation (what is on the bottom) color/alpha,
alpha is the processed pixel's alpha value
and RGB represents one of the pixel's red, green, or blue color values
However, OpenGL can only handle a limited variety of additional factors to go with the source or destination values (RGB_s and RGB_d in the color equation) (see here), the relevant ones in this case being GL_ONE, GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA. We can specify the alpha formula correctly using those options, but the best we can do for RGB is:
RGB_f = alpha_s x RGB_s + RGB_d x (1 - alpha_s)
Which completely lacks the destination's alpha component (alpha_d). Note that this formula is equivalent to the correct one if \alpha_d = 1. In other words, when rendering onto a framebuffer which has no alpha channel (such as the window backbuffer), this is fine, otherwise it will produce incorrect results.
To solve that problem and achieve correct alpha blending if alpha_d is NOT equal to 1, we need some gnarly workarounds. The original (first) formula above can be rewritten to
alpha_f x RGB_f = alpha_s x RGB_s + alpha_d x RGB_d x (1 - alpha_s)
if we accept the fact that the result color values will be too dark (they will be multiplied by the result alpha color). This gets rid of the division already. To get the correct RGB value, one would have to divide the result RGB value by the result alpha value, however, as it turns out that conversion usually never needed. We introduce a new symbol (pmaRGB) which denotes RGB values which are generally too dark because they have been multiplied by their corresponding pixel's alpha value.
pmaRGB_f = alpha_s x RGB_s + alpha_d x RGB_d x (1 - alpha_s)
We can also get rid of the problematic alpha_d factor by ensuring that ALL of the destination image's RGB values have been multiplied with their respective alpha values at some point. For example, if we wanted the background color (1.0, 0.5, 0, 0.3), we do not clear the framebuffer with that color, but with (0.3, 0.15, 0, 0.3) instead. In other words, we are doing one of the steps that the GPU would have to do already in advance, because the GPU can only handle one factor. If we are rendering to an existing texture, we have to ensure that it was created with premultiplied alpha. The result of our blending operations will always be textures that also have premultiplied alpha, so we can keep rendering things onto there and always be sure that the destination does have premultiplied alpha. If we are rendering to a semi-transparent texture, the semi-transparent pixels will always be too dark, depending on their alpha value (0 alpha meaning black, 1 alpha meaning the correct color). If we are rendering to a buffer which has no alpha channel (like the back buffer we use for actually displaying things), alpha_f is implicitly 1, so the premultiplied RGB values are equal to the correctly blended RGB values. This is the current formula:
pmaRGB_f = alpha_s x RGB_s + pmaRGB_d x (1 - alpha_s)
This function can be used when the source does not yet have premultiplied alpha (for example, if the source is a regular image that came out of an image processing program, with an alpha channel that is correctly blended with no premultiplied alpha).
There is a reason we might want to get rid of \alpha_s as well, and use premultiplied alpha for the source as well:
pmaRGB_f = pmaRGB_s + pmaRGB_d x (1 - alpha_s)
This formula needs to be taken if the source happens to have premultiplied alpha - because then the source pixel values are all pmaRGB instead of RGB. This is always going to be the case if we are rending to an offscreen buffer with an alpha channel using the above method. It may also be reasonable to have all texture assets stored with premultiplied alpha by default so that this formula can always be taken.
To recap, to calculate the alpha value, we always use this formula:
alpha_f = alpha_s + alpha_d x (1 - alpha_s)
, which corresponds to (GL_ONE, GL_ONE_MINUS_SRC_ALPHA). To calculate the RGB color values, if the source does not have premultiplied alpha applied to its RGB values, we use
pmaRGB_f = alpha_s x RGB_s + pmaRGB_d x (1 - alpha_s)
, which corresponds to (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA). If it does have premultiplied alpha applied to it, we use
pmaRGB_f = pmaRGB_s + pmaRGB_d x (1 - alpha_s)
, which corresponds to (GL_ONE, GL_ONE_MINUS_SRC_ALPHA).
What that practically means in OpenGL: When rendering to a framebuffer with alpha channel, switch to the correct blending function accordingly and make sure that the FBO's texture always has premultiplied alpha applied to its RGB values. Note that the correct blending function may potentially be different for each rendered object, according to whether or not the source has premultiplied alpha. Example: We want a background [1, 0, 0, 0.1], and render an object with color [1, 1, 1, 0.5] onto it.
// Clear with the premultiplied version of the real background color - the texture (which is always the destination in all blending operations) now complies with the "destination must always have premultiplied alpha" convention.
glClearColor(0.1f, 0.0f, 0.0f, 0.1f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
//
// Option 1 - source either already has premultiplied alpha for whatever reason, or we can easily ensure that it has
//
{
// Set the drawing color to the premultiplied version of the real drawing color.
glColor4f(0.5f, 0.5f, 0.5f, 0.5f);
// Set the blending equation according to "blending source with premultiplied alpha".
glEnable(GL_BLEND);
glBlendFuncSeparate(GL_ONE, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
glBlendEquationSeparate(GL_ADD, GL_ADD);
}
//
// Option 2 - source does not have premultiplied alpha
//
{
// Set the drawing color to the original version of the real drawing color.
glColor4f(1.0f, 1.0f, 1.0f, 0.5f);
// Set the blending equation according to "blending source with premultiplied alpha".
glEnable(GL_BLEND);
glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
glBlendEquationSeparate(GL_ADD, GL_ADD);
}
// --- draw the thing ---
glDisable(GL_BLEND);
In either case, the resulting texture has premultiplied alpha. Here are 2 possibilities what we might want to do with this texture:
If we want to export it as an image that is correctly alpha blended (as per the SourceOver definition), we need to get its RGBA data and explicitly divide each RGB value by the corresponding pixel's alpha value.
If we want to render it onto the backbuffer (whose background color shall be (0, 0, 0.5)), we proceed as we would normally (for this example, we additionally want to modulate the texture with (0, 0, 1, 0.8)):
// The back buffer has 100 % alpha.
glClearColor(0.0f, 0.0f, 0.5f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
// The color with which the texture is drawn - the modulating color's RGB values also need premultiplied alpha
glColor4f(0.0f, 0.0f, 0.8f, 0.8f);
// Set the blending equation according to "blending source with premultiplied alpha".
glEnable(GL_BLEND);
glBlendFuncSeparate(GL_ONE, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
glBlendEquationSeparate(GL_ADD, GL_ADD);
// --- draw the texture ---
glDisable(GL_BLEND);
Technically, the result will have premultiplied alpha applied to it. However, because the result alpha will always be 1 for each pixel, the premultiplied RGB values are always equal to the correctly blended RGB values.
To achieve the same in SFML:
renderTexture.clear(sf::Color(25, 0, 0, 25));
sf::RectangleShape rect;
sf::RenderStates rs;
// Assuming the object has premultiplied alpha - or we can easily make sure that it has
{
rs.blendMode = sf::BlendMode(sf::BlendMode::One, sf::BlendMode::OneMinusSrcAlpha);
rect.setFillColor(sf::Color(127, 127, 127, 127));
}
// Assuming the object does not have premultiplied alpha
{
rs.blendMode = sf::BlendAlpha; // This is a shortcut for the constructor with the correct blending parameters for this type
rect.setFillColor(sf::Color(255, 255, 255, 127));
}
// --- align the rect ---
renderTexture.draw(rect, rs);
And the likewise to draw the renderTexture onto the backbuffer
// premultiplied modulation color
renderTexture_sprite.setColor(sf::Color(0, 0, 204, 204));
window.clear(sf::Color(0, 0, 127, 255));
sf::RenderStates rs;
rs.blendMode = sf::BlendMode(sf::BlendMode::One, sf::BlendMode::OneMinusSrcAlpha);
window.draw(renderTexture_sprite, rs);
Unfortunately, this is not possible with SDL afaik (at least not on the GPU as part of the rendering process). Unlike SFML, which exposes fine-grained control over the blending mode to the user, SDL does not allow setting the individual blending function components - it only has SDL_BLENDMODE_BLEND hardcoded with glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA).
I am working on a game project using LibGDX. Now I'm facing a problem I can't understand; let me explain what I've got before coming to the actual problem:
Static background C (tiled)
Moving background B (tiled)
Moving background A (tiled)
Characters, entities
Walls (tiled)
HUD
The render loop goes as follows:
Render 2 to 5 in previous list to a transparent FBO
Render the lightmap to another FBO (using box2dLights)
Blend both products together using a custom shader
Draw the static background
Draw the blended texture from step 3
Draw the HUD
The problem I'm facing is that I obtain a white sprite when rendering background B (not a white block; it is more like a rgba(1, 1, 1, x). Then the background B is blended correctly with the lightmap (render step 3), but I cannot get its real color, just white:
The background B should appear dark blue and with a texture; it appears as a white sprite (blended with the lightmap). Behind background A, at the left of the picture (and also where the "SHADOW" text is), you can still see background B in a purplish tone, result of the blending with the light map, but please note it has no texture and it seems to me like if the texture is just plain white + alpha.
The render code:
#Override
public void render(float delta) {
Gdx.gl.glClearColor(0f, 0f, 0f, 1f);
Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);
/* 1. Update Physics and lights */
if (worldFixedStep(delta)) {
rayHandler.update();
}
updateCameras();
cameraMatrixCopy.set(camera.combined);
rayHandler.setCombinedMatrix(cameraMatrixCopy.scale(Globals.BOX_TO_WORLD, Globals.BOX_TO_WORLD, 1.0f), camera.position.x,
camera.position.y, camera.viewportWidth * camera.zoom * Globals.BOX_TO_WORLD,
camera.viewportHeight * camera.zoom * Globals.BOX_TO_WORLD);
rayHandler.render();
final Texture lightMap = rayHandler.getLightMapTexture();
fbo.begin();
{
Gdx.gl.glClearColor(0f, 0f, 0f, 0f);
Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);
/* 2. Draw the backgrounds */
batch.enableBlending();
batch.setBlendFunction(GL20.GL_ONE, GL20.GL_ONE_MINUS_SRC_ALPHA);
batch.setProjectionMatrix(bgCamera2.combined); //Background B
batch.begin();
bTileMapRenderer.setView(bgCamera2);
bTileMapRenderer.renderTileLayer(tilesBg2Layer);
batch.end();
batch.setProjectionMatrix(bgCamera1.combined); //Background A
batch.begin();
bTileMapRenderer.setView(bgCamera1);
bTileMapRenderer.renderTileLayer(tilesBg1Layer);
batch.end();
batch.setBlendFunction(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA); //FIXME this is where to touch to avoid gray alpha'ed borders.
batch.setProjectionMatrix(camera.combined);
batch.begin();
(...) //Draw everything else that needs to be blended with the light map
batch.end();
}
fbo.end();
/* 3. Render the static background (background C) */
batch.setProjectionMatrix(bgCameraStatic.combined);
batch.disableBlending();
batch.begin();
bTileMapRenderer.setView(bgCameraStatic);
bTileMapRenderer.renderTileLayer(tilesBg3Layer);
batch.end();
/* 4. Blend the frame buffer's texture with the light map in a fancy way */
Gdx.gl20.glActiveTexture(GL20.GL_TEXTURE0);
fboRegion.getTexture().bind();
Gdx.gl20.glActiveTexture(GL20.GL_TEXTURE1);
lightMap.bind();
Gdx.gl20.glEnable(Gdx.gl20.GL_BLEND);
Gdx.gl20.glBlendFunc(Gdx.gl20.GL_SRC_ALPHA, Gdx.gl20.GL_ONE_MINUS_SRC_ALPHA);
lightShader.begin();
lightShader.setUniformf("ambient_color", bgColor[0], bgColor[1], bgColor[2]);
lightShader.setUniformi("u_texture0", 0);
lightShader.setUniformi("u_texture1", 1);
fullScreenQuad.render(lightShader, GL20.GL_TRIANGLE_FAN, 0, 4);
lightShader.end();
Gdx.gl20.glDisable(Gdx.gl20.GL_BLEND);
hud.draw();
}
I just can't understand why this background is drawn white but still with its alpha data. The image is a premultiplied alpha texture, but again I cannot see why could this affect the color rendering.
Any help would be much appreciated.
Cheers
I have an DirectX 11 C++ application that displays two rectangles with textures and some text.
Both textures are taken from TGA resources (with alpha channel added).
When I run the program, I get the result:
What's wrong? Take a closer look:
The corners of the rectangles are transparent (and they should be). The rest of textures are set to be 30% opacity (and it works well too).
But, when one texture (let's call it texture1) is over another (texture2):
The corners of texture1 are transparent.
But behind them I see the background of window, instead of texture2.
In other words, transparency of texture interacts with the background of window, not with the textures behind it.
What have I done wrong? What part of my program can be responsible for it? Blending options, render states, shader code...?
In my shader, I set:
technique10 RENDER{
pass P0{
SetVertexShader(CompileShader( vs_4_0, VS()));
SetPixelShader(CompileShader( ps_4_0, PS()));
SetBlendState(SrcAlphaBlendingAdd, float4(0.0f, 0.0f, 0.0f, 0.0f),
0xFFFFFFFF);
}
}
P.s.
Of course, when I change the background of window from blue to another colour, the elements still have the transparency (the corners aren't blue).
edit:
According to #ComicSansMS (+ for nick, anyway ;p ), I've tried to change to order of render elements (I've also moved the smaller texture a bit, to check if the error remains):
The smaller texture is now behind the bigger one. But the problem with corners remains (now it appears on the second texture). I am almost sure that I draw the rectangle behind BEFORE I render the rectangle above (I see the code's lines order).
My depth stencil:
//initialize the description of the stencil state
ZeroMemory(depthStencilsDescs, sizeof(*depthStencilsDescs));
//set up the description of the stencil state
depthStencilsDescs->DepthEnable = true;
depthStencilsDescs->DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
depthStencilsDescs->DepthFunc = D3D11_COMPARISON_LESS;
depthStencilsDescs->StencilEnable = true;
depthStencilsDescs->StencilReadMask = 0xFF;
depthStencilsDescs->StencilWriteMask = 0xFF;
//stencil operations if pixel is front-facing
depthStencilsDescs->FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
depthStencilsDescs->FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_INCR;
depthStencilsDescs->FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
depthStencilsDescs->FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
//stencil operations if pixel is back-facing
depthStencilsDescs->BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
depthStencilsDescs->BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_DECR;
depthStencilsDescs->BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
depthStencilsDescs->BackFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
//create the depth stencil state
result = device->CreateDepthStencilState(depthStencilsDescs, depthStencilState2D);
The render function:
...
//clear the back buffer
context->ClearRenderTargetView(myRenderTargetView, backgroundColor); //backgroundColor
//clear the depth buffer to 1.0 (max depth)
context->ClearDepthStencilView(depthStencilView, D3D11_CLEAR_DEPTH, 1.0f, 0);
context->OMSetDepthStencilState(depthStencilState2D, 1);
context->VSSetShader(getVertexShader(), NULL, 0);
context->PSSetShader(getPixelShader(), NULL, 0);
for(...){
rectangles[i]->render();
}
The blend state:
D3D11_BLEND_DESC blendDesc;
ZeroMemory(&blendDesc, sizeof(D3D11_BLEND_DESC) );
blendDesc.AlphaToCoverageEnable = false;
blendDesc.IndependentBlendEnable = false;
blendDesc.RenderTarget[0].BlendEnable = true;
blendDesc.RenderTarget[0].SrcBlend = D3D11_BLEND_SRC_ALPHA;
blendDesc.RenderTarget[0].DestBlend = D3D11_BLEND_INV_SRC_ALPHA;
blendDesc.RenderTarget[0].BlendOp = D3D11_BLEND_OP_ADD;
blendDesc.RenderTarget[0].SrcBlendAlpha = D3D11_BLEND_ONE;
blendDesc.RenderTarget[0].DestBlendAlpha = D3D11_BLEND_ONE;
blendDesc.RenderTarget[0].BlendOpAlpha = D3D11_BLEND_OP_ADD;
blendDesc.RenderTarget[0].RenderTargetWriteMask = D3D11_COLOR_WRITE_ENABLE_ALL ;
ID3D11BlendState * blendState;
if (FAILED(device->CreateBlendState(&blendDesc, &blendState))){
}
context->OMSetBlendState(blendState,NULL,0xffffffff);
Your draw order is probably wrong.
Blending does not interact with the pixels of the object behind it, it interacts with the pixels that are currently in the frame buffer.
So if you draw the rectangle in front before the rectangle in the back, its blending operation will interact with what is in the frame buffer at that point (that is, the background).
The solution is obviously to sort your objects by their depth in view space and draw from back-to-front, although that is sometimes easier said than done (especially when allowing arbitrary overlaps).
The other problem seems to be that you draw both rectangles at the same depth value. Your depth test is set to D3D11_COMPARISON_LESS, so as soon as a triangle is drawn on a pixel, the other triangle will fail the depth test for that pixel and won't get drawn at all. This is explains the results you get when swapping the drawing order.
Note that if you draw objects back-to-front there is no need to perform a depth test at all, so you might just want to switch it off in this case. Alternatively, you can try to arrange the objects along the depth axis by giving them different Z values, or just switch to D3D11_COMPARISON_LESS_EQUAL for the depth test.