I know this problem may seem silly, but I'm having problems when using alpha channel values to blend textures, if my alpha values goes from 1.0 to nearly 0.501, the object fades slowly , once it gets to 0.5 or lower it simply vanishes. Here's two print screens that shows it:
Alpha set to 0.501
Alpha set to 0.5
I wanted to be able to see the tree above even at around 0.1 alpha, even though it would be barely visible and mostly transparent, instead of it just vanishing suddenly. As follows is my current code for my blend state:
D3D11_BLEND_DESC bd;
ZeroMemory(&bd, sizeof(D3D11_BLEND_DESC));
bd.RenderTarget->BlendEnable = true;
bd.AlphaToCoverageEnable = true;
bd.RenderTarget->RenderTargetWriteMask = D3D11_COLOR_WRITE_ENABLE_ALL;
bd.RenderTarget->SrcBlend = D3D11_BLEND_SRC_ALPHA;
bd.RenderTarget->DestBlend = D3D11_BLEND_INV_SRC_ALPHA;
bd.RenderTarget->SrcBlendAlpha = D3D11_BLEND_INV_DEST_ALPHA;
bd.RenderTarget->DestBlendAlpha = D3D11_BLEND_ONE;
bd.RenderTarget->BlendOp = D3D11_BLEND_OP_ADD;
bd.RenderTarget->BlendOpAlpha = D3D11_BLEND_OP_ADD;
hr = m_pDevice->CreateBlendState(&bd, &m_pBlendStateON);
if (FAILED(hr))
return Log("Failed to create blend state."); // Log is just a function to register errors on my app.
My blend factor is defined as follows: float BlendFactor[4] = { 0,0,0,0 };, and my sample mask uses the default value UINT SampleMask = 0xffffffff;
If anyone knows what I could do to make the transparency fade slowly from 1.0f to 0.0f values it would be a great help.
EDIT: I've found out that if I disable AlphaToCoverageEnable it will not cull colors until it gets to a 0.0f alpha value, but don't know what to do because I need the AlphaToCoverageEnable or else this squares from the tree's branches will show up, is there any way I can change the threshold so AlphaToCoverageEnable only culls the colors if alpha is actually 0.0f?
Related
I am totally lost now. Have been trying to read the backbuffer inside a vertex shader for days with no luck whatsoever.
I'm trying to read the vertexes position from the backbuffer and it's neighboring pixels. (I'm trying to count how many black pixels are around a vertex, and if there are any color that vertex red in the pixel shader). I've created a separate ID3D11Texture2D and an SRV to go with the backBuffer. I copy the backbuffer into this SRV's resource. Bind the SRV using VSSetShaderResources but just can't seem to be able to read from it inside the vertex shader.
I will share some code here from the creation of these elements as well as include some RenderDoc screenshots that keep showing that the SRV is being bound to the VS stage and has the right texture associated with it but every Load or []operator or tex2dlod or SampleLevel(i bound a SamplerState too)
just keeps returning a single 1.0 value with the rest of the float4 never being returned, meaning i only get a float1 back. I will also include a renderdoc capture file if anyone wants to take a look.
This is a simple scene from tutorial 42 on the rastertek.com site, there is a ground plane with a cube and a sphere on it :
https://i.imgur.com/cbVC48E.gif
// Here is some code when creating the secondary texture and SRV that houses a //backBuffer
// Get the pointer to the back buffer.
result = m_swapChain->GetBuffer(0, __uuidof(ID3D11Texture2D), (LPVOID*)&backBufferPtr);
if(FAILED(result))
{
MessageBox((*(hwnd)), L"Get the pointer to the back buffer FAILED", L"Error", MB_OK);
return false;
}
// Create another texture2d that we will use to make an SRV out of, and this texture2d will be used to copy the backbuffer to so we can read it in a shader
D3D11_TEXTURE2D_DESC bbDesc;
backBufferPtr->GetDesc(&bbDesc);
bbDesc.MipLevels = 1;
bbDesc.ArraySize = 1;
bbDesc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
bbDesc.Usage = D3D11_USAGE_DEFAULT;
bbDesc.MiscFlags = 0;
bbDesc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
result = m_device->CreateTexture2D(&bbDesc, NULL, &m_backBufferTx2D);
if (FAILED(result))
{
MessageBox((*(m_hwnd)), L"Create a Tx2D for backbuffer SRV FAILED", L"Error", MB_OK);
return false;
}
D3D11_SHADER_RESOURCE_VIEW_DESC descSRV;
ZeroMemory(&descSRV, sizeof(descSRV));
descSRV.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
descSRV.ViewDimension = D3D11_SRV_DIMENSION_TEXTURE2D;
descSRV.Texture2D.MipLevels = 1;
descSRV.Texture2D.MostDetailedMip = 0;
result = GetDevice()->CreateShaderResourceView(m_backBufferTx2D, &descSRV, &m_backBufferSRV);
if (FAILED(result))
{
MessageBox((*(m_hwnd)), L"Creating BackBuffer SRV FAILED.", L"Error", MB_OK);
return false;
}
// Create the render target view with the back buffer pointer.
result = m_device->CreateRenderTargetView(backBufferPtr, NULL, &m_renderTargetView);
First I render the scene in all white and then I copy that to the SRV and bind it for the next shader that's supposed to sample it. I'm expecting to get a float4(1.0, 1.0, 1.0, 1.0) value returned when i sample the backbuffer with the vertex's on screen position
https://i.imgur.com/N9CYg8c.png
as shown on the top left in the event browser, there were three drawindexed calls for rendering everything in white and then a CopyResource.
I've selected the next (fourth) DrawIndexed and on the right side outlined in red are the inputs for this next shader clearly showing that the backBuffer has been successfully bound to the vertex shader.
And now for the part that's giving me trouble
https://i.imgur.com/ENuXk0n.png
I'm gonna be debugging this top-left vertex as shown on the screenshot,
the vertex Shader has a
Texture2D prevBackBuffer: register(t0);
written at the top
https://i.imgur.com/8cihNsq.png
When trying to sample the left neighboring pixel
this line of code returns newCoord = float2(158, 220)
when entering these pixel values in the texture view i get this pixel
https://i.imgur.com/DT72Fl1.png
so the coordinates are ok so far, and as outlined i'm expecting to get a float4(0.0, 0.0, 0.0, 1,0) returned when i sample this pixel
(I'm trying to count how many black pixels are around a vertex, and if there are any color that vertex red in the pixel shader)
AND YET, when i sample that pixel right after altering the pixel coordinates since load counts pixels from bottom left so i need
newCoord = float2(158, 379), i get this
https://i.imgur.com/8SuwOzz.png
why is this, even if it's out of range, load should return all zeros, since I'm not sure about the whole load counts from bottom left thing I tried sampling using the top left coordinates (158, 220) but end up getting 0.0, ?, ?, ?
I'm completely stumped and have no idea what to try next. I've tried using a sample state :
// Create a clamp texture sampler state description.
samplerDesc.Filter = D3D11_FILTER_MIN_MAG_MIP_LINEAR;
samplerDesc.AddressU = D3D11_TEXTURE_ADDRESS_CLAMP;
samplerDesc.AddressV = D3D11_TEXTURE_ADDRESS_CLAMP;
samplerDesc.AddressW = D3D11_TEXTURE_ADDRESS_CLAMP;
samplerDesc.MipLODBias = 0.0f;
samplerDesc.MaxAnisotropy = 1;
samplerDesc.ComparisonFunc = D3D11_COMPARISON_ALWAYS;
samplerDesc.BorderColor[0] = 0;
samplerDesc.BorderColor[1] = 0;
samplerDesc.BorderColor[2] = 0;
samplerDesc.BorderColor[3] = 0;
samplerDesc.MinLOD = 0;
samplerDesc.MaxLOD = D3D11_FLOAT32_MAX;
// Create the texture sampler state.
result = device->CreateSamplerState(&samplerDesc, &m_sampleStateClamp);
but still never get a proper float4 back when reading the texture.
Any ideas, suggestions, I'll take anything at this point.
Oh and here's a RenderDoc file of the frame i was examining :
http://www.mediafire.com/file/1bfiqdpjkau4l0n/my_capture.rdc/file
So from my experience, reading from the back buffer is not really an operation that you want to be doing in the first place. If you have to do any operation on the rendered scene, the best way to do that is to render the scene to an intermediate texture, perform the operation on that texture, then render the final scene to the back buffer. This is generally how things like dynamic shadows are done - the scene is rendered from the perspective of the light, and the resulting buffer is interpreted to get a shadow value that is then applied to the final scene (this is also why dynamic light sources are limited in commercial game engines - they're rather expensive to use).
A similar idea can be applied here. First, render the whole scene to an intermediate texture, bound as a render target view (where the pixel format is specified by you, the programmer). Next, rebind that intermediate texture as a shader resource view, and render the scene again, using the edge detection shader and the real back buffer (where the pixel format is defined by the hardware).
This, fundamentally, is what I believe the issue is - a back buffer is a device dependent resource, and its format can change depending on the hardware. Therefore, using it from a shader is not safe, as you don't always know what the format will be. A device independent resource, on the other hand, will always have the same format, and you can safely use it however you like from a shader.
I wasn't able to get sampling an SRV in the vertex shader to work
but what i was able to get working
is using a backBuffer.SampleLevel inside a compute shader
I also had to change the sampler to something like this :
D3D11_SAMPLER_DESC samplerDesc;
samplerDesc.Filter = D3D11_FILTER_MIN_MAG_MIP_POINT;
samplerDesc.AddressU = D3D11_TEXTURE_ADDRESS_BORDER;
samplerDesc.AddressV = D3D11_TEXTURE_ADDRESS_BORDER;
samplerDesc.AddressW = D3D11_TEXTURE_ADDRESS_BORDER;
samplerDesc.MipLODBias = 0.0f;
samplerDesc.MaxAnisotropy = 1;
samplerDesc.ComparisonFunc = D3D11_COMPARISON_ALWAYS;
samplerDesc.BorderColor[0] = 0.5f;
samplerDesc.BorderColor[1] = 0.5f;
samplerDesc.BorderColor[2] = 0.5f;
samplerDesc.BorderColor[3] = 0.5f;
samplerDesc.MinLOD = 0;
samplerDesc.MaxLOD = 0;
I'm currently working on a D3D project and want to implement directional shadow mapping. I set everything up according to the Microsoft Guide, but it just doesn't work.
I've created a 2D texture object, a depth stencil view and a shader resource view and set them up using the following descriptions:
D3D11_TEXTURE2D_DESC shadowMapDesc;
ZeroMemory(&shadowMapDesc, sizeof(D3D11_TEXTURE2D_DESC));
shadowMapDesc.Width = width;
shadowMapDesc.Height = height;
shadowMapDesc.MipLevels = 1;
shadowMapDesc.ArraySize = 1;
shadowMapDesc.Format = DXGI_FORMAT_R24G8_TYPELESS;
shadowMapDesc.SampleDesc.Count = 1;
shadowMapDesc.SampleDesc.Quality = 0;
shadowMapDesc.Usage = D3D11_USAGE_DEFAULT;
shadowMapDesc.BindFlags = D3D11_BIND_DEPTH_STENCIL | D3D11_BIND_SHADER_RESOURCE;
shadowMapDesc.CPUAccessFlags = 0;
shadowMapDesc.MiscFlags = 0;
ID3D11Device& d3ddev = dev.getD3DDevice();
uint32_t *initData = new uint32_t[width * height];
ZeroMemory(initData, sizeof(uint32_t) * width * height);
D3D11_SUBRESOURCE_DATA data;
ZeroMemory(&data, sizeof(D3D11_SUBRESOURCE_DATA));
data.pSysMem = initData;
data.SysMemPitch = sizeof(uint32_t) * width;
data.SysMemSlicePitch = 0;
HRESULT hr = d3ddev.CreateTexture2D(&shadowMapDesc, &data, &texture_);
D3D11_DEPTH_STENCIL_VIEW_DESC depthStencilViewDesc;
ZeroMemory(&depthStencilViewDesc, sizeof(D3D11_DEPTH_STENCIL_VIEW_DESC));
depthStencilViewDesc.Format = DXGI_FORMAT_D24_UNORM_S8_UINT;
depthStencilViewDesc.ViewDimension = D3D11_DSV_DIMENSION_TEXTURE2D;
depthStencilViewDesc.Texture2D.MipSlice = 0;
hr = d3ddev.CreateDepthStencilView(texture_, &depthStencilViewDesc, &stencilView_);
D3D11_SHADER_RESOURCE_VIEW_DESC shaderResourceViewDesc;
ZeroMemory(&shaderResourceViewDesc, sizeof(D3D11_SHADER_RESOURCE_VIEW_DESC));
shaderResourceViewDesc.Format = DXGI_FORMAT_R24_UNORM_X8_TYPELESS;
shaderResourceViewDesc.ViewDimension = D3D11_SRV_DIMENSION_TEXTURE2D;
shaderResourceViewDesc.Texture2D.MipLevels = 1;
shaderResourceViewDesc.Texture2D.MostDetailedMip = 0;
hr = d3ddev.CreateShaderResourceView(texture_, &shaderResourceViewDesc, &shaderView_);
Between these steps there is additional error checking, but all the create-functions return successfully. I then bind the texture, render my scene and unbind the texture using the following functions:
void D3DDepthTexture2D::bindAsTarget(D3DDevice& dev)
{
dev.getDeviceContext().ClearDepthStencilView(stencilView_, D3D11_CLEAR_DEPTH | D3D11_CLEAR_STENCIL, 1.0f, 0);
// Bind target
dev.getDeviceContext().OMSetRenderTargets(0, 0, stencilView_);
// Set viewport
dev.setViewport(static_cast<float>(width_), static_cast<float>(height_), 0.0f, 0.0f);
}
void D3DDepthTexture2D::unbindAsTarget(D3DDevice& dev, float width, float height)
{
// Unbind target
dev.resetRenderTarget();
// Reset viewport
dev.setViewport(width, height, 0.0f, 0.0f);
}
My render-to-depth-texture routine basically looks like this (removing all the unnecessary details):
camera = buildCameraFromLight(light);
setCameraCBuffer(camera);
bindTexture();
activateShader();
for(Object j : objects) {setTransformationCBuffer(j); renderObject(j);}
deactivateShader();
unbindTexture();
Rendering the scene from the light's perspective to the normal render target (screen) results in the proper image (both the actual image and just rendering the depth values). I use a simple vertex shader that just transforms the vertices and a pixel shader that does nothing at all OR returns the depth values (I tried both, doesn't change anything about the end result since we don't care about the color buffer).
After clearing the texture and rendering to it, I render it onto a quad to my screen, but all I get is a red square - so the depth value is 1.0f, the value I've cleared the texture to. I'm really at a loss for what to do, I tried everything, implemented every possible solution from online tutorials or changed things around on my own, but nothing helps. Here's a list of all the things I already checked:
All FAILED(hr)-calls return false, no error message is printed to the console
I tested whether the geometry gets transformed properly by rendering the geometry and their depth values (z / w) to screen, which worked and looked correct
I tested calculating the depth values in the fragment shader and rendering to a normal render target (basically trying to render my color buffer to texture) instead of a depth stencil texture, but that didn't work either, red square
I tested different formats and format combinations for the shadow map and the views, which either caused the creation to fail or didn't change a thing
I checked whether any call between setting and unsetting my texture as the render target during the render call resetted the depth stencil target to something else - not the case
I debugged my texture-to-screen/quad rendering routine already and it works properly with other textures, so I am in fact seeing what the depth texture looks like
I changed the geometry and camera perspective around to see whether that makes anything visible in the depth texture - it doesn't
I came across this similar StackOverflow problem and checked whether my default depth stencil buffer had the same dimensions, AA settings etc. as my texture - and it does (count 1, quality 0)
I really don't know what's up, I've been trying to debug this for hours and hours. I hope someone here can give me any advice on what I'm doing wrong or what I could try to fix this. I'm using C++11 with Direct3D11.
Note: I can't debug any of this using NSight or any Visual Studio tools since they don't seem to work properly with my system right now and I don't have any administrative rights to fix any of it. I just have to deal with it for now. I hope the given information and code samples are enough to provide a rough idea of what I could also try to make this work.
Thanks in advance.
I got NSight to work and debugged the whole thing with that. Turns out the depth texture was properly created and filled with the depth and stencil data and I just forgot that all the depth information is stored in the first channel - so I ignored the g and b data and used 1.0 for a and it worked. Using the g and b channels somehow made the whole thing red (maybe someone wants to add to this and explain why).
Debugging this got much easier once I could observe the texture that is present in the shader - I should've used a debugging tool like NSight or RenderDoc way earlier. Thanks to #EgorShkorov for the advice.
I'm trying to set up a two-stage render of objects in a 3D engine I'm working on written in C++ with DirectX9 to facilitate transparency (and other things). I thought it was all working nicely until I noticed some dodgyness on the edge of objects rendered before objects using this two stage method.
The two stage method is simple:
Draw model to off-screen ("side") texture of same size using same zbuffer (no MSAA is used anywhere)
Draw off-screen ("side") texture over the top of the main render target with a suitable blend and no alpha test or write
In the image below the left view is with the two stage render of the gray object (a lamppost) with the body in-front of it rendered directly to the target texture. The right view is with the two-stage render disabled, so both are rendered directly onto the target surface.
On close inspection it is as if the side texture is offset by exactly 1 pixel "down" and 1 pixel "right" when rendered over the target surface (but is rendered correctly in-place). This can be seen in an overlay of the off screen texture (which I get my program to write out to a bitmap file via D3DXSaveTextureToFile) over a screen shot below.
One last image so you can see where the edge in the side texture is coming from (it's because rendering to the side texture does use z test). Left is screen short, right is side texture (as overlaid above).
All this leads me to believe that my "overlaying" isn't very effective. The code that renders the side texture over the main render target is shown below (note that the same viewport is used for all scene rendering (on and off screen)). The "effect" object is an instance of a thin wrapper over LPD3DXEFFECT, with the "effect" field (sorry about shoddy naming) being a LPD3DXEFFECT itself.
void drawSideOver(LPDIRECT3DDEVICE9 dxDevice, drawData* ddat)
{ // "ddat" drawdata contains lots of render state information, but all we need here is the handles for the targetSurface and sideSurface
D3DXMATRIX idMat;
D3DXMatrixIdentity(&idMat); // create identity matrix
dxDevice->SetRenderTarget(0, ddat->targetSurface); // switch to targetSurface
dxDevice->SetRenderState(D3DRS_ZENABLE, false); // disable z test and z write
dxDevice->SetRenderState(D3DRS_ZWRITEENABLE, false);
vertexOver overVerts[4]; // create square
overVerts[0] = vertexOver(-1, -1, 0, 0, 1);
overVerts[1] = vertexOver(-1, 1, 0, 0, 0);
overVerts[2] = vertexOver(1, -1, 0, 1, 1);
overVerts[3] = vertexOver(1, 1, 0, 1, 0);
effect.setTexture(ddat->sideTex); // use side texture as shader texture ("tex")
effect.effect->SetTechnique("over"); // change to "over" technique
effect.setViewProj(&idMat); // set viewProj to identity matrix so 1/-1 map directly
effect.effect->CommitChanges();
setAlpha(dxDevice); // this sets up the alpha blending which works fine
UINT numPasses, pass;
effect.effect->Begin(&numPasses, 0);
effect.effect->BeginPass(0);
dxDevice->SetVertexDeclaration(vertexDecOver);
dxDevice->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP, 2, overVerts, sizeof(vertexOver));
effect.effect->EndPass();
effect.effect->End();
dxDevice->SetRenderState(D3DRS_ZENABLE, true); // revert these so we don't mess everything up drawn after this
dxDevice->SetRenderState(D3DRS_ZWRITEENABLE, true);
}
The C++ side definition for the VertexOver struct and constructor (HLSL side shown below somewhere):
struct vertexOver
{
public:
float x;
float y;
float z;
float w;
float tu;
float tv;
vertexOver() { }
vertexOver(float xN, float yN, float zN, float tuN, float tvN)
{
x = xN;
y = yN;
z = zN;
w = 1.0;
tu = tuN;
tv = tvN;
}
};
Inefficiency in re-creating and passing the vertices down to the GPU each draw aside, what I really want to know is why this method doesn't quite work, and if there are any better methods for overlaying textures like this with an alpha blend that won't exhibit this issue
I figured that the texture sampling may matter somewhat in this matter, but messing about with options didn't seem to help much (for example, using a LINEAR filter just makes it fuzzy as you might expect implying that the offset isn't as clear-cut as a 1 pixel discrepancy). Shader code:
struct VS_Input_Over
{
float4 pos : POSITION0;
float2 txc : TEXCOORD0;
};
struct VS_Output_Over
{
float4 pos : POSITION0;
float2 txc : TEXCOORD0;
float4 altPos : TEXCOORD1;
};
struct PS_Output
{
float4 col : COLOR0;
};
Texture tex;
sampler texSampler = sampler_state { texture = <tex>;magfilter = NONE; minfilter = NONE; mipfilter = NONE; AddressU = mirror; AddressV = mirror;};
// side/over shaders (these make up the "over" technique (pixel shader version 2.0)
VS_Output_Over VShade_Over(VS_Input_Over inp)
{
VS_Output_Over outp = (VS_Output_Over)0;
outp.pos = mul(inp.pos, viewProj);
outp.altPos = outp.pos;
outp.txc = inp.txc;
return outp;
}
PS_Output PShade_Over(VS_Output_Over inp)
{
PS_Output outp = (PS_Output)0;
outp.col = tex2D(texSampler, inp.txc);
return outp;
}
I've looked about for a "Blended Blit" or something but I can't find anything, and other related searches have only brought up forums implying that rendering a quad with an orthographic projection is the way to go about doing this.
Sorry if I've given far too much detail for this issue but it's both interesting and infuriating and any feedback would be greatly appreciated.
It looks for me that you problem is the mapping of texels to pixels. You must offset a screen-aligned quad with a half pixel to match the texels direct to the screenpixels. This issue is explaines here: Directly Mapping Texels to Pixels (MSDN)
For anyone else hitting a similar wall, my specific problem solved by adjusting the U and V values of the verticies sent to the GPU for the overlaid texture triangles thus:
for (int i = 0; i < 4; i++)
{
overVerts[i].tu += 0.5 / (float)ddat->targetVp->Width; // ddat->targetVp is the viewport in use, and the viewport is the same size as the texture
overVerts[i].tv += 0.5 / (float)ddat->targetVp->Height;
}
See Directly Mapping Texels to Pixels as provided by Gnietschow's answer for an explanation as to why this makes sense.
I have a texture and was curious as to what the texture sampler should be for sampling the sprite texture? I am using DirectX11, though if you know what it should be for DX9/10, I believe it is transferable.
I tried
AddressU = D3D11_TEXTURE_ADDRESS_WRAP
AddressV = D3D11_TEXTURE_ADDRESS_WRAP
AddressW = D3D11_TEXTURE_ADDRESS_WRAP
ComparisonFunc = D3D11_COMPARISON_NEVER
Filter = D3D11_FILTER_MIN_MAG_MIP_POINT
MaxAnisotropy = 1;
MaxLOD = D3D11_FLOAT32_MAX;
MinLOD = 0;
MipLODBias = 0;
Although when rendering, there appeared to be artifacts and it did not seem as clear as it should be.
This is an example of what the artifcats are. The top text with a light blue background you can see artifacts (for example, the A and C). The bottom text with the black background is the origin image.
I'm trying to replicate the Photoshop filter multiply with Direct3D. I've been reading and googling about the different render states and I've got the effect almost working. The problem is that it's ignoring the alpha value of the textures.
Here's an image that explains the sitution:
http://www.kloonigames.com/petri/stackoverflow_doesnt_allow_.jpg
I found one solution to this, which was to save the images with no transparency and white background. But I'm not satisfied with this solution. The problem is that I really need to use the alpha value. I want to fade out the images gradually. And I cannot do this if the blending mode is ignoring the alpha value.
So the question is how to render the images with alpha?
Here's the blending mode code:
dev->SetRenderState(D3DRS_ALPHABLENDENABLE, TRUE);
dev->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_ZERO);
dev->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_SRCCOLOR);
Edit added the SetTextureStageState
dev->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
dev->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
dev->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);
dev->SetTextureStageState(0, D3DTSS_ALPHAARG2, D3DTA_DIFFUSE);
You can achieve this effect in one step by premultipling alpha in your pixel shader, or by using textures with pre-multiplied alpha.
For example if you have 3 possible blend operations for a shader, and you want each one to take alpha into account.
Blend = ( src.rgb * src.a ) + ( dest.rgb * (1-src.a) )
Add = ( src.rgb * src.a ) + ( dest.rgb )
Multiply = (src.rgb * dest.rgb * src.a) + (dest.rgb * (1-src.a) )
You'll notice that Multiply is impossible with a single pass because there are two operations on the source color. But if you premultiply alpha in your shader you can extract the alpha component from the blending operation and it becomes possible to blend all three operations in the same shader.
In your pixel shader you can pre-multiply alpha manually. Or use a tool like DirectXTex texconv to modify your textures.
return float4(color.rgb*color.a, color.a);
The operations become:
Blend = ( src.rgb ) + ( dest.rgb * (1-src.a) )
Add = ( src.rgb ) + ( dest.rgb )
Multiply = ( src.rgb * dest.rgb ) + (dest.rgb * (1-src.a) )
It sounds like you want:
dst.rgb = (src.a * src.rgb) * ((1 - src.a) * dst.rgb)
You would use D3DRS_BLENDOP to do that, but unfortunately there isn't a D3DBLENDOP_MULTIPLY. I don't think this operation is possible without a fragment shader.
OK this is not as simple as you would think. I would use an Effect & two renderTargets for this...
I'm amusing your using one render pass to try to do this, which will not work.
Photoshop has layers & each layers have an alpha channel. BTW it would be nice to know what kind of app your making.
So first in D3D I would create 2 RGBA_32bit renderTargets of the same size as your window & clear them to color white. Make it an array like so (new RenderTarget[2];) for swapping.
Now set the blending state to (AlphaFunc=Add, Src=SrcAlpha, Dst=InvSrcAlpha). For the first circle you draw it into renderTarget[0] using renderTarget[1] as a texture/sampler input source. You will render the circle with an Effect that will take the circles color & multiply it with renderTarget[1]'s sampler color. After you draw circle one you swap the renderTarget[0] with renderTarget[1] by simple indexing, so now renderTarget[1] is the one you draw to & renderTarget[0] is the one you sample from. Then you repeat the drawing process for circle 2 & so on.
After you draw ever circle you copy the last drawn renderTarget to the backBuffer & present the scene.
Here is an example of logically how you would do it. If you need reference for coding http://www.codesampler.com/ is a good place.
void TestLayering()
{
bool rtIndex = false;
RenderTarget* renderTarget = new RenderTarget[2];
Effect effect = new Effect("multiplyEffect.fx");
effect.Enable();
BlendingFunc = Add;
BlendingSource = SrcAlpha;
BlendingDest = InvSrcAlpha;
for(int i = 0; i != circleCount; ++i)
{
renderTarget[rtIndex].EnableAsRenderTarget();
renderTarget[!rtIndex].EnableAsSampler();
circle[i].Draw();
rtIndex = !rtIndex;
}
//Use D3D9's StretchRect for this...
backBuffer.CopyFromSurface(renderTarget[rtIndex]);
}
//Here is the effects pixel shader
float4 PS_Main(InStruct In) : COLOR
{
float4 backGround = tex2D(someSampler, In.UV);
return circleColor * backGround;
}
dev->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_DESTCOLOR);
dev->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);
Will do the trick. You cannot use the 'alpha' from the diffuse vertex color anymore though. Setting a low alpha on the vertex colors will actually brighten your overlaying pixels.