I am writing a DLL plugin that will read the desktop frame buffer (whole screen) and render it directly into a Texture2D pointer that is passed in. The goal is to keep everything in video memory (and avoid the cost of copying back to system memory and back to video memory).
I am able to pass the Texture2D (showing up as a ID3D11Texture2D), but I am having issues grabbing the desktop frame buffer with D3D11. D3D9 offered GetFrontBufferData() but it seems D3D11 solution is to use GetBuffer().
My issue is about getting the IDXGISwapChain. Because I want to read the desktop frame buffer and simply read the content via GetBuffer() and display it on the ID3D11Texture2D. I am getting a ID3D11Device I do not understand how to get its IDXGISwapChain.
I have code as follows to get the frame buffer and put it on the Texture:
ID3D11Texture2D* src = (ID3D11Texture2D*)g_TexturePointer;
ID3D11Texture2D* dst = NULL;
HRESULT hr = swapchain->GetBuffer(0, __uuidof(ID3D11Texture2D), (LPVOID*)&dst);
g_devCon->CopyResource(dst, src);
Here I actually created my own swapchain using D3D11CreateDeviceAndSwapChain() but I wonder if it is necessary as I already have a ID3D11Device.
The CopyResource() also seems to fail.
Not necessarily all desktop content would be rendered with D3D11. DXGI is the underlying system for all graphics on Windows, so you will definitely need to use it in some way to get a capture of the desktop. However, D3D11 is built on DXGI (for example, ID3D11Texture2D supports the IDXGIResource interface). The code sample below shows how you can capture the output of an entire monitor into a D3D11 staging texture:
// IDXGIOutput* poutput = ...; // from DXGIAdapter::EnumOutputs.
// Get description of desktop.
DXGI_OUTPUT_DESC outdesc;
poutput->GetDesc(&outdesc);
// Create destination texture, sized same as desktop.
D3D11_TEXTURE2D_DESC texDesc;
memset(&texDesc, 0, sizeof(texDesc));
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
texDesc.BindFlags = 0;
texDesc.Width = outdesc.DesktopCoordinates.right - outdesc.DesktopCoordinates.left;
texDesc.Height = outdesc.DesktopCoordinates.bottom - outdesc.DesktopCoordinates.top;
texDesc.MipLevels = 1;
texDesc.SampleDesc = { 1, 0 };
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.ArraySize = 1;
ID3D11Texture2D* destinationTexture = 0;
pDevice->CreateTexture2D(&texDesc, 0, &destinationTexture); // check HRESULT.
// Get IDXGIResource from texture.
IDXGIResource* destinationResource=0;
destinationTexture->QueryInterface(IID_PPV_ARGS(&destinationResource)); // check HRESULT.
// Get data.
IDXGIOutput1* poutput1;
poutput->QueryInterface(IID_PPV_ARGS(&poutput1)); // check HRESULT.
poutput1->TakeOwnership(pDevice, TRUE);
poutput1->GetDisplaySurfaceData1(destinationResource); // check HRESULT.
poutput1->ReleaseOwnership();
// Now use destinationTexture, it contains the contents of the desktop.
Unfortunately, it has the nasty side effect of turning the output black during the IDXGIOutput::TakeOwnership call. However, without this call, the GetDiplaySurfaceData1 will fail. Depending on your situation, this may be acceptable.
Related
I am currently trying to load images from disk into memory using WIC. I used the MSDN documentation here to write my code.
Everything works fine for loading PNG images. Loading JPEG images works without any error but does not produce the correct result! The interesting thing is, that when I convert the image from JPEG to PNG (using irfan view) the error persists.
Consider the following test image:
As you can see the image got shrunk in the x direction and all color information is gone. However, when you zoom in you can see that there is still some color present, but it doesn’t look like expected:
(Problems persist when uploading the image as texture to the GPU)
I have striped out my WIC loading code omitting error handling and resource freeing for readability:
// CoInitialize(NULL) called in main
// WIC Factory
IWICImagingFactory* ptrFactory = NULL;
CoCreateInstance(CLSID_WICImagingFactory, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&ptrFactory));
// Open decoder
IWICBitmapDecoder* ptrDecoder = NULL;
ptrFactory->CreateDecoderFromFilename(fileName, NULL, GENERIC_READ, WICDecodeMetadataCacheOnDemand, &ptrDecoder)
// Get first frame
IWICBitmapFrameDecode* ptrFrameDecoder = NULL;
ptrDecoder->GetFrame(0, &ptrFrameDecoder));
// Read formate
WICPixelFormatGUID frameNativeFormat;
ptrFrameDecoder->GetPixelFormat(&frameNativeFormat);
// Computing target format selectConverterWic(...) and selectConverterDx(...) are just some very basic functions for selecting the right convert to formate by iterating over an std::array witch contains the mappings suggested by the MSDC article mentioned above
WICPixelFormatGUID targetFormat;
DXGI_FORMAT targetFormatDx;
if (!selectConverterDx(frameNativeFormat, &targetFormatDx)) {
// Try to find WIC to WIC converter
selectConverterWic(frameNativeFormat, &targetFormat)
selectConverterDx(targetFormat, &targetFormatDx)
}else {
memcpy(&targetFormat, &frameNativeFormat, sizeof(GUID));
}
// Get format info
IWICComponentInfo* ptrCmpInfo = NULL;
ptrFactory->CreateComponentInfo(targetFormat, &ptrCmpInfo);
// Get type
WICComponentType ptrCmpType;
ptrCmpInfo->GetComponentType(&ptrCmpType);
// Get info from type
IWICPixelFormatInfo* ptrFormatInfo = NULL;
ptrCmpInfo->QueryInterface(IID_PPV_ARGS(&ptrFormatInfo));
// Get BBP
UINT uiBitsPerPixel;
ptrFormatInfo->GetBitsPerPixel(&uiBitsPerPixel);
// ID3D12Device->CheckFeatureSupport(...) and fallback omitted
// Image size
UINT width, height;
ptrFrameDecoder->GetSize(&width, &height);
// Tempory memory allocation
UINT rowPitch = (width * uiBitsPerPixel + 7) / 8;
SIZE_T imageSize = (SIZE_T)rowPitch * (SIZE_T)height;
void* workMemory = malloc(imageSize);
// Check if direct copy is possible
if (memcmp(&frameNativeFormat, &targetFormat, sizeof(GUID)) == 0){
ptrFrameDecoder->CopyPixels(NULL, rowPitch, (UINT)imageSize, (BYTE*)workMemory);
}else{
// Format conversion (Got never hit by a jpeg file; I tried to force it but results weren't right asswell)
IWICFormatConverter* ptrFormatConverter = NULL;
ptrFactory->CreateFormatConverter(&ptrFormatConverter);
ptrFormatConverter->Initialize(ptrFrameDecoder, targetFormat, WICBitmapDitherTypeErrorDiffusion, NULL, 0, WICBitmapPaletteTypeCustom);
ptrFormatConverter->CopyPixels(NULL, rowPitch, (UINT)imageSize, (BYTE*)workMemory);
}
// I inspected workMemory and got the result you saw above
// Some more code for copying data to user supplied parameters
Thanks in advance for your help!
The code above is totally fine and works! The problem was in the omitted DirectX 12 Feature support check:
ID3D12Device->CheckFeatureSupport(...)
I forgot an inversion which resulted that the following code snipped was executed
memcpy(&frameNativeFormat, &GUID_WICPixelFormat32bppRGBA, sizeof(GUID));
targetFormatDx = DXGI_FORMAT_R8G8B8A8_UNORM;
uiBitsPerPixel = 32;
The second bug was that I override "frameNativeFormat" instead of "targetFormat" which resulted that no conversion was executed (At least for JPEGs).
Fixing both issues gave me a decent texture loading algorithm (At least for Windows)
A little background: I'm attempting to make a Windows (10) application which makes the screen look like an old CRT monitor, scanlines, blur, and all. I'm using this official Microsoft screen capture demo as a starting point: At this stage I can capture a window, and display it back in a new mouse-through window as if it were the original window.
I am attempting to use the CRT-Royale CRT shaders which are generally considered the best CRT shaders; these are available in .cg format. I transpile them with cgc to hlsl, then compile the hlsl files to compiled shader byte code with fxc. I am able to successfully load the compiled shaders and create the pixel shader. I then set the pixel shader in the d3d context. I then attempt to copy the capture surface frame to a pixel shader resource and set the created shaders resource. All of this builds and runs, but I do not see any difference in the output image and am not sure how to proceed. Below is the relevant code. I am not a c++ developer and am making this as a personal project which I plan on open sourcing once I have a primitive working version. Any advice is appreciated, thanks.
SimpleCapture::SimpleCapture(
IDirect3DDevice const& device,
GraphicsCaptureItem const& item)
{
m_item = item;
m_device = device;
// Set up
auto d3dDevice = GetDXGIInterfaceFromObject<ID3D11Device>(m_device);
d3dDevice->GetImmediateContext(m_d3dContext.put());
auto size = m_item.Size();
m_swapChain = CreateDXGISwapChain(
d3dDevice,
static_cast<uint32_t>(size.Width),
static_cast<uint32_t>(size.Height),
static_cast<DXGI_FORMAT>(DirectXPixelFormat::B8G8R8A8UIntNormalized),
2);
// ADDED THIS
HRESULT hr1 = D3DReadFileToBlob(L"crt-royale-first-pass-ps_4_0.fxc", &ps_1_buffer);
HRESULT hr = d3dDevice->CreatePixelShader(
ps_1_buffer->GetBufferPointer(),
ps_1_buffer->GetBufferSize(),
nullptr,
&ps_1
);
m_d3dContext->PSSetShader(
ps_1,
nullptr,
0
);
// END OF ADDED CHANGES
// Create framepool, define pixel format (DXGI_FORMAT_B8G8R8A8_UNORM), and frame size.
m_framePool = Direct3D11CaptureFramePool::Create(
m_device,
DirectXPixelFormat::B8G8R8A8UIntNormalized,
2,
size);
m_session = m_framePool.CreateCaptureSession(m_item);
m_lastSize = size;
m_frameArrived = m_framePool.FrameArrived(auto_revoke, { this, &SimpleCapture::OnFrameArrived });
}
void SimpleCapture::OnFrameArrived(
Direct3D11CaptureFramePool const& sender,
winrt::Windows::Foundation::IInspectable const&)
{
auto newSize = false;
{
auto frame = sender.TryGetNextFrame();
auto frameContentSize = frame.ContentSize();
if (frameContentSize.Width != m_lastSize.Width ||
frameContentSize.Height != m_lastSize.Height)
{
// The thing we have been capturing has changed size.
// We need to resize our swap chain first, then blit the pixels.
// After we do that, retire the frame and then recreate our frame pool.
newSize = true;
m_lastSize = frameContentSize;
m_swapChain->ResizeBuffers(
2,
static_cast<uint32_t>(m_lastSize.Width),
static_cast<uint32_t>(m_lastSize.Height),
static_cast<DXGI_FORMAT>(DirectXPixelFormat::B8G8R8A8UIntNormalized),
0);
}
{
auto frameSurface = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());
com_ptr<ID3D11Texture2D> backBuffer;
check_hresult(m_swapChain->GetBuffer(0, guid_of<ID3D11Texture2D>(), backBuffer.put_void()));
// ADDED THIS
D3D11_TEXTURE2D_DESC txtDesc = {};
txtDesc.MipLevels = txtDesc.ArraySize = 1;
txtDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
txtDesc.SampleDesc.Count = 1;
txtDesc.Usage = D3D11_USAGE_IMMUTABLE;
txtDesc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
auto d3dDevice = GetDXGIInterfaceFromObject<ID3D11Device>(m_device);
ID3D11Texture2D *tex;
d3dDevice->CreateTexture2D(&txtDesc, NULL,
&tex);
frameSurface.copy_to(&tex);
d3dDevice->CreateShaderResourceView(
tex,
nullptr,
srv_1
);
auto texture = srv_1;
m_d3dContext->PSSetShaderResources(0, 1, texture);
// END OF ADDED CHANGES
m_d3dContext->CopyResource(backBuffer.get(), frameSurface.get());
}
}
DXGI_PRESENT_PARAMETERS presentParameters = { 0 };
m_swapChain->Present1(1, 0, &presentParameters);
... // Truncated
Shaders define how things are drawn. However, you don't draw anything - you just copy, which is why the shader doesn't do anything.
What you should do is to remove the CopyResource call, and instead draw a full screen quad on the back buffer (Which requires you to create a vertex buffer that you can bind, then set the back buffer as render target, and finally call Draw/DrawIndexed to actually render something, which then will invoke the shader).
Also - since I'm not sure whether you already do this and just stripped it from the shown code - functions like CreatePixelShader don't return HRESULTs just for the fun of it - you should check what is actually returned, because DirectX silently returns most errors and expects you to handle them, instead of crashing your program.
Hi I just start with DirectX under CoreWindow^ + C++/CLI, all things seems to be ok until I want to start using a Mulitsampling.
This example is rendering the simple triangle
Working example without AA
When I fill the SwapChain struct like this:
UINT m4xMsaaQuality;
dev->CheckMultisampleQualityLevels(DXGI_FORMAT_B8G8R8A8_UNORM,4, &m4xMsaaQuality);
// set up the swap chain description
DXGI_SWAP_CHAIN_DESC1 scd = { 0 };
scd.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT; // how the swap chain should be used
scd.BufferCount =2;
scd.Format = DXGI_FORMAT_B8G8R8A8_UNORM; // the most common swap chain format
scd.SwapEffect = DXGI_SWAP_EFFECT_FLIP_DISCARD; // the recommended flip mode
scd.SampleDesc.Count = 4; // >1 enable anti-aliasing
scd.SampleDesc.Quality = m4xMsaaQuality-1;
CoreWindow^ Window = CoreWindow::GetForCurrentThread(); // get the window pointer
// create the swap chain
dxgiFactory->CreateSwapChainForCoreWindow(
dev.Get(), // address of the device
reinterpret_cast<IUnknown*>(Window), // address of the window
&scd, // address of the swap chain description
nullptr, // advanced
&swapchain); // address of the new swap chain pointer
// get a pointer directly to the back buffer
ComPtr<ID3D11Texture2D> backbuffer;
swapchain->GetBuffer(0, __uuidof(ID3D11Texture2D), (&backbuffer));`
the
dxgiFactory->CreateSwapChainForCoreWindow(
dev.Get(), // address of the device
reinterpret_cast<IUnknown*>(Window), // address of the window
&scd, // address of the swap chain description
nullptr, // advanced
&swapchain);
return "nullptr",
I checked that m4xMsaaQuality is equal "17" so scd.SampleDesc.Quality =16
How I should fill up the SwapChain struct?
Flip swap effects are not compatible with a multisampling surface. You need to create a non-msaa swap chain and explicitly resolve your msaa render to the swapchain buffers.
YES! Finally I menage to use this multisampling, I realize that
ResolveSubresource
Copy a multisampled resource into a non-multisampled resource.
That why I don't have to recreating one more time the RenderViewTarget, just I can use my.
I still have question is there is some way to increase the anti-aliasing? The maximum sample what I can put in offScreenSurfaceDesc.SampleDesc.Count is 8.
Init_DirectX with MSAA enable
Here is working solution.
Thank you for your answer, I read in MSDN about Multisampling in UWP app. But still I can't to render the smooth triangle. I changed the way of swapping - now should be correct for UWP application. I used also the ResolveSubresource
Can you take a look? The Window is complete black without a triangle. So far I finished with this code, Mostly I suspect the render target bacause when I uncomment this:
dev->CreateRenderTargetView(backbuffer.Get(), &renderTargetViewDesc, &rendertarget);
And in Render Method [line:210, 214] use "renderTarget"
[210] devcon->OMSetRenderTargets(1, rendertarget.GetAddressOf(), nullptr);
[214]devcon->ClearRenderTargetView(rendertarget.Get(), color);
Everything back to normal (of course without MSAA)
Init_DirectX - render fail
I want to understand DXGI Desktop Duplication. I have read a lot and this is the code I copied from parts of the DesktopDuplication sample on the Microsoft Website. My plan is to get the Buffer or Array from the DesktopImage because I want to make a new Texture for an other program. I hope somebody can explain me what I can do to get it.
void DesktopDublication::GetFrame(_Out_ FRAME_DATA* Data, _Out_ bool* Timeout)
{
IDXGIResource* DesktopResource = nullptr;
DXGI_OUTDUPL_FRAME_INFO FrameInfo;
// Get new frame
HRESULT hr = m_DeskDupl->AcquireNextFrame(500, &FrameInfo, &DesktopResource);
if (hr == DXGI_ERROR_WAIT_TIMEOUT)
{
*Timeout = true;
}
*Timeout = false;
if (FAILED(hr))
{
}
// If still holding old frame, destroy it
if (m_AcquiredDesktopImage)
{
m_AcquiredDesktopImage->Release();
m_AcquiredDesktopImage = nullptr;
}
// QI for IDXGIResource
hr = DesktopResource->QueryInterface(__uuidof(ID3D11Texture2D), reinterpret_cast<void **>(&m_AcquiredDesktopImage));
DesktopResource->Release();
DesktopResource = nullptr;
if (FAILED(hr))
{
}
// Get metadata
if (FrameInfo.TotalMetadataBufferSize)
{
// Old buffer too small
if (FrameInfo.TotalMetadataBufferSize > m_MetaDataSize)
{
if (m_MetaDataBuffer)
{
delete[] m_MetaDataBuffer;
m_MetaDataBuffer = nullptr;
}
m_MetaDataBuffer = new (std::nothrow) BYTE[FrameInfo.TotalMetadataBufferSize];
if (!m_MetaDataBuffer)
{
m_MetaDataSize = 0;
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
m_MetaDataSize = FrameInfo.TotalMetadataBufferSize;
}
UINT BufSize = FrameInfo.TotalMetadataBufferSize;
// Get move rectangles
hr = m_DeskDupl->GetFrameMoveRects(BufSize, reinterpret_cast<DXGI_OUTDUPL_MOVE_RECT*>(m_MetaDataBuffer), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
Data->MoveCount = BufSize / sizeof(DXGI_OUTDUPL_MOVE_RECT);
BYTE* DirtyRects = m_MetaDataBuffer + BufSize;
BufSize = FrameInfo.TotalMetadataBufferSize - BufSize;
// Get dirty rectangles
hr = m_DeskDupl->GetFrameDirtyRects(BufSize, reinterpret_cast<RECT*>(DirtyRects), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
Data->DirtyCount = BufSize / sizeof(RECT);
Data->MetaData = m_MetaDataBuffer;
}
Data->Frame = m_AcquiredDesktopImage;
Data->FrameInfo = FrameInfo;
}
If I'm understanding you correctly, you want to get the current desktop image, duplicate it into a private texture, and then render that private texture onto your window. I would start by reading up on Direct3D 11 and learning how to render a scene, as you will need D3D to do anything with the texture object you get from DXGI. This, this, and this can get you started on D3D11. I would also spend some time reading through the source of the sample you copied your code from, as it completely explains how to do this. Here is the link to the full source code for that sample.
To actually get the texture data and render it out, you need to do the following:
1). Create a D3D11 Device object and a Device Context.
2). Write and compile a Vertex and Pixel shader for the graphics card, then load them into your application.
3). Create an Input Layout object and set it to the device.
4). Initialize the required Blend, Depth-Stencil, and Rasterizer states for the device.
5). Create a Texture object and a Shader Resource View object.
6). Acquire the Desktop Duplication texture using the above code.
7). Use CopyResource to copy the data into your texture.
8). Render that texture to the screen.
This will capture all data displayed on one of the desktops to your texture. It does not do processing on the dirty rects of the desktop. It does not do processing on moved regions. This is bare bones 'capture the desktop and display it elsewhere' code.
If you want to get more in depth, read the linked resources and study the sample code, as the sample basically does what you're asking for.
Since tacking this onto my last answer didn't feel quite right, I decided to create a second.
If you want to read the desktop data to a file, you need a D3D11 Device object, a texture object with the D3D11_USAGE_STAGING flag set, and a method of converting the RGBA pixel data of the desktop texture to whatever it is you want. The basic procedure is a simplified version of the one in my original answer:
1). Create a D3D11 Device object and a Device Context.
2). Create a Staging Texture with the same format as the Desktop Texture.
3). Use CopyResource to copy the Desktop Texture into your Staging Texture.
4). Use ID3D11DeviceContext::Map() to get a pointer to the data contained in the Staging Texture.
Make sure you know how Map works and make sure you can write out image files from a single binary stream. There may also be padding in the image buffer, so be aware you may also need to filter that out. Additionally, make sure you Unmap the buffer instead of calling free, as the buffer given to you almost certainly does not belong to the CRT.
I am trying to extract images out of a mp4 video stream. After looking stuff up, it seems like the proper way of doing that is using Media Foundations in C++ and open the frame/read stuff out of it.
There's very little by way of documentation and samples, but after some digging, it seems like some people have had success in doing this by reading frames into a texture and copying the content of that texture to a memory-readable texture (I am not even sure if I am using the correct terms here). Trying what I found though gives me errors and I am probably doing a bunch of stuff wrong.
Here's a short piece of code from where I try to do that (project itself attached at the bottom).
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))
);
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(
m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)
);
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(600 * 400 * 3);
memcpy(buffer, mapped.pData,600 * 400 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0);
// and the present it to the screen
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->Present(1, 0)
);
}
The error I get is:
First-chance exception at 0x76814B32 in App1.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x07AFF60C. HRESULT:0x80070057
I am not really sure how to pursue it further it since, like I said, there's very little docs about it.
Here's the modified sample I am working off of. This question is specific for WinRT (Windows 8 apps).
UPDATE success!! see edit at bottom
Some partial success, but maybe enough to answer your question. Please read on.
On my system, debugging the exception showed that the OnTimer() function failed when attempting to call TransferVideoFrame(). The error it gave was InvalidArgumentException.
So, a bit of Googling led to my first discovery - there is apparently a bug in NVIDIA drivers - which means the video playback seems to fail with 11 and 10 feature levels.
So my first change was in function CreateDX11Device() as follows:
static const D3D_FEATURE_LEVEL levels[] = {
/*
D3D_FEATURE_LEVEL_11_1,
D3D_FEATURE_LEVEL_11_0,
D3D_FEATURE_LEVEL_10_1,
D3D_FEATURE_LEVEL_10_0,
*/
D3D_FEATURE_LEVEL_9_3,
D3D_FEATURE_LEVEL_9_2,
D3D_FEATURE_LEVEL_9_1
};
Now TransferVideoFrame() still fails, but gives E_FAIL (as an HRESULT) instead of an invalid argument.
More Googling led to my second discovery -
Which was an example showing use of TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see you already had some code in OnTimer() similar to this but which was not used, so I guess you'd found the same link.
Anyway, I now used this code to get the video frame:
ComPtr <ID3D11Texture2D> spTextureDst;
m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst));
m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor);
After doing this, I see that TransferVideoFrame() succeeds (good!) but calling Map() on your copied texture - m_spCopyTexture - fails because that texture wasn't created with CPU read access.
So, I just used your read/write m_spRenderTexture as the target of the copy instead because that has the correct flags and, due to the previous change, I was no longer using it.
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3);
memcpy(buffer, mapped.pData,176 * 144 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0);
Now, on my system, the OnTimer() function does not fail. Video frames are rendered to the texture and the pixel data is copied out successfully to the memory buffer.
Before looking to see if there are further problems, maybe this is a good time to see if you can make the same progress as I have so far. If you comment on this answer with more info, I will edit the answer to add any more help if possible.
EDIT
Changes made to texture description in FramePlayer::CreateBackBuffers()
//make first texture cpu readable
D3D11_TEXTURE2D_DESC texDesc = {0};
texDesc.Width = 176;
texDesc.Height = 144;
texDesc.MipLevels = 1;
texDesc.ArraySize = 1;
texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
texDesc.SampleDesc.Count = 1;
texDesc.SampleDesc.Quality = 0;
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.BindFlags = 0;
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.MiscFlags = 0;
MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture));
Note also that there's a memory leak that needs to be cleared up sometime (I'm sure you're aware) - the memory allocated in the following line is never freed:
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test
SUCCESS
I have now succeeded in saving an individual frame, but now without the use of the copy texture.
First, I downloaded the latest version of the DirectXTex Library, which provides DX11 texture helper functions, for example to extract an image from a texture and to save to file. The instructions for adding the DirectXTex library to your solution as an existing project need to be followed carefully, taking note of the changes needed for Windows 8 Store Apps.
Once, the above library is included, referenced and built, add the following #include's to FramePlayer.cpp
#include "..\DirectXTex\DirectXTex.h" // nb - use the relative path you copied to
#include <wincodec.h>
Finally, the central section of code in FramePlayer::OnTimer() needs to be similar to the following. You will see I just save to the same filename each time so this will need amending to add e.g. a frame number to the name
// new frame available at the media engine so get it
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)));
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor));
// capture an image from the DX11 texture
DirectX::ScratchImage pImage;
HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage);
if (SUCCEEDED(hr))
{
// get the image object from the wrapper
const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0);
// set some place to save the image frame
StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder;
Platform::String ^szPath = dataFolder->Path + "\\frame.png";
// save the image to file
hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data());
}
// and the present it to the screen
MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0));
I don't have time right now to take this any further but I'm very pleased with what I have achieved so far :-))
Can you take a fresh look and update your results in comments?
Look at the Video Thumbnail Sample and the Source Reader documentation.
You can find sample code under SDK Root\Samples\multimedia\mediafoundation\VideoThumbnail
I think OpenCV may help you.
OpenCV offers api to capture frames from camera or video files.
You can download it here http://opencv.org/downloads.html.
The following is a demo I writed with "OpenCV 2.3.1".
#include "opencv.hpp"
using namespace cv;
int main()
{
VideoCapture cap("demo.avi"); // open a video to capture
if (!cap.isOpened()) // check if succeeded
return -1;
Mat frame;
namedWindow("Demo", CV_WINDOW_NORMAL);
// Loop to capture frame and show on the window
while (1) {
cap >> frame;
if (frame.empty())
break;
imshow("Demo", frame);
if (waitKey(33) >= 0) // pause 33ms every frame
break;
}
return 0;
}