JPEG Decoding problems with WIC - c++

I am currently trying to load images from disk into memory using WIC. I used the MSDN documentation here to write my code.
Everything works fine for loading PNG images. Loading JPEG images works without any error but does not produce the correct result! The interesting thing is, that when I convert the image from JPEG to PNG (using irfan view) the error persists.
Consider the following test image:
As you can see the image got shrunk in the x direction and all color information is gone. However, when you zoom in you can see that there is still some color present, but it doesn’t look like expected:
(Problems persist when uploading the image as texture to the GPU)
I have striped out my WIC loading code omitting error handling and resource freeing for readability:
// CoInitialize(NULL) called in main
// WIC Factory
IWICImagingFactory* ptrFactory = NULL;
CoCreateInstance(CLSID_WICImagingFactory, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&ptrFactory));
// Open decoder
IWICBitmapDecoder* ptrDecoder = NULL;
ptrFactory->CreateDecoderFromFilename(fileName, NULL, GENERIC_READ, WICDecodeMetadataCacheOnDemand, &ptrDecoder)
// Get first frame
IWICBitmapFrameDecode* ptrFrameDecoder = NULL;
ptrDecoder->GetFrame(0, &ptrFrameDecoder));
// Read formate
WICPixelFormatGUID frameNativeFormat;
ptrFrameDecoder->GetPixelFormat(&frameNativeFormat);
// Computing target format selectConverterWic(...) and selectConverterDx(...) are just some very basic functions for selecting the right convert to formate by iterating over an std::array witch contains the mappings suggested by the MSDC article mentioned above
WICPixelFormatGUID targetFormat;
DXGI_FORMAT targetFormatDx;
if (!selectConverterDx(frameNativeFormat, &targetFormatDx)) {
// Try to find WIC to WIC converter
selectConverterWic(frameNativeFormat, &targetFormat)
selectConverterDx(targetFormat, &targetFormatDx)
}else {
memcpy(&targetFormat, &frameNativeFormat, sizeof(GUID));
}
// Get format info
IWICComponentInfo* ptrCmpInfo = NULL;
ptrFactory->CreateComponentInfo(targetFormat, &ptrCmpInfo);
// Get type
WICComponentType ptrCmpType;
ptrCmpInfo->GetComponentType(&ptrCmpType);
// Get info from type
IWICPixelFormatInfo* ptrFormatInfo = NULL;
ptrCmpInfo->QueryInterface(IID_PPV_ARGS(&ptrFormatInfo));
// Get BBP
UINT uiBitsPerPixel;
ptrFormatInfo->GetBitsPerPixel(&uiBitsPerPixel);
// ID3D12Device->CheckFeatureSupport(...) and fallback omitted
// Image size
UINT width, height;
ptrFrameDecoder->GetSize(&width, &height);
// Tempory memory allocation
UINT rowPitch = (width * uiBitsPerPixel + 7) / 8;
SIZE_T imageSize = (SIZE_T)rowPitch * (SIZE_T)height;
void* workMemory = malloc(imageSize);
// Check if direct copy is possible
if (memcmp(&frameNativeFormat, &targetFormat, sizeof(GUID)) == 0){
ptrFrameDecoder->CopyPixels(NULL, rowPitch, (UINT)imageSize, (BYTE*)workMemory);
}else{
// Format conversion (Got never hit by a jpeg file; I tried to force it but results weren't right asswell)
IWICFormatConverter* ptrFormatConverter = NULL;
ptrFactory->CreateFormatConverter(&ptrFormatConverter);
ptrFormatConverter->Initialize(ptrFrameDecoder, targetFormat, WICBitmapDitherTypeErrorDiffusion, NULL, 0, WICBitmapPaletteTypeCustom);
ptrFormatConverter->CopyPixels(NULL, rowPitch, (UINT)imageSize, (BYTE*)workMemory);
}
// I inspected workMemory and got the result you saw above
// Some more code for copying data to user supplied parameters
Thanks in advance for your help!

The code above is totally fine and works! The problem was in the omitted DirectX 12 Feature support check:
ID3D12Device->CheckFeatureSupport(...)
I forgot an inversion which resulted that the following code snipped was executed
memcpy(&frameNativeFormat, &GUID_WICPixelFormat32bppRGBA, sizeof(GUID));
targetFormatDx = DXGI_FORMAT_R8G8B8A8_UNORM;
uiBitsPerPixel = 32;
The second bug was that I override "frameNativeFormat" instead of "targetFormat" which resulted that no conversion was executed (At least for JPEGs).
Fixing both issues gave me a decent texture loading algorithm (At least for Windows)

Related

Forcing to WIC to treat image data as RGB

I have a image as a BYTE array(in RGB format) and I needed to encode it. I did, but colors have changed.
/*Create stream and initialized*/
hr = piFactory->CreateStream(&piStream);
hr = piStream->InitializeFromFilename(L"..\\test.jpg",GENERIC_WRITE);
/*created an encoder. I want to save JPG*/
hr = piFactory->CreateEncoder(GUID_ContainerFormatJpeg, NULL, &piEncoder);
did some manipulations, also tried to set pixel format.
hr = piBitmapFrame->SetPixelFormat(&formatGUID);
but it always set to "GUID_WICPixelFormat24bppBGR"
and i write image data as here.
/*pBitmap contains image data*/
hr = piBitmapFrame->WritePixels(lHeight, cbStride, cbBufferSize, pBitmap);
the problem was colors of the image has changed, and I found that if i change RGB to BGR then the output looks fine. so i did this
for(int i = 0; i < DataSize; i += 3)
{
inBuff[i] = pBitmap[i+2];
inBuff[i + 1] = pBitmap[i+1];
inBuff[i + 2] = pBitmap[i];
}
pBitmap = inBuff
But i dont want to expend more time here looping through whole image.
I need to tell WIC "treat the data as RGB(or BGR)".
is that possible? if it is then how?
You can create a bitmap first. at that time you could say the format it needs to be. Here is "24bppRGB" format:
hr = piFactory->CreateBitmapFromMemory(
lWidth,
lHeight,
GUID_WICPixelFormat24bppRGB,
cbStride,
cbBufferSize,
pBitmap,
&piBitmapSrc
);
This is how You write image data into the frame:
hr = piBitmapFrame->WritePixels(lHeight, cbStride, cbBufferSize, pBitmap);
Instead write image to the frame like this (because WritePixels() doesn't accept bitmap):
hr = piBitmapFrame->WriteSource(
piBitmapSrc,
NULL
);
And that help to avoid the loop. I'm not sure what happens inside the CreateBitmapFromMemory() though. Since you are concerned with time, I don't know if this will help (it still makes your code better).
Found a way.
This forum says,
This is covered on MSDN
The JPEG encoder encodes three formats:
GUID_WICPixelFormat8bppGray
GUID_WICPixelFormat24bppBGR
GUID_WICPixelFormat32bppCMYK
If your source data is in a different format, you need to use IWICFormatConverter as part of your chain.

D3D11 screen desktop copy to ID3D11Texture2D

I am writing a DLL plugin that will read the desktop frame buffer (whole screen) and render it directly into a Texture2D pointer that is passed in. The goal is to keep everything in video memory (and avoid the cost of copying back to system memory and back to video memory).
I am able to pass the Texture2D (showing up as a ID3D11Texture2D), but I am having issues grabbing the desktop frame buffer with D3D11. D3D9 offered GetFrontBufferData() but it seems D3D11 solution is to use GetBuffer().
My issue is about getting the IDXGISwapChain. Because I want to read the desktop frame buffer and simply read the content via GetBuffer() and display it on the ID3D11Texture2D. I am getting a ID3D11Device I do not understand how to get its IDXGISwapChain.
I have code as follows to get the frame buffer and put it on the Texture:
ID3D11Texture2D* src = (ID3D11Texture2D*)g_TexturePointer;
ID3D11Texture2D* dst = NULL;
HRESULT hr = swapchain->GetBuffer(0, __uuidof(ID3D11Texture2D), (LPVOID*)&dst);
g_devCon->CopyResource(dst, src);
Here I actually created my own swapchain using D3D11CreateDeviceAndSwapChain() but I wonder if it is necessary as I already have a ID3D11Device.
The CopyResource() also seems to fail.
Not necessarily all desktop content would be rendered with D3D11. DXGI is the underlying system for all graphics on Windows, so you will definitely need to use it in some way to get a capture of the desktop. However, D3D11 is built on DXGI (for example, ID3D11Texture2D supports the IDXGIResource interface). The code sample below shows how you can capture the output of an entire monitor into a D3D11 staging texture:
// IDXGIOutput* poutput = ...; // from DXGIAdapter::EnumOutputs.
// Get description of desktop.
DXGI_OUTPUT_DESC outdesc;
poutput->GetDesc(&outdesc);
// Create destination texture, sized same as desktop.
D3D11_TEXTURE2D_DESC texDesc;
memset(&texDesc, 0, sizeof(texDesc));
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
texDesc.BindFlags = 0;
texDesc.Width = outdesc.DesktopCoordinates.right - outdesc.DesktopCoordinates.left;
texDesc.Height = outdesc.DesktopCoordinates.bottom - outdesc.DesktopCoordinates.top;
texDesc.MipLevels = 1;
texDesc.SampleDesc = { 1, 0 };
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.ArraySize = 1;
ID3D11Texture2D* destinationTexture = 0;
pDevice->CreateTexture2D(&texDesc, 0, &destinationTexture); // check HRESULT.
// Get IDXGIResource from texture.
IDXGIResource* destinationResource=0;
destinationTexture->QueryInterface(IID_PPV_ARGS(&destinationResource)); // check HRESULT.
// Get data.
IDXGIOutput1* poutput1;
poutput->QueryInterface(IID_PPV_ARGS(&poutput1)); // check HRESULT.
poutput1->TakeOwnership(pDevice, TRUE);
poutput1->GetDisplaySurfaceData1(destinationResource); // check HRESULT.
poutput1->ReleaseOwnership();
// Now use destinationTexture, it contains the contents of the desktop.
Unfortunately, it has the nasty side effect of turning the output black during the IDXGIOutput::TakeOwnership call. However, without this call, the GetDiplaySurfaceData1 will fail. Depending on your situation, this may be acceptable.

Problems to understand DXGI DirectX 11 Desktop Duplication to get a Buffer or Array

I want to understand DXGI Desktop Duplication. I have read a lot and this is the code I copied from parts of the DesktopDuplication sample on the Microsoft Website. My plan is to get the Buffer or Array from the DesktopImage because I want to make a new Texture for an other program. I hope somebody can explain me what I can do to get it.
void DesktopDublication::GetFrame(_Out_ FRAME_DATA* Data, _Out_ bool* Timeout)
{
IDXGIResource* DesktopResource = nullptr;
DXGI_OUTDUPL_FRAME_INFO FrameInfo;
// Get new frame
HRESULT hr = m_DeskDupl->AcquireNextFrame(500, &FrameInfo, &DesktopResource);
if (hr == DXGI_ERROR_WAIT_TIMEOUT)
{
*Timeout = true;
}
*Timeout = false;
if (FAILED(hr))
{
}
// If still holding old frame, destroy it
if (m_AcquiredDesktopImage)
{
m_AcquiredDesktopImage->Release();
m_AcquiredDesktopImage = nullptr;
}
// QI for IDXGIResource
hr = DesktopResource->QueryInterface(__uuidof(ID3D11Texture2D), reinterpret_cast<void **>(&m_AcquiredDesktopImage));
DesktopResource->Release();
DesktopResource = nullptr;
if (FAILED(hr))
{
}
// Get metadata
if (FrameInfo.TotalMetadataBufferSize)
{
// Old buffer too small
if (FrameInfo.TotalMetadataBufferSize > m_MetaDataSize)
{
if (m_MetaDataBuffer)
{
delete[] m_MetaDataBuffer;
m_MetaDataBuffer = nullptr;
}
m_MetaDataBuffer = new (std::nothrow) BYTE[FrameInfo.TotalMetadataBufferSize];
if (!m_MetaDataBuffer)
{
m_MetaDataSize = 0;
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
m_MetaDataSize = FrameInfo.TotalMetadataBufferSize;
}
UINT BufSize = FrameInfo.TotalMetadataBufferSize;
// Get move rectangles
hr = m_DeskDupl->GetFrameMoveRects(BufSize, reinterpret_cast<DXGI_OUTDUPL_MOVE_RECT*>(m_MetaDataBuffer), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
Data->MoveCount = BufSize / sizeof(DXGI_OUTDUPL_MOVE_RECT);
BYTE* DirtyRects = m_MetaDataBuffer + BufSize;
BufSize = FrameInfo.TotalMetadataBufferSize - BufSize;
// Get dirty rectangles
hr = m_DeskDupl->GetFrameDirtyRects(BufSize, reinterpret_cast<RECT*>(DirtyRects), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
Data->DirtyCount = BufSize / sizeof(RECT);
Data->MetaData = m_MetaDataBuffer;
}
Data->Frame = m_AcquiredDesktopImage;
Data->FrameInfo = FrameInfo;
}
If I'm understanding you correctly, you want to get the current desktop image, duplicate it into a private texture, and then render that private texture onto your window. I would start by reading up on Direct3D 11 and learning how to render a scene, as you will need D3D to do anything with the texture object you get from DXGI. This, this, and this can get you started on D3D11. I would also spend some time reading through the source of the sample you copied your code from, as it completely explains how to do this. Here is the link to the full source code for that sample.
To actually get the texture data and render it out, you need to do the following:
1). Create a D3D11 Device object and a Device Context.
2). Write and compile a Vertex and Pixel shader for the graphics card, then load them into your application.
3). Create an Input Layout object and set it to the device.
4). Initialize the required Blend, Depth-Stencil, and Rasterizer states for the device.
5). Create a Texture object and a Shader Resource View object.
6). Acquire the Desktop Duplication texture using the above code.
7). Use CopyResource to copy the data into your texture.
8). Render that texture to the screen.
This will capture all data displayed on one of the desktops to your texture. It does not do processing on the dirty rects of the desktop. It does not do processing on moved regions. This is bare bones 'capture the desktop and display it elsewhere' code.
If you want to get more in depth, read the linked resources and study the sample code, as the sample basically does what you're asking for.
Since tacking this onto my last answer didn't feel quite right, I decided to create a second.
If you want to read the desktop data to a file, you need a D3D11 Device object, a texture object with the D3D11_USAGE_STAGING flag set, and a method of converting the RGBA pixel data of the desktop texture to whatever it is you want. The basic procedure is a simplified version of the one in my original answer:
1). Create a D3D11 Device object and a Device Context.
2). Create a Staging Texture with the same format as the Desktop Texture.
3). Use CopyResource to copy the Desktop Texture into your Staging Texture.
4). Use ID3D11DeviceContext::Map() to get a pointer to the data contained in the Staging Texture.
Make sure you know how Map works and make sure you can write out image files from a single binary stream. There may also be padding in the image buffer, so be aware you may also need to filter that out. Additionally, make sure you Unmap the buffer instead of calling free, as the buffer given to you almost certainly does not belong to the CRT.

C++ Direct3D9 GetFrontBufferData with 16 bits color Depth

I am currently developing a little screenshot application which records both of my screen's desktop in a file.
I am using the GetFrontBufferData() function and it is working great.
Unfortunately when changing the screen color depth from 32 to 16 bits (to perform some tests) I have a bad image (purple image with changed resolution) and the recorded screenshot has a very poor quality:
Does someone know if there is a way to use GetFrontBufferData() with a 16 bits screen ?
edit:
My init direct3D:
ZeroMemory(&d3dPresentationParameters,sizeof(D3DPRESENT_PARAMETERS));//Fills a block of memory with zeros.
d3dPresentationParameters.Windowed = TRUE;
d3dPresentationParameters.Flags = D3DPRESENTFLAG_LOCKABLE_BACKBUFFER;
d3dPresentationParameters.BackBufferFormat = d3dFormat;//d3dDisplayMode.Format;//D3DFMT_A8R8G8B8;
d3dPresentationParameters.BackBufferCount = 1;
d3dPresentationParameters.BackBufferHeight = gScreenRect.bottom = uiHeight;
d3dPresentationParameters.BackBufferWidth = gScreenRect.right = uiWidth;
d3dPresentationParameters.MultiSampleType = D3DMULTISAMPLE_NONE;
d3dPresentationParameters.MultiSampleQuality = 0;
d3dPresentationParameters.SwapEffect = D3DSWAPEFFECT_DISCARD;
d3dPresentationParameters.hDeviceWindow = hWnd;
d3dPresentationParameters.PresentationInterval = D3DPRESENT_INTERVAL_DEFAULT;
d3dPresentationParameters.FullScreen_RefreshRateInHz = D3DPRESENT_RATE_DEFAULT;
The thread I use to capture screenshots:
CreateOffscreenPlainSurface(uiWidth, uiHeight, D3DFMT_A8R8G8B8, D3DPOOL_SYSTEMMEM, pBackBuffer, NULL)) != D3D_OK )
{
DBG("Error: CreateOffscreenPlainSurface failed = 0x%x", iRes);
break;
}
GetFrontBufferData(0, pCaptureSurface)) != D3D_OK)
{
DBG("Error: GetFrontBufferData failed = 0x%x", iRes);
break;
}
//D3DXSaveSurfaceToFile("Desktop.bmp", D3DXIFF_BMP, pBackBuffer,NULL, NULL); //Test purposes
ZeroMemory(lockedRect, sizeof(D3DLOCKED_RECT));
LockRect(lockedRect, NULL, D3DLOCK_READONLY)) != D3D_OK )
{
DBG("Error: LockRect failed = 0x%x", iRes);
break;
}
if( (iRes = UnlockRect()) != D3D_OK )
{
DBG("Error: UnlockRect failed = 0x%x", iRes);
break;
}
/**/
This code is perfectly working with 32 bits color depth but not with 16bits.
When creating the device I create 2 devices for both screens (iScreenNber). This is also working in 32bits (not in 16).
When saving the captured screenshot into 2 bmp files for testing (in 16 bits), I have one screen which represents the main display perfectly and the other screen is black.
When using memcpy to use pData, I have the above screenshot with purple color and bad resolution
edit2:
I noticed the following:
When saving Offscreen surface to a BMP file, I get the main display (on 1.bmp) which is refreshed each frame (so it is working just fine). For the second display, I just get the first frame then nothing more.
Quoting MSDN for GetFrontBufferData "The buffer pointed to by pDestSurface will be filled with a representation of the front buffer, converted to the standard 32 bits per pixel format D3DFMT_A8R8G8B8." I guess this is a problem for 16 bits color depth.
The first problem comes from the memcpy which does not handle properly the 16 bits color depth and I still don't know why ----> Help needed for this !!
Second problem is the second display which is not working and I don't why either
What am I doing wrong here ? I just get a black image on my Desktop N°xx.bmp file
Thank you very much for your help.
This is how I create a surface to capture screenshots:
IDirect3DSurface9* pCaptureSurface = NULL;
HRESULT hr = pD3DDevice->CreateOffscreenPlainSurface(
D3DPresentParams.BackBufferWidth,
D3DPresentParams.BackBufferHeight,
D3DPresentParams.BackBufferFormat,
D3DPOOL_SYSTEMMEM,
&pCaptureSurface,
NULL);
pD3DDevice->GetFrontBufferData(0, pCaptureSurface);
If you didn't store D3DPresentParams anywhere, you can use IDirect3DDevice9::GetDisplayMode to obtain width, height and format of your swap chain. All operations of resizing and format conversion you can perform after capturing a front buffer. Also, as I know, display format doesn't support alpha channel, so it typically is D3DFMT_X8R8G8B8, not D3DFMT_A8R8G8B8.
Update:
Actually, you try to capture a whole screen by using d3d device, without rendering anything. A purpose of d3d/opengl is to create or process images and do it GPU-accelerated. Taking a screenshot is just copying some video memory, it doesn't use all GPU power. So, using any GPU API brings no significant gain. Moreover, when you capture front buffer rendered not by yourself, strange things occur, you see. To extend your app you may capture image by GDI and then load it into texture and do any GPU postprocessing.
So i found some answers to my problem.
1) Second monitor wasn't working and I was unable to capture screenshot from it in 16 bits
This comes from the memcpy(..) line in the code. Because I am working with a 16 bits monitor, when executing the memcpy, the surface memory is corrupt and this leads to a black screen.
I still didn't find the solution for this but I'm working on.
2) The colors of the screenshot are wrong
This is, without any surprise, due to the 16 bits color depth. Because I am using GetFrontBufferData, and I am quoting Microsoft: The buffer pointed to by pDestSurface will be filled with a representation of the front buffer, converted to the standard 32 bits per pixel format D3DFMT_A8R8G8B8. This means, if I want to use the pixel data from LockRect(...), I have to "re-convert" my data into 16 bits mode. Therefore, I need to convert my pData data from D3DFMT_A8R8G8B8 to D3DFMT_R5G6B5 which is pretty simple.
3) How to debug the application ?
Thanks to your comments, I've been told that I should analyze pScreeInfo->pData content when I was in 16bits (thanks to Niello). Therefore, I've created a simple method using raw data from pScreeInfo->pData and copying in a .bmp:
HRESULT hr;
DWORD dwBytesRead;
UINT uiSize = 1920 * 1080 * 4;
HANDLE hFile;
hFile = CreateFile(TEXT("data.raw"), GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
BOOL bOk = ReadFile(hFile, pData, uiSize, &dwBytesRead, NULL);
if(!bOk)
exit(0);
pTexture = NULL;
hr = pScreenInfo->g_pD3DDevice->CreateTexture(width, height, 1, 0, D3DFMT_A8R8G8B8, D3DPOOL_MANAGED, &pTexture, NULL);
D3DLOCKED_RECT lockedRect;
hr = pTexture->LockRect(0, &lockedRect, NULL, D3DLOCK_READONLY);
memcpy(lockedRect.pBits, pData, lockedRect.Pitch * height);
hr = pTexture->UnlockRect(0);
hr = D3DXSaveTextureToFile(test, D3DXIFF_BMP, pTexture,NULL);
bOk = CloseHandle(hFile);
SAFE_RELEASE(pTexture);
This piece of code allowed me to notice that pData data was correct and I could get a good .bmp file at the end which means that GetFrontBufferData(...) was correctly working and the problem comes from the memcpy(...)
4) Remaining problems
I am still trying to know how I can solve the memcpy issue to see where the problem comes from. This is the last problem since the colors are good now (thanks to the 32bits to 16 bits conversion)
Thank everybody for your helpful comments !

How do I grab frames from a video stream on Windows 8 modern apps?

I am trying to extract images out of a mp4 video stream. After looking stuff up, it seems like the proper way of doing that is using Media Foundations in C++ and open the frame/read stuff out of it.
There's very little by way of documentation and samples, but after some digging, it seems like some people have had success in doing this by reading frames into a texture and copying the content of that texture to a memory-readable texture (I am not even sure if I am using the correct terms here). Trying what I found though gives me errors and I am probably doing a bunch of stuff wrong.
Here's a short piece of code from where I try to do that (project itself attached at the bottom).
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))
);
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(
m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)
);
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(600 * 400 * 3);
memcpy(buffer, mapped.pData,600 * 400 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0);
// and the present it to the screen
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->Present(1, 0)
);
}
The error I get is:
First-chance exception at 0x76814B32 in App1.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x07AFF60C. HRESULT:0x80070057
I am not really sure how to pursue it further it since, like I said, there's very little docs about it.
Here's the modified sample I am working off of. This question is specific for WinRT (Windows 8 apps).
UPDATE success!! see edit at bottom
Some partial success, but maybe enough to answer your question. Please read on.
On my system, debugging the exception showed that the OnTimer() function failed when attempting to call TransferVideoFrame(). The error it gave was InvalidArgumentException.
So, a bit of Googling led to my first discovery - there is apparently a bug in NVIDIA drivers - which means the video playback seems to fail with 11 and 10 feature levels.
So my first change was in function CreateDX11Device() as follows:
static const D3D_FEATURE_LEVEL levels[] = {
/*
D3D_FEATURE_LEVEL_11_1,
D3D_FEATURE_LEVEL_11_0,
D3D_FEATURE_LEVEL_10_1,
D3D_FEATURE_LEVEL_10_0,
*/
D3D_FEATURE_LEVEL_9_3,
D3D_FEATURE_LEVEL_9_2,
D3D_FEATURE_LEVEL_9_1
};
Now TransferVideoFrame() still fails, but gives E_FAIL (as an HRESULT) instead of an invalid argument.
More Googling led to my second discovery -
Which was an example showing use of TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see you already had some code in OnTimer() similar to this but which was not used, so I guess you'd found the same link.
Anyway, I now used this code to get the video frame:
ComPtr <ID3D11Texture2D> spTextureDst;
m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst));
m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor);
After doing this, I see that TransferVideoFrame() succeeds (good!) but calling Map() on your copied texture - m_spCopyTexture - fails because that texture wasn't created with CPU read access.
So, I just used your read/write m_spRenderTexture as the target of the copy instead because that has the correct flags and, due to the previous change, I was no longer using it.
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3);
memcpy(buffer, mapped.pData,176 * 144 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0);
Now, on my system, the OnTimer() function does not fail. Video frames are rendered to the texture and the pixel data is copied out successfully to the memory buffer.
Before looking to see if there are further problems, maybe this is a good time to see if you can make the same progress as I have so far. If you comment on this answer with more info, I will edit the answer to add any more help if possible.
EDIT
Changes made to texture description in FramePlayer::CreateBackBuffers()
//make first texture cpu readable
D3D11_TEXTURE2D_DESC texDesc = {0};
texDesc.Width = 176;
texDesc.Height = 144;
texDesc.MipLevels = 1;
texDesc.ArraySize = 1;
texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
texDesc.SampleDesc.Count = 1;
texDesc.SampleDesc.Quality = 0;
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.BindFlags = 0;
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.MiscFlags = 0;
MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture));
Note also that there's a memory leak that needs to be cleared up sometime (I'm sure you're aware) - the memory allocated in the following line is never freed:
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test
SUCCESS
I have now succeeded in saving an individual frame, but now without the use of the copy texture.
First, I downloaded the latest version of the DirectXTex Library, which provides DX11 texture helper functions, for example to extract an image from a texture and to save to file. The instructions for adding the DirectXTex library to your solution as an existing project need to be followed carefully, taking note of the changes needed for Windows 8 Store Apps.
Once, the above library is included, referenced and built, add the following #include's to FramePlayer.cpp
#include "..\DirectXTex\DirectXTex.h" // nb - use the relative path you copied to
#include <wincodec.h>
Finally, the central section of code in FramePlayer::OnTimer() needs to be similar to the following. You will see I just save to the same filename each time so this will need amending to add e.g. a frame number to the name
// new frame available at the media engine so get it
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)));
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor));
// capture an image from the DX11 texture
DirectX::ScratchImage pImage;
HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage);
if (SUCCEEDED(hr))
{
// get the image object from the wrapper
const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0);
// set some place to save the image frame
StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder;
Platform::String ^szPath = dataFolder->Path + "\\frame.png";
// save the image to file
hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data());
}
// and the present it to the screen
MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0));
I don't have time right now to take this any further but I'm very pleased with what I have achieved so far :-))
Can you take a fresh look and update your results in comments?
Look at the Video Thumbnail Sample and the Source Reader documentation.
You can find sample code under SDK Root\Samples\multimedia\mediafoundation\VideoThumbnail
I think OpenCV may help you.
OpenCV offers api to capture frames from camera or video files.
You can download it here http://opencv.org/downloads.html.
The following is a demo I writed with "OpenCV 2.3.1".
#include "opencv.hpp"
using namespace cv;
int main()
{
VideoCapture cap("demo.avi"); // open a video to capture
if (!cap.isOpened()) // check if succeeded
return -1;
Mat frame;
namedWindow("Demo", CV_WINDOW_NORMAL);
// Loop to capture frame and show on the window
while (1) {
cap >> frame;
if (frame.empty())
break;
imshow("Demo", frame);
if (waitKey(33) >= 0) // pause 33ms every frame
break;
}
return 0;
}