I have working solution to extract frames from a video in c++ at github. Problem is its very slow. What I am doing is I am using a timer and playing video and whenever frame is ready I convert it into bitmap and saves it and seek to next position . This is not the right approach I think, there must be another way of pulling out frames. Please go through Github project and suggest any changes.
following is my Timer function
if (m_spMediaEngine != nullptr)
{
LONGLONG pts;
if (m_spMediaEngine->OnVideoStreamTick(&pts) == S_OK)
{
// new frame available at the media engine so get it
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(
m_d3dDevice->CreateTexture2D(
&CD3D11_TEXTURE2D_DESC(
DXGI_FORMAT_B8G8R8A8_UNORM,
m_rcTarget.right, // Width
m_rcTarget.bottom, // Height
1, // MipLevels
1, // ArraySize
D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET
),
nullptr,
&spTextureDst
)
);
if (FAILED(
m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), nullptr, &m_rcTarget, &m_bkgColor)
))
{
return;
}
Position = Position + interval;
SetPlaybackPosition(Position);
ComPtr<IDXGISurface2> surface;
MEDIA::ThrowIfFailed(
spTextureDst.Get()->QueryInterface(
__uuidof(IDXGISurface2), &surface)
);
D2D1_BITMAP_PROPERTIES1 bitmapProperties =
D2D1::BitmapProperties1(
D2D1_BITMAP_OPTIONS_TARGET | D2D1_BITMAP_OPTIONS_CANNOT_DRAW,
D2D1::PixelFormat(DXGI_FORMAT_B8G8R8A8_UNORM, D2D1_ALPHA_MODE_PREMULTIPLIED),
96,
96
);
m_d2dContext->CreateBitmapFromDxgiSurface(surface.Get(), &bitmapProperties, &bitmap);
SaveBitmapToFile();
}
}
My Question is : Is this the right and only way of extracting frames ?
I would do something along the lines of converting it to a hashsum and storing that, which should speed it up. For example, you could create a class to hold a specific instance of a hash, (or create a linked list for those hashes), and then extract a frame and hash it using dhash http://www.hackerfactor.com/blog/index.php?/archives/2013/01/21.html
Related
I'm trying to get rect. of diffent area between two Bitmap* objects. When i pass 2 bitmap* it can run lockbits for first frame but it cannot do it for second bitmap.
Rect GetRecDifference(Bitmap* currentFrame, Bitmap* previousFrame) {
if (previousFrame == NULL) {
previousFrame = currentFrame;
}
else {
return Rect(0, 0, 0, 0);
}
BitmapData* bd1 = new BitmapData;
Rect rect1(0, 0, currentFrame->GetWidth(), currentFrame->GetHeight());
currentFrame->LockBits(&rect1, ImageLockModeRead, PixelFormat32bppARGB, bd1);
BitmapData* bd2 = new BitmapData;
Rect rect2(0, 0, previousFrame->GetWidth(), previousFrame->GetHeight());
previousFrame->LockBits(&rect2, ImageLockModeRead, PixelFormat32bppARGB, bd2);
It can run for first one (bd1*) load status ok and last result is shows ok. but when it comes to bd2 it shows status as WrongState(8).
Is this because i copy current pointer to previous one ? What's reason can be for wrong state error ? Do i need to clear some parts from memory ?
The problem is that you are trying to lock the same image twice, this
previousFrame = currentFrame;
means that both your pointers are pointing to the same image.
Instead you need a scheme that keeps two images in memory at once. Something like the following
Bitmap* current = NULL;
Bitmap* previous = NULL;
while (something)
{
current = getNextImage(); // get the new image
if (current && previous)
{
// process current and previous images
...
}
delete previous; // delete the previous image, not needed anymore
previous = current; // save the current image as the previous
}
delete previous; // one image left over, delete it as well
Not the only way to do it, but hopefully you get the idea.
A little background: I'm attempting to make a Windows (10) application which makes the screen look like an old CRT monitor, scanlines, blur, and all. I'm using this official Microsoft screen capture demo as a starting point: At this stage I can capture a window, and display it back in a new mouse-through window as if it were the original window.
I am attempting to use the CRT-Royale CRT shaders which are generally considered the best CRT shaders; these are available in .cg format. I transpile them with cgc to hlsl, then compile the hlsl files to compiled shader byte code with fxc. I am able to successfully load the compiled shaders and create the pixel shader. I then set the pixel shader in the d3d context. I then attempt to copy the capture surface frame to a pixel shader resource and set the created shaders resource. All of this builds and runs, but I do not see any difference in the output image and am not sure how to proceed. Below is the relevant code. I am not a c++ developer and am making this as a personal project which I plan on open sourcing once I have a primitive working version. Any advice is appreciated, thanks.
SimpleCapture::SimpleCapture(
IDirect3DDevice const& device,
GraphicsCaptureItem const& item)
{
m_item = item;
m_device = device;
// Set up
auto d3dDevice = GetDXGIInterfaceFromObject<ID3D11Device>(m_device);
d3dDevice->GetImmediateContext(m_d3dContext.put());
auto size = m_item.Size();
m_swapChain = CreateDXGISwapChain(
d3dDevice,
static_cast<uint32_t>(size.Width),
static_cast<uint32_t>(size.Height),
static_cast<DXGI_FORMAT>(DirectXPixelFormat::B8G8R8A8UIntNormalized),
2);
// ADDED THIS
HRESULT hr1 = D3DReadFileToBlob(L"crt-royale-first-pass-ps_4_0.fxc", &ps_1_buffer);
HRESULT hr = d3dDevice->CreatePixelShader(
ps_1_buffer->GetBufferPointer(),
ps_1_buffer->GetBufferSize(),
nullptr,
&ps_1
);
m_d3dContext->PSSetShader(
ps_1,
nullptr,
0
);
// END OF ADDED CHANGES
// Create framepool, define pixel format (DXGI_FORMAT_B8G8R8A8_UNORM), and frame size.
m_framePool = Direct3D11CaptureFramePool::Create(
m_device,
DirectXPixelFormat::B8G8R8A8UIntNormalized,
2,
size);
m_session = m_framePool.CreateCaptureSession(m_item);
m_lastSize = size;
m_frameArrived = m_framePool.FrameArrived(auto_revoke, { this, &SimpleCapture::OnFrameArrived });
}
void SimpleCapture::OnFrameArrived(
Direct3D11CaptureFramePool const& sender,
winrt::Windows::Foundation::IInspectable const&)
{
auto newSize = false;
{
auto frame = sender.TryGetNextFrame();
auto frameContentSize = frame.ContentSize();
if (frameContentSize.Width != m_lastSize.Width ||
frameContentSize.Height != m_lastSize.Height)
{
// The thing we have been capturing has changed size.
// We need to resize our swap chain first, then blit the pixels.
// After we do that, retire the frame and then recreate our frame pool.
newSize = true;
m_lastSize = frameContentSize;
m_swapChain->ResizeBuffers(
2,
static_cast<uint32_t>(m_lastSize.Width),
static_cast<uint32_t>(m_lastSize.Height),
static_cast<DXGI_FORMAT>(DirectXPixelFormat::B8G8R8A8UIntNormalized),
0);
}
{
auto frameSurface = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());
com_ptr<ID3D11Texture2D> backBuffer;
check_hresult(m_swapChain->GetBuffer(0, guid_of<ID3D11Texture2D>(), backBuffer.put_void()));
// ADDED THIS
D3D11_TEXTURE2D_DESC txtDesc = {};
txtDesc.MipLevels = txtDesc.ArraySize = 1;
txtDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
txtDesc.SampleDesc.Count = 1;
txtDesc.Usage = D3D11_USAGE_IMMUTABLE;
txtDesc.BindFlags = D3D11_BIND_SHADER_RESOURCE;
auto d3dDevice = GetDXGIInterfaceFromObject<ID3D11Device>(m_device);
ID3D11Texture2D *tex;
d3dDevice->CreateTexture2D(&txtDesc, NULL,
&tex);
frameSurface.copy_to(&tex);
d3dDevice->CreateShaderResourceView(
tex,
nullptr,
srv_1
);
auto texture = srv_1;
m_d3dContext->PSSetShaderResources(0, 1, texture);
// END OF ADDED CHANGES
m_d3dContext->CopyResource(backBuffer.get(), frameSurface.get());
}
}
DXGI_PRESENT_PARAMETERS presentParameters = { 0 };
m_swapChain->Present1(1, 0, &presentParameters);
... // Truncated
Shaders define how things are drawn. However, you don't draw anything - you just copy, which is why the shader doesn't do anything.
What you should do is to remove the CopyResource call, and instead draw a full screen quad on the back buffer (Which requires you to create a vertex buffer that you can bind, then set the back buffer as render target, and finally call Draw/DrawIndexed to actually render something, which then will invoke the shader).
Also - since I'm not sure whether you already do this and just stripped it from the shown code - functions like CreatePixelShader don't return HRESULTs just for the fun of it - you should check what is actually returned, because DirectX silently returns most errors and expects you to handle them, instead of crashing your program.
I am currently developing a little screenshot application which records both of my screen's desktop in a file.
I am using the GetFrontBufferData() function and it is working great.
Unfortunately when changing the screen color depth from 32 to 16 bits (to perform some tests) I have a bad image (purple image with changed resolution) and the recorded screenshot has a very poor quality:
Does someone know if there is a way to use GetFrontBufferData() with a 16 bits screen ?
edit:
My init direct3D:
ZeroMemory(&d3dPresentationParameters,sizeof(D3DPRESENT_PARAMETERS));//Fills a block of memory with zeros.
d3dPresentationParameters.Windowed = TRUE;
d3dPresentationParameters.Flags = D3DPRESENTFLAG_LOCKABLE_BACKBUFFER;
d3dPresentationParameters.BackBufferFormat = d3dFormat;//d3dDisplayMode.Format;//D3DFMT_A8R8G8B8;
d3dPresentationParameters.BackBufferCount = 1;
d3dPresentationParameters.BackBufferHeight = gScreenRect.bottom = uiHeight;
d3dPresentationParameters.BackBufferWidth = gScreenRect.right = uiWidth;
d3dPresentationParameters.MultiSampleType = D3DMULTISAMPLE_NONE;
d3dPresentationParameters.MultiSampleQuality = 0;
d3dPresentationParameters.SwapEffect = D3DSWAPEFFECT_DISCARD;
d3dPresentationParameters.hDeviceWindow = hWnd;
d3dPresentationParameters.PresentationInterval = D3DPRESENT_INTERVAL_DEFAULT;
d3dPresentationParameters.FullScreen_RefreshRateInHz = D3DPRESENT_RATE_DEFAULT;
The thread I use to capture screenshots:
CreateOffscreenPlainSurface(uiWidth, uiHeight, D3DFMT_A8R8G8B8, D3DPOOL_SYSTEMMEM, pBackBuffer, NULL)) != D3D_OK )
{
DBG("Error: CreateOffscreenPlainSurface failed = 0x%x", iRes);
break;
}
GetFrontBufferData(0, pCaptureSurface)) != D3D_OK)
{
DBG("Error: GetFrontBufferData failed = 0x%x", iRes);
break;
}
//D3DXSaveSurfaceToFile("Desktop.bmp", D3DXIFF_BMP, pBackBuffer,NULL, NULL); //Test purposes
ZeroMemory(lockedRect, sizeof(D3DLOCKED_RECT));
LockRect(lockedRect, NULL, D3DLOCK_READONLY)) != D3D_OK )
{
DBG("Error: LockRect failed = 0x%x", iRes);
break;
}
if( (iRes = UnlockRect()) != D3D_OK )
{
DBG("Error: UnlockRect failed = 0x%x", iRes);
break;
}
/**/
This code is perfectly working with 32 bits color depth but not with 16bits.
When creating the device I create 2 devices for both screens (iScreenNber). This is also working in 32bits (not in 16).
When saving the captured screenshot into 2 bmp files for testing (in 16 bits), I have one screen which represents the main display perfectly and the other screen is black.
When using memcpy to use pData, I have the above screenshot with purple color and bad resolution
edit2:
I noticed the following:
When saving Offscreen surface to a BMP file, I get the main display (on 1.bmp) which is refreshed each frame (so it is working just fine). For the second display, I just get the first frame then nothing more.
Quoting MSDN for GetFrontBufferData "The buffer pointed to by pDestSurface will be filled with a representation of the front buffer, converted to the standard 32 bits per pixel format D3DFMT_A8R8G8B8." I guess this is a problem for 16 bits color depth.
The first problem comes from the memcpy which does not handle properly the 16 bits color depth and I still don't know why ----> Help needed for this !!
Second problem is the second display which is not working and I don't why either
What am I doing wrong here ? I just get a black image on my Desktop N°xx.bmp file
Thank you very much for your help.
This is how I create a surface to capture screenshots:
IDirect3DSurface9* pCaptureSurface = NULL;
HRESULT hr = pD3DDevice->CreateOffscreenPlainSurface(
D3DPresentParams.BackBufferWidth,
D3DPresentParams.BackBufferHeight,
D3DPresentParams.BackBufferFormat,
D3DPOOL_SYSTEMMEM,
&pCaptureSurface,
NULL);
pD3DDevice->GetFrontBufferData(0, pCaptureSurface);
If you didn't store D3DPresentParams anywhere, you can use IDirect3DDevice9::GetDisplayMode to obtain width, height and format of your swap chain. All operations of resizing and format conversion you can perform after capturing a front buffer. Also, as I know, display format doesn't support alpha channel, so it typically is D3DFMT_X8R8G8B8, not D3DFMT_A8R8G8B8.
Update:
Actually, you try to capture a whole screen by using d3d device, without rendering anything. A purpose of d3d/opengl is to create or process images and do it GPU-accelerated. Taking a screenshot is just copying some video memory, it doesn't use all GPU power. So, using any GPU API brings no significant gain. Moreover, when you capture front buffer rendered not by yourself, strange things occur, you see. To extend your app you may capture image by GDI and then load it into texture and do any GPU postprocessing.
So i found some answers to my problem.
1) Second monitor wasn't working and I was unable to capture screenshot from it in 16 bits
This comes from the memcpy(..) line in the code. Because I am working with a 16 bits monitor, when executing the memcpy, the surface memory is corrupt and this leads to a black screen.
I still didn't find the solution for this but I'm working on.
2) The colors of the screenshot are wrong
This is, without any surprise, due to the 16 bits color depth. Because I am using GetFrontBufferData, and I am quoting Microsoft: The buffer pointed to by pDestSurface will be filled with a representation of the front buffer, converted to the standard 32 bits per pixel format D3DFMT_A8R8G8B8. This means, if I want to use the pixel data from LockRect(...), I have to "re-convert" my data into 16 bits mode. Therefore, I need to convert my pData data from D3DFMT_A8R8G8B8 to D3DFMT_R5G6B5 which is pretty simple.
3) How to debug the application ?
Thanks to your comments, I've been told that I should analyze pScreeInfo->pData content when I was in 16bits (thanks to Niello). Therefore, I've created a simple method using raw data from pScreeInfo->pData and copying in a .bmp:
HRESULT hr;
DWORD dwBytesRead;
UINT uiSize = 1920 * 1080 * 4;
HANDLE hFile;
hFile = CreateFile(TEXT("data.raw"), GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
BOOL bOk = ReadFile(hFile, pData, uiSize, &dwBytesRead, NULL);
if(!bOk)
exit(0);
pTexture = NULL;
hr = pScreenInfo->g_pD3DDevice->CreateTexture(width, height, 1, 0, D3DFMT_A8R8G8B8, D3DPOOL_MANAGED, &pTexture, NULL);
D3DLOCKED_RECT lockedRect;
hr = pTexture->LockRect(0, &lockedRect, NULL, D3DLOCK_READONLY);
memcpy(lockedRect.pBits, pData, lockedRect.Pitch * height);
hr = pTexture->UnlockRect(0);
hr = D3DXSaveTextureToFile(test, D3DXIFF_BMP, pTexture,NULL);
bOk = CloseHandle(hFile);
SAFE_RELEASE(pTexture);
This piece of code allowed me to notice that pData data was correct and I could get a good .bmp file at the end which means that GetFrontBufferData(...) was correctly working and the problem comes from the memcpy(...)
4) Remaining problems
I am still trying to know how I can solve the memcpy issue to see where the problem comes from. This is the last problem since the colors are good now (thanks to the 32bits to 16 bits conversion)
Thank everybody for your helpful comments !
We are working with the Kinect to track faces for a schoolproject. We have set up Visual Studio 2012, and all the test programs are working correctly. However we are trying to run this code and it gives us an error. After many attempts to fix the code, it gives the following error:
"The application was unable to start correctly (0xc000007b).Click OK to close the application.
The good thing is that it's finally running. The bad thing is that the compiler doesn't throw any errors other than this vague error.
We are completely lost and we hope that someone can help us or point us into the right direction. Thanks in advance for helping us.
The code:
#include "stdafx.h"
#include <iostream>
#include <Windows.h>
#include <NuiApi.h>
#include <FaceTrackLib.h>
#include <NuiSensor.h>
using namespace std;
HANDLE rgbStream;
HANDLE depthStream;
INuiSensor* sensor;
#define width 640
#define height 480
bool initKinect() {
// Get a working kinect sensor
int numSensors;
if (NuiGetSensorCount(&numSensors) < 0 || numSensors < 1) return false;
if (NuiCreateSensorByIndex(0, &sensor) < 0) return false;
// Initialize sensor
sensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_DEPTH | NUI_INITIALIZE_FLAG_USES_COLOR);
sensor->NuiImageStreamOpen(
NUI_IMAGE_TYPE_COLOR, // Depth camera or rgb camera?
NUI_IMAGE_RESOLUTION_640x480, // Image resolution
0, // Image stream flags, e.g. near mode
2, // Number of frames to buffer
NULL, // Event handle
&rgbStream);
// --------------- END CHANGED CODE -----------------
return true;
}
BYTE* dataEnd;
USHORT* dataEndD;
void getKinectDataD(){
NUI_IMAGE_FRAME imageFrame;
NUI_LOCKED_RECT LockedRect;
if (sensor->NuiImageStreamGetNextFrame(rgbStream, 0, &imageFrame) < 0) return;
INuiFrameTexture* texture = imageFrame.pFrameTexture;
texture->LockRect(0, &LockedRect, NULL, 0);
const USHORT* curr = (const USHORT*)LockedRect.pBits;
const USHORT* dataEnding = curr + (width*height);
if (LockedRect.Pitch != 0)
{
const BYTE* curr = (const BYTE*)LockedRect.pBits;
dataEnd = (BYTE*)(curr + (width*height) * 4);
}
while (curr < dataEnding) {
// Get depth in millimeters
USHORT depth = NuiDepthPixelToDepth(*curr++);
dataEndD = (USHORT*)depth;
// Draw a grayscale image of the depth:
// B,G,R are all set to depth%256, alpha set to 1.
}
texture->UnlockRect(0);
sensor->NuiImageStreamReleaseFrame(rgbStream, &imageFrame);
}
// This example assumes that the application provides
// void* cameraFrameBuffer, a buffer for an image, and that there is a method
// to fill the buffer with data from a camera, for example
// cameraObj.ProcessIO(cameraFrameBuffer)
int main(){
initKinect();
// Create an instance of a face tracker
IFTFaceTracker* pFT = FTCreateFaceTracker();
if (!pFT)
{
// Handle errors
}
// Initialize cameras configuration structures.
// IMPORTANT NOTE: resolutions and focal lengths must be accurate, since it affects tracking precision!
// It is better to use enums defined in NuiAPI.h
// Video camera config with width, height, focal length in pixels
// NUI_CAMERA_COLOR_NOMINAL_FOCAL_LENGTH_IN_PIXELS focal length is computed for 640x480 resolution
// If you use different resolutions, multiply this focal length by the scaling factor
FT_CAMERA_CONFIG videoCameraConfig = { 640, 480, NUI_CAMERA_COLOR_NOMINAL_FOCAL_LENGTH_IN_PIXELS };
// Depth camera config with width, height, focal length in pixels
// NUI_CAMERA_COLOR_NOMINAL_FOCAL_LENGTH_IN_PIXELS focal length is computed for 320x240 resolution
// If you use different resolutions, multiply this focal length by the scaling factor
FT_CAMERA_CONFIG depthCameraConfig = { 320, 240, NUI_CAMERA_DEPTH_NOMINAL_FOCAL_LENGTH_IN_PIXELS };
// Initialize the face tracker
HRESULT hr = pFT->Initialize(&videoCameraConfig, &depthCameraConfig, NULL, NULL);
if (FAILED(hr))
{
// Handle errors
}
// Create a face tracking result interface
IFTResult* pFTResult = NULL;
hr = pFT->CreateFTResult(&pFTResult);
if (FAILED(hr))
{
// Handle errors
}
// Prepare image interfaces that hold RGB and depth data
IFTImage* pColorFrame = FTCreateImage();
IFTImage* pDepthFrame = FTCreateImage();
if (!pColorFrame || !pDepthFrame)
{
// Handle errors
}
// Attach created interfaces to the RGB and depth buffers that are filled with
// corresponding RGB and depth frame data from Kinect cameras
pColorFrame->Attach(640, 480, dataEnd, FTIMAGEFORMAT_UINT8_R8G8B8, 640 * 3);
pDepthFrame->Attach(320, 240, dataEndD, FTIMAGEFORMAT_UINT16_D13P3, 320 * 2);
// You can also use Allocate() method in which case IFTImage interfaces own their memory.
// In this case use CopyTo() method to copy buffers
FT_SENSOR_DATA sensorData;
sensorData.ZoomFactor = 1.0f; // Not used must be 1.0
bool isFaceTracked = false;
// Track a face
while (true)
{
// Call Kinect API to fill videoCameraFrameBuffer and depthFrameBuffer with RGB and depth data
getKinectDataD();
// Check if we are already tracking a face
if (!isFaceTracked)
{
// Initiate face tracking.
// This call is more expensive and searches the input frame for a face.
hr = pFT->StartTracking(&sensorData, NULL, NULL, pFTResult);
if (SUCCEEDED(hr))
{
isFaceTracked = true;
}
else
{
// No faces found
isFaceTracked = false;
}
}
else
{
// Continue tracking. It uses a previously known face position.
// This call is less expensive than StartTracking()
hr = pFT->ContinueTracking(&sensorData, NULL, pFTResult);
if (FAILED(hr))
{
// Lost the face
isFaceTracked = false;
}
}
// Do something with pFTResult like visualize the mask, drive your 3D avatar,
// recognize facial expressions
}
// Clean up
pFTResult->Release();
pColorFrame->Release();
pDepthFrame->Release();
pFT->Release();
return 0;
}
We figured it out we used the wrong dll indeed, it runs without errors now. But we ran in to an another problem, we have no clue how to use the pFTResult and retrieve the face angles with use of "getFaceRect". Does somebody know how?
I am trying to extract images out of a mp4 video stream. After looking stuff up, it seems like the proper way of doing that is using Media Foundations in C++ and open the frame/read stuff out of it.
There's very little by way of documentation and samples, but after some digging, it seems like some people have had success in doing this by reading frames into a texture and copying the content of that texture to a memory-readable texture (I am not even sure if I am using the correct terms here). Trying what I found though gives me errors and I am probably doing a bunch of stuff wrong.
Here's a short piece of code from where I try to do that (project itself attached at the bottom).
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))
);
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(
m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)
);
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(600 * 400 * 3);
memcpy(buffer, mapped.pData,600 * 400 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0);
// and the present it to the screen
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->Present(1, 0)
);
}
The error I get is:
First-chance exception at 0x76814B32 in App1.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x07AFF60C. HRESULT:0x80070057
I am not really sure how to pursue it further it since, like I said, there's very little docs about it.
Here's the modified sample I am working off of. This question is specific for WinRT (Windows 8 apps).
UPDATE success!! see edit at bottom
Some partial success, but maybe enough to answer your question. Please read on.
On my system, debugging the exception showed that the OnTimer() function failed when attempting to call TransferVideoFrame(). The error it gave was InvalidArgumentException.
So, a bit of Googling led to my first discovery - there is apparently a bug in NVIDIA drivers - which means the video playback seems to fail with 11 and 10 feature levels.
So my first change was in function CreateDX11Device() as follows:
static const D3D_FEATURE_LEVEL levels[] = {
/*
D3D_FEATURE_LEVEL_11_1,
D3D_FEATURE_LEVEL_11_0,
D3D_FEATURE_LEVEL_10_1,
D3D_FEATURE_LEVEL_10_0,
*/
D3D_FEATURE_LEVEL_9_3,
D3D_FEATURE_LEVEL_9_2,
D3D_FEATURE_LEVEL_9_1
};
Now TransferVideoFrame() still fails, but gives E_FAIL (as an HRESULT) instead of an invalid argument.
More Googling led to my second discovery -
Which was an example showing use of TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see you already had some code in OnTimer() similar to this but which was not used, so I guess you'd found the same link.
Anyway, I now used this code to get the video frame:
ComPtr <ID3D11Texture2D> spTextureDst;
m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst));
m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor);
After doing this, I see that TransferVideoFrame() succeeds (good!) but calling Map() on your copied texture - m_spCopyTexture - fails because that texture wasn't created with CPU read access.
So, I just used your read/write m_spRenderTexture as the target of the copy instead because that has the correct flags and, due to the previous change, I was no longer using it.
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3);
memcpy(buffer, mapped.pData,176 * 144 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0);
Now, on my system, the OnTimer() function does not fail. Video frames are rendered to the texture and the pixel data is copied out successfully to the memory buffer.
Before looking to see if there are further problems, maybe this is a good time to see if you can make the same progress as I have so far. If you comment on this answer with more info, I will edit the answer to add any more help if possible.
EDIT
Changes made to texture description in FramePlayer::CreateBackBuffers()
//make first texture cpu readable
D3D11_TEXTURE2D_DESC texDesc = {0};
texDesc.Width = 176;
texDesc.Height = 144;
texDesc.MipLevels = 1;
texDesc.ArraySize = 1;
texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
texDesc.SampleDesc.Count = 1;
texDesc.SampleDesc.Quality = 0;
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.BindFlags = 0;
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.MiscFlags = 0;
MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture));
Note also that there's a memory leak that needs to be cleared up sometime (I'm sure you're aware) - the memory allocated in the following line is never freed:
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test
SUCCESS
I have now succeeded in saving an individual frame, but now without the use of the copy texture.
First, I downloaded the latest version of the DirectXTex Library, which provides DX11 texture helper functions, for example to extract an image from a texture and to save to file. The instructions for adding the DirectXTex library to your solution as an existing project need to be followed carefully, taking note of the changes needed for Windows 8 Store Apps.
Once, the above library is included, referenced and built, add the following #include's to FramePlayer.cpp
#include "..\DirectXTex\DirectXTex.h" // nb - use the relative path you copied to
#include <wincodec.h>
Finally, the central section of code in FramePlayer::OnTimer() needs to be similar to the following. You will see I just save to the same filename each time so this will need amending to add e.g. a frame number to the name
// new frame available at the media engine so get it
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)));
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor));
// capture an image from the DX11 texture
DirectX::ScratchImage pImage;
HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage);
if (SUCCEEDED(hr))
{
// get the image object from the wrapper
const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0);
// set some place to save the image frame
StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder;
Platform::String ^szPath = dataFolder->Path + "\\frame.png";
// save the image to file
hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data());
}
// and the present it to the screen
MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0));
I don't have time right now to take this any further but I'm very pleased with what I have achieved so far :-))
Can you take a fresh look and update your results in comments?
Look at the Video Thumbnail Sample and the Source Reader documentation.
You can find sample code under SDK Root\Samples\multimedia\mediafoundation\VideoThumbnail
I think OpenCV may help you.
OpenCV offers api to capture frames from camera or video files.
You can download it here http://opencv.org/downloads.html.
The following is a demo I writed with "OpenCV 2.3.1".
#include "opencv.hpp"
using namespace cv;
int main()
{
VideoCapture cap("demo.avi"); // open a video to capture
if (!cap.isOpened()) // check if succeeded
return -1;
Mat frame;
namedWindow("Demo", CV_WINDOW_NORMAL);
// Loop to capture frame and show on the window
while (1) {
cap >> frame;
if (frame.empty())
break;
imshow("Demo", frame);
if (waitKey(33) >= 0) // pause 33ms every frame
break;
}
return 0;
}