I want to extract raw frames or bitmaps from a video that I'm playing in my C++ console application using C++/WinRT APIs. I'm simply using CopyFrameToVideoSurface to copy the video's frame to a IDirect3DSurface. But, it just crashes my program (which works fine, if I don't set up this frame extracting callback). My goal is to render this frame buffer somewhere else to display the video.
Frame extracting code
(see complete project here: https://github.com/harmonoid/libwinmedia/tree/stackoverflow)
IDirect3DSurface surface = IDirect3DSurface();
Streams::IBuffer buffer = Streams::IBuffer();
DLLEXPORT void PlayerSetFrameEventHandler(
int32_t player_id, void (*callback)(uint8_t* buffer, int32_t size,
int32_t width, int32_t height)) {
g_media_players.at(player_id).IsVideoFrameServerEnabled(true);
g_media_players.at(player_id)
.VideoFrameAvailable([=](auto, const auto& args) -> void {
g_media_players.at(player_id).CopyFrameToVideoSurface(surface);
SoftwareBitmap bitmap =
SoftwareBitmap::CreateCopyFromSurfaceAsync(surface).get();
bitmap.CopyToBuffer(buffer);
(*callback)(buffer.data(), buffer.Length(), bitmap.PixelWidth(),
bitmap.PixelHeight());
});
}
You may simply build this shared library using cmake --build .
For testing the crash, you can compile following example (also present on the link repo):
https://github.com/harmonoid/libwinmedia/blob/stackoverflow/examples/frame_extractor.cpp
#include <cstdio>
#include "../include/internal.hpp"
int32_t main() {
using namespace Internal;
// Create a list of medias.
const char* media_uris[] = {
"http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/"
"ForBiggerJoyrides.mp4"};
const int media_ids[] = {0};
// Create a player instance.
PlayerCreate(0);
// Set frame callback (comment out the code to prevent crash from happening).
PlayerSetFrameEventHandler(
0, [](uint8_t*, int32_t, int32_t width, int32_t height) {
printf("Video width: %d, Video height: %d.", width, height);
});
// Open list of medias.
PlayerOpen(0, 1, media_uris, media_ids);
// Start playing the player.
PlayerPlay(0);
// Prevent console from closing.
getchar();
return 0;
}
I will be really helped, if I can get help to fix the code or any other working method for extracting the frames or video bitmaps using winrt::Windows::Media::Playback::MediaPlayer.
Thankyou 🙏.
Following is the stacktrace of the crash:
Related
I am working on a C++/CLI program using Win32 Media Foundation and Windows Forms. I was able to get my USB webcams data into a BYTE* array in C++ and pass the data to my managed code to display in a picturebox.
Now I want to be able to do a crossfade between 2 cameras, but the code I wrote in C++ doesn't do it smoothly. I am experimenting with the Direct2D ID2D1 crossfade effect to see if it would look any better.
Apparently the ID2D1Effect interface ID2D1Effect::SetInput method wants the data as a ID2D1Image*. I could really use some help to figure out how to convert a frame of RGB32 webcam data pointed to by a BYTE* pointer to ID2D1Image*.
Here is what I have so far:
CrossFade.h
#pragma once
#include <comip.h>
#include <atlcomcli.h>
#include <d2d1_1.h>
#include <d2d1effects_2.h>
#pragma comment(lib,"d2d1.lib")
#pragma comment(lib,"dxguid.lib")
class CrossFade
{
ID2D1Factory* pD2DFactory;
ID2D1HwndRenderTarget* pRT;
public:
//Data in my webcam class
BYTE * srcData = nullptr;
BYTE * dstData = nullptr;
//Methods
CrossFade(HWND hwndPgm, int iWidth, int iHeight);
~CrossFade();
void Draw(ID2D1Image* pBitmapSrc, ID2D1Image* pBitmapDest, float dWeight);
};
CrossFade.cpp
#include "CrossFade.h"
CrossFade::CrossFade(HWND hwndPgm, int iWidth, int iHeight)
{
HRESULT hr = S_OK;
pD2DFactory = NULL;
pRT = NULL;
D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, &pD2DFactory);
//hwndPgm references the picturbox handle. I may need to change this to a panel handle according to some posts.
hr = pD2DFactory->CreateHwndRenderTarget(D2D1::RenderTargetProperties(),
D2D1::HwndRenderTargetProperties(hwndPgm, D2D1::SizeU(iWidth, iHeight)), &pRT);
}
CrossFade::~CrossFade()
{
}
void CrossFade::Draw(float dWeight)
{
if (srcData == nullptr || dstData == nullptr) return; //Data not ready
//I know this is incorrect, I'm just trying to indicate I need the BYTE* data to be converted to ID2D1Image* data
ID2D1Image* pBitmapSrc = srcData;
ID2D1Image* pBitmapDest = dstData;
CComQIPtr<ID2D1DeviceContext> spDeviceContext = pRT;
CComPtr<ID2D1Effect> spCrossFadeEffect;
spDeviceContext->CreateEffect(CLSID_D2D1CrossFade, &spCrossFadeEffect);
// set source and destination bitmaps
spCrossFadeEffect->SetInput(0, pBitmapSrc);
spCrossFadeEffect->SetInput(1, pBitmapDest);
spCrossFadeEffect->SetValue(D2D1_CROSSFADE_PROP_WEIGHT, static_cast<FLOAT>(dWeight));
spDeviceContext->DrawImage(spCrossFadeEffect);
}
Once this is working, I would also like to figure out how to use animation to create a smooth transition. if I include afxanimationcontroller.h I get an error WINDOWS.H is already included. MFC apps must not #include<Windows.h>.
Is there something other than MFC I could use to animate the crossfade? I'm not using MFC otherwise.
may be you can use this function to convert your BTYE* data to a ID2D1Bitmap1 interface which inherits from ID2D1Image.
HRESULT CreateBitmap(
D2D1_SIZE_U size,
const void *sourceData,
UINT32 pitch,
const D2D1_BITMAP_PROPERTIES1 *bitmapProperties,
ID2D1Bitmap1 **bitmap
);
I am making an app in flutter that uses the camera of the device to do filtering with opencv over current video preview.
I am using the camera plugin
With startImageStream I am obtaining the frames of the video
I am doing the filter in C++, so I use ffi to send the information to the filter. Here is the header of the function in c++
__attribute__((visibility("default"))) __attribute__((used))
uint8_t* process_image(int32_t width, int32_t height, uint8_t *bytes)
I have a native_opencv.dart as follows
import 'dart:ffi' as ffi;
import 'dart:io';
import 'package:ffi/ffi.dart';
// C function signatures
typedef _process_image_func = ffi.Pointer<ffi.Uint8> Function(ffi.Int32 width, ffi.Int32 height, ffi.Pointer<ffi.Uint8> bytes);
// Dart function signatures
typedef _ProcessImageFunc = ffi.Pointer<ffi.Uint8> Function(int width, int height, ffi.Pointer<ffi.Uint8> bytes);
// Getting a library that holds needed symbols
ffi.DynamicLibrary _lib = Platform.isAndroid
? ffi.DynamicLibrary.open('libnative_opencv.so')
: ffi.DynamicLibrary.process();
// Looking for the functions
final _ProcessImageFunc _processImage = _lib
.lookup<ffi.NativeFunction<_process_image_func>>('process_image')
.asFunction();
ffi.Pointer<ffi.Uint8> processImage(int width, int height, ffi.Pointer<ffi.Uint8> bytes)
{
return _processImage(width, height, bytes);
}
Here I am stuck. I need to return the frames of the video processed in the filter in C++ to the app and display it on the screen.
I thought I could use CameraController and feed it with an array of bytes, but I can't figure out how to do that (if possible).
This is what I have until now in main.dart for that part:
void _initializeCamera() async {
// Get list of cameras of the device
List<CameraDescription> cameras = await availableCameras();
// Create the CameraController
_camera = new CameraController(cameras[0], ResolutionPreset.veryHigh);
_camera.initialize().then((_) async{
// Start ImageStream
await _camera.startImageStream((CameraImage image) => _processCameraImage(image));
});
}
Future<void> _processCameraImage(CameraImage image) async
{
Pointer<Uint8> p = allocate(count: _savedImage.planes[0].bytes.length);
// Assign the planes data to the pointers of the image
Uint8List pointerList = p.asTypedList(_savedImage.planes[0].bytes.length);
pointerList.setRange(0, _savedImage.planes[0].bytes.length, _savedImage.planes[0].bytes);
// Get the pointer of the data returned from the function to a List
Pointer<Uint8> afterP = processImage(_savedImage.width, _savedImage.height, p);
List imgData = afterP.asTypedList((_savedImage.width * _savedImage.height));
// Generate image from the converted data
imglib.Image img = imglib.Image.fromBytes(_savedImage.height, _savedImage.width, imgData);
}
I don't know how to show the filtered frames of the preview video from the camera on the app screen.
Any help is appreciated, thank you.
The problem is simple enough, i have a code that generates a pixel buffer. Now i need to present this pixel buffer instead of saving image and then analyzing it after.
What would be the solution to:
Open window
Replace all pixels in this window with my pixels RGB888
So far suggestion were: To use opengl, create a vertex buffer for a rect covering a window, and use pixel shader to draw your pixels. Which clearly is not the best way to swap pixel buffers in window.
Platform: Ubuntu 18
You can also display bitmapped images in a window pretty easily with SFML. In fact, it seems considerably faster than CImg in my other answer. I am no expert in this, but the following code does what you seem to want:
// g++ -std=c++11 main.cpp $(pkg-config --libs --cflags sfml-graphics sfml-window)
#include <SFML/Graphics.hpp>
#include <iostream>
#include <cstdint>
int main()
{
const unsigned width = 1024;
const unsigned height= 768;
// create the window
sf::RenderWindow window(sf::VideoMode(width, height), "Some Funky Title");
// create a texture
sf::Texture texture;
texture.create(width, height);
// Create a pixel buffer to fill with RGBA data
unsigned char *pixbuff = new unsigned char[width * height * 4];
// Create uint32_t pointer to above for easy access as RGBA
uint32_t * intptr = (uint32_t *)pixbuff;
// The colour we will fill the window with
unsigned char red = 0;
unsigned char blue = 255;
// run the program as long as the window is open
int frame = 0;
while (window.isOpen())
{
// check all the window's events that were triggered since the last iteration of the loop
sf::Event event;
while (window.pollEvent(event))
{
// "close requested" event: we close the window
if (event.type == sf::Event::Closed)
window.close();
}
// clear the window with black color
window.clear(sf::Color::Black);
// Create RGBA value to fill screen with.
// Increment red and decrement blue on each cycle. Leave green=0, and make opaque
uint32_t RGBA;
RGBA = (red++ << 24) | (blue-- << 16) | 255;
// Stuff data into buffer
for(int i=0;i<width*height;i++){
intptr[i] = RGBA;
}
// Update screen
texture.update(pixbuff);
sf::Sprite sprite(texture);
window.draw(sprite);
// end the current frame
window.display();
std::cout << "Frame: " << frame << std::endl;
frame++;
if(frame==1000)break;
}
return 0;
}
On my Mac, I achieved the following frame rates:
700 fps # 640x480 resolution
384 fps # 1024x768 resolution
You can/could create and fill a texture off-screen in a second thread if you want to improve performance, but this is already pretty fast.
Keywords: C++, Image Processing, display, bitmapped graphics, pixel buffer, SFML, imshow, prime.
You could use CImg which is a small, fast, modern C++ library. It is "header only" so no complicated linking or dependencies.
// http://cimg.eu/reference/group__cimg__tutorial.html
#include <iostream>
#include <string>
#include "CImg.h"
using namespace cimg_library;
int main(int argc,char **argv) {
const unsigned char white[] = { 255,255,255 };
const int width = 320;
const int height = 240;
// Create 3-channel RGB image
CImg<> img(width,height,1,3);
// Create main window
CImgDisplay main_window(img,"Random Data",0);
int frame = 0;
while (!main_window.is_closed()) {
// Fill image with random noise
img.rand(0,255);
// Draw in frame counter
std::string text = "Frame: " + std::to_string(frame);
img.draw_text(10,10,text.c_str(),white,0,1,32);
main_window.display(img);
frame++;
std::cout << "Frame: " << frame << std::endl;
}
}
Here it is in action - the quality is not best because random data is poorly compressible and Stack Overflow has a 2MB image limit. It is good in real-life.
Note that as I am using X11 underneath here, the compilation command must define cimg_display so will look something like:
g++ -Dcimg_display=1 -std=c++11 -I /opt/X11/include -L /opt/X11/lib -lx11 ...
Note also that I am using img.rand() to fill the image with data, you will want to get img.data() which is a pointer to the pixel buffer and then memcpy() your image data into the buffer at that address.
Note that I also did some stuff with writing to the framebuffer directly in another answer. That was in Python but it is easily adapted.
Meta Context:
I'm currently working on a game that utilizes opencv as a substitute for ordinary inputs (keyboard, mouse, etc...). I'm using Unity3D's C# scripts and opencv in C++ via DllImports. My goal is to create an image inside my game coming from opencv.
Code Context:
As done usually in OpenCV, I'm using Mat to represent my image. This is the way that I'm exporting the image bytes:
cv::Mat _currentFrame;
...
extern "C" byte * EXPORT GetRawImage()
{
return _currentFrame.data;
}
And this is how i'm importing from C#:
[DllImport ("ImageInputInterface")]
private static extern IntPtr GetRawImage ();
...
public static void GetRawImageBytes (ref byte[] result, int arrayLength) {
IntPtr a = GetRawImage ();
Marshal.Copy(a, result, 0, arrayLength);
FreeBuffer(a);
}
Judging by the way I understand OpenCV, I expect the byte array to be structured in this way when serialized in a uchar pointer:
b1, g1, r1, b2, g2, r2, ...
I'm converting this BGR array to a RGB array using:
public static void BGR2RGB(ref byte[] buffer) {
byte swap;
for (int i = 0; i < buffer.Length; i = i + 3) {
swap = buffer[i];
buffer[i] = buffer[i + 2];
buffer[i + 2] = swap;
}
}
Finally, I'm using Unity's LoadRawTextureData to load the bytes to a texture:
this.tex = new Texture2D(
ImageInputInterface.GetImageWidth(),
ImageInputInterface.GetImageHeight(),
TextureFormat.RGB24,
false
);
...
ImageInputInterface.GetRawImageBytes(ref ret, ret.Length);
ImageInputInterface.BGR2RGB(ref ret);
tex.LoadRawTextureData(ret);
tex.Apply();
Results:
The final image seems to be scattered in someway, it resembles some shapes, but it seems to triple the shapes as well. This is me holding my hand in front of the camera:
[Me, my hand and the camera]
Doing some tests, I concluded that I decoded the channels correctly, since, using my phone to emit RGB light, I can reproduce the colors from the real world:
[Red Test]
[Blue Test]
[Green Test]
There are also some strange lines in the image:
[Spooky Lines]
There is also my face to compare these images to:
[My face in front of the camera]
Questions:
Since I'm able to correctly decode the color channels, what have I assumed wrong in decoding the OpenCV array? It's that I don't know how the Unity's LoadRawTextureData works, or have I decoded something in the wrong way?
How is the OpenCV Mat.data array structured?
UPDATE
Thanks to #Programmer, his solution worked like magic.
[Me Happy]
I changed his script a little, there was no need to do some stuff. And in my case i needed to use BGR2RGBA, not RGB2RGBA:
extern "C" void EXPORT GetRawImage( byte *data, int width, int height )
{
cv::Mat resizedMat( height, width, _currentFrame.type() );
cv::resize( _currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC );
cv::Mat argbImg;
cv::cvtColor( resizedMat, argbImg, CV_BGR2RGBA );
std::memcpy( data, argbImg.data, argbImg.total() * argbImg.elemSize() );
}
Use SetPixels32 instead of LoadRawTextureData. Instead of returning the array data from C++, do that from C#. Create Color32 array and pin it in c# with GCHandle.Alloc, send the address of the pinned Color32 array to C++, use cv::resize to resize the cv::Mat to match the size of pixels sent from C#. You must do this step or expect some error or issues.
Finally, convert cv::Mat from RGB to ARGB then use std::memcpy to update the array from C++. The SetPixels32 function can then be used to load that updated Color32 array into Texture2D. This is how I do it and it has been working for me without any issues. There might be other better ways to do it but I have never found one.
C++:
cv::Mat _currentFrame;
void GetRawImageBytes(unsigned char* data, int width, int height)
{
//Resize Mat to match the array passed to it from C#
cv::Mat resizedMat(height, width, _currentFrame.type());
cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);
//You may not need this line. Depends on what you are doing
cv::imshow("Nicolas", resizedMat);
//Convert from RGB to ARGB
cv::Mat argb_img;
cv::cvtColor(resizedMat, argb_img, CV_RGB2BGRA);
std::vector<cv::Mat> bgra;
cv::split(argb_img, bgra);
std::swap(bgra[0], bgra[3]);
std::swap(bgra[1], bgra[2]);
std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}
C#:
Attach to any GameObject with a Renderer and you should see the cv::Mat displayed and updated on that Object every frame. Code is commented if confused:
using System;
using System.Runtime.InteropServices;
using UnityEngine;
public class Test : MonoBehaviour
{
[DllImport("ImageInputInterface")]
private static extern void GetRawImageBytes(IntPtr data, int width, int height);
private Texture2D tex;
private Color32[] pixel32;
private GCHandle pixelHandle;
private IntPtr pixelPtr;
void Start()
{
InitTexture();
gameObject.GetComponent<Renderer>().material.mainTexture = tex;
}
void Update()
{
MatToTexture2D();
}
void InitTexture()
{
tex = new Texture2D(512, 512, TextureFormat.ARGB32, false);
pixel32 = tex.GetPixels32();
//Pin pixel32 array
pixelHandle = GCHandle.Alloc(pixel32, GCHandleType.Pinned);
//Get the pinned address
pixelPtr = pixelHandle.AddrOfPinnedObject();
}
void MatToTexture2D()
{
//Convert Mat to Texture2D
GetRawImageBytes(pixelPtr, tex.width, tex.height);
//Update the Texture2D with array updated in C++
tex.SetPixels32(pixel32);
tex.Apply();
}
void OnApplicationQuit()
{
//Free handle
pixelHandle.Free();
}
}
I am trying to extract images out of a mp4 video stream. After looking stuff up, it seems like the proper way of doing that is using Media Foundations in C++ and open the frame/read stuff out of it.
There's very little by way of documentation and samples, but after some digging, it seems like some people have had success in doing this by reading frames into a texture and copying the content of that texture to a memory-readable texture (I am not even sure if I am using the correct terms here). Trying what I found though gives me errors and I am probably doing a bunch of stuff wrong.
Here's a short piece of code from where I try to do that (project itself attached at the bottom).
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))
);
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(
m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)
);
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(600 * 400 * 3);
memcpy(buffer, mapped.pData,600 * 400 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0);
// and the present it to the screen
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->Present(1, 0)
);
}
The error I get is:
First-chance exception at 0x76814B32 in App1.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x07AFF60C. HRESULT:0x80070057
I am not really sure how to pursue it further it since, like I said, there's very little docs about it.
Here's the modified sample I am working off of. This question is specific for WinRT (Windows 8 apps).
UPDATE success!! see edit at bottom
Some partial success, but maybe enough to answer your question. Please read on.
On my system, debugging the exception showed that the OnTimer() function failed when attempting to call TransferVideoFrame(). The error it gave was InvalidArgumentException.
So, a bit of Googling led to my first discovery - there is apparently a bug in NVIDIA drivers - which means the video playback seems to fail with 11 and 10 feature levels.
So my first change was in function CreateDX11Device() as follows:
static const D3D_FEATURE_LEVEL levels[] = {
/*
D3D_FEATURE_LEVEL_11_1,
D3D_FEATURE_LEVEL_11_0,
D3D_FEATURE_LEVEL_10_1,
D3D_FEATURE_LEVEL_10_0,
*/
D3D_FEATURE_LEVEL_9_3,
D3D_FEATURE_LEVEL_9_2,
D3D_FEATURE_LEVEL_9_1
};
Now TransferVideoFrame() still fails, but gives E_FAIL (as an HRESULT) instead of an invalid argument.
More Googling led to my second discovery -
Which was an example showing use of TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see you already had some code in OnTimer() similar to this but which was not used, so I guess you'd found the same link.
Anyway, I now used this code to get the video frame:
ComPtr <ID3D11Texture2D> spTextureDst;
m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst));
m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor);
After doing this, I see that TransferVideoFrame() succeeds (good!) but calling Map() on your copied texture - m_spCopyTexture - fails because that texture wasn't created with CPU read access.
So, I just used your read/write m_spRenderTexture as the target of the copy instead because that has the correct flags and, due to the previous change, I was no longer using it.
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3);
memcpy(buffer, mapped.pData,176 * 144 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0);
Now, on my system, the OnTimer() function does not fail. Video frames are rendered to the texture and the pixel data is copied out successfully to the memory buffer.
Before looking to see if there are further problems, maybe this is a good time to see if you can make the same progress as I have so far. If you comment on this answer with more info, I will edit the answer to add any more help if possible.
EDIT
Changes made to texture description in FramePlayer::CreateBackBuffers()
//make first texture cpu readable
D3D11_TEXTURE2D_DESC texDesc = {0};
texDesc.Width = 176;
texDesc.Height = 144;
texDesc.MipLevels = 1;
texDesc.ArraySize = 1;
texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
texDesc.SampleDesc.Count = 1;
texDesc.SampleDesc.Quality = 0;
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.BindFlags = 0;
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.MiscFlags = 0;
MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture));
Note also that there's a memory leak that needs to be cleared up sometime (I'm sure you're aware) - the memory allocated in the following line is never freed:
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test
SUCCESS
I have now succeeded in saving an individual frame, but now without the use of the copy texture.
First, I downloaded the latest version of the DirectXTex Library, which provides DX11 texture helper functions, for example to extract an image from a texture and to save to file. The instructions for adding the DirectXTex library to your solution as an existing project need to be followed carefully, taking note of the changes needed for Windows 8 Store Apps.
Once, the above library is included, referenced and built, add the following #include's to FramePlayer.cpp
#include "..\DirectXTex\DirectXTex.h" // nb - use the relative path you copied to
#include <wincodec.h>
Finally, the central section of code in FramePlayer::OnTimer() needs to be similar to the following. You will see I just save to the same filename each time so this will need amending to add e.g. a frame number to the name
// new frame available at the media engine so get it
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)));
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor));
// capture an image from the DX11 texture
DirectX::ScratchImage pImage;
HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage);
if (SUCCEEDED(hr))
{
// get the image object from the wrapper
const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0);
// set some place to save the image frame
StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder;
Platform::String ^szPath = dataFolder->Path + "\\frame.png";
// save the image to file
hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data());
}
// and the present it to the screen
MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0));
I don't have time right now to take this any further but I'm very pleased with what I have achieved so far :-))
Can you take a fresh look and update your results in comments?
Look at the Video Thumbnail Sample and the Source Reader documentation.
You can find sample code under SDK Root\Samples\multimedia\mediafoundation\VideoThumbnail
I think OpenCV may help you.
OpenCV offers api to capture frames from camera or video files.
You can download it here http://opencv.org/downloads.html.
The following is a demo I writed with "OpenCV 2.3.1".
#include "opencv.hpp"
using namespace cv;
int main()
{
VideoCapture cap("demo.avi"); // open a video to capture
if (!cap.isOpened()) // check if succeeded
return -1;
Mat frame;
namedWindow("Demo", CV_WINDOW_NORMAL);
// Loop to capture frame and show on the window
while (1) {
cap >> frame;
if (frame.empty())
break;
imshow("Demo", frame);
if (waitKey(33) >= 0) // pause 33ms every frame
break;
}
return 0;
}