Convert from Direct3D11CaptureFrame to OpenCV's cv::Mat - c++

Question

I'm trying to convert from a Direct3D11CaptureFrame, obtained as in the example from Microsoft's Screen Capture documentation (and verified to be correct by saving it on disk), to a cv::Mat. I am converting the frame's surface into a ID3D11Texture2D in the following way, and it seems to work:
void processFrame(const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame &frame) {
winrt::com_ptr<Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> access =
frame.Surface().as<Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
winrt::com_ptr<ID3D11Texture2D> frameSurface;
winrt::check_hresult(access->GetInterface(winrt::guid_of<ID3D11Texture2D>(), frameSurface.put_void()));
(...)
}
From this point, looking at opencv's d3d interop samples, I see I have two options. The first one consists in using cv::directx::convertFromD3D11Texture2D such as
void processFrame(const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame &frame) {
winrt::com_ptr<Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> access =
frame.Surface().as<Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
winrt::com_ptr<ID3D11Texture2D> frameSurface;
winrt::check_hresult(access->GetInterface(winrt::guid_of<ID3D11Texture2D>(), frameSurface.put_void()));
cv::Mat img;
// throws exception in the following line
//
cv::directx::convertFromD3D11Texture2D(frameSurface.get(), img);
}
But it throws an exception after calling the destructor of the input array (frameSurface.get()). This method is used for GPU work in the code examples, but I haven't seen it stated as exclusively for GPU in the documentation.
The second option goes through a couple more hoops:
void processFrame(const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame &frame, int height, int width) {
auto access =
frame.Surface().as<Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
winrt::com_ptr<ID3D11Texture2D> frameSurface;
winrt::check_hresult(
access->GetInterface(winrt::guid_of<ID3D11Texture2D>(), frameSurface.put_void()));
ID3D11Device *deviceD3D;
frameSurface->GetDevice(&deviceD3D);
ID3D11DeviceContext *m_pD3D11Ctx;
deviceD3D->GetImmediateContext(&m_pD3D11Ctx);
auto subResource = ::D3D11CalcSubresource(0, 0, 1);
D3D11_MAPPED_SUBRESOURCE mappedTex;
auto r = m_pD3D11Ctx->Map(frameSurface.get(), subResource, D3D11_MAP_WRITE_DISCARD, 0, &mappedTex);
// FAILS here
//
if (FAILED(r)) {
throw std::runtime_error("surface mapping failed!");
}
cv::Mat m(height, width, CV_8UC4, mappedTex.pData, mappedTex.RowPitch);
}
... but it fails when mapping the context to the subresource.
I have been tweaking everything that I could think of given my inexperience with DirectX and trying to see what I am missing, but I can't find a solution for any of the two options. Does anyone see my mistakes or has a better solution for my problem?

Answer 1

1.you need new D3D11_TEXTURE2D_DESC and set: CPUAccessFlags = D3D11_CPU_ACCESS_READ
2.use that D3D11_TEXTURE2D_DESC to create a new CreateTexture2D
3.copy frameSurface to your new CreateTexture2D
void SimpleCapture::OnFrameArrived(
Direct3D11CaptureFramePool const& sender,
winrt::Windows::Foundation::IInspectable const&) {
auto newSize = false;
auto frame = sender.TryGetNextFrame();
auto frameContentSize = frame.ContentSize();
D3D11_TEXTURE2D_DESC desc;
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
desc.SampleDesc = { 1,0 };
desc.Usage = D3D11_USAGE_STAGING;
desc.BindFlags = 0;
desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
desc.MiscFlags = 0;
if (frameContentSize.Width != m_lastSize.Width ||
frameContentSize.Height != m_lastSize.Height) {
newSize = true;
m_lastSize = frameContentSize;
}
desc.Width = m_lastSize.Width;
desc.Height = m_lastSize.Height;
ID3D11Texture2D* stagingTexture;
d3dDevice.get()->CreateTexture2D(&desc, nullptr, &stagingTexture);
auto frameSurface = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());
m_d3dContext->CopyResource(stagingTexture, frameSurface.get());
D3D11_MAPPED_SUBRESOURCE mappedTex;
m_d3dContext->Map(stagingTexture, 0, D3D11_MAP_READ, 0, &mappedTex);
m_d3dContext->Unmap(stagingTexture, 0);
cv::Mat c = cv::Mat(desc.Height, desc.Width, CV_8UC4, mappedTex.pData, mappedTex.RowPitch);
cv::imshow("show", c);
stagingTexture->Release();
DXGI_PRESENT_PARAMETERS presentParameters = { 0 };
if (newSize)
{
m_framePool.Recreate(
m_device,
DirectXPixelFormat::B8G8R8A8UIntNormalized,
2,
m_lastSize);
}
}