AcquiredBuffer.Attach(Buffer.MetaData.pSurface);
IDXGISurface* surface;
DXGI_MAPPED_RECT cpuImage;
AcquiredBuffer->QueryInterface(IID_PPV_ARGS(&surface));
surface->Unmap();
hr = surface->Map(&cpuImage, DXGI_MAP_READ);
if (FAILED(hr)) {
}
else {
av_frame_make_writable(frame);
memcpy(frame->data[0], cpuImage.pBits, cpuImage.Pitch);
}
surface->Unmap();
The map fails. Maybe the IDXGISurface is accesible only from GPU
I've tried a lot of methods to make it CPU accesible.
Another solution would be to encode frame with ffmpeg directly on GPU , so i don't need to copy it to cpu, but not sure how i could do that.
Related
I am currently trying to load images from disk into memory using WIC. I used the MSDN documentation here to write my code.
Everything works fine for loading PNG images. Loading JPEG images works without any error but does not produce the correct result! The interesting thing is, that when I convert the image from JPEG to PNG (using irfan view) the error persists.
Consider the following test image:
As you can see the image got shrunk in the x direction and all color information is gone. However, when you zoom in you can see that there is still some color present, but it doesn’t look like expected:
(Problems persist when uploading the image as texture to the GPU)
I have striped out my WIC loading code omitting error handling and resource freeing for readability:
// CoInitialize(NULL) called in main
// WIC Factory
IWICImagingFactory* ptrFactory = NULL;
CoCreateInstance(CLSID_WICImagingFactory, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&ptrFactory));
// Open decoder
IWICBitmapDecoder* ptrDecoder = NULL;
ptrFactory->CreateDecoderFromFilename(fileName, NULL, GENERIC_READ, WICDecodeMetadataCacheOnDemand, &ptrDecoder)
// Get first frame
IWICBitmapFrameDecode* ptrFrameDecoder = NULL;
ptrDecoder->GetFrame(0, &ptrFrameDecoder));
// Read formate
WICPixelFormatGUID frameNativeFormat;
ptrFrameDecoder->GetPixelFormat(&frameNativeFormat);
// Computing target format selectConverterWic(...) and selectConverterDx(...) are just some very basic functions for selecting the right convert to formate by iterating over an std::array witch contains the mappings suggested by the MSDC article mentioned above
WICPixelFormatGUID targetFormat;
DXGI_FORMAT targetFormatDx;
if (!selectConverterDx(frameNativeFormat, &targetFormatDx)) {
// Try to find WIC to WIC converter
selectConverterWic(frameNativeFormat, &targetFormat)
selectConverterDx(targetFormat, &targetFormatDx)
}else {
memcpy(&targetFormat, &frameNativeFormat, sizeof(GUID));
}
// Get format info
IWICComponentInfo* ptrCmpInfo = NULL;
ptrFactory->CreateComponentInfo(targetFormat, &ptrCmpInfo);
// Get type
WICComponentType ptrCmpType;
ptrCmpInfo->GetComponentType(&ptrCmpType);
// Get info from type
IWICPixelFormatInfo* ptrFormatInfo = NULL;
ptrCmpInfo->QueryInterface(IID_PPV_ARGS(&ptrFormatInfo));
// Get BBP
UINT uiBitsPerPixel;
ptrFormatInfo->GetBitsPerPixel(&uiBitsPerPixel);
// ID3D12Device->CheckFeatureSupport(...) and fallback omitted
// Image size
UINT width, height;
ptrFrameDecoder->GetSize(&width, &height);
// Tempory memory allocation
UINT rowPitch = (width * uiBitsPerPixel + 7) / 8;
SIZE_T imageSize = (SIZE_T)rowPitch * (SIZE_T)height;
void* workMemory = malloc(imageSize);
// Check if direct copy is possible
if (memcmp(&frameNativeFormat, &targetFormat, sizeof(GUID)) == 0){
ptrFrameDecoder->CopyPixels(NULL, rowPitch, (UINT)imageSize, (BYTE*)workMemory);
}else{
// Format conversion (Got never hit by a jpeg file; I tried to force it but results weren't right asswell)
IWICFormatConverter* ptrFormatConverter = NULL;
ptrFactory->CreateFormatConverter(&ptrFormatConverter);
ptrFormatConverter->Initialize(ptrFrameDecoder, targetFormat, WICBitmapDitherTypeErrorDiffusion, NULL, 0, WICBitmapPaletteTypeCustom);
ptrFormatConverter->CopyPixels(NULL, rowPitch, (UINT)imageSize, (BYTE*)workMemory);
}
// I inspected workMemory and got the result you saw above
// Some more code for copying data to user supplied parameters
Thanks in advance for your help!
The code above is totally fine and works! The problem was in the omitted DirectX 12 Feature support check:
ID3D12Device->CheckFeatureSupport(...)
I forgot an inversion which resulted that the following code snipped was executed
memcpy(&frameNativeFormat, &GUID_WICPixelFormat32bppRGBA, sizeof(GUID));
targetFormatDx = DXGI_FORMAT_R8G8B8A8_UNORM;
uiBitsPerPixel = 32;
The second bug was that I override "frameNativeFormat" instead of "targetFormat" which resulted that no conversion was executed (At least for JPEGs).
Fixing both issues gave me a decent texture loading algorithm (At least for Windows)
This is the syntax of the SDL_CreateTextureFromSurface function:
SDL_Texture* SDL_CreateTextureFromSurface(SDL_Renderer* renderer, SDL_Surface* surface)
However, I'm confused why we need to pass a renderer*? I thought we need a renderer* only when drawing the texture?
You need SDL_Renderer to get information about the applicable constraints:
maximum supported size
pixel format
And probably something more..
In addition to the answer by plaes..
Under the hood, SDL_CreateTextureFromSurface calls SDL_CreateTexture, which itself also needs a Renderer, to create a new texture with the same size as the passed in surface.
Then the the SDL_UpdateTexture function is called on the new created texture to load(copy) the pixel data from the surface you passed in to SDL_CreateTextureFromSurface. If the formats between the passed-in surface differ from what the renderer supports, more logic happens to ensure correct behavior.
The Renderer itself is needed for SDL_CreateTexture because its the GPU that handles and stores textures (most of the time) and the Renderer is supposed to be an abstraction over the GPU.
A surface never needs a Renderer since its loaded in RAM and handled by the CPU.
You can find out more about how these calls work if you look at SDL_render.c from the SDL2 source code.
Here is some code inside SDL_CreateTextureFromSurface:
texture = SDL_CreateTexture(renderer, format, SDL_TEXTUREACCESS_STATIC,
surface->w, surface->h);
if (!texture) {
return NULL;
}
if (format == surface->format->format) {
if (SDL_MUSTLOCK(surface)) {
SDL_LockSurface(surface);
SDL_UpdateTexture(texture, NULL, surface->pixels, surface->pitch);
SDL_UnlockSurface(surface);
} else {
SDL_UpdateTexture(texture, NULL, surface->pixels, surface->pitch);
}
}
I am writing a DLL plugin that will read the desktop frame buffer (whole screen) and render it directly into a Texture2D pointer that is passed in. The goal is to keep everything in video memory (and avoid the cost of copying back to system memory and back to video memory).
I am able to pass the Texture2D (showing up as a ID3D11Texture2D), but I am having issues grabbing the desktop frame buffer with D3D11. D3D9 offered GetFrontBufferData() but it seems D3D11 solution is to use GetBuffer().
My issue is about getting the IDXGISwapChain. Because I want to read the desktop frame buffer and simply read the content via GetBuffer() and display it on the ID3D11Texture2D. I am getting a ID3D11Device I do not understand how to get its IDXGISwapChain.
I have code as follows to get the frame buffer and put it on the Texture:
ID3D11Texture2D* src = (ID3D11Texture2D*)g_TexturePointer;
ID3D11Texture2D* dst = NULL;
HRESULT hr = swapchain->GetBuffer(0, __uuidof(ID3D11Texture2D), (LPVOID*)&dst);
g_devCon->CopyResource(dst, src);
Here I actually created my own swapchain using D3D11CreateDeviceAndSwapChain() but I wonder if it is necessary as I already have a ID3D11Device.
The CopyResource() also seems to fail.
Not necessarily all desktop content would be rendered with D3D11. DXGI is the underlying system for all graphics on Windows, so you will definitely need to use it in some way to get a capture of the desktop. However, D3D11 is built on DXGI (for example, ID3D11Texture2D supports the IDXGIResource interface). The code sample below shows how you can capture the output of an entire monitor into a D3D11 staging texture:
// IDXGIOutput* poutput = ...; // from DXGIAdapter::EnumOutputs.
// Get description of desktop.
DXGI_OUTPUT_DESC outdesc;
poutput->GetDesc(&outdesc);
// Create destination texture, sized same as desktop.
D3D11_TEXTURE2D_DESC texDesc;
memset(&texDesc, 0, sizeof(texDesc));
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
texDesc.BindFlags = 0;
texDesc.Width = outdesc.DesktopCoordinates.right - outdesc.DesktopCoordinates.left;
texDesc.Height = outdesc.DesktopCoordinates.bottom - outdesc.DesktopCoordinates.top;
texDesc.MipLevels = 1;
texDesc.SampleDesc = { 1, 0 };
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.ArraySize = 1;
ID3D11Texture2D* destinationTexture = 0;
pDevice->CreateTexture2D(&texDesc, 0, &destinationTexture); // check HRESULT.
// Get IDXGIResource from texture.
IDXGIResource* destinationResource=0;
destinationTexture->QueryInterface(IID_PPV_ARGS(&destinationResource)); // check HRESULT.
// Get data.
IDXGIOutput1* poutput1;
poutput->QueryInterface(IID_PPV_ARGS(&poutput1)); // check HRESULT.
poutput1->TakeOwnership(pDevice, TRUE);
poutput1->GetDisplaySurfaceData1(destinationResource); // check HRESULT.
poutput1->ReleaseOwnership();
// Now use destinationTexture, it contains the contents of the desktop.
Unfortunately, it has the nasty side effect of turning the output black during the IDXGIOutput::TakeOwnership call. However, without this call, the GetDiplaySurfaceData1 will fail. Depending on your situation, this may be acceptable.
I want to understand DXGI Desktop Duplication. I have read a lot and this is the code I copied from parts of the DesktopDuplication sample on the Microsoft Website. My plan is to get the Buffer or Array from the DesktopImage because I want to make a new Texture for an other program. I hope somebody can explain me what I can do to get it.
void DesktopDublication::GetFrame(_Out_ FRAME_DATA* Data, _Out_ bool* Timeout)
{
IDXGIResource* DesktopResource = nullptr;
DXGI_OUTDUPL_FRAME_INFO FrameInfo;
// Get new frame
HRESULT hr = m_DeskDupl->AcquireNextFrame(500, &FrameInfo, &DesktopResource);
if (hr == DXGI_ERROR_WAIT_TIMEOUT)
{
*Timeout = true;
}
*Timeout = false;
if (FAILED(hr))
{
}
// If still holding old frame, destroy it
if (m_AcquiredDesktopImage)
{
m_AcquiredDesktopImage->Release();
m_AcquiredDesktopImage = nullptr;
}
// QI for IDXGIResource
hr = DesktopResource->QueryInterface(__uuidof(ID3D11Texture2D), reinterpret_cast<void **>(&m_AcquiredDesktopImage));
DesktopResource->Release();
DesktopResource = nullptr;
if (FAILED(hr))
{
}
// Get metadata
if (FrameInfo.TotalMetadataBufferSize)
{
// Old buffer too small
if (FrameInfo.TotalMetadataBufferSize > m_MetaDataSize)
{
if (m_MetaDataBuffer)
{
delete[] m_MetaDataBuffer;
m_MetaDataBuffer = nullptr;
}
m_MetaDataBuffer = new (std::nothrow) BYTE[FrameInfo.TotalMetadataBufferSize];
if (!m_MetaDataBuffer)
{
m_MetaDataSize = 0;
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
m_MetaDataSize = FrameInfo.TotalMetadataBufferSize;
}
UINT BufSize = FrameInfo.TotalMetadataBufferSize;
// Get move rectangles
hr = m_DeskDupl->GetFrameMoveRects(BufSize, reinterpret_cast<DXGI_OUTDUPL_MOVE_RECT*>(m_MetaDataBuffer), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
Data->MoveCount = BufSize / sizeof(DXGI_OUTDUPL_MOVE_RECT);
BYTE* DirtyRects = m_MetaDataBuffer + BufSize;
BufSize = FrameInfo.TotalMetadataBufferSize - BufSize;
// Get dirty rectangles
hr = m_DeskDupl->GetFrameDirtyRects(BufSize, reinterpret_cast<RECT*>(DirtyRects), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
}
Data->DirtyCount = BufSize / sizeof(RECT);
Data->MetaData = m_MetaDataBuffer;
}
Data->Frame = m_AcquiredDesktopImage;
Data->FrameInfo = FrameInfo;
}
If I'm understanding you correctly, you want to get the current desktop image, duplicate it into a private texture, and then render that private texture onto your window. I would start by reading up on Direct3D 11 and learning how to render a scene, as you will need D3D to do anything with the texture object you get from DXGI. This, this, and this can get you started on D3D11. I would also spend some time reading through the source of the sample you copied your code from, as it completely explains how to do this. Here is the link to the full source code for that sample.
To actually get the texture data and render it out, you need to do the following:
1). Create a D3D11 Device object and a Device Context.
2). Write and compile a Vertex and Pixel shader for the graphics card, then load them into your application.
3). Create an Input Layout object and set it to the device.
4). Initialize the required Blend, Depth-Stencil, and Rasterizer states for the device.
5). Create a Texture object and a Shader Resource View object.
6). Acquire the Desktop Duplication texture using the above code.
7). Use CopyResource to copy the data into your texture.
8). Render that texture to the screen.
This will capture all data displayed on one of the desktops to your texture. It does not do processing on the dirty rects of the desktop. It does not do processing on moved regions. This is bare bones 'capture the desktop and display it elsewhere' code.
If you want to get more in depth, read the linked resources and study the sample code, as the sample basically does what you're asking for.
Since tacking this onto my last answer didn't feel quite right, I decided to create a second.
If you want to read the desktop data to a file, you need a D3D11 Device object, a texture object with the D3D11_USAGE_STAGING flag set, and a method of converting the RGBA pixel data of the desktop texture to whatever it is you want. The basic procedure is a simplified version of the one in my original answer:
1). Create a D3D11 Device object and a Device Context.
2). Create a Staging Texture with the same format as the Desktop Texture.
3). Use CopyResource to copy the Desktop Texture into your Staging Texture.
4). Use ID3D11DeviceContext::Map() to get a pointer to the data contained in the Staging Texture.
Make sure you know how Map works and make sure you can write out image files from a single binary stream. There may also be padding in the image buffer, so be aware you may also need to filter that out. Additionally, make sure you Unmap the buffer instead of calling free, as the buffer given to you almost certainly does not belong to the CRT.
I am trying to extract images out of a mp4 video stream. After looking stuff up, it seems like the proper way of doing that is using Media Foundations in C++ and open the frame/read stuff out of it.
There's very little by way of documentation and samples, but after some digging, it seems like some people have had success in doing this by reading frames into a texture and copying the content of that texture to a memory-readable texture (I am not even sure if I am using the correct terms here). Trying what I found though gives me errors and I am probably doing a bunch of stuff wrong.
Here's a short piece of code from where I try to do that (project itself attached at the bottom).
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst))
);
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(
m_spMediaEngine->TransferVideoFrame(m_spRenderTexture.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor)
);
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spCopyTexture.Get(),0,0,0,0,m_spRenderTexture.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
m_spDX11DeviceContext->Map(m_spCopyTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(600 * 400 * 3);
memcpy(buffer, mapped.pData,600 * 400 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spCopyTexture.Get(),0);
// and the present it to the screen
MEDIA::ThrowIfFailed(
m_spDX11SwapChain->Present(1, 0)
);
}
The error I get is:
First-chance exception at 0x76814B32 in App1.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x07AFF60C. HRESULT:0x80070057
I am not really sure how to pursue it further it since, like I said, there's very little docs about it.
Here's the modified sample I am working off of. This question is specific for WinRT (Windows 8 apps).
UPDATE success!! see edit at bottom
Some partial success, but maybe enough to answer your question. Please read on.
On my system, debugging the exception showed that the OnTimer() function failed when attempting to call TransferVideoFrame(). The error it gave was InvalidArgumentException.
So, a bit of Googling led to my first discovery - there is apparently a bug in NVIDIA drivers - which means the video playback seems to fail with 11 and 10 feature levels.
So my first change was in function CreateDX11Device() as follows:
static const D3D_FEATURE_LEVEL levels[] = {
/*
D3D_FEATURE_LEVEL_11_1,
D3D_FEATURE_LEVEL_11_0,
D3D_FEATURE_LEVEL_10_1,
D3D_FEATURE_LEVEL_10_0,
*/
D3D_FEATURE_LEVEL_9_3,
D3D_FEATURE_LEVEL_9_2,
D3D_FEATURE_LEVEL_9_1
};
Now TransferVideoFrame() still fails, but gives E_FAIL (as an HRESULT) instead of an invalid argument.
More Googling led to my second discovery -
Which was an example showing use of TransferVideoFrame() without using CreateTexture2D() to pre-create the texture. I see you already had some code in OnTimer() similar to this but which was not used, so I guess you'd found the same link.
Anyway, I now used this code to get the video frame:
ComPtr <ID3D11Texture2D> spTextureDst;
m_spDX11SwapChain->GetBuffer (0, IID_PPV_ARGS (&spTextureDst));
m_spMediaEngine->TransferVideoFrame (spTextureDst.Get (), nullptr, &m_rcTarget, &m_bkgColor);
After doing this, I see that TransferVideoFrame() succeeds (good!) but calling Map() on your copied texture - m_spCopyTexture - fails because that texture wasn't created with CPU read access.
So, I just used your read/write m_spRenderTexture as the target of the copy instead because that has the correct flags and, due to the previous change, I was no longer using it.
//copy the render target texture to the readable texture.
m_spDX11DeviceContext->CopySubresourceRegion(m_spRenderTexture.Get(),0,0,0,0,spTextureDst.Get(),0,NULL);
m_spDX11DeviceContext->Flush();
//Map the readable texture;
D3D11_MAPPED_SUBRESOURCE mapped = {0};
HRESULT hr = m_spDX11DeviceContext->Map(m_spRenderTexture.Get(),0,D3D11_MAP_READ,0,&mapped);
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3);
memcpy(buffer, mapped.pData,176 * 144 * 3);
//unmap so we can copy during next update.
m_spDX11DeviceContext->Unmap(m_spRenderTexture.Get(),0);
Now, on my system, the OnTimer() function does not fail. Video frames are rendered to the texture and the pixel data is copied out successfully to the memory buffer.
Before looking to see if there are further problems, maybe this is a good time to see if you can make the same progress as I have so far. If you comment on this answer with more info, I will edit the answer to add any more help if possible.
EDIT
Changes made to texture description in FramePlayer::CreateBackBuffers()
//make first texture cpu readable
D3D11_TEXTURE2D_DESC texDesc = {0};
texDesc.Width = 176;
texDesc.Height = 144;
texDesc.MipLevels = 1;
texDesc.ArraySize = 1;
texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
texDesc.SampleDesc.Count = 1;
texDesc.SampleDesc.Quality = 0;
texDesc.Usage = D3D11_USAGE_STAGING;
texDesc.BindFlags = 0;
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ | D3D11_CPU_ACCESS_WRITE;
texDesc.MiscFlags = 0;
MEDIA::ThrowIfFailed(m_spDX11Device->CreateTexture2D(&texDesc,NULL,&m_spRenderTexture));
Note also that there's a memory leak that needs to be cleared up sometime (I'm sure you're aware) - the memory allocated in the following line is never freed:
void* buffer = ::CoTaskMemAlloc(176 * 144 * 3); // sizes changed for my test
SUCCESS
I have now succeeded in saving an individual frame, but now without the use of the copy texture.
First, I downloaded the latest version of the DirectXTex Library, which provides DX11 texture helper functions, for example to extract an image from a texture and to save to file. The instructions for adding the DirectXTex library to your solution as an existing project need to be followed carefully, taking note of the changes needed for Windows 8 Store Apps.
Once, the above library is included, referenced and built, add the following #include's to FramePlayer.cpp
#include "..\DirectXTex\DirectXTex.h" // nb - use the relative path you copied to
#include <wincodec.h>
Finally, the central section of code in FramePlayer::OnTimer() needs to be similar to the following. You will see I just save to the same filename each time so this will need amending to add e.g. a frame number to the name
// new frame available at the media engine so get it
ComPtr<ID3D11Texture2D> spTextureDst;
MEDIA::ThrowIfFailed(m_spDX11SwapChain->GetBuffer(0, IID_PPV_ARGS(&spTextureDst)));
auto rcNormalized = MFVideoNormalizedRect();
rcNormalized.left = 0;
rcNormalized.right = 1;
rcNormalized.top = 0;
rcNormalized.bottom = 1;
MEDIA::ThrowIfFailed(m_spMediaEngine->TransferVideoFrame(spTextureDst.Get(), &rcNormalized, &m_rcTarget, &m_bkgColor));
// capture an image from the DX11 texture
DirectX::ScratchImage pImage;
HRESULT hr = DirectX::CaptureTexture(m_spDX11Device.Get(), m_spDX11DeviceContext.Get(), spTextureDst.Get(), pImage);
if (SUCCEEDED(hr))
{
// get the image object from the wrapper
const DirectX::Image *pRealImage = pImage.GetImage(0, 0, 0);
// set some place to save the image frame
StorageFolder ^dataFolder = ApplicationData::Current->LocalFolder;
Platform::String ^szPath = dataFolder->Path + "\\frame.png";
// save the image to file
hr = DirectX::SaveToWICFile(*pRealImage, DirectX::WIC_FLAGS_NONE, GUID_ContainerFormatPng, szPath->Data());
}
// and the present it to the screen
MEDIA::ThrowIfFailed(m_spDX11SwapChain->Present(1, 0));
I don't have time right now to take this any further but I'm very pleased with what I have achieved so far :-))
Can you take a fresh look and update your results in comments?
Look at the Video Thumbnail Sample and the Source Reader documentation.
You can find sample code under SDK Root\Samples\multimedia\mediafoundation\VideoThumbnail
I think OpenCV may help you.
OpenCV offers api to capture frames from camera or video files.
You can download it here http://opencv.org/downloads.html.
The following is a demo I writed with "OpenCV 2.3.1".
#include "opencv.hpp"
using namespace cv;
int main()
{
VideoCapture cap("demo.avi"); // open a video to capture
if (!cap.isOpened()) // check if succeeded
return -1;
Mat frame;
namedWindow("Demo", CV_WINDOW_NORMAL);
// Loop to capture frame and show on the window
while (1) {
cap >> frame;
if (frame.empty())
break;
imshow("Demo", frame);
if (waitKey(33) >= 0) // pause 33ms every frame
break;
}
return 0;
}