I am having to build a video topology manually, which includes using loading and configuring the mpeg2videoextension (decoder). Otherwise the default topoloader fails to resolve the video stream automatically. I am using the default topology loader to resolve the rest of the topology.
Since I am loading the decoder manually, the docs say that I am responsible to get the decoder the hardware acceleration manager. (This decoder is D3D11 Aware). If I create a DXGI device, then create manager in code, I can pass the manager to the decoder, and it seems to work.
The docs also say, however that "In a Media Session scenario, the video renderer creates the Direct3D 11 device."
If this is the case, how do I get a handle to that device? I assume I should be using that device in the device manager to pass into the decoder.
I'm going around in circles. All of the sample code uses IDirect3DDeviceManager9. I am unable to get those samples to work. So I decided to use 11. But I can't find any sample code that uses 11.
Can someone point me in the right direction?
Microsoft does not give a good solution for this challenge. Indeed, standard video renderer for Media Foundation is EVR and it is "aware" of Direct3D 9 only. So you cannot combine it with the decoder using common DXGI device manager. Newer Microsoft applications use a different Direct3D 11 aware renderer, which is not published as an API: you can take advantage of these rendering services as a part of wrapping APIs such as UWP or HTML5 media element playing video. MPEG-2 decoder extension targets primarily these scanarios leaving you with a problem if you are plugging this into older Media Foundation topologies.
I can think of a few solutions to this problems, none of which sound exactly perfect:
Stop using EVR and use DX11VideoRenderer instead: Microsoft gives a starting point with this sample and you are own your own to establish required wiring to share DXGI device manager.
Use multiple Direct3D devices and transfer video frames between the two; there should be graphics API interop to help transfer in efficient way, but overall this looks a sort of stupid work as of 2020 even though doable. This path looks more or less acceptable if you can accept performance hit from transfer through system memory, which makes things a tad easier to implement.
Stop using MPEG-2 decoder extension and implement your own decoder on top of lower level DXVA2 API and implement hardware assisted decoder without fallback to software, in which case you have more control over using GPU services and fitting to renderer's device.
Related
I would like to use DirectX 12 to load each frame of an H264 file into a texture and render it. There is however little to no information on doing this, and the Microsoft website has limited superficial documentation.
Media Foundation has plenty of examples and offers Hardware Enabled decoding. Is the Media Foundation a wrapper around DirectX or is it doing something else?
If not, how much less optimised would the Media Foundation equivalent be in comparison to a DX 12 approach?
Essentially, what are the big differences between Media Foundation and DirectX12 Video Decoding?
I am already using DirectX 12 in my engine so this is specifically regarding DX12.
Thanks in advance.
Hardware video decoding comes from DXVA (DXVA2) API. It's DirectX 11 evolution is D3D11 Video Device part of D3D11 API. Microsoft provides wrappers over hardware accelerated decoders in the format of Media Foundation API primitives, such as H.264 Video Decoder. This decoder is offering use of hardware decoding capabilities as well as fallback to software decoding scenario.
Note that even though Media Foundation is available for UWP development, your options are limited and you are not offered primitives like mentioned transform directly. However if you use higher level APIs (Media Foundation Source Reader API in particular) you can leverage hardware accelerated video decoding in your UWP application.
Media Foundation implementation offers interoperability with Direct3D 11, in the part of video encoding/decoding in particular, but not Direct3D 12. You will not be able to use Media Foundation and DirectX 12 together out of the box. You will either have to implement Direct3D 11/12 interop to transfer the data between the APIs (or, where applicable, use shared access to the same GPU data).
Or alternatively you will have to step down to underlying ID3D12VideoDevice::CreateVideoDecoder which is further evolution of mentioned DXVA2 and Direct3D 11 video decoding APIs with similar usage.
Unfortunately if Media Foundation is notoriously known for poor documentation and hard-to-start development, Direct3D 12 video decoding has zero information and you will have to enjoy a feeling of a pioneer.
Either way all the mentioned are relatively thin wrappers over hardware assisted video decoding implementation with the same great performance. I would recommend taking Media Foundation path and implement 11/12 interop if/when it becomes necessary.
You will get a lot of D3D12 errors caused by Media Foundation if you pass a D3D12 device to IMFDXGIDeviceManager::ResetDevice.
The errors could be avoided if you call IMFSourceReader::ReadSample slowly. It doesn't matter that you adopt sync or async mode to use this method. And, how slowly it should be depends on the machine that runs the program. I use ::Sleep(1) between ReadSample calls for sync mode playing a stream from network, and ::Sleep(3) for sync mode playing a local mp4 file on my machine.
Don't ask who I am. My name is 'the pioneer'.
I have a windows only Direct2D application and would like to implement a video playback system for cutscenes. These files are mp4 but the format can be changed, if need be.
It seems like DirectShow is the advised way to render video/audio on windows.
Now how do I let DirectShow render the video frames to my Direct2D render target?
The VMR-9 filter looks like the best route, but I can't seem to find an elegant way of integrating it into my application
There is no Direct2D/DirectShow interoperability layer in Windows. To fit these two technologies you would have to copy data between the APIs in a rather inefficient way (and this will still take some time to develop the fitting).
With H.264/HEVC MP4 video files you would be better off using Media Foundation to read and decode frames, then load them into Direct2D bitmaps and display in your application. Performance wise it is possible to transfer video frames to Direct2D bitmaps via GPU at reasonable cost and with reasonable development effort, but even if you make a shortcut and do integration roughly and inefficiently it will be on par with DirectShow.
I recommend to start with looking at reading and decoding video frames with Media Foundation Source Reader API. Once you get familiar with fitting the technologies, you will take next step and optimize the transfer by using GPU capacity and interop between Direct3D and Direct2D.
I'm developing USB camera streaming Desktop application using MediaFoundation SourceReader technique. The camera is having USB3.0 support and gives 60fps for 1080p MJPG video format resolution.
I used Software MJPEG Decoder MFT to convert MJPG to YUY2 frames and then converted into the RGB32 frame to draw on the window. Instead of 60fps, I'm able to render only 30fps on the window when using this software decoder. I have posted a question on this site and got some suggestion to use Intel Hardware MJPEG Decoder MFT to solve frame drop issue.
I have faced an error 0xC00D36B5 - MF_E_NOTACCEPTING when calling IMFTransform::ProcessInput() method. To solve this error, MSDN suggested using IMFTranform interface asynchronously. So, I used IMFMediaEventGenerator interface to GetEvent for every In/Out sample. Successfully, I can process only one input sample and then continuously IMFMediaEventGenerator:: GetEvent() methods returns MF_E_NO_EVENTS_AVAILABLE error(GetEvent() is synchronous).
I have tried to configure an asynchronous callback for SourceReader as well as IMFTransform but MFAsyncCallback:: Invoke method is not invoking, hence I planned to use GetEvent method.
Am I missing anything?If Yes, Someone guides me to use Intel Hardware Decoder into my project?
Intel Hardware MJPEG Decoder MFT is an asynchronous MFT and if you are managing it directly, you are responsible to apply asynchronous model. You seem to be doing this but you don't provide information that allows nailing the problem down. Yes, you are supposed to use event model described in ProcessInput, ProcessOutput sections of the article linked above. As you get the first frame, you should debug further to make it work with smooth continuous processing.
When you use APIs like media session our source reader, you have Media Foundation itself dealing with the MFTs. It is capable of doing synchronous and asynchronous consumption when appropriate. In this case, however, you don't do IMFTransform calls and even from your vague description it comes you are doing it wrong way.
Pretty self explanatory. Microsoft had DirectShow for DirectX 9, but using DirectShow with DX11 is a COM nightmare beyond words. Is there a standard for video rendering I haven't heard of, or perhaps a free third-party library for this purpose?
Edit: Thanks to Mgetz, I am aware of Microsoft's attempt at a solution, Media Foundation. However, it's limited to Windows 8+, which I would much prefer to avoid.
This may not exactly match your requirement, but for your GOAL, you may take a look on ffmpeg, libx264 and theora(for ogg sound) or faad(decode aac).
I have done using ffmpeg to open container(3gp/mp4 is simple to implment yourself btw if full GPL licence is a concern), libx264 to decode to frame and upload to opengl texture, performance is good (on mac pro it can render 50 fps for 1080p without optimization) and by getting your hand dirty you can have fun doing stupid things with the texture and 3d transforms.
Media Foundation says that it "enables the development of applications and components for using digital media on Windows Vista and later."
So, it looks like it should work for Vista, Windows 7, and Windows 8.
There is DirectX Video Acceleration 2.0 which has a fabulous API, the DXVA-HD (after one has seen VMR9's API, especially with that custom allocator/presenter for renderless drawing, every other API is fabulous :) )
Have a look at: https://msdn.microsoft.com/en-us/library/windows/desktop/ee663586(v=vs.85).aspx
Also, there is a sample in: https://msdn.microsoft.com/en-us/library/windows/desktop/dd756740(v=vs.85).aspx
Windows 7 is the minimum supported windows version
You will not believe how straight forward it is with this API to have it decode the video into your texture.
I need to use DirectShow (C++) for recording a webcam and saving the data to a file.
I really don't know how DirectShow works, this is a "stage" (working experience), but at school we didn't study it.
I think the best way to implement this could be:
List the video devices connected to the computer
Select the correct camera (there will be only one)
Retrieve the video
Save it to a file
Now there are two problems:
Where can I find a good reference book or how do I start?
The saved video shouldn't be too big, does DirectShow provide a way to compress it?
I won't use OpenCV because sometime it doesn't work properly (It doesn't find the camera).
Are there any high level wrapper that could help?
EDIT: the program won't have a window, it will run in background called by a dll.
Where can I find a good reference book or how do I start?
DirectShow introduction material
The saved video shouldn't be too big, does DirectShow provide a way to compress it?
Yes it provides capabilities to attach codecs, that needs to be installed in the system. These are typically third party codecs (for reasons beyond the scope of brief answer). You might want to record into Windows Media files to not depend on third party codecs. SWee more on MSDN: Choosing a Compression Filter.