I would like to use DirectX 12 to load each frame of an H264 file into a texture and render it. There is however little to no information on doing this, and the Microsoft website has limited superficial documentation.
Media Foundation has plenty of examples and offers Hardware Enabled decoding. Is the Media Foundation a wrapper around DirectX or is it doing something else?
If not, how much less optimised would the Media Foundation equivalent be in comparison to a DX 12 approach?
Essentially, what are the big differences between Media Foundation and DirectX12 Video Decoding?
I am already using DirectX 12 in my engine so this is specifically regarding DX12.
Thanks in advance.
Hardware video decoding comes from DXVA (DXVA2) API. It's DirectX 11 evolution is D3D11 Video Device part of D3D11 API. Microsoft provides wrappers over hardware accelerated decoders in the format of Media Foundation API primitives, such as H.264 Video Decoder. This decoder is offering use of hardware decoding capabilities as well as fallback to software decoding scenario.
Note that even though Media Foundation is available for UWP development, your options are limited and you are not offered primitives like mentioned transform directly. However if you use higher level APIs (Media Foundation Source Reader API in particular) you can leverage hardware accelerated video decoding in your UWP application.
Media Foundation implementation offers interoperability with Direct3D 11, in the part of video encoding/decoding in particular, but not Direct3D 12. You will not be able to use Media Foundation and DirectX 12 together out of the box. You will either have to implement Direct3D 11/12 interop to transfer the data between the APIs (or, where applicable, use shared access to the same GPU data).
Or alternatively you will have to step down to underlying ID3D12VideoDevice::CreateVideoDecoder which is further evolution of mentioned DXVA2 and Direct3D 11 video decoding APIs with similar usage.
Unfortunately if Media Foundation is notoriously known for poor documentation and hard-to-start development, Direct3D 12 video decoding has zero information and you will have to enjoy a feeling of a pioneer.
Either way all the mentioned are relatively thin wrappers over hardware assisted video decoding implementation with the same great performance. I would recommend taking Media Foundation path and implement 11/12 interop if/when it becomes necessary.
You will get a lot of D3D12 errors caused by Media Foundation if you pass a D3D12 device to IMFDXGIDeviceManager::ResetDevice.
The errors could be avoided if you call IMFSourceReader::ReadSample slowly. It doesn't matter that you adopt sync or async mode to use this method. And, how slowly it should be depends on the machine that runs the program. I use ::Sleep(1) between ReadSample calls for sync mode playing a stream from network, and ::Sleep(3) for sync mode playing a local mp4 file on my machine.
Don't ask who I am. My name is 'the pioneer'.
Related
I am having to build a video topology manually, which includes using loading and configuring the mpeg2videoextension (decoder). Otherwise the default topoloader fails to resolve the video stream automatically. I am using the default topology loader to resolve the rest of the topology.
Since I am loading the decoder manually, the docs say that I am responsible to get the decoder the hardware acceleration manager. (This decoder is D3D11 Aware). If I create a DXGI device, then create manager in code, I can pass the manager to the decoder, and it seems to work.
The docs also say, however that "In a Media Session scenario, the video renderer creates the Direct3D 11 device."
If this is the case, how do I get a handle to that device? I assume I should be using that device in the device manager to pass into the decoder.
I'm going around in circles. All of the sample code uses IDirect3DDeviceManager9. I am unable to get those samples to work. So I decided to use 11. But I can't find any sample code that uses 11.
Can someone point me in the right direction?
Microsoft does not give a good solution for this challenge. Indeed, standard video renderer for Media Foundation is EVR and it is "aware" of Direct3D 9 only. So you cannot combine it with the decoder using common DXGI device manager. Newer Microsoft applications use a different Direct3D 11 aware renderer, which is not published as an API: you can take advantage of these rendering services as a part of wrapping APIs such as UWP or HTML5 media element playing video. MPEG-2 decoder extension targets primarily these scanarios leaving you with a problem if you are plugging this into older Media Foundation topologies.
I can think of a few solutions to this problems, none of which sound exactly perfect:
Stop using EVR and use DX11VideoRenderer instead: Microsoft gives a starting point with this sample and you are own your own to establish required wiring to share DXGI device manager.
Use multiple Direct3D devices and transfer video frames between the two; there should be graphics API interop to help transfer in efficient way, but overall this looks a sort of stupid work as of 2020 even though doable. This path looks more or less acceptable if you can accept performance hit from transfer through system memory, which makes things a tad easier to implement.
Stop using MPEG-2 decoder extension and implement your own decoder on top of lower level DXVA2 API and implement hardware assisted decoder without fallback to software, in which case you have more control over using GPU services and fitting to renderer's device.
I have a windows only Direct2D application and would like to implement a video playback system for cutscenes. These files are mp4 but the format can be changed, if need be.
It seems like DirectShow is the advised way to render video/audio on windows.
Now how do I let DirectShow render the video frames to my Direct2D render target?
The VMR-9 filter looks like the best route, but I can't seem to find an elegant way of integrating it into my application
There is no Direct2D/DirectShow interoperability layer in Windows. To fit these two technologies you would have to copy data between the APIs in a rather inefficient way (and this will still take some time to develop the fitting).
With H.264/HEVC MP4 video files you would be better off using Media Foundation to read and decode frames, then load them into Direct2D bitmaps and display in your application. Performance wise it is possible to transfer video frames to Direct2D bitmaps via GPU at reasonable cost and with reasonable development effort, but even if you make a shortcut and do integration roughly and inefficiently it will be on par with DirectShow.
I recommend to start with looking at reading and decoding video frames with Media Foundation Source Reader API. Once you get familiar with fitting the technologies, you will take next step and optimize the transfer by using GPU capacity and interop between Direct3D and Direct2D.
I am searching hardware accelerated(GPU) based video scaling.I found a extensive discussion in following threads
How to use hardware video scalers? and
Hardware Accelerated Image Scaling in windows using C++
I try to stick with MFT based scaling because i also using H.264 Encoder MFT in my application.
We have two option for MFT based solution-
1. Video Resizer DSP
2. Video Processor MFT
But both these methods used MF_SA_D3D_AWARE. As mentioned below:
A video MFT has the attribute MF_SA_D3D_AWARE.aspx which can be used to query whether it supports DirectX 3D hardware acceleration, and this can be enabled by sending it the MFT_MESSAGE_SET_D3D_MANAGER message.
and MF_SA_D3D_AWARE is supported in Windows 8 onwards.
Is their any MFT for scaling which uses Hardware acceleration in windows 7 ?
I haven't investigated other two options(MFCreateVideoRenderer and IDirectXVideoProcessor::VideoProcessBlt) mentioned in How to use hardware video scalers? that it will support in windows 7 or not. But I am actively looking for MFT option on priority.
Under Windows 7, i will recommand you to use IDXVAHD_VideoProcessor
You have a sample here : DXVA-HD Sample
But i think that if you use a simple DirectXDevice9 with a DirectXTexture9, the scaling result will be the same. There is no reason that dedicated scaling process only apply to video file process. I think they are the same for both (game and video file).
The lonely thing i saw, is that you can setup the constriction mode : DXVAHD_BLT_STATE_CONSTRICTION_DATA , wich apply to downscaling, not really to upscaling.
Pretty self explanatory. Microsoft had DirectShow for DirectX 9, but using DirectShow with DX11 is a COM nightmare beyond words. Is there a standard for video rendering I haven't heard of, or perhaps a free third-party library for this purpose?
Edit: Thanks to Mgetz, I am aware of Microsoft's attempt at a solution, Media Foundation. However, it's limited to Windows 8+, which I would much prefer to avoid.
This may not exactly match your requirement, but for your GOAL, you may take a look on ffmpeg, libx264 and theora(for ogg sound) or faad(decode aac).
I have done using ffmpeg to open container(3gp/mp4 is simple to implment yourself btw if full GPL licence is a concern), libx264 to decode to frame and upload to opengl texture, performance is good (on mac pro it can render 50 fps for 1080p without optimization) and by getting your hand dirty you can have fun doing stupid things with the texture and 3d transforms.
Media Foundation says that it "enables the development of applications and components for using digital media on Windows Vista and later."
So, it looks like it should work for Vista, Windows 7, and Windows 8.
There is DirectX Video Acceleration 2.0 which has a fabulous API, the DXVA-HD (after one has seen VMR9's API, especially with that custom allocator/presenter for renderless drawing, every other API is fabulous :) )
Have a look at: https://msdn.microsoft.com/en-us/library/windows/desktop/ee663586(v=vs.85).aspx
Also, there is a sample in: https://msdn.microsoft.com/en-us/library/windows/desktop/dd756740(v=vs.85).aspx
Windows 7 is the minimum supported windows version
You will not believe how straight forward it is with this API to have it decode the video into your texture.
Is there a good solution for playing a compressed video in OpenGL?
It needs to
Be cross-platform (Windows and MacOSX)
Render to a texture (preferably but not 100% needed)
Cost less than Bink
Any ideas?
Qt can be used to render widgets (including a video player) in an OpenGL scene. It has a multimedia framework called phonon that can play video and audio.
See this demo video.
Qt is cross-platform and is now licensed under LGPL.
I recommend the Theora video format.
Here are the benefits:
Totally open, free and patent-unencoumbered specification
Free working library implementation (encoder/decoder) and source-code examples, available under a BSD-style license
Not too shabby documentation.
Portable
The decoder lets you decode to R'G'B', which can easily be uploaded with an OpenGL buffer object and fetched in a shader via a sampler.
if you mean by solution that you can build/code it, i can suggest quicktime (easy on mac with cocoa, strange on windows but it works) or you can checkout mplayer/vlc sources and try to integrate that. there are a lot of demos about this on the web.
since you need cross platform, i guess gstreamer, video4linux and directshow are nothing for you. but there are video players that support different backends on different platforms - like openFrameworks