I am wrting a media player on Windows with C++ and D3D11 APIs. I have decoded video frames by GPU. And I convert the NV12 frames to RGB for swap chain presenting by using VideoProcessorBlt. The images are displayed on a window successfully.
But when I open the Windows Task Manager, it seems that the 3D, Video Decode, Video Processing modules of the GPU all are used. The utilization rate of the three modules keeps at about 7%.
Contrast with Chrome or Edge browser, the browsers can play an MP4 file just using Video Decode module and Video Processing module. The utilization rate of the 3D module is 0%.
How do they implement this? I am wondering how to render a frame efficiently. Thanks.
Related
I'm about to generate 2D and 3D music animations and render them to video using C++. I was thinking about using OpenGL, but I've read that, unfortunately, it is being discontinued in favour of Vulkan, which seems to offer higher performance using a GPU, but is also a lower-level API, making it more difficult to learn. I still have almost no knowledge in both OpenGL and Vulkan, beginning to learn now.
My question is:
is there a way to encode the Vulkan render output (showing a window or not) into a video file, preferentially through FFPMEG? If so, how could I do that?
Requisites:
Speed: the decrease in performance should be nearly that of encoding the video only, not much more than that (e.g. by having to save lossless frames as images first and then encoding a video from them).
Controllable FPS and resolution: the video fps and frame resolution can be freely chosen.
Reliability, reproducibility: running a code that gives a same Vulkan output twice should result in 2 equal videos independently of the system, i.e. no dropping frames, async problems (I want to sync with audio) or whatsoever. The chosen video fps should stay fixed (e.g. 60 fps), no matter if the computer can render 300 or 3 fps.
What I found out so far:
An example of taking "screenshots" from Vulkan output: it writes to a ppm image at the end, which is a binary uncompressed image file.
An encoder for rendering videos from OpenGL output, which is what I want, but using OpenGL in that case.
That Khronos includes in the Vulkan API a video subset.
A video tool to decode, demux, process videos using FFMPEG and Vulkan.
That is possible to render the output into a buffer without the need of a screen to display it.
First of all, ffmpeg is a framework used for video encoding and decoding. Second, if you have no experience with any of the GPU rendering API you should start with OpenGL. Vulkan is very low-level and complicated. OpenGL will be here for a very long time and will not be immediately replaced with Vulkan.
The off-screen rendering option you mentioned is probably the best one. It doesn't really matter though, you can also use the image from the framebuffer. The image is just a matrix of RGBA pixels. You need these data as the input for the video encoding. Please take a look at how ffmpeg works. You need to send the rendered frame data in the encoder which produces video packets that are stored in a video file. You need to chose a container (mp4, mkv, avi,...) and video format (h265, av1, vp9,...). You can of course implement a frame limiter and render the scene with a constant framerate or just pick the frames that have a constant timestep.
The performance problem happens, when you transfer the data from RAM to GPU memory and vice versa. For example, when downloading the rendered image from the buffer and passing it to the CPU encoder. Therefore, the most optimal approach would be with Vulkan, using the new video extension and directly sending the rendered frames in the HW accelerated encoder without any transfers from the GPU memory. You can also run the encoder in a different thread to make it work asynchronously.
But honestly, it's not trivial. The most simple solution (not realtime) for you to create a video from 3D render would be to:
Create a fixed FPS game loop
Make screenshots of the scene by downloading the framebuffer data in OGL or Vulkan
Process the frames by ffmpeg binary to create a video file
Another hack would be to use a screen recording software (OBS, Fraps, etc.) to create the video form your 3D app.
I'm totally new to OpenCv library and I'm implementing a simple client server application using Opencv and python. Here the client captures the video from the webcam and sends it to the server. I need to compress the video frame in order to reduce the bandwidth usage. As I could find we can save the frame in to a JPEG which is a loosy compression a technique. But using the provided method I have to write the frame into and JPEG image. What I need is without writing to an image rendering the low quality(compressed frame). What i'm currently doing is writing to a JPEG and reading it again. two IO cycles per a single frame is not efficient at all. Can anyone suggest a better solution?
cv2.imwrite('imageName.jpg', frame, [int(cv2.IMWRITE_JPEG_QUALITY), 90])
newFrame=cv2.imread('imageName.jpg')
cv2.imshow('preview',newFrame);
(frame= current image frame I captured,
newFrame=loading the saved image in to the programme)
I'm developing a multimedia streaming application for Desktop using SourceReader MediaFoundation technique.
I'm using USB camera device to show streaming. The camera supports 2-video formats: YUY2 and MJPG.
For 1980x1080p YUY2 video resolution, receiving only 48fps for 60fps. I fetched YUY2-RGB32 conversion from MSDN page and using in my application (Note: I didn't use any transform filter for color conversion).
For MJPG video format, I used MJPEG Decoder MFT to convert MJPG - YUY2 - RGB32 and then displaying on the window using Direct3D9. For specific resolution, I'm facing framerate drops from 60fps to 30fps(Ex: 1920x1080 60fps but drawing only 30-33fps).
Two ways, I verified in Graphedit to confirm about the filter:
Added MJPEG Decompressor filter and built the graph for MJPG video format to check fps for FullHD resolution and its showing 28fps for 60fps.
Added AVI Decompressor filter and built the graph for MJPG video format to check fps for FullHD resolution and its showing 60fps.
I have searched on many sites to find AVI decompressor for media foundation but no luck.
Anyone confirm, is there any filter available in MFT?
Microsoft ships [recent versions of] Windows with stock Motion JPEG decoders:
MJPEG Decompressor Filter for DirectShow
MJPEG Decoder MFT for Media Foundation
To my best knowledge those do not share codebases, however both are not supposed to be performance efficient decoders.
Your using GraphEdit means you are trying DirectShow decoders and AVI Decompressor is supposedly using another (Video for Windows) codec which you did not identify.
For Media Foundation, you might be able to use Intel Hardware M-JPEG Decoder MFT or NVIDIA MJPEG Video Decoder MFT is you have respective hardware and drivers. Presumably, vendor specific decoders deliver better performance, and also have higher priority compared to generic software peers. Other than this, for an MFT form factor you might need to look at commercial decoders and/or custom developed, as the API itself is not so much popular to offer a wide range of options.
I have recently purchased an Orbbec Astra camera, which uses the same technology and produces the same style depth map as a Microsoft Kinect.
What would be the correct file format to save the depth map frames, How would I go about saving the videos recorded?
I have been able to load a stream but am not sure what format the frames should be saved in so that i can load them for testing at a later stage and still have all the same information.
I am using OpenNI2, OpenCV3.1.0 and C++.
I am looking for ways to layer multiple 1080p Videos with transparency on Windows in C++ and DirectX or Opengl. The videos will start at different moments in time. Ideally the videos can be blended with another render target with other game content, so the resulting video texture should contain transparent pixels.
Can this be done with EVR and hardware acceleration? Which codecs are supported? http://en.wikipedia.org/wiki/Media_Foundation mentions transparency, but does not answer my questions. It sounds as if all the videos have to start at the same time and the resulting video texture has no transparency.
TIA
Christoph
This are my research results from around 03/14 with no definitive answer to this problem.
I did not try the mentioned possibility in Media Foundation, since it sounded as if the result has no transparency.
I was able to use a second gray scale video to mask the rgb video inside a shader. This can be done with a separate video stream, but syncing is needed. Moreover it is possible to encode a video with two frames side by side, but many HW accelerated video codecs do not allow this, WMF being the exception. Performance is not great but I was able to play 3 1080p30 videos simultaneously.
On a side note, to my surprise Flash was able to play 5+ 1080p30 videos with transparency simultaneously. The flash video codec allows alpha values, but I managed only inside flash to use them.