I'm about to generate 2D and 3D music animations and render them to video using C++. I was thinking about using OpenGL, but I've read that, unfortunately, it is being discontinued in favour of Vulkan, which seems to offer higher performance using a GPU, but is also a lower-level API, making it more difficult to learn. I still have almost no knowledge in both OpenGL and Vulkan, beginning to learn now.
My question is:
is there a way to encode the Vulkan render output (showing a window or not) into a video file, preferentially through FFPMEG? If so, how could I do that?
Requisites:
Speed: the decrease in performance should be nearly that of encoding the video only, not much more than that (e.g. by having to save lossless frames as images first and then encoding a video from them).
Controllable FPS and resolution: the video fps and frame resolution can be freely chosen.
Reliability, reproducibility: running a code that gives a same Vulkan output twice should result in 2 equal videos independently of the system, i.e. no dropping frames, async problems (I want to sync with audio) or whatsoever. The chosen video fps should stay fixed (e.g. 60 fps), no matter if the computer can render 300 or 3 fps.
What I found out so far:
An example of taking "screenshots" from Vulkan output: it writes to a ppm image at the end, which is a binary uncompressed image file.
An encoder for rendering videos from OpenGL output, which is what I want, but using OpenGL in that case.
That Khronos includes in the Vulkan API a video subset.
A video tool to decode, demux, process videos using FFMPEG and Vulkan.
That is possible to render the output into a buffer without the need of a screen to display it.
First of all, ffmpeg is a framework used for video encoding and decoding. Second, if you have no experience with any of the GPU rendering API you should start with OpenGL. Vulkan is very low-level and complicated. OpenGL will be here for a very long time and will not be immediately replaced with Vulkan.
The off-screen rendering option you mentioned is probably the best one. It doesn't really matter though, you can also use the image from the framebuffer. The image is just a matrix of RGBA pixels. You need these data as the input for the video encoding. Please take a look at how ffmpeg works. You need to send the rendered frame data in the encoder which produces video packets that are stored in a video file. You need to chose a container (mp4, mkv, avi,...) and video format (h265, av1, vp9,...). You can of course implement a frame limiter and render the scene with a constant framerate or just pick the frames that have a constant timestep.
The performance problem happens, when you transfer the data from RAM to GPU memory and vice versa. For example, when downloading the rendered image from the buffer and passing it to the CPU encoder. Therefore, the most optimal approach would be with Vulkan, using the new video extension and directly sending the rendered frames in the HW accelerated encoder without any transfers from the GPU memory. You can also run the encoder in a different thread to make it work asynchronously.
But honestly, it's not trivial. The most simple solution (not realtime) for you to create a video from 3D render would be to:
Create a fixed FPS game loop
Make screenshots of the scene by downloading the framebuffer data in OGL or Vulkan
Process the frames by ffmpeg binary to create a video file
Another hack would be to use a screen recording software (OBS, Fraps, etc.) to create the video form your 3D app.
I am wrting a media player on Windows with C++ and D3D11 APIs. I have decoded video frames by GPU. And I convert the NV12 frames to RGB for swap chain presenting by using VideoProcessorBlt. The images are displayed on a window successfully.
But when I open the Windows Task Manager, it seems that the 3D, Video Decode, Video Processing modules of the GPU all are used. The utilization rate of the three modules keeps at about 7%.
Contrast with Chrome or Edge browser, the browsers can play an MP4 file just using Video Decode module and Video Processing module. The utilization rate of the 3D module is 0%.
How do they implement this? I am wondering how to render a frame efficiently. Thanks.
I'm developing a multimedia streaming application for Desktop using SourceReader MediaFoundation technique.
I'm using USB camera device to show streaming. The camera supports 2-video formats: YUY2 and MJPG.
For 1980x1080p YUY2 video resolution, receiving only 48fps for 60fps. I fetched YUY2-RGB32 conversion from MSDN page and using in my application (Note: I didn't use any transform filter for color conversion).
For MJPG video format, I used MJPEG Decoder MFT to convert MJPG - YUY2 - RGB32 and then displaying on the window using Direct3D9. For specific resolution, I'm facing framerate drops from 60fps to 30fps(Ex: 1920x1080 60fps but drawing only 30-33fps).
Two ways, I verified in Graphedit to confirm about the filter:
Added MJPEG Decompressor filter and built the graph for MJPG video format to check fps for FullHD resolution and its showing 28fps for 60fps.
Added AVI Decompressor filter and built the graph for MJPG video format to check fps for FullHD resolution and its showing 60fps.
I have searched on many sites to find AVI decompressor for media foundation but no luck.
Anyone confirm, is there any filter available in MFT?
Microsoft ships [recent versions of] Windows with stock Motion JPEG decoders:
MJPEG Decompressor Filter for DirectShow
MJPEG Decoder MFT for Media Foundation
To my best knowledge those do not share codebases, however both are not supposed to be performance efficient decoders.
Your using GraphEdit means you are trying DirectShow decoders and AVI Decompressor is supposedly using another (Video for Windows) codec which you did not identify.
For Media Foundation, you might be able to use Intel Hardware M-JPEG Decoder MFT or NVIDIA MJPEG Video Decoder MFT is you have respective hardware and drivers. Presumably, vendor specific decoders deliver better performance, and also have higher priority compared to generic software peers. Other than this, for an MFT form factor you might need to look at commercial decoders and/or custom developed, as the API itself is not so much popular to offer a wide range of options.
I have recently purchased an Orbbec Astra camera, which uses the same technology and produces the same style depth map as a Microsoft Kinect.
What would be the correct file format to save the depth map frames, How would I go about saving the videos recorded?
I have been able to load a stream but am not sure what format the frames should be saved in so that i can load them for testing at a later stage and still have all the same information.
I am using OpenNI2, OpenCV3.1.0 and C++.
I'm creating an augmented reality application in OpenGL where I want to augment a video stream captured by a Kinect with virtual objects. I found some running code using fixed function pipeline OpenGL that creates a glTexImage2D, fills it with the image data and then creates a GL_QUAD with glTexCoord2f to fill the screen.
I'm looking for an optimized solution using modern, shader-based OpenGL only which is also capable of handling HD video streams.
I guess what I hope for as an answer to my question is a list of possibilities on how rendering a camera video stream can be achieved in OpenGL from which I can select the one that best fits my needs.