I am doing some preliminary work to make a rendering pipeline and I am investigating whether OpenGL is a good option for my use case: from a markup language I need to generate a video, ideally using opengl which already implements most of the primitives I need.
Is there a way to, instead of (or additionally to) updating a framebuffer, to make an mp4 video file using nvenc, without copying data back and forth between the GPU's and main memory?
The nvenc SDK page[1] on the NVidia website suggests that it can, as the current header graphic is of a game being streamed. (Even if it's a Direct3D game, same chip underneath.) A quick search for "nvenc share buffer with OpenGL" turned up a number of people apparently combining the two.
Runs on Linux and MS Windows only, so no joy if you have a Mac.
Hope this helps.
Related
I've read a lot of posts describing how people use AVAssetReader or AVPlayerItemVideoOutput to get video frames as raw pixel data from a video file, which they then use to upload to an OpenGL texture. However, this seems to create the needless step of decoding the video frames with the CPU (as opposed to the graphics card), as well as creating unnecessary copies of the pixel data.
Is there a way to let AVFoundation own all aspects of the video playback process, but somehow also provide access to an OpenGL texture ID it created, which can just be drawn into an OpenGL context as necessary? Has anyone come across anything like this?
In other words, something like this pseudo code:
initialization:
open movie file, providing an opengl context;
get opengl texture id;
every opengl loop:
draw texture id;
If you were to use the Video Decode Acceleration Framework on OS X, it will give you a CVImageBufferRef when you "display" decoded frames, which you can call CVOpenGLTextureGetName (...) on to use as a native texture handle in OpenGL software.
This of course is lower level than your question, but it is definitely possible for certain video formats. This is the only technique that I have personal experience with. However, I believe QTMovie also has similar functionality at a much higher level, and would likely provide the full range of features you are looking for.
I wish I could comment on AVFoundation, but I have not done any development work on OS X since 10.6. I imagine the process ought to be similar though, it should be layered on top of CoreVideo.
I just can't find them anywhere. The most important part for me is the hardware acceleration, and I have no idea if there is a performance or openGL version compatibility requirement that the video card has to follow.
The minimum system requirements will depend alot more on the application that you are writing than what SDL2 does.
If you just create a standard window and render SDL will use what it can find and what it thinks is best either OpenGL, OpenGL ES, Direct3D or use the old style software rendering for machines that can't do any of the other. So if a computer can support an OS that SDL runs on then you will almost always (I just said almost since there can possible be exceptions) be able to run these type of apps (Video card not a requirement, but having one will greatly increase programs drawing speed).
You can also be creating a OpenGL application directly and then it depends on what type of context you are making what the video card has to support.
You can find most of the information here: http://wiki.libsdl.org/moin.fcg/MigrationGuide
under the Video section. It's actually how to port from 1.2 to 2.0 , but it explains the new Video Pipeline pretty well.
Hope thats what you were looking for.
I am trying to play a .flv file in the GLUT window using OpenGL and C++ in Linux, but I'm not sure where to start.
Is it possible to do this? If so, how?
Make sure you mean .flv not .swf.
It's quite easy. Decode the video with something like libavcodec and you can use raw frames as textures.
If you really want to do this, check out the source code of Gnash. They've a renderer that use OpenGL. However, rendering is just a small part of the job, you also have to decode audio/video, run actionscript, etc.. in order to run a flash file.
It so complicated that even Adobe didn't manage to make it right :)
If you want to play just some video, look at #Banthar's answer, otherwise:
OpenGL is a no-frills drawing API. It gives you the computer equivalent of "pens and brushes" to draw on some framebuffer. Period. No higher level functionality than that.
Flash it a really complex thing. It consists of a vector geometry object system, a script engine (ActionScript), provides sound and video de-/compression etc. All this must be supported by a SWF player. ATM there's only one fully featured SWF player and that's the one Adobe makes. There are free alternatives, but the are behind the official flash players by several major versions (Lightspark, Gnash).
So the most viable thing to do was loading the Flash player browser plugin in your program through the plugin interface, provide it, what a browser provided to a plugin (DOM, HTTP transport, etc.) and have the plugin render to a offscreen buffer which you then copy to OpenGL context. But that's not very efficient.
TL;DR: Complicated as sh**, and probably not worth the effort.
Is it possible to allocate some memory on the GPU without cuda?
i'm adding some more details...
i need to get the video frame decoded from VLC and have some compositing functions on the video; I'm doing so using the new SDL rendering capabilities.
All works fine until i have to send the decoded data to the sdl texture... that part of code is handled by standard malloc which is slow for video operations.
Right now i'm not even sure that using gpu video will actually help me
Let's be clear: are you are trying to accomplish real time video processing? Since your latest update changed the problem considerably, I'm adding another answer.
The "slowness" you are experiencing could be due to several reasons. In order get the "real-time" effect (in the perceptual sense), you must be able to process the frame and display it withing 33ms (approximately, for a 30fps video). This means you must decode the frame, run the compositing functions (as you call) on it, and display it on the screen within this time frame.
If the compositing functions are too CPU intensive, then you might consider writing a GPU program to speed up this task. But the first thing you should do is determine where the bottleneck of your application is exactly. You could strip your application momentarily to let it decode the frames and display them on the screen (do not execute the compositing functions), just to see how it goes. If its slow, then the decoding process could be using too much CPU/RAM resources (maybe a bug on your side?).
I have used FFMPEG and SDL for a similar project once and I was very happy with the result. This tutorial shows to do a basic video player using both libraries. Basically, it opens a video file, decodes the frames and renders them on a surface for displaying.
You can do this via Direct3D 11 Compute Shaders or OpenCL. These are similar in spirit to CUDA.
Yes, it is. You can allocate memory in the GPU through OpenGL textures.
Only indirectly through a graphics framework.
You can use OpenGL which is supported by virtually every computer.
You could use a vertex buffer to store your data. Vertex buffers are usually used to store points for rendering, but you can easily use it to store an array of any kind. Unlike textures, their capacity is only limited by the amount of graphics memory available.
http://www.songho.ca/opengl/gl_vbo.html has a good tutorial on how to read and write data to vertex buffers, you can ignore everything about drawing the vertex buffer.
What is the best/easiest way to display a video (with sound!) in an application using XAudio2 and Direct3D9/10?
At the very least it needs to be able to stream potentially larger videos, and take care of the fact that the windows aspect ratio may differ from the videos (eg by adding letter boxes), although ideally Id like the ability to embed the video into a 3D scene.
I could of course work out a way to load each frame into a texture, discarding/reusing the textures once rendered, and playing the audio separately through XAudio2, however as well as writing a loader for at least one format, ive also got to deal with stuff like synchronising the video and audio components, so hopefully there is an eaier solution available or even a ready made free one with a suitable lisence (commercial distribution in binary form, dynamic linking is fine in the case of say LGPL).
In Windows SDK, there is a DirectShow example for rendering video to texture. It handles audio output too.
But there are limitations and I can't honestly call it easy.
Have you looked at Bink video? Its what lots of games use for video playback. Works great and you don't have to code all that video stuff yourself from scratch.