I am developing mobile games on windows. Our image resources are in the PVR-TC 4 format. When we run our game on simulator, images are decoded by CPU which is really slow, as our PC graphic card don't support GPU decode. Is it possible to make PC OpenGL support PVR-TC or ETC hardware decode?
You cannot force an implementation to implement a particular extension or image format.
Your best bet is to convert the images yourself offline. That is, instead of loading images of a format your hardware can't handle, load images of the format that it can.
After all, it's not like the images are originally in PVRTC format, right? They were originally authored in a regular format like PNG or whatever, then converted to PVRTC. So just add another conversion for S3TC or whatever format desktop hardware actually supports.
Related
What I am doing is trying to record the screen in windows XP and Win7. I got the bitmap by using DirectX's interface CreateOffscreenPlainSurface and GetFrontBufferData. I need to encode the bitmap into a H.264 format video. The problem is the bitmap captured is in format D3DFMT_A8R8G8B8, but the H.264 Video Encoder can only support MFVideoFormat_I420, MFVideoFormat_IYUV, MFVideoFormat_NV12, MFVideoFormat_YUY2 and MFVideoFormat_YV12 as input. My question is do I need to transfer the format myself(I do not want to)? Are there any other better solutions for this?
The input format corresponds to MFVideoFormat_ARGB32.
Stock OS component that handles the conversion is Video Processor MFT. I don't see availability information in the footer of MSDN article, however I am under impression that this MFT comes with Windows Vista, just like the whole Media Foundation API.
In Windows XP there has been a similar Color Converter DSP which offers really close services, and exposes a really close interface of DirectX Media Object (DMO). It is available in all more recent operating systems, however it is software only and never leverages GPU capability for the conversion.
These both can handle the requested format conversion for you.
Also for the reference, H.264 Video Encoder was introduced with Windows 7 only.
I need to decode video but my video player only supports RGB8 pixel format. So I'm looking into how to do pixel format conversion in the GPU, preferably in the decoding process, but if not possible, after it.
I've found How to set decode pixel format in libavcodec? which explains how to decode video on ffmpeg to an specific pixel format, as long as it's suported by the codec.
Basically, get_format() is a function which chooses, from a list of supported pixel formats from the codec, a pixel format for the decoded video. My questions are:
Is this list of supported codec output formats the same for all computers? For example, if my codec is for H264, then it will always give me the same list on all computers? (assuming same ffmpeg version of all computers)
If I choose any of these supported pixel formats, will the pixel format conversion always happen in the GPU?
If some of the pixel format conversions won't happen in the GPU, then my question is: does sws_scale() function converts in the GPU or CPU?
It depends. First, H264 is just a Codec standard. While libx264 or openh264 are implementing this standard you can guess that each implementation supports different formats. But let's assume (as you did in your question) you are using the same implementation on different machines then yes there might be still cases where different machines support different formats. Take H264_AMF for example. You will need an AMD graphics card to use the codec and the supported formats will depend on your graphics card as well.
Decoding will generally happen on your CPU unless you explicitly specify a hardware decoder. See this example for Hardware decoding: https://github.com/FFmpeg/FFmpeg/blob/release/4.1/doc/examples/hw_decode.c
When using Hardware decoding you are heavily relying on your machine. And each Hardware encoder will output their own (proprietary) frame format e.g. NV12 for a Nvida Graphics Card. Now comes the tricky part. The encoded frames will remain on your GPU memory which means you might be able to reuse the avframe buffer to do the pixel conversion using OpenCL/GL. But achieving GPU zero-copy when working with different frameworks is not that easy and I don't have enough knowledge to help you there. So what I would do is to download the decoded frame from the GPU via av_hwframe_transfer_data like in the example.
From this point on it doesn't make much of a difference if you used hardware or software decoding.
To my knowledge sws_scale isn't using hardware acceleration. Since it's not accepting "hwframes". If you want to do color conversion on Hardware Level you might wanna take a look at OpenCV you can use GPUMat there then upload your frame, call cvtColor and download it again.
Some general remarks:
Almost any image operation scaling etc. is faster on your GPU, but uploading and downloading the data can take ages. For single operations, it's often not worth using your GPU.
In your case, I would try to work with CPU decoding and CPU color conversion first. Just make sure to use well threaded and vectorized algorithms like OpenCV or Intel IPP. If you still lack performance then you can think about Hardware Acceleration.
I'm looking for a way to display videos in a 2D game. The videos need to support an alpha-channel so they can be overlayed on top of the other game elements.
Currently I just have a series of PNG files which are decompressed and then flipped through for the animation. This works, but it is a massive memory hog; a 1024x1024 animation that is 5 seconds long at 24 frames per second takes up well over 400MB. And I'm targeting embedded systems, so this is really not good.
I've been looking for some video codecs that can support these requirements, yet so far all I've really been able to come up with that support RGBA are licensed under GPL, so we can't use them in a commercial product.
Any such beast(s) out there?
Most codecs don't support an alpha channel - the only one I can think of is the QuickTime animation codec, which isn't very popular.
If you only need binary alpha channel (transparent or not) then setting the top bit of one of the color channels is a common approach.
If these are animation type frames then something like MJPEG might work well and there are lots of LGPL licensed mjpeg libs
What is the best/easiest way to display a video (with sound!) in an application using XAudio2 and Direct3D9/10?
At the very least it needs to be able to stream potentially larger videos, and take care of the fact that the windows aspect ratio may differ from the videos (eg by adding letter boxes), although ideally Id like the ability to embed the video into a 3D scene.
I could of course work out a way to load each frame into a texture, discarding/reusing the textures once rendered, and playing the audio separately through XAudio2, however as well as writing a loader for at least one format, ive also got to deal with stuff like synchronising the video and audio components, so hopefully there is an eaier solution available or even a ready made free one with a suitable lisence (commercial distribution in binary form, dynamic linking is fine in the case of say LGPL).
In Windows SDK, there is a DirectShow example for rendering video to texture. It handles audio output too.
But there are limitations and I can't honestly call it easy.
Have you looked at Bink video? Its what lots of games use for video playback. Works great and you don't have to code all that video stuff yourself from scratch.
In the past I've saved RGB images (generated from physical simulations) as 8-bits/channel PPM or PNG or JPEG.
Now I want to preserve the dynamic range of the simulation output, which means saving a floating point image and then treating conversion to 8-bits/channel as a post-processing step (so I can tweak the conversion to 8-bit without running the lengthy simulation again).
Has a "standard" floating point image format emerged ?
Good free supporting libraries/viewers/manipulation tools, preferably available in Debian, would be a bonus.
Have you looked into Radiance RGBE (.hdr) and OpenEXR (.exr). RGBE has some source code here. NVIDIA and ATI both support EXR data in their graphics cards. There are source code and binaries from the OpenEXR download page. ILM created OpenEXR and it has wide support. OpenEXR has support for 16 and 32 bit floating point per channel, and is what most people use these days, unless they've written their own format.
The Pixel Image Editor for linux has EXR support for editing, too.
pfstools is also necessary if you're going to work with HDR on linux. Its a set of command line programs for reading, writing and manipulating HDR and has Qt and OpenGL viewers.
Theres also jpeg2exr for linux
Heres some other debian packages for OpenEXR viewers.
Based on this, it looks like theres also a Gimp plugin somewhere.
It looks like the modern incarnation of FITS would fit your stated needs, but I would also suggest you consider using a 2D histogram structure from one of the good analysis packages in wide use by the physics community: ROOT or AIDA are the modern ones that I am familiar with.
NB: It's been more than a decade since I used FITS for anything, but I recall it begin a nice and flexible way to store fairly raw data.
For future reference, also rather widespread is the TIFF format. You can use the free and open-source LibTIFF for I/O.