WIC HEIF HEIC image decoding C++ - c++

Windows Imaging Component is used for decoding heif images. However extra apps from the microsoft store (heif image extension, hevc video extensions) are required for a successful decoding of the images.
Without them the WIC api returns blank image. Now I want to be able to programatically determine whether the heif file can be correctly decoded.
I have tried to locate the existence of required decoder type using DXVA Checker which is supposed to be WIC_HEIF_Decoder. But I can't find it registered anywhere.
There is a GUID key however CLSID_WICHeifDecoder documented here which I think can be registered in the system even if the decoder is missing.
Does anyone have any idea how to do this?

If you plan to use WIC decoder, you should rather use WIC API to check the availability and not Media Foundation API, even if both decoders are known - at the moment - to be packaged together and belong to the same Windows Store application (extension).
You should be able to use IWICImagingFactory::CreateComponentEnumerator to enumerate decoders and identify if HEIF is among the available ones.
Microsoft HEIF Decoder
Class Identifier: CLSID_WICHeifDecoder
Signing Status: WICComponentSigned
Author: Microsoft
Vendor Identifier: {F0E749CA-EDEF-4589-A73A-EE0E626A2A2B}
Version: 1.0.0.0
Spec Version: 1.0.0.0
Friendly Name: Microsoft HEIF Decoder
IWICBitmapCodecInfo:
Container Format: GUID_ContainerFormatHeif
Pixel Formats: GUID_WICPixelFormat32bppBGR
Color Management Version: 1.0.0.0
MIME Types: image/heic,image/heif,image/avci,image/heic-sequence,image/heif-sequence,image/avcs,image/avif,image/avif-sequence
File Extensions: .heic,.heif,.avci,.heics,.heifs,.avcs,.avif,.avifs
Patterns: 576
You might prefer to use IWICImagingFactory::CreateDecoder with GUID_ContainerFormatHeif because you should be more interested in ability to decode rather that in specific decoder implementation.
Check for specific CLSID is trivial, using either registry query or direct CoCreateInstance call but it does not make much sense for the specified task.

Since the HEIF decoders are still recognized on Windows that can't decode them this is the best hack imo:
In order to decode HEIF images HEVC video extension should be installed on the machine.
So the right check is to see if there is any decoding type matching the HEVC input
MFStartup(MF_VERSION);
IMFActivate** activate {};
unsigned int count {};
// Set the HEVC GUID
MFT_REGISTER_TYPE_INFO input;
input.guidMajorType = MFMediaType_Video;
input.guidSubtype = MFVideoFormat_HEVC;
// Get all available output types for HEVC input
MFTEnumEx(MFT_CATEGORY_VIDEO_DECODER, MFT_ENUM_FLAG_SORTANDFILTER | MFT_ENUM_FLAG_SYNCMFT, &input, nullptr, &activate, &count);
// Release interface pointers
for (size_t i = 0; i < count; i++) {
activate[i]->Release();
}
CoTaskMemFree(activate);
MFShutdown();
return (count > 0);

Related

DirectShow universal media decoder

I am new to DirectShow API.
I want to decode a media file and get uncompressed RGB video frames using DirectShow.
I noted that all such operations should be completed through a GraphBuilder. Also, every the processing block is called a filter and there are many different filters for different media files. For example, for decoding H264 we should use "Microsoft MPEG-2 Video Decoder", for AVI files "AVI Splitter Filter" etc.
I would like to know if there is a general way (decoder) that can handle all those file types?
I would really appreciate if someone can point out an example that goes from importing a local file to decoding it into uncompressed RGB frames. All the examples I found are dealing with window handles and they just configure it and call pGraph->run(). I have also surfed through Windows SDK samples, but couldn't find useful samples.
Thanks very much in advance.
Universal DirectShow decoder in general is against the concept of DirectShow API. The whole idea is that individual filters are responsible for individual task (esp. decoding certain encoding or demultiplexing certain container format). The registry of the filters and Intelligent Connect let one to have the filters built in chain to do certain requested processing, in particular decoding from compressed format to 24-bit RGB for video.
From this standpoint you don't need a universal decoder and it is not expected that such decoder exists. However, such decoder (or close) does exist and it's a ffdshow or one of its derivatives. Presently, you might want to look at LAVFilters, for example. They wrap FFmpeg, which itself can handle many formats, and connect it to DirectShow API so that, as as filter, ffdshow could handle many formats/encodings.
There is no general rule to use or not use such codec pack, in most cases you take into consideration various factors and decide what to do. If your application handles various scenarios, a good starting point into graph building would be Overview of Graph Building.
My goal is to accomplish the task using DirectShow in order to have no external dependencies. Do you know a particular example that does uncompressing frames for some file type?
Your request is too broad and in the same time typical and, to some extent, fairy simple. If you spend some time playing with GraphEdit SDK tool, or rather GraphStudioNext, which is a more powerful version of the former, you will be able to build filter graph interactively, also render media files of different types and see what filters participate in rendering. You can accomplish the very same programmatically too, since the interactive actions basically all have matching API calls individually.
You will be able to see that specific formats are handled by different filters and Intelligent Connect mentioned above is building chains of filters in combinations in order to satisfy the requests and get the pipeline together.
Default use case is playback, and if you want to get video rendered to 24/32-bit RGB, your course of actions is pretty much similar: you are to build a graph, which just needs to terminate with something else. More flexible, sophisticated and typical for advanced development approach is to supply a custom video renderer filter and accept decompressed RGB frames on it.
A simple and so much popular version of the solution is to use Sample Grabber filter, initialize it to accept RGB, setup a callback on it so that your SampleCB callback method is called every time RGB frame is decompressed, and use Sample Grabber in the graph. (You will find really a lot of attempts to accomplish that if you search open source code and/or web for keywords ISampleGrabber, ISampleGrabberCB, SampleCB or BufferCB, MEDIASUBTYPE_RGB24).
Using the Sample Grabber
DirectShow: Examples for Using SampleGrabber for Grabbing a Frame and Building a VU Meter
Another more or less popular approach is to setup a playback pipeline, play a file, and read back frames from video presenter. This is suggested in another answer to the question, is relatively easy to do, and does the job if you don't have performance requirement and requirements to extract every single frame. That is, it is a good way to get a random RGB frame from the feed but not every/all frames. See related on this:
Different approaches on getting captured video frames in DirectShow
You are looking for vmr9 example in DirectShow library.
In your Windows SDK's install, look for this example:
Microsoft SDKs\Windows\v7.0\Samples\multimedia\directshow\vmr9\windowless\windowless.sln
And search this function: CaptureImage, in this method, see IVMRWindowlessControl9::GetCurrentImage, is exactly what you want.
This method captures a video frame in bitmap format (RGB).
Next, this is a copy of CaptureImage code:
BOOL CaptureImage(LPCTSTR szFile)
{
HRESULT hr;
if(pWC && !g_bAudioOnly)
{
BYTE* lpCurrImage = NULL;
// Read the current video frame into a byte buffer. The information
// will be returned in a packed Windows DIB and will be allocated
// by the VMR.
if(SUCCEEDED(hr = pWC->GetCurrentImage(&lpCurrImage)))
{
BITMAPFILEHEADER hdr;
DWORD dwSize, dwWritten;
LPBITMAPINFOHEADER pdib = (LPBITMAPINFOHEADER) lpCurrImage;
// Create a new file to store the bitmap data
HANDLE hFile = CreateFile(szFile, GENERIC_WRITE, FILE_SHARE_READ, NULL,
CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);
if (hFile == INVALID_HANDLE_VALUE)
return FALSE;
// Initialize the bitmap header
dwSize = DibSize(pdib);
hdr.bfType = BFT_BITMAP;
hdr.bfSize = dwSize + sizeof(BITMAPFILEHEADER);
hdr.bfReserved1 = 0;
hdr.bfReserved2 = 0;
hdr.bfOffBits = (DWORD)sizeof(BITMAPFILEHEADER) + pdib->biSize +
DibPaletteSize(pdib);
// Write the bitmap header and bitmap bits to the file
WriteFile(hFile, (LPCVOID) &hdr, sizeof(BITMAPFILEHEADER), &dwWritten, 0);
WriteFile(hFile, (LPCVOID) pdib, dwSize, &dwWritten, 0);
// Close the file
CloseHandle(hFile);
// The app must free the image data returned from GetCurrentImage()
CoTaskMemFree(lpCurrImage);
// Give user feedback that the write has completed
TCHAR szDir[MAX_PATH];
GetCurrentDirectory(MAX_PATH, szDir);
// Strip off the trailing slash, if it exists
int nLength = (int) _tcslen(szDir);
if (szDir[nLength-1] == TEXT('\\'))
szDir[nLength-1] = TEXT('\0');
Msg(TEXT("Captured current image to %s\\%s."), szDir, szFile);
return TRUE;
}
else
{
Msg(TEXT("Failed to capture image! hr=0x%x"), hr);
return FALSE;
}
}
return FALSE;
}

Media Foundation AMR decode

I have a file with .amr extension, and I want to get it's sample rate and number of channels using Microsoft Media Foundation. Further, I want to decode and get the uncompressed data.
I can successfully get those from .aac .mp4 and other file types but not from from a .amr file (or 3.gp file which contains .amr track).
So, for other types I do:
IMFSourceReader *m_pReader;
IMFMediaType *m_pAudioType;
MFCreateSourceReaderFromURL(filePath, NULL, &m_pReader);
m_pReader->SetStreamSelection(MF_SOURCE_READER_ALL_STREAMS, false);
m_pReader->SetStreamSelection(MF_SOURCE_READER_FIRST_AUDIO_STREAM, true);
m_pReader->GetCurrentMediaType(MF_SOURCE_READER_FIRST_AUDIO_STREAM, &m_pAudioType);
UINT32 numChannels,sampleRate;
m_pAudioType->GetUINT32(MF_MT_AUDIO_NUM_CHANNELS, &numChannels);
m_pAudioType->GetUINT32(MF_MT_AUDIO_SAMPLES_PER_SECOND, &sampleRate);
Consider there are no any errors during this code.
For .amr files, some garbage is being written in the numChannels and sampleRate.
Does anyone have experience with this and knows how to recognize and/or get proper channels and sample rate for further decoding?
BTW, Windows Media Player plays this file with no problems.
Thanks in advance.
So I found out that it supports decoding for .amr files not encoding.
Just before we get this properties:
UINT32 numChannels,sampleRate;
m_pAudioType->GetUINT32(MF_MT_AUDIO_NUM_CHANNELS, &numChannels);
m_pAudioType->GetUINT32(MF_MT_AUDIO_SAMPLES_PER_SECOND, &sampleRate);
We have to set a new media type to our Source Reader
m_pAudioType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Audio)
m_pAudioType->SetGUID(MF_MT_SUBTYPE, MFAudioFormat_Float)
m_pReader->SetCurrentMediaType(MF_SOURCE_READER_FIRST_AUDIO_STREAM, NULL, m_pAudioType);

list file extensions supported by OpenCV

In OpenCV, I see imread() and VideoCapture() both take a string to a file path of multiple extensions. Is there a way to get a list of extensions that are supported by them? For example, getting a list of "jpg", "png", "mov", "mpg", etc.? I assume this is system dependent and others have needed to query this at runtime.
Furthermore, how is support determined? If have something like the below code and the Mat I get back always seems partially corrupted (I can see a bit of the image). It doesn't seem to change regardless of the frame number I ask for. I can play this video in my video player "totem", but I'm not even sure if totem and OpenCV are even using the same codec for this file.
Mat fromVideo(std::string _videoPath, int frame) {
VideoCapture capture(_videoPath);
Mat f;
for (int i = 0; i < frame; i++) {
capture >> f;
}
return f;
}
For imread() (more info here):
Windows bitmaps - *.bmp, *.dib (always supported)
JPEG files - *.jpeg, *.jpg, *.jpe (see the Notes section)
JPEG 2000 files - *.jp2 (see the Notes section)
Portable Network Graphics - *.png (see the Notes section)
Portable image format - *.pbm, *.pgm, *.ppm (always supported)
Sun rasters - *.sr, *.ras (always supported)
TIFF files - *.tiff, *.tif (see the Notes section)
For VideoCapture():
AVI files - *.avi
It seems that AVI is the only format with decent cross-platform support. See here for more info.
Use the method cv::VideoCapture::isOpened() to make sure that the constructor was successful in initializing the VideoCapture object.
Note that even if it was possible to get a list of supported container formats from OpenCV (AVI, MKV for instance) with their typical filename extensions, you would still need to know the exact list of supported codecs (and even then the exact file you want to open might be corrupted, etc...). So a list of filename extensions is not enough to accurately describe what is internally supported by OpenCV, and the simplest solution at the OpenCV API level is this isOpened() method.
Just update:
cv::VideoCapture cap("D:\\test.mp4")
works for me.

Raw Audio File to AAC using Windows Media Foundation on Windows 7

Thanks for taking some time to read my question.
I'm developping a C++ application using Qt and windows API.
I'm recording the microphone output in small 10s audio files in raw format, and I want to convert them to aac format.
I have tried to read as many things as I could, and thought it would be a great idea to start from windows media foundation transcode API.
Problem is, I can't seem to use a .raw or .pcm file in the "CreateObjectFromUrl" function, and so I'm pretty much stuck here for the moment. It keeps on failing. The hr return code equals 3222091460. I have tried to pass an .mp3 file to the function and of course it works, so no url-human-failure involved.
MF_OBJECT_TYPE ObjectType = MF_OBJECT_INVALID;
IMFSourceResolver* pSourceResolver = NULL;
IUnknown* pUnkSource = NULL;
// Create the source resolver.
hr = MFCreateSourceResolver(&pSourceResolver);
if (FAILED(hr))
{
qDebug() << "Failed !";
}
// Use the source resolver to create the media source.
hr = pSourceResolver->CreateObjectFromURL(
sURL, // URL of the source.
MF_RESOLUTION_MEDIASOURCE, // Create a source object.
NULL, // Optional property store.
&ObjectType, // Receives the created object type.
&pUnkSource // Receives a pointer to the media source.
);
The MFCreateSourceResolver works fine, but CreateObjectFromURL does not succeed :(
So I have two questions for you folks :
Is it possible to encode raw audio files to aac files using windows media foundation ?
If yes, what should I read to accomplish what I want ?
I want to point out that I can't just use ffmpeg or libav because I can't afford any license for my software, and don't want it to be under the GPL license. But if there are alternatives to windows media foundations to encode raw audio files to aac, I would be glad to hear them.
And finally, sorry for my bad english, this is obviously not my native language and I'm sorry if I made your eyes bleed. (and happy if I made you laugh)
Have a nice day
The hr return code equals 3222091460
Those are HRESULT codes. Use this "ShowHresult" tool to have them conveniently decoded for you. The code means 0xC00D36C4 MF_E_UNSUPPORTED_BYTESTREAM_TYPE "The byte stream type of the given URL is unsupported."
The problem is basically that there is no support for these raw files, .WAV is a good source for raw audio - the file holds both format descriptor and the payload.
You can obviously read data from the raw audio file yourself and compress into AAC using Media Foundation's AAC Encoder via its IMFTransform interface. This is reasonably easy and you have AAC data on the output to e.g. write into raw .AAC.
Alternate options to Media Foundation is DirectShow (there are suitable codecs, though I thought it might be not so easy to start), libfaac, FFmpeg's libavcodec (available under LGPL, not GPL).

Virtual Webcam Driver

I want to develop a virtual webcam driver which from User mode I'll pass image to it and it will display as webcam output.
I don't want to use DirectX filter and CSourceStream etc. Because they don't work on some programs which doesn't use DirectX for capturing webcam image.
I have to write a kernel mode device driver so.
Any ideas? I tried testcap from DDK samples, but it doesn't process image from user mode and doesn't get any input, just it displays 7 colors in webcam...
Any help would be greatly appreciated.
Thanks
Thank you all!
I tried code from here:
http://tmhare.mvps.org/downloads.htm (find Capture Source Filter)
It worked well when I compiled it in Yahoo, MSN, but it crashed AIM, Internet Explorer Flash Webcam, Firefox Flash webcam and Skype... I got crash in QueryInterface after 8 time call to that, I found it with tracing it with a lot of tricks..
Now I know, it crashes on 8th call to
HRESULT CVCamStream::QueryInterface(REFIID riid, void **ppv)
8th call when it reaches to last if, I mean:
return CSourceStream::QueryInterface(riid, ppv);
It's in 17th line of Filters.cpp
Why do you think I'm getting crash??
Thank you all for guiding me to find correct solution which is DirectShow, not driver
There are several APIs from Microsoft which provide access to image data.
Twain: Used for single image capture from scanners, etc.
WIA: This seems to have degenerated to a single image codec library.
VfW: A very old (Win16) API which really works only Video-File encoding/decoding, but has support for some video acquisition.
DirectShow: previously part in the DirectX SDK, currently in the Platform SDK. This is the place to go for current (general) streaming solutions.
Windows Media/Media Foundation: This seems more to be geared at video playback/reencoding.
Manufacturer Specific Libraries: Pylon/Halcon/Imaging Control/...
DirectShow specific :
To create image acquisition devices under windows, you have to provide either a device (driver) which implements the streamclasses interfaces (or newer Avstream) or you have to write a usermode COM object which has to be added to the VideoInputCategory enumerator.
The Avstream sample provides everything for a real image acquisition device. Only the lower layer for the actual device really is missing.
If you can design a device, you should either create it DCAM or UVC compatible. For both there are built-in drivers supplied by windows.
How to write a software source device :
You have to create a DirectShow filter which provides at least one output pin and register this under the VideoInputCategory. There may be several interfaces certain applications require from a capture application, but these depend on the application itself. Simple applications to try out filters are GraphEdit and AMCap which are supplied in the Plattform SDK.
Some code :
#include <InitGuid.h>
#include <streams.h>
const AMOVIESETUP_MEDIATYPE s_VideoPinType =
{
&MEDIATYPE_Video, // Major type
&MEDIATYPE_NULL // Minor type
};
const AMOVIESETUP_PIN s_VideoOutputPin =
{
L"Output", // Pin string name
FALSE, // Is it rendered
TRUE, // Is it an output
FALSE, // Can we have none
FALSE, // Can we have many
&CLSID_NULL, // Connects to filter
NULL, // Connects to pin
1, // Number of types
&s_VideoPinType // Pin details
};
const AMOVIESETUP_FILTER s_Filter =
{
&CLSID_MyFilter, // Filter CLSID
L"bla", // String name
MERIT_DO_NOT_USE, // Filter merit
1, // Number pins
&s_VideoOutputPin // Pin details
};
REGFILTER2 rf2;
rf2.dwVersion = 1;
rf2.dwMerit = MERIT_DO_NOT_USE;
rf2.cPins = 1;
rf2.rgPins = s_Filter.lpPin;
HRESULT hr = pFilterMapper->RegisterFilter( CLSID_MyFilter, _FriendlyName.c_str(), 0,
&CLSID_VideoInputDeviceCategory, _InstanceID.c_str(), &rf2 );
if( FAILED( hr ) )
{
return false;
}
std::wstring inputCat = GUIDToWString( CLSID_VideoInputDeviceCategory );
std::wstring regPath = L"CLSID\\" + inputCat + L"\\Instance";
win32_utils::CRegKey hKeyInstancesDir;
LONG rval = openKey( HKEY_CLASSES_ROOT, regPath, KEY_WRITE, hKeyInstancesDir );
if( rval == ERROR_SUCCESS )
{
win32_utils::CRegKey hKeyInstance;
rval = createKey( hKeyInstancesDir, _InstanceID, KEY_WRITE, hKeyInstance );
....
_InstanceID is a GUID created for this 'virtual device' entry.
You can not decide how other program would call your driver. Most of programs will use DirectShow. Some would use the win3.x technology VFW. Many new programs, including Windows XP's scanner and camera wizard, may call you via the WIA interface. If you do not want to implement all that, you need to at least provide the DirectShow interface via WDM and let vfwwdm32.dll gives you a VFW interface, or write your own VFW driver.