Playing music with Media Engine on Windows 8 - c++

I am porting a Windows Phone 8 app to Windows 8 and it seems that Media Engine library works differently.
Here is my initialization code that works on WP8:
DX::ThrowIfFailed(
MFStartup(MF_VERSION)
);
ComPtr<IMFMediaEngineClassFactory> mediaEngineFactory;
ComPtr<IMFAttributes> mediaEngineAttributes;
// Create the class factory for the Media Engine.
DX::ThrowIfFailed(
CoCreateInstance(CLSID_MFMediaEngineClassFactory, nullptr, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&mediaEngineFactory))
);
// Define configuration attributes.
DX::ThrowIfFailed(
MFCreateAttributes(&mediaEngineAttributes, 1)
);
ComPtr<MediaEngineNotify> notify = Make<MediaEngineNotify>();
ComPtr<IUnknown> unknownNotify;
DX::ThrowIfFailed(
notify.As(&unknownNotify)
);
DX::ThrowIfFailed(
mediaEngineAttributes->SetUnknown(MF_MEDIA_ENGINE_CALLBACK, unknownNotify.Get())
);
// Create the Media Engine.
DX::ThrowIfFailed(
mediaEngineFactory->CreateInstance(0, mediaEngineAttributes.Get(), &m_mediaEngine)
);
CreateInstance() throws 0xc00d36e6 exception ( MF_E_ATTRIBUTENOTFOUND ).
I've tried to search for samples for Media Engine playback of mp3s but can only find Video Playback samples.
Any ideas?

I figured out the missing attribute. I had to add MF_MEDIA_ENGINE_AUDIO_CATEGORY to specify what M the Media Engine is actually doing. Here is an example that works for both WP8 and WIN8:
#if PLATFORM_WINDOWS8
DX::ThrowIfFailed(
MFStartup(MF_VERSION)
);
#endif
ComPtr<IMFMediaEngineClassFactory> mediaEngineFactory;
ComPtr<IMFAttributes> mediaEngineAttributes;
// Create the class factory for the Media Engine.
DX::ThrowIfFailed(
CoCreateInstance(CLSID_MFMediaEngineClassFactory, nullptr, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&mediaEngineFactory))
);
// Define configuration attributes.
DX::ThrowIfFailed(
MFCreateAttributes(&mediaEngineAttributes, 1)
);
ComPtr<MediaEngineNotify> notify = Make<MediaEngineNotify>();
ComPtr<IUnknown> unknownNotify;
DX::ThrowIfFailed(
notify.As(&unknownNotify)
);
DX::ThrowIfFailed(
mediaEngineAttributes->SetUnknown(MF_MEDIA_ENGINE_CALLBACK, unknownNotify.Get())
);
DWORD flags=0;
#if PLATFORM_WINDOWS8
DX::ThrowIfFailed(
mediaEngineAttributes->SetUINT32(MF_MEDIA_ENGINE_AUDIO_CATEGORY, AudioCategory_GameMedia)
);
flags = MF_MEDIA_ENGINE_AUDIOONLY | MF_MEDIA_ENGINE_REAL_TIME_MODE;
#endif
// Create the Media Engine.
DX::ThrowIfFailed(
mediaEngineFactory->CreateInstance(flags, mediaEngineAttributes.Get(), &m_mediaEngine)
);

Related

DirectX: Get InfoQueue before Device

Is there a way to get the InfoQueue or set the break parameters before the Device is created?
Right now i am creating the Device and then getting the InfoQueue, but any messages that are emitted before that point are going to be ignored and buried in the output window.
ID3D11Device* pDevice;
//...Create Device...
ID3D11InfoQueue* pInfoQueue;
pDevice->QueryInterface(__uuidof(ID3D11InfoQueue), &pInfoQueue);
pInfoQueue->SetBreakOnSeverity(D3D11_MESSAGE_SEVERITY_ERROR, TRUE);
And I want something like:
ID3D11InfoQueue* pInfoQueue;
//...Get InfoQueue...
pInfoQueue->SetBreakOnSeverity(D3D11_MESSAGE_SEVERITY_ERROR, TRUE);
ID3D11Device* pDevice;
//...Create Device...
Given the documentation for ID3D11InfoQueue saying that you get the interface pointer via a QueryInterface call on the device I would say the answer is 'no'.
What you want to do is enable DXGI debugging which will provide debug information for the device & swapchain creation.
#include <dxgidebug.h>
#if defined(_DEBUG)
Microsoft::WRL::ComPtr<IDXGIInfoQueue> dxgiInfoQueue;
typedef HRESULT (WINAPI * LPDXGIGETDEBUGINTERFACE)(REFIID, void ** );
HMODULE dxgidebug = LoadLibraryEx( L"dxgidebug.dll", nullptr, LOAD_LIBRARY_SEARCH_SYSTEM32 );
if ( dxgidebug )
{
auto dxgiGetDebugInterface = reinterpret_cast<LPDXGIGETDEBUGINTERFACE>(
reinterpret_cast<void*>( GetProcAddress( dxgidebug, "DXGIGetDebugInterface" ) ) );
if ( SUCCEEDED( dxgiGetDebugInterface( IID_PPV_ARGS( dxgiInfoQueue.GetAddressOf() ) ) ) )
{
dxgiInfoQueue->SetBreakOnSeverity( DXGI_DEBUG_ALL, DXGI_INFO_QUEUE_MESSAGE_SEVERITY_ERROR, true );
dxgiInfoQueue->SetBreakOnSeverity( DXGI_DEBUG_ALL, DXGI_INFO_QUEUE_MESSAGE_SEVERITY_CORRUPTION, true );
}
}
#endif
See this blog post and this one

How does one play multiple sounds simultaneously on XAudio2?

I am currently trying to make a game app in Windows with XAudio2 and I cannot figure out how to make the application not block when playing a sound. I tried calling a new thread in the samples in this repository.
But it will just cause an error. I tried passing a reference to the mastering voice in the function but then it just raises a "XAudio2: Must create a mastering voice first" error. Am I missing something? I am just trying to make it play two sounds at once and build from there. I went over the documentation but it's very vague.
XAudio2 is a non-blocking API. To play two sounds simultaneously, you need two 'source voices' and one 'mastering voice' at a minimum.
DX::ThrowIfFailed(
CoInitializeEx( nullptr, COINIT_MULTITHREADED )
);
Microsoft::WRL::ComPtr<IXAudio2> pXAudio2;
// Note that only IXAudio2 (and APOs) are COM reference counted
DX::ThrowIfFailed(
XAudio2Create( pXAudio2.GetAddressOf(), 0 )
);
IXAudio2MasteringVoice* pMasteringVoice = nullptr;
DX::ThrowIfFailed(
pXAudio2->CreateMasteringVoice( &pMasteringVoice )
);
IXAudio2SourceVoice* pSourceVoice1 = nullptr;
DX::ThrowIfFailed(
pXaudio2->CreateSourceVoice( &pSourceVoice1, &wfx ) )
// The default 'pSendList' will be just to the pMasteringVoice
);
IXAudio2SourceVoice* pSourceVoice2 = nullptr;
DX::ThrowIfFailed(
pXaudio2->CreateSourceVoice( &pSourceVoice2, &wfx) )
// Doesn't have to be same format as other source voice
// And doesn't have to match the mastering voice either
);
DX::ThrowIfFailed(
pSourceVoice1->SubmitSourceBuffer( &buffer )
);
DX::ThrowIfFailed(
pSourceVoice2->SubmitSourceBuffer( &buffer /* could be different WAV data or not */)
);
DX::ThrowIfFailed(
pSourceVoice1->Start( 0 );
);
DX::ThrowIfFailed(
pSourceVoice2->Start( 0 );
);
You should take a look at the samples on GitHub as well as DirectX Tool Kit for Audio
If you wanted to ensure both source voices started at precisely the same time, you'd use:
DX::ThrowIfFailed(
pSourceVoice1->Start( 0, 1 );
);
DX::ThrowIfFailed(
pSourceVoice2->Start( 0, 1 );
);
DX::ThrowIfFailed(
pSourceVoice2->CommitChanges( 1 );
);
If you want to play multiple sounds simultaneously have a playSound function and launch various threads to play your various sounds each one of a certain source voice.
XAudio2 will take care of mapping each sound to available channels (or if you have a more advanced system you can specify the mapping yourself using IXAudio2Voice::SetOutputMatrix).
void playSound( IXAudio2SourceVoice* sourceVoice )
{
BOOL isPlayingSound = TRUE;
XAUDIO2_VOICE_STATE soundState = {0};
HRESULT hres = sourceVoice->Start( 0u );
while ( SUCCEEDED( hres ) && isPlayingSound )
{// loop till sound completion
sourceVoice->GetState( &soundState );
isPlayingSound = ( soundState.BuffersQueued > 0 ) != 0;
Sleep( 100 );
}
}
For example to play two sounds simultaneously:
IXAudio2SourceVoice* pSourceVoice1 = nullptr;
IXAudio2SourceVoice* pSourceVoice2 = nullptr;
// setup the source voices, load the sounds etc..
std::thread thr1{ playSound, pSourceVoice1 };
std::thread thr2{ playSound, pSourceVoice2 };
thr1.join();
thr2.join();

How do I encode raw 48khz/32bits PCM to FLAC using Microsoft Media Foundation?

I created a SinkWriter that is able to encode video and audio using Microsoft's Media Foundation Platform.
Video is working fine so far but I have some troubles with audio only.
My PCM source has a sample rate of 48828hz, 32 bits per sample and is mono.
Everything is working well so far except for FLAC.
For instance the MP3 output is working more or less but has a wrong format. Regarding to MSDN (MP3 Audio Encoder) the MP3 encoder only supports 16 bits per sample as input. My PCM source as descriped above has 32 bits per sample.
However the export with MP3 is working cause the MF Platform seems like to have some kind of fallback and is using the MPEG Audio layer 1/2 (mpga) with 2 Channels, 32khz and a bitrate of 320kb/s.
Things start to get weird when I set the MF_MT_SUBTYPE to MFAudioFormat_FLAC. The export is working too but the quality of the audio is aweful. There's a lot of noise but I am able to recognize the audio. Regarding to VLC the FLAC file has a sample rate of 44,1khz, 8 bits per sample and is mono.
Does this mean the FLAC codec isn't able to work with the PCM I provide?
Has anyone had the same problem and was able to fix it?
Update
After doing some more research about this problem it seems like that my PCM Audio with a resolution of 32 Bit is too high. So currently I am trying to convert the 32 Bit PCM to 24 Bit for FLAC and 16 Bit for MP3 but with no luck so far. I keep you updated if I make some progress.
--------
Update 2
I've created a minimal example app that shows the problem I am facing.
It reads the 48khz32bit wave file and tries to encode it to flac.
When executing the hr = pSinkWriter->BeginWriting(); command I get the error 0xc00d36b4 whice means The data specified for the media type is invalid, inconsistent, or not supported by this object.
What am I doing wrong here?
#include "stdafx.h"
#include <windows.h>
#include <windowsx.h>
#include <comdef.h>
#include <mfapi.h>
#include <mfidl.h>
#include <mfreadwrite.h>
#include <Mferror.h>
#pragma comment(lib, "ole32")
#pragma comment(lib, "mfplat")
#pragma comment(lib, "mfreadwrite")
#pragma comment(lib, "mfuuid")
using namespace System;
int main(array<System::String ^> ^args)
{
HRESULT hr = CoInitializeEx(0, COINIT_MULTITHREADED);
hr = MFStartup(MF_VERSION);
IMFMediaType *pMediaType;
IMFMediaType *pMediaTypeOut;
IMFSourceReader *pSourceReader;
IMFAttributes *pAttributes;
IMFSinkWriter *pSinkWriter;
hr = MFCreateSourceReaderFromURL(
L"C:\\Temp\\48khz32bit.wav",
NULL,
&pSourceReader
);
hr = MFCreateAttributes(&pAttributes, 1);
hr = pAttributes->SetGUID(
MF_TRANSCODE_CONTAINERTYPE,
MFTranscodeContainerType_WAVE
);
hr = MFCreateSinkWriterFromURL(
L"C:\\Temp\\foo.flac",
NULL,
pAttributes,
&pSinkWriter
);
hr = pSourceReader->GetCurrentMediaType(
MF_SOURCE_READER_FIRST_AUDIO_STREAM,
&pMediaType);
hr = MFCreateMediaType(&pMediaTypeOut);
hr = pMediaTypeOut->SetGUID(
MF_MT_MAJOR_TYPE,
MFMediaType_Audio
);
hr = pMediaTypeOut->SetGUID(
MF_MT_SUBTYPE,
MFAudioFormat_FLAC
);
hr = pMediaTypeOut->SetUINT32(
MF_MT_AUDIO_SAMPLES_PER_SECOND,
48000
);
hr = pMediaTypeOut->SetUINT32(
MF_MT_AUDIO_NUM_CHANNELS,
1
);
hr = pMediaTypeOut->SetUINT32(
MF_MT_AUDIO_BITS_PER_SAMPLE,
32
);
hr = pMediaTypeOut->SetUINT32(
MF_MT_AUDIO_AVG_BYTES_PER_SECOND,
(((32 + 7) / 8) * 1) * 48000
);
hr = pMediaTypeOut->SetUINT32(
MF_MT_AUDIO_BLOCK_ALIGNMENT,
((32 + 7) / 8) * 1
);
DWORD nWriterStreamIndex = -1;
hr = pSinkWriter->AddStream(pMediaTypeOut, &nWriterStreamIndex);
hr = pSinkWriter->BeginWriting();
_com_error err(hr);
LPCTSTR errMsg = err.ErrorMessage();
for (;;)
{
DWORD nStreamIndex, nStreamFlags;
LONGLONG nTime;
IMFSample *pSample;
hr = pSourceReader->ReadSample(
MF_SOURCE_READER_FIRST_AUDIO_STREAM,
0,
&nStreamIndex,
&nStreamFlags,
&nTime,
&pSample);
if (pSample)
{
OutputDebugString(L"Write sample...\n");
hr = pSinkWriter->WriteSample(
nWriterStreamIndex,
pSample
);
}
if (nStreamFlags & MF_SOURCE_READERF_ENDOFSTREAM)
{
break;
}
}
hr = pSinkWriter->Finalize();
return 0;
}
--------
Update 3
I added the solution as answer.
--------
Initialize SinkWriter
HRESULT SinkWriter::InitializeSinkWriter(IMFSinkWriter **ppWriter, DWORD *pStreamIndex, DWORD *pAudioStreamIndex, LPCWSTR filename)
{
*ppWriter = NULL;
*pStreamIndex = NULL;
*pAudioStreamIndex = NULL;
IMFSinkWriter *pSinkWriter = NULL;
// Attributes
IMFAttributes *pAttributes;
HRESULT hr = S_OK;
DX::ThrowIfFailed(
MFCreateAttributes(
&pAttributes,
3
)
);
#if defined(ENABLE_HW_ACCELERATION)
CComPtr<ID3D11Device> device;
D3D_FEATURE_LEVEL levels[] = { D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0 };
#if defined(ENABLE_HW_DRIVER)
DX::ThrowIfFailed(
D3D11CreateDevice(
nullptr,
D3D_DRIVER_TYPE_HARDWARE,
nullptr,
(0 * D3D11_CREATE_DEVICE_SINGLETHREADED) | D3D11_CREATE_DEVICE_VIDEO_SUPPORT,
levels,
ARRAYSIZE(levels),
D3D11_SDK_VERSION,
&device,
nullptr,
nullptr
)
);
const CComQIPtr<ID3D10Multithread> pMultithread = device;
pMultithread->SetMultithreadProtected(TRUE);
#else
DX::ThrowIfFailed(
D3D11CreateDevice(
nullptr,
D3D_DRIVER_TYPE_NULL,
nullptr,
D3D11_CREATE_DEVICE_SINGLETHREADED,
levels,
ARRAYSIZE(levels),
D3D11_SDK_VERSION,
&device,
nullptr,
nullptr)
);
#endif
UINT token;
CComPtr<IMFDXGIDeviceManager> pManager;
DX::ThrowIfFailed(
MFCreateDXGIDeviceManager(
&token,
&pManager
)
);
DX::ThrowIfFailed(
pManager->ResetDevice(
device,
token
)
);
DX::ThrowIfFailed(
pAttributes->SetUnknown(
MF_SOURCE_READER_D3D_MANAGER,
pManager
)
);
DX::ThrowIfFailed(
pAttributes->SetUINT32(
MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS,
TRUE
)
);
#if (WINVER >= 0x0602)
DX::ThrowIfFailed(
pAttributes->SetUINT32(
MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING,
TRUE
)
);
#endif
#else
DX::ThrowIfFailed(
pAttributes->SetUINT32(
MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS,
TRUE
)
);
DX::ThrowIfFailed(
pAttributes->SetUINT32(
MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING,
TRUE
)
);
#endif
DX::ThrowIfFailed(
MFCreateSinkWriterFromURL(
filename,
NULL,
pAttributes,
&pSinkWriter
)
);
if (m_vFormat != VideoFormat::SWFV_NONE)
{
DX::ThrowIfFailed(
InitializeVideoCodec(
pSinkWriter,
pStreamIndex
)
);
}
if (m_audFormat != AudioFormat::SWAF_NONE)
{
DX::ThrowIfFailed(
InitializeAudioCodec(
pSinkWriter,
pAudioStreamIndex
)
);
}
// Tell the sink writer to start accepting data.
DX::ThrowIfFailed(
pSinkWriter->BeginWriting()
);
// Return the pointer to the caller.
*ppWriter = pSinkWriter;
(*ppWriter)->AddRef();
SAFE_RELEASE(pSinkWriter);
return hr;
}
Initialize Audio Codec
HRESULT SinkWriter::InitializeAudioCodec(IMFSinkWriter *pSinkWriter, DWORD *pStreamIndex)
{
// Audio media types
IMFMediaType *pAudioTypeOut = NULL;
IMFMediaType *pAudioTypeIn = NULL;
DWORD audioStreamIndex;
HRESULT hr = S_OK;
// Set the output audio type.
DX::ThrowIfFailed(
MFCreateMediaType(
&pAudioTypeOut
)
);
DX::ThrowIfFailed(
pAudioTypeOut->SetGUID(
MF_MT_MAJOR_TYPE,
MFMediaType_Audio
)
);
DX::ThrowIfFailed(
pAudioTypeOut->SetGUID(
MF_MT_SUBTYPE,
AUDIO_SUBTYPE
)
);
DX::ThrowIfFailed(
pSinkWriter->AddStream(
pAudioTypeOut,
&audioStreamIndex
)
);
// Set the input audio type
DX::ThrowIfFailed(
MFCreateMediaType(
&pAudioTypeIn
)
);
DX::ThrowIfFailed(
pAudioTypeIn->SetGUID(
MF_MT_MAJOR_TYPE,
AUDIO_MAJOR_TYPE
)
);
DX::ThrowIfFailed(
pAudioTypeIn->SetGUID(
MF_MT_SUBTYPE,
MFAudioFormat_PCM
)
);
DX::ThrowIfFailed(
pAudioTypeIn->SetUINT32(
MF_MT_AUDIO_NUM_CHANNELS,
AUDIO_NUM_CHANNELS
)
);
DX::ThrowIfFailed(
pAudioTypeIn->SetUINT32(
MF_MT_AUDIO_BITS_PER_SAMPLE,
AUDIO_BITS_PER_SAMPLE
)
);
DX::ThrowIfFailed(
pAudioTypeIn->SetUINT32(
MF_MT_AUDIO_BLOCK_ALIGNMENT,
AUDIO_BLOCK_ALIGNMENT
)
);
DX::ThrowIfFailed(
pAudioTypeIn->SetUINT32(
MF_MT_AUDIO_SAMPLES_PER_SECOND,
AUDIO_SAMPLES_PER_SECOND
)
);
DX::ThrowIfFailed(
pAudioTypeIn->SetUINT32(
MF_MT_AUDIO_AVG_BYTES_PER_SECOND,
AUDIO_AVG_BYTES_PER_SECOND
)
);
DX::ThrowIfFailed(
pSinkWriter->SetInputMediaType(
audioStreamIndex,
pAudioTypeIn,
NULL
)
);
*pStreamIndex = audioStreamIndex;
SAFE_RELEASE(pAudioTypeOut);
SAFE_RELEASE(pAudioTypeIn);
return hr;
}
Push audio data
HRESULT SinkWriter::PushAudio(UINT32* data)
{
HRESULT hr = S_FALSE;
if (m_isInitializing)
{
return hr;
}
IMFSample *pSample = NULL;
IMFMediaBuffer *pBuffer = NULL;
BYTE *pMem = NULL;
size_t cbBuffer = m_bufferLength * sizeof(short);
// Create a new memory buffer.
hr = MFCreateMemoryBuffer(cbBuffer, &pBuffer);
// Lock the buffer and copy the audio frame to the buffer.
if (SUCCEEDED(hr))
{
hr = pBuffer->Lock(&pMem, NULL, NULL);
}
if (SUCCEEDED(hr))
{
CopyMemory(pMem, data, cbBuffer);
}
if (pBuffer)
{
pBuffer->Unlock();
}
if (m_vFormat == VideoFormat::SWFV_NONE && m_audFormat == AudioFormat::SWAF_WAV)
{
DWORD cbWritten = 0;
if (SUCCEEDED(hr))
{
hr = m_pByteStream->Write(pMem, cbBuffer, &cbWritten);
}
if (SUCCEEDED(hr))
{
m_cbWrittenByteStream += cbWritten;
}
}
else
{
// Set the data length of the buffer.
if (SUCCEEDED(hr))
{
hr = pBuffer->SetCurrentLength(cbBuffer);
}
// Create media sample and add the buffer to the sample.
if (SUCCEEDED(hr))
{
hr = MFCreateSample(&pSample);
}
if (SUCCEEDED(hr))
{
hr = pSample->AddBuffer(pBuffer);
}
// Set the timestamp and the duration.
if (SUCCEEDED(hr))
{
hr = pSample->SetSampleTime(m_cbRtStartVideo);
}
if (SUCCEEDED(hr))
{
hr = pSample->SetSampleDuration(m_cbRtDurationVideo);
}
// Send the sample to the Sink Writer
if (SUCCEEDED(hr))
{
hr = m_pSinkWriter->WriteSample(m_audioStreamIndex, pSample);
}
/*if (SUCCEEDED(hr))
{
m_cbRtStartAudio += m_cbRtDurationAudio;
}*/
SAFE_RELEASE(pSample);
SAFE_RELEASE(pBuffer);
}
return hr;
}
So, Microsoft introduced a FLAC Media Foundation Transform (MFT) Encoder CLSID_CMSFLACEncMFT in Windows 10, but the codec remains undocumented at the moment.
Supported Media Formats in Media Foundation is similarly out of date and does not reflect presence of recent additions.
I am not aware of any comment on this, and my opinion is that the codec is added for internal use but the implementation is merely a standard Media Foundation components without licensing restrictions, so the codecs are unrestricted too by, for example, field of use limitations.
This stock codec seems to be limited to 8, 16 and 24 bit PCM input options (that is, not 32 bits/sample - you need to resample respectively). The codec is capable to accept up to 8 channels and flexible samples per second rate (48828 Hz is okay).
Even though the codec (transform) seems to be working, if you want to produce a file, you also need a suitable container format (multiplexer) which is compatible with MFAudioFormat_FLAC (the identifier has 7 results on Google Search at the moment of the post, which basically means noone is even aware of the codec). Outdated documentation does not reflect actual support for FLAC in stock media sinks.
I borrowed a custom media sink that writes a raw MFT output payload into a file, and such FLAC output is playable as the FLAC frames contain necessary information to parse the bitstream for playback.
For the reference, the file itself is: 20180224-175524.flac.
An obvious candidate among stock media sinks WAVE Media Sink is unable to accept FLAC input. Nevertheless it potentially could, the implementation is presumably limited to simpler audio formats.
AVI media sink might possibly take FLAC audio, but it seems to be impossible to create an audio only AVI.
Among other media sink there is however a media sink which can process FLAC: MPEG-4 File Sink. Again, despite the outdated documentation, the media sink takes FLAC input, so you should be able to create .MP4 files with FLAC audio track.
Sample file: 20180224-184012.mp4. "FLAC (framed)"
To sum it up:
FLAC encoder MFT is present in Windows 10 and is available for use; lacks proper documentation though
One needs to take care of conversion of input to compatible format (no direct support for 32-bit PCM)
It is possible to manage MFT directly and consume MFT output, then obtain FLAC bitstream
Alternatively, it is possible to use stock MP4 media sink to produce output with FLAC audio track
Alternatively, it is possible to develop a custom media sink and consume FLAC bitstream from upstream encoder connection
Potentially, the codec is compatible with Transcode API, however the restrictions above apply. The container type needs to be MFTranscodeContainerType_MPEG4 in particular.
The codec is apparently compatible with Media Session API, presumably it is good for use with Sink Writer API either.
In your code as you attempt to use Sink Writer API you should similarly either have MP4 output with input possibly converted to compatible format in your code (compatible PCM or compatible FLAC with encoder MFT managed on your side). Knowing that MP4 media sink overall is capable to create FLAC audio track you should be able to debug fine details in your code and fit the components to work together.
Finally I was able to solve the problem. It wasn't that hard to be honest. But that is always the case if you know how to achieve something ;).
I created a copy and paste example below to give an idea how to implement FLAC encoding with Microsoft Media Foundation.
The missing piece of the puzzle was the MFTranscodeGetAudioOutputAvailableTypes. This function lists all the available output formats from an audio encoder.
If you are not sure what MFTs are supported by the operation system you can call MFTEnumEx function first. This gives you a list of all the available MFTs. In my case with windows 10 there's the FLAC MFT that is defined like this.
Name: Microsoft FLAC Audio Encoder MFT
Input Types: 1 items:
Audio-PCM
Class identifier: 128509e9-c44e-45dc-95e9-c255b8f466a6
Output Types: 1 items:
Audio-0000f1ac-0000-0010-8000-00aa00389b71
Transform Flags: 1
Transform Category: Audio Encoder
So the next thing I did was to create the source reader and get the current media type. The important values for me are sample rate, bit rate and channels.
Then I created a GetOutputMediaTypes function that needs the requested audio format, sample rate, bit rate, channels and a reference to the IMFMediaType.
The MFTranscodeGetAudioOutputAvailableTypes function returns all available types for the MFAudioFormat_flac GUID.
After getting the count of the available media types with hr = pAvailableTypes->GetElementCount(&dwMTCount); I am able to iterate through them and check if a type is supporting my request. If that's the case I return the media type.
The last part is the easiest one.
First add the output media type to the sinkwriter to get the stream index.
DWORD dwWriterStreamIndex = -1;
// Add the stream
hr = pSinkWriter->AddStream(
pOuputMediaType,
&dwWriterStreamIndex
);
Then set the input type and call pSinkWriter->BeginWriting(); so the sinkwriter starts to accepting data.
// Set input media type
hr = pSinkWriter->SetInputMediaType(
dwWriterStreamIndex,
pInputType,
NULL
);
// Tell the sink writer to accept data
hr = pSinkWriter->BeginWriting();
If the output and input media type is correctly set, BeginWriting should return 0 as HRESULT.
We should get no error because we are using the media type the function MFTranscodeGetAudioOutputAvailableTypes is providing.
The last step is to read all samples from the source reader and write it through the sinkwriter into the flac container.
Done :)
I hope I could help with this answer.
Also thanks to Roman R.
Update
This sample is only working with Audio-PCM formats from 4 bits to 24 bits. If you want to encode an 32 Bit Audio-PCM you have to resample it first and then encode it.
--------
Here's the minimal example app.
#include <windows.h>
#include <windowsx.h>
#include <atlstr.h>
#include <comdef.h>
#include <exception>
#include <mfapi.h>
#include <mfplay.h>
#include <mfreadwrite.h>
#include <mmdeviceapi.h>
#include <Audioclient.h>
#include <mferror.h>
#include <Wmcodecdsp.h>
#pragma comment(lib, "mf.lib")
#pragma comment(lib, "mfplat.lib")
#pragma comment(lib, "mfplay.lib")
#pragma comment(lib, "mfreadwrite.lib")
#pragma comment(lib, "mfuuid.lib")
#pragma comment(lib, "wmcodecdspuuid")
inline void ThrowIfFailed(HRESULT hr)
{
if (FAILED(hr))
{
// Get the error message
_com_error err(hr);
LPCTSTR errMsg = err.ErrorMessage();
OutputDebugString(L"################################## ERROR ##################################\n");
OutputDebugString(errMsg);
OutputDebugString(L"\n################################## ----- ##################################\n");
CStringA sb(errMsg);
// Set a breakpoint on this line to catch DirectX API errors
throw std::exception(sb);
}
}
template <class T> void SafeRelease(T **ppT)
{
if (*ppT)
{
(*ppT)->Release();
*ppT = nullptr;
}
}
using namespace System;
HRESULT GetOutputMediaTypes(
GUID cAudioFormat,
UINT32 cSampleRate,
UINT32 cBitPerSample,
UINT32 cChannels,
IMFMediaType **ppType
)
{
// Enumerate all codecs except for codecs with field-of-use restrictions.
// Sort the results.
DWORD dwFlags =
(MFT_ENUM_FLAG_ALL & (~MFT_ENUM_FLAG_FIELDOFUSE)) |
MFT_ENUM_FLAG_SORTANDFILTER;
IMFCollection *pAvailableTypes = NULL; // List of audio media types.
IMFMediaType *pAudioType = NULL; // Corresponding codec.
HRESULT hr = MFTranscodeGetAudioOutputAvailableTypes(
cAudioFormat,
dwFlags,
NULL,
&pAvailableTypes
);
// Get the element count.
DWORD dwMTCount;
hr = pAvailableTypes->GetElementCount(&dwMTCount);
// Iterate through the results and check for the corresponding codec.
for (DWORD i = 0; i < dwMTCount; i++)
{
hr = pAvailableTypes->GetElement(i, (IUnknown**)&pAudioType);
GUID majorType;
hr = pAudioType->GetMajorType(&majorType);
GUID subType;
hr = pAudioType->GetGUID(MF_MT_SUBTYPE, &subType);
if (majorType != MFMediaType_Audio || subType != MFAudioFormat_FLAC)
{
continue;
}
UINT32 sampleRate = NULL;
hr = pAudioType->GetUINT32(
MF_MT_AUDIO_SAMPLES_PER_SECOND,
&sampleRate
);
UINT32 bitRate = NULL;
hr = pAudioType->GetUINT32(
MF_MT_AUDIO_BITS_PER_SAMPLE,
&bitRate
);
UINT32 channels = NULL;
hr = pAudioType->GetUINT32(
MF_MT_AUDIO_NUM_CHANNELS,
&channels
);
if (sampleRate == cSampleRate
&& bitRate == cBitPerSample
&& channels == cChannels)
{
// Found the codec.
// Jump out!
break;
}
}
// Add the media type to the caller
*ppType = pAudioType;
(*ppType)->AddRef();
SafeRelease(&pAudioType);
return hr;
}
int main(array<System::String ^> ^args)
{
HRESULT hr = S_OK;
// Initialize com interface
ThrowIfFailed(
CoInitializeEx(0, COINIT_MULTITHREADED)
);
// Start media foundation
ThrowIfFailed(
MFStartup(MF_VERSION)
);
IMFMediaType *pInputType = NULL;
IMFSourceReader *pSourceReader = NULL;
IMFMediaType *pOuputMediaType = NULL;
IMFSinkWriter *pSinkWriter = NULL;
// Create source reader
hr = MFCreateSourceReaderFromURL(
L"C:\\Temp\\48khz24bit.wav",
NULL,
&pSourceReader
);
// Create sink writer
hr = MFCreateSinkWriterFromURL(
L"C:\\Temp\\foo.flac",
NULL,
NULL,
&pSinkWriter
);
// Get media type from source reader
hr = pSourceReader->GetCurrentMediaType(
MF_SOURCE_READER_FIRST_AUDIO_STREAM,
&pInputType
);
// Get sample rate, bit rate and channels
UINT32 sampleRate = NULL;
hr = pInputType->GetUINT32(
MF_MT_AUDIO_SAMPLES_PER_SECOND,
&sampleRate
);
UINT32 bitRate = NULL;
hr = pInputType->GetUINT32(
MF_MT_AUDIO_BITS_PER_SAMPLE,
&bitRate
);
UINT32 channels = NULL;
hr = pInputType->GetUINT32(
MF_MT_AUDIO_NUM_CHANNELS,
&channels
);
// Try to find a media type that is fitting.
hr = GetOutputMediaTypes(
MFAudioFormat_FLAC,
sampleRate,
bitRate,
channels,
&pOuputMediaType);
DWORD dwWriterStreamIndex = -1;
// Add the stream
hr = pSinkWriter->AddStream(
pOuputMediaType,
&dwWriterStreamIndex
);
// Set input media type
hr = pSinkWriter->SetInputMediaType(
dwWriterStreamIndex,
pInputType,
NULL
);
// Tell the sink writer to accept data
hr = pSinkWriter->BeginWriting();
// Forever alone loop
for (;;)
{
DWORD nStreamIndex, nStreamFlags;
LONGLONG nTime;
IMFSample *pSample;
// Read through the samples until...
hr = pSourceReader->ReadSample(
MF_SOURCE_READER_FIRST_AUDIO_STREAM,
0,
&nStreamIndex,
&nStreamFlags,
&nTime,
&pSample);
if (pSample)
{
OutputDebugString(L"Write sample...\n");
hr = pSinkWriter->WriteSample(
dwWriterStreamIndex,
pSample
);
}
// ... we are at the end of the stream...
if (nStreamFlags & MF_SOURCE_READERF_ENDOFSTREAM)
{
// ... and jump out.
break;
}
}
// Call finalize to finish writing.
hr = pSinkWriter->Finalize();
// Done :D
return 0;
}

DirectX11 2 window rendering

How can I render my objects with DirectX into 2 separated windows?
You need to create one SwapChain and RenderTargetView for every window.
1 if you created your device via CreateDeviceAndSwapChain you need to obtain IDXGIFactory first
IDXGIDevice * device;
d3ddevice->QueryInterface(__uuidof(IDXGIDevice), (void**)&device);
IDXGIAdapter * adapter;
device->GetParent(__uuidof(IDXGIAdapter), (void**)&adapter);
IDXGIFactory * factory;
adapter->GetParent(__uuidof(DDXGIFactory), (void**)&factory);
With DXGIFactory you can create additional swapchain for new window
factory->CreateSwapChain(g_pd3dDevice, &sd, &g_pSwapChain2);
then create a render target view
ID3D11Texture2D* pBackBuffer = NULL;
hr = g_pSwapChain2->GetBuffer( 0, __uuidof( ID3D11Texture2D ), ( LPVOID* )&pBackBuffer );
if( FAILED( hr ) )
return hr;
hr = g_pd3dDevice->CreateRenderTargetView( pBackBuffer, NULL, &g_pRenderTargetView );
pBackBuffer->Release();
if( FAILED( hr ) )
return hr;
And finally just set your render target(s) and Draw something!
g_immediateContext->OMSetRenderTargets(1, &g_RenderTargetView, NULL);
....
I hope this has been helpful.
Best regards Quest :)

GmfBridge doesn't connect sink filter with source filter

I am trying to use GmfBridge library to dynamically change source filters from cam to file and vice versa. All functions return S_OK (well, almost all - pMediaControlOutput->Run() returns S_FALSE actually, but in msdn said that this can be ok too), so I've assumed that all is ok but data isn't transfered to the other side of the bridge. I use GraphStudio to connect to remote graph and all seems to be ok - all filters in both graph are connected as it should be. But while playing I receive following messages in bridge log file:
0: Started 2011-09-10 15:58:38
0: Buffer minimum 100
0: Added stream 1: 畡楤o, Decompressed Only, Discard mode
1: Sink 0x7ae0ca8 has 1 pins
1: Sink filter 0x7ae0cf8 in graph 0x193bf0
2: Source 0x7ae1030 has 1 pins
2: Source filter 0x7ae1080 in graph 0x194c18
14: ReceiveConnection Aware: false
14: Bridging 0x194c18 to 0x193bf0
14: Pin 0x7ae3438 disconnect
25: Source 0x7ae1030 pause from 0
25: Source pin 0x7ae3618 active
2234: Pin 0x7ae3438 receive 0x721ec68
2234: Sink pin 0x7ae3438 disconnected: discarding 0x721ec68
3389: Pin 0x7ae3438 receive 0x721ec68
3389: Sink pin 0x7ae3438 disconnected: discarding 0x721ec68
3940: Pin 0x7ae3438 receive 0x721ec68
3940: Sink pin 0x7ae3438 disconnected: discarding 0x721ec68
4440: Pin 0x7ae3438 receive 0x721ec68
So as you can see, left graph isn't connected to right one despite BridgeGraphs() returned S_OK and media sample is discarded. Below there is my code. Where am I going wrong?
// Create graphs
HRESULT hr = m_graphInput.CreateInstance(CLSID_FilterGraph);
ATLASSERT( SUCCEEDED( hr ) );
hr = m_graphOutput.CreateInstance(CLSID_FilterGraph);
ATLASSERT( SUCCEEDED( hr ) );
// Get IMediaControl interfaces
hr = m_graphInput.QueryInterface( IID_IMediaControl, (void**)&pMediaControlInput );
ATLASSERT( SUCCEEDED( hr ) );
hr = m_graphOutput.QueryInterface( IID_IMediaControl, (void**)&pMediaControlOutput );
ATLASSERT( SUCCEEDED( hr ) );
// Get builder interfaces
hr = m_graphInput.QueryInterface( IID_IGraphBuilder, (void**)&pBuilderInput );
ATLASSERT( SUCCEEDED( hr ) );
hr = m_graphOutput.QueryInterface( IID_IGraphBuilder, (void**)&pBuilderOutput );
ATLASSERT( SUCCEEDED( hr ) );
// Load source filter (on sink side)
LocateFilter( SOURCE_FILTER_NAME, CLSID_LegacyAmFilterCategory, &inputDevice );
hr = m_graphInput->AddFilter( inputDevice, SOURCE_FILTER_NAME );
ATLASSERT( SUCCEEDED( hr ) );
// Load render filter (on bridge's source side)
LocateFilter( _T( "Default DirectSound Device" ), CLSID_AudioRendererCategory, &audioOutputPreview );
hr = m_graphOutput->AddFilter( audioOutputPreview, _T( "Default DirectSound Device" ) );
ATLASSERT( SUCCEEDED( hr ) );
// Init bridge
bridge.CreateInstance( __uuidof(GMFBridgeController) );
hr = bridge->SetBufferMinimum( 100 );
ATLASSERT( SUCCEEDED( hr ) );
hr = bridge->AddStream( false, eUncompressed, false );
ATLASSERT( SUCCEEDED( hr ) );
// Add sink filter and connect to input graph
IUnknownPtr pSinkFilter;
{
hr = bridge->InsertSinkFilter( m_graphInput, (IUnknown**)&pSinkFilter );
ATLASSERT( SUCCEEDED( hr ) );
// Using own functions get pins
IPin* pInAudio = CPinController::getOutputPin( inputDevice, _T("Audio"));
IPin* pOutAudio = CPinController::getInputPin( pSinkFilter );
hr = pBuilderInput->Connect( pOutAudio, pInAudio );
ATLASSERT( SUCCEEDED( hr ) );
}
// Add output filter and connect to output graph
IUnknownPtr pFeederFilter;
{
hr = bridge->InsertSourceFilter( pSinkFilter, m_graphOutput, &pFeederFilter );
ATLASSERT( SUCCEEDED( hr ) );
// Get pins
IPin* pInAudio = CPinController::getOutputPin( pFeederFilter/*, _T("Audio")*/);
IPin* pOutAudio = CPinController::getInputPin( audioOutputPreview );
hr = pBuilderOutput->Connect( pInAudio, pOutAudio );
ATLASSERT( SUCCEEDED( hr ) );
}
// Run left
hr = pMediaControlInput->Run();
ATLASSERT( SUCCEEDED( hr ) );
// Run right
hr = pMediaControlOutput->Run();
ATLASSERT( SUCCEEDED( hr ) );
hr = bridge->BridgeGraphs( m_graphOutput, m_graphInput );
ATLASSERT( SUCCEEDED( hr ) );
It's really ridiculous but about a few minutes ago after a day of searching we have found the answer. All deal was about really huge hole in GmfBridge. I was giving wrong interfaces here (there are graphs instead of bridge's sink and source filters) 'coz function needed pointers to IUnknown:
hr = bridge->BridgeGraphs( m_graphOutput, m_graphInput );
And in GmfBridge library this situation wasn't handled properly - there is no "else" brunch to handle error and function returns hr which was set in begin to S_OK:
HRESULT STDMETHODCALLTYPE BridgeGraphs(
/* [in] */ IUnknown *pSourceGraphSinkFilter,
/* [in] */ IUnknown *pRenderGraphSourceFilter)
{
HRESULT hr = S_OK;
...
// if we are not given both filters, then
// we need do nothing
IBridgeSinkPtr pSink = pSourceGraphSinkFilter;
IBridgeSourcePtr pSource = pRenderGraphSourceFilter;
if ((pSink != NULL) && (pSource != NULL))
{
...
}
return hr;
}
So as you can see it just says there is nothing wrong and then it just do nothing! I think it's good idea to notify authors of the lib about this bug.
Hope this info will help someone.