How to play audio using Sink Writer from Microsoft Media Foundation - c++

Question

I'm trying to create an audio visualizer using Microsoft Media Foundation. For this I need to intercept the samples and simultaneously play them. Using a Media Session with Topology and a sample-grabber sink seems impractical and over-complicated, hence I'm trying to use a combination of a Sink Reader and Sink Writer for this (see the right half of the image on Overview of the Media Foundation Architecture). Unfortunately, Audio/Video Playback does not really explain how to do this. The book Developing Microsoft Media Foundation Applications contains a source-to-sink loop on page 92, but that still does not really help me.
Creating the Source Reader works fine and I'm reading nonzero samples. Writing them to the Sink Writer (which uses the Streaming Audio Renderer) does not give me any errors, but I don't hear anything. I tried multiple things like selecting other media types and explicitly selecting the rendering device (although I only have one, as it indicated), but to no avail. Note that playing audio using a Media Session works fine, though!
I based my code on this question: Play audio from file to speaker with Media Foundation.
This is my code at this moment:
#include <iostream>
#include <cassert>
#include <mfidl.h>
#include <mfapi.h>
#include <mfreadwrite.h>
#include <Mferror.h>
#pragma comment(lib, "mf")
#pragma comment(lib, "mfplat")
#pragma comment(lib, "mfreadwrite")
#include <winrt/base.h>
#pragma comment(lib, "windowsapp")
void winHr(const HRESULT result) { winrt::check_hresult(result); }
template<class T>
struct ComPtr : winrt::com_ptr<T>
{
auto operator&() noexcept { return this->put(); }
operator T*() noexcept
{
assert(this->get());
return this->get();
}
};
int main() noexcept
{
winHr(CoInitializeEx(nullptr, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE));
winHr(MFStartup(MF_VERSION));
{
ComPtr<IMFSourceReader> reader;
winHr(MFCreateSourceReaderFromURL(
LR"(test.wav)",
nullptr, &reader));
constexpr auto inStreamIndex = MF_SOURCE_READER_FIRST_AUDIO_STREAM;
// Select only the audio stream
winHr(reader->SetStreamSelection(MF_SOURCE_READER_ALL_STREAMS, false));
winHr(reader->SetStreamSelection(inStreamIndex, true));
ComPtr<IMFMediaSink> mediaSink;
winHr(MFCreateAudioRenderer(nullptr, &mediaSink));
ComPtr<IMFSinkWriter> writer;
{
ComPtr<IMFStreamSink> streamSink;
winHr(mediaSink->GetStreamSinkByIndex(0, &streamSink));
ComPtr<IMFMediaTypeHandler> typeHandler;
winHr(streamSink->GetMediaTypeHandler(&typeHandler));
ComPtr<IMFMediaType> inputType;
winHr(reader->GetCurrentMediaType(inStreamIndex, &inputType));
ComPtr<IMFMediaType> closestSupportedType;
const auto result = typeHandler->IsMediaTypeSupported(inputType, &closestSupportedType);
if (result == MF_E_INVALIDMEDIATYPE)
{
if (!closestSupportedType)
{
std::cerr << "Media type not supported" << std::endl;
winHr(mediaSink->Shutdown());
goto end; //:o
}
winHr(reader->SetCurrentMediaType(inStreamIndex, nullptr, closestSupportedType));
winHr(typeHandler->SetCurrentMediaType(closestSupportedType));
winHr(MFCreateSinkWriterFromMediaSink(mediaSink, nullptr, &writer));
winHr(writer->SetInputMediaType(0, closestSupportedType, nullptr));
}
else {
winHr(result);
winHr(reader->SetCurrentMediaType(inStreamIndex, nullptr, inputType));
winHr(typeHandler->SetCurrentMediaType(inputType));
winHr(MFCreateSinkWriterFromMediaSink(mediaSink, nullptr, &writer));
winHr(writer->SetInputMediaType(0, inputType, nullptr));
}
}
winHr(writer->BeginWriting());
while (true)
{
ComPtr<IMFSample> sample;
DWORD streamFlags;
MFTIME timestamp;
winHr(reader->ReadSample(inStreamIndex, 0, nullptr, &streamFlags, &timestamp, &sample));
if (streamFlags & MF_SOURCE_READERF_ENDOFSTREAM)
{
winHr(writer->NotifyEndOfSegment(0));
break;
}
if (streamFlags & MF_SOURCE_READERF_STREAMTICK)
winHr(writer->SendStreamTick(0, timestamp));
if (!sample) continue;
winHr(sample->SetSampleTime(timestamp));
winHr(writer->WriteSample(0, sample));
}
winHr(writer->Flush(0));
std::cout << "(Press enter to stop)" << std::endl;
std::cin.get();
winHr(writer->Finalize());
writer.attach(nullptr);
winHr(mediaSink->Shutdown());
}
end:
winHr(MFShutdown());
CoUninitialize();
}
Just to be clear: when I run this it prints (Press enter to stop) and I can hear from the noise (read: distortions from electronic signals) from my headphones that I can deduce that for a short moment an audio port was opened and then closed, but no actual audio is played. How can I get this to work?
Edit 1: I just fixed that if result != MF_E_INVALIDMEDIATYPE I didn't set the media type, but now I often (but not always, for some reason) get MF_E_TOPO_CODEC_NOT_FOUND at the line winHr(writer->SetInputMediaType(0, inputType, nullptr));. Why would this be? (Still no audio is played in any case.)
Edit 2: Apparently it matters when I create the writer, so now I do that only at the last moment, but now I get the "Media type not supported" error. Maybe I need to manually pick some media type but I will look in to this later -- unless someone knows the answer.

Answer 1

I modified your code to work differently in the following manner:
1. Enumerates the Audio Sink output media types, until it finds a supported one.
2. Sets this media type to the Reader in order to force it to use Audio Resampler DSP (this is what IMFMediaTopology does).
Here is the code, it plays back the input wav file properly. Let me know if it works for you.
#include <iostream>
#include <cassert>
#include <mfidl.h>
#include <mfapi.h>
#include <mfreadwrite.h>
#include <Mferror.h>
#pragma comment(lib, "mf")
#pragma comment(lib, "mfplat")
#pragma comment(lib, "mfreadwrite")
#include <winrt/base.h>
#pragma comment(lib, "windowsapp")
void winHr(const HRESULT result) { winrt::check_hresult(result); }
template<class T>
struct ComPtr : winrt::com_ptr<T>
{
auto operator&() noexcept { return this->put(); }
operator T*() noexcept
{
assert(this->get());
return this->get();
}
};
int main() noexcept
{
winHr(CoInitializeEx(nullptr, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE));
winHr(MFStartup(MF_VERSION));
{
ComPtr<IMFSourceReader> reader;
winHr(MFCreateSourceReaderFromURL(
LR"(test.wav)",
nullptr, &reader));
constexpr auto inStreamIndex = MF_SOURCE_READER_FIRST_AUDIO_STREAM;
// Select only the audio stream
winHr(reader->SetStreamSelection(MF_SOURCE_READER_ALL_STREAMS, false));
winHr(reader->SetStreamSelection(inStreamIndex, true));
ComPtr<IMFMediaSink> mediaSink;
winHr(MFCreateAudioRenderer(nullptr, &mediaSink));
ComPtr<IMFSinkWriter> writer;
{
ComPtr<IMFStreamSink> streamSink;
winHr(mediaSink->GetStreamSinkByIndex(0, &streamSink));
ComPtr<IMFMediaTypeHandler> typeHandler;
winHr(streamSink->GetMediaTypeHandler(&typeHandler));
DWORD dwCount = 0;
ComPtr<IMFMediaType> inputType;
winHr(typeHandler->GetMediaTypeCount(&dwCount));
for (INT i = 0; i < dwCount; i++)
{
inputType.attach(nullptr);
winHr(typeHandler->GetMediaTypeByIndex(i, &inputType));
if (SUCCEEDED(typeHandler->IsMediaTypeSupported(inputType, NULL)))
break;
}
//ComPtr<IMFMediaType> inputType;
//winHr(reader->GetCurrentMediaType(inStreamIndex, &inputType));
winHr(reader->SetCurrentMediaType(inStreamIndex, NULL, inputType));
//ComPtr<IMFMediaType> closestSupportedType;
//const auto result = typeHandler->IsMediaTypeSupported(inputType, &closestSupportedType);
//if (result == MF_E_INVALIDMEDIATYPE)
//{
// if (!closestSupportedType)
// {
// std::cerr << "Media type not supported" << std::endl;
// winHr(mediaSink->Shutdown());
// goto end; //:o
// }
// winHr(reader->SetCurrentMediaType(inStreamIndex, nullptr, closestSupportedType));
// winHr(typeHandler->SetCurrentMediaType(closestSupportedType));
// winHr(MFCreateSinkWriterFromMediaSink(mediaSink, nullptr, &writer));
// winHr(writer->SetInputMediaType(0, closestSupportedType, nullptr));
//}
//else
{
//winHr(result);
//winHr(reader->SetCurrentMediaType(inStreamIndex, nullptr, inputType));
winHr(typeHandler->SetCurrentMediaType(inputType));
winHr(MFCreateSinkWriterFromMediaSink(mediaSink, nullptr, &writer));
winHr(writer->SetInputMediaType(0, inputType, nullptr));
}
}
winHr(writer->BeginWriting());
while (true)
{
ComPtr<IMFSample> sample;
DWORD streamFlags;
MFTIME timestamp;
winHr(reader->ReadSample(inStreamIndex, 0, nullptr, &streamFlags, &timestamp, &sample));
if (streamFlags & MF_SOURCE_READERF_ENDOFSTREAM)
{
winHr(writer->NotifyEndOfSegment(0));
break;
}
if (streamFlags & MF_SOURCE_READERF_STREAMTICK)
winHr(writer->SendStreamTick(0, timestamp));
if (!sample)
continue;
// SetSampleTime is redundant
//winHr(sample->SetSampleTime(timestamp));
winHr(writer->WriteSample(0, sample));
}
// Flush shouldn't be called!
// winHr(writer->Flush(0));
std::cout << "(Press enter to stop)" << std::endl;
std::cin.get();
winHr(writer->Finalize());
writer.attach(nullptr);
winHr(mediaSink->Shutdown());
}
end:
winHr(MFShutdown());
CoUninitialize();
}

Answer 2

This code works just fine for me with wav files (Win7, default sound card) :
//----------------------------------------------------------------------------------------------
// Main.cpp
//----------------------------------------------------------------------------------------------
#pragma once
#define WIN32_LEAN_AND_MEAN
#define STRICT
#pragma comment(lib, "mfplat")
#pragma comment(lib, "mfreadwrite")
#pragma comment(lib, "mf")
#pragma comment(lib, "mfuuid")
//----------------------------------------------------------------------------------------------
// Microsoft Windows SDK for Windows 7
#include <WinSDKVer.h>
#include <new>
#include <windows.h>
#include <strsafe.h>
#include <mfapi.h>
#include <mfidl.h>
#include <mferror.h>
#include <mfreadwrite.h>
template <class T> inline void SAFE_RELEASE(T*& p){
if(p){
p->Release();
p = NULL;
}
}
#define AUDIO_FILE L"C:\\Project\\Media\\Audio\\test.wav"
HRESULT ProcessAudio(const WCHAR*);
void main(){
HRESULT hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE);
if(SUCCEEDED(hr)){
hr = MFStartup(MF_VERSION, MFSTARTUP_LITE);
if(SUCCEEDED(hr)){
hr = ProcessAudio(AUDIO_FILE);
hr = MFShutdown();
}
CoUninitialize();
}
}
HRESULT ProcessAudio(const WCHAR*){
IMFSourceReader* pSourceReader = NULL;
IMFMediaType* pType = NULL;
DWORD dwMediaTypeIndex = 0;
IMFMediaSink* pAudioSink = NULL;
IMFStreamSink* pStreamSink = NULL;
IMFMediaTypeHandler* pMediaTypeHandler = NULL;
IMFSinkWriter* pSinkWriter = NULL;
HRESULT hr = MFCreateSourceReaderFromURL(AUDIO_FILE, NULL, &pSourceReader);
// Check native media type
if(SUCCEEDED(hr))
hr = pSourceReader->GetNativeMediaType((DWORD)MF_SOURCE_READER_FIRST_AUDIO_STREAM, dwMediaTypeIndex, &pType);
SAFE_RELEASE(pType);
// Get current media type
if(SUCCEEDED(hr))
hr = pSourceReader->GetCurrentMediaType(dwMediaTypeIndex, &pType);
if(SUCCEEDED(hr))
hr = MFCreateAudioRenderer(NULL, &pAudioSink);
if(SUCCEEDED(hr))
hr = pAudioSink->GetStreamSinkByIndex(0, &pStreamSink);
if(SUCCEEDED(hr))
hr = pStreamSink->GetMediaTypeHandler(&pMediaTypeHandler);
if(FAILED(hr = pMediaTypeHandler->IsMediaTypeSupported(pType, NULL))){
SAFE_RELEASE(pType);
// This is a compatible type with my soundcard
// MF_MT_MAJOR_TYPE MFMediaType_Audio
// MF_MT_SUBTYPE MFAudioFormat_PCM
// MF_MT_AUDIO_NUM_CHANNELS 2
// MF_MT_AUDIO_SAMPLES_PER_SECOND 48000
// MF_MT_AUDIO_BLOCK_ALIGNMENT 4
// MF_MT_AUDIO_AVG_BYTES_PER_SECOND 192000
// MF_MT_AUDIO_BITS_PER_SAMPLE 16
// MF_MT_ALL_SAMPLES_INDEPENDENT 1
// MF_MT_AUDIO_PREFER_WAVEFORMATEX 1
hr = MFCreateMediaType(&pType);
if(SUCCEEDED(hr))
hr = pType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Audio);
if(SUCCEEDED(hr))
hr = pType->SetGUID(MF_MT_SUBTYPE, MFAudioFormat_PCM);
if(SUCCEEDED(hr))
hr = pType->SetUINT32(MF_MT_AUDIO_NUM_CHANNELS, 2);
if(SUCCEEDED(hr))
hr = pType->SetUINT32(MF_MT_AUDIO_SAMPLES_PER_SECOND, 48000);
if(SUCCEEDED(hr))
hr = pType->SetUINT32(MF_MT_AUDIO_BLOCK_ALIGNMENT, 4);
if(SUCCEEDED(hr))
hr = pType->SetUINT32(MF_MT_AUDIO_AVG_BYTES_PER_SECOND, 192000);
if(SUCCEEDED(hr))
hr = pType->SetUINT32(MF_MT_AUDIO_BITS_PER_SAMPLE, 16);
if(SUCCEEDED(hr))
hr = pType->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, TRUE);
if(SUCCEEDED(hr))
hr = pType->SetUINT32(MF_MT_AUDIO_PREFER_WAVEFORMATEX, TRUE);
}
if(SUCCEEDED(hr))
hr = pMediaTypeHandler->SetCurrentMediaType(pType);
if(SUCCEEDED(hr))
hr = MFCreateSinkWriterFromMediaSink(pAudioSink, NULL, &pSinkWriter);
if(SUCCEEDED(hr))
hr = pSinkWriter->BeginWriting();
BOOL bProcess = (hr == S_OK ? TRUE : FALSE);
DWORD streamIndex;
DWORD flags;
LONGLONG llTimeStamp;
IMFSample* pSample = NULL;
while(bProcess){
hr = pSourceReader->ReadSample((DWORD)MF_SOURCE_READER_FIRST_AUDIO_STREAM, 0, &streamIndex, &flags, &llTimeStamp, &pSample);
if(SUCCEEDED(hr) && (flags == 0)){
if(pSample){
hr = pSinkWriter->WriteSample(0, pSample);
SAFE_RELEASE(pSample);
}
}
else{
bProcess = FALSE;
}
}
if(pSinkWriter)
pSinkWriter->Finalize();
if(pAudioSink)
pAudioSink->Shutdown();
SAFE_RELEASE(pSample);
SAFE_RELEASE(pSinkWriter);
SAFE_RELEASE(pMediaTypeHandler);
SAFE_RELEASE(pStreamSink);
SAFE_RELEASE(pAudioSink);
SAFE_RELEASE(pType);
SAFE_RELEASE(pSourceReader);
return hr;
}
If you face problem with MF_E_TOPO_CODEC_NOT_FOUND, share your file. Perhaps extra code is needed in this case.
EDIT
Do you have more than one sound card ?
Share your audio file, so we can see what's going on.
My little hack with Sleep(40), is just here to tell you that doing "while(true)" without pausing have no sens (your processor is working too much). Off course 40 should be better decided. 40 is the minimum processing time processor preemption on windows OS. That my choice for this sample code. We can talk about the best way to choose this value. I'm waiting for your suggestion.