XAudio2 - Source voice hangs when active audio device gets removed - c++

Question

I have a problem I am not able to solve. My application should be able to switch the default audio device during runtime. To achieve this I am using XAudio2 from the DirectXTK.
I implemented the IMMNotificationClient into my audio class to be able to react on default device changes.
For instance when the default device changes I stop my current source voice, reset the audio engine and start the source voice again. Everything works as expected.
However when my default device is a USB soundcard and I unplug it during the source voice is playing the application freezes.
The reason for this is that the source voice hangs when stopping the voice. Sometimes also when flushing the source buffer. Seems like the source voice could not be stopped anymore when the audio device was removed the source voice was using.
Does someone had the same problem and was able to solve this?
Here's the function I am using to reset the audio engine.
HRESULT DX::XAudioEngine::ResetAudioDevice()
{
HRESULT hr = S_OK;
this->m_retryAudio = TRUE;
if (SUCCEEDED(hr) && m_pSourceVoice)
{
hr = m_pSourceVoice->Stop();
}
if (SUCCEEDED(hr))
{
hr = m_pSourceVoice->FlushSourceBuffers();
}
if (m_audEngine && m_pSourceVoice)
{
// Get the source voice back from the smart pointer
IXAudio2SourceVoice* ptmpSrcVoice = nullptr;
ptmpSrcVoice = m_pSourceVoice.release();
// Destroy the voice
ptmpSrcVoice->DestroyVoice();
}
m_audEngine->Reset(&m_waveFormat, NULL);
// Create source voice
IXAudio2SourceVoice* ptmpSrcVoice = nullptr;
m_audEngine->AllocateVoice(
(WAVEFORMATEX*)&m_waveFormat,
SoundEffectInstance_Default,
false,
&ptmpSrcVoice
);
// Add source voice to smart pointer
m_pSourceVoice.reset(ptmpSrcVoice);
// Set the input volume
if (this->m_inputVolume != 1.0f) {
hr = m_pSourceVoice->SetVolume(this->m_inputVolume);
}
hr = m_pSourceVoice->Start(0);
this->m_retryAudio = FALSE;
return hr;
}