Broken spatialization when using Resonance Audio from FMOD Low Level API - audio-player

I am struggling to enable Resonance Audio sound spatialization when using Resonance Audio plugin for FMOD and trying to setup DSPs manually, without FMOD Studio.
My code for plugin loading and DSPs' setup:
auto system = audio->fmod->system; //instance of FMOD::System
unsigned int resHandle;
CHECK_ERR(system->loadPlugin("../lib/resonanceaudio.dll", &resHandle, 0));
//0 = Resonance Audio Listener
//1 = Resonance Audio Soundfield
//2 = Resonance Audio Source
unsigned int listenerPlugin, sourcePlugin;
system->getNestedPlugin(resHandle, 0, &listenerPlugin);
system->getNestedPlugin(resHandle, 2, &sourcePlugin);
FMOD::DSP* listenerDsp;
CHECK_ERR(system->createDSPByPlugin(listenerPlugin, &listenerDsp));
FMOD::DSP* sourceDsp;
CHECK_ERR(system->createDSPByPlugin(sourcePlugin, &sourceDsp));
//This a channel group routed from the Master group
//I want to spatialize all sounds which are played in this group
FMOD::ChannelGroup *worldGroup = nullptr;
system->createChannelGroup("World", &worldGroup);
FMOD::ChannelGroup *masterGroup = nullptr;
system->getMasterChannelGroup(&masterGroup);
masterGroup->addGroup(worldGroup);
//Adding Resonance Audio dsps to the group
worldGroup->addDsp(0, sourceDsp);
worldGroup->addDsp(1, listenerDsp);
//Setting listener's position at (0, 0, 0)
system->set3DListenerAttributes(0, FMOD_VECTOR{0, 0, 0}, 0, FMOD_VECTOR{0, 0, 1}, FMOD_VECTOR{0, 1, 0});
<loading sound>
FMOD::Channel* channel = nullptr;
CHECK_ERR(system->playSound(sound, worldGroup, true, &channel));
channel->setMode(FMOD_3D);
channel->set3DAttributes(FMOD_VECTOR{4, 0, 3}, nullptr);
channel->setPaused(false);
<somewhere in update loop>
system->update();
But after all of this I dont hear any audio at all.
I assume the mistake is on my side so I tried to repeat the same setup in FMOD Studio. As I did in code I placed Resonance Audio Source before the Resonance Audio Listener at the master track of the event and everything worked correctly (at least in FMOD Studio, didnt try it in game).
How can I fix this issue?

From what I think you should be calling:
System::getMasterChannelGroup
To get an instance of the master channel group and then assigning the Resonance audio DSPs to the master channel group.
//This a channel group routed from the Master group
//I want to spatialize all sounds which are played in this group
FMOD::ChannelGroup *worldGroup>;
<worldGroup setup>
From this I was not entirely sure what you meant as I cannot see where you have setup the ChannelGroup routing from the master channel. If the solution I have provided is incorrect, then please provide further insight into the code I have mentioned above.

Related

Canon SDK (EDSDK) capture region of specified size for video stream

I am very new to the EDSDK so sorry for maybe weird question in some places.
Is it possible to access a video stream and perform some operations on it using the SDK? I need this to capture very thin region (ROI) of a specified size (for example 3840x10 px) for each frame in the stream. Don`t understand this as compression of a frame, aspect ratios are not needed to follow. These changes in theory should increase fps, because the region will be very thin (Should they?).
I found the code snippet below from the official documentation, although it seems this causes only to send a signal for starting and stopping video rec, without accessing the stream.
EdsUInt32 record_start = 4; // Begin movie shooting
err = EdsSetPropertyData(cameraRef, kEdsPropID_Record, 0, sizeof(record_start), &record_start);
EdsUInt32 record_stop = 0; // End movie shooting
err = EdsSetPropertyData(cameraRef, kEdsPropID_Record, 0, sizeof(record_stop), &record_stop);
I would be very thanksful for any suggestions and help. Please feel free to ask any additional information!
This sdk doesnt allow you to directly get access to hi res streams like industrial cams would. You can access over USB ~960x640 liveview images in sequential JPGs. Movie recording can only be done to internal card, and after stopping transfering the result. Outside of this SDk, use of an external HDMI recorder gives access to a near realtime feed at max FullHD1080p, depending on model and not always “clean”.

Why pjmedia_snd_port_create(...) fails initializing the sound port?

I am trying to initialise a sound port from pjsip and pjsua with the standart pjmedia_snd_port_create but the result is always not successful.
pj_caching_pool_init(&cp, &pj_pool_factory_default_policy, 0);
pool = pj_pool_create(&cp.factory,
"pool1",
4000,
4000,
NULL);
pjmedia_snd_port *snd_port1 = NULL;
status = pjmedia_snd_port_create(pool, id1, id1, clock_rate,
channel_count, samples_per_frame,
bits_per_sample,
0, &snd_port1);
My device id1 is 0, as i got it from the audio device manager. I`ve tried with the -1 for defaults but always fails me. I have endpoint created with the pjsua2 api from a C++ class, the lib is OK and running, I can create conference bridges also but the sound port fails me. A bit of a hint will be great.
I`ve fixed it with initialising loop, I guess it has nothing to do with the setup, what I need is actually to register all of my playback and rec devices from my hardware.

Record in linux with QAudioInput and play it in windows

For my purpose, I want to record sounds in raw format(only samples), 8kHz, 16bit(little endian) and 1 channel. Then, I would like to transfer those samples to the windows and play it with QAudioOutput. So I have two separated programs: one for recording voice with QAudioInput, and other one gives a file which is contained some samples, then I play it with QAudioOutput. Below is my source code for creating QAudioInput and QAudioOutput.
//Initialize audio
void AudioBuffer::initializeAudio()
{
m_format.setFrequency(8000); //set frequency to 8000
m_format.setChannels(1); //set channels to mono
m_format.setSampleSize(16); //set sample sze to 16 bit
m_format.setSampleType(QAudioFormat::UnSignedInt ); //Sample type as usigned integer sample
m_format.setByteOrder(QAudioFormat::LittleEndian); //Byte order
m_format.setCodec("audio/pcm"); //set codec as simple audio/pcm
QAudioDeviceInfo infoIn(QAudioDeviceInfo::defaultInputDevice());
if (!infoIn.isFormatSupported(m_format))
{
//Default format not supported - trying to use nearest
m_format = infoIn.nearestFormat(m_format);
}
QAudioDeviceInfo infoOut(QAudioDeviceInfo::defaultOutputDevice());
if (!infoOut.isFormatSupported(m_format))
{
//Default format not supported - trying to use nearest
m_format = infoOut.nearestFormat(m_format);
}
createAudioInput();
createAudioOutput();
}
void AudioBuffer::createAudioOutput()
{
m_audioOutput = new QAudioOutput(m_Outputdevice, m_format, this);
}
void AudioBuffer::createAudioInput()
{
if (m_input != 0) {
disconnect(m_input, 0, this, 0);
m_input = 0;
}
m_audioInput = new QAudioInput(m_Inputdevice, m_format, this);
}
These programs work well in windows and Linux separately. However, it has a lot of noise when I record a voice in Linux and play it in windows.
I figure out captured samples in windows and Linux are different. First picture is related to captured sound in Linux and second one for windows.
Captured sound in Linux:
Captured sound in Windows:
A bit more on details is that silence in Windows and Linux is different. I tried many things including swapping bytes, even though I set little endian in both platforms.
Now, I am in doubt about alsa configuration. Are there any missed settings?
Do you think it will be better if I record voice directly without using QAudioInput?
The voice is UnSignedInt, but sample value has both negative and positive value! It seems you had trouble in capturing. Change QAudioFormat::UnSignedInt to QAudioFormat::SignedInt.

How to set bus on apple's AudioUnit 3D Mixer?

I'm trying to figure out how to query or setup which sound is connected to which bus input of the 3D Mixer. Say I have an explosion sound, is there a way to specifically set which bus that sound will be located on for future reference?
Assuming you are working with an AUGraph(), this is pretty straightforward:
Just connect the node to the desired busNumber of the mixer.
// connect the file player's output to the mixer's input
checkResult( AUGraphConnectNodeInput(audioGraph,
filePlayerNode,
0,
mixerNode,
1, // this is the first bus #1 "AUGraphConnectNodeInput(filePlayer:out -> mixer:in");

Best DirectShow way to capture image from web cam preview ? SampleGrabber is deprecated

I have developed DirectShow C++ app which successfully previews web cam view into provided window. Now I want to capture image from this live web cam preview. I have used graph manager, ICaptureGraphBuilder2, IMoniker etc. for that.
I have searched and found following options:
WIA & Sample Grabber.
Many recommends using SampleGrabber but as per MS's msdn document SampleGrabber is deprecated and one should not use. And I don't want to use WIA API.
So which is the best DirectShow way to capture image from live web cam preview?
Here is a quote from DxSnap sample from DirectShow.NET library:
Use DirectShow to take snapshots from the Still pin of a capture
device. Note the MS encourages you to use WIA for this, but if you
want to do in with DirectShow and C#, here's how.
Note that this sample will only work with devices that output
uncompressed video as RBG24. This will include most webcams, but
probably zero tv tuners.
This is C# code, but you should get the idea as the interfaces are all the same. And there are other samples on how to use Sample Grabber Filter in C++.
Sample Grabber is deprecated, the headers are removed from a couple of latest SDKs, however the runtime components are all there and are going to be there for a long time, or otherwise a multitude of application would be broken (e.g. Video Chat in browser hosted GMail is using Sample Grabber). So basically Sample Grabber is still an easy way to capture snapshots from a web camera, or if you alternatively prefer to follow the latest MS APIs - you would want to look into Media Foundation (09 Jul 2016 update: new Windows Server installations might need one to add "Media Foundation" and/or "Desktop Experience" features to make Media Foundation API available along with DirectShow, and DirectShow Editing Services, Sample Grabber is a part of which. Default installation does not offer qedit.dll out of the box).
Also in C++ you certainly don't have to use Sample Grabber Filter. You can develop a custom filter using DirectShow BaseClasses to be a custom transformation filter or a custom renderer, which which accept incoming video feed and export the frames from the DirectShow pipeline. Another option is to use Sample Grabber sample source code from one of the older SDKs (which is not exact source for OS Sample Grabber, but it is doing the same thing). The point however that Sample Grabber shipped with Windows is still a good option.
Listed on Microsoft's website is an example of how to capture a frame using IVMRWindowlessControl9::GetCurrentImage ... Here's one way of doing it:
IBaseFilter* vmr9ptr; // I'm assuming that you got this pointer already
IVMRWindowlessControl9* controlPtr = NULL;
vmr9ptr->QueryInterface(IID_IVMRWindowlessControl9, (void**)controlPtr);
assert ( controlPtr != NULL );
// Get the current frame
BYTE* lpDib = NULL;
hr = controlPtr->GetCurrentImage(&lpDib);
// If everything is okay, we can create a BMP
if (SUCCEEDED(hr))
{
BITMAPINFOHEADER* pBMIH = (BITMAPINFOHEADER*) lpDib;
DWORD bufSize = pBMIH->biSizeImage;
// Let's create a bmp
BITMAPFILEHEADER bmpHdr;
BITMAPINFOHEADER bmpInfo;
size_t hdrSize = sizeof(bmpHdr);
size_t infSize = sizeof(bmpInfo);
memset(&bmpHdr, 0, hdrSize);
bmpHdr.bfType = ('M' << 8) | 'B';
bmpHdr.bfOffBits = static_cast<DWORD>(hdrSize + infSize);
bmpHdr.bfSize = bmpHdr.bfOffBits + bufSize;
// Builder the bit map info.
memset(&bmpInfo, 0, infSize);
bmpInfo.biSize = static_cast<DWORD>(infSize);
bmpInfo.biWidth = pBMIH->biWidth;
bmpInfo.biHeight = pBMIH->biHeight;
bmpInfo.biPlanes = pBMIH->biPlanes;
bmpInfo.biBitCount = pBMIH->biBitCount;
// boost::shared_arrays are awesome!
boost::shared_array<BYTE> buf(new BYTE[bmpHdr.bfSize]);//(lpDib);
memcpy(buf.get(), &bmpHdr, hdrSize); // copy the header
memcpy(buf.get() + hdrSize, &bmpInfo, infSize); // now copy the info block
memcpy(buf.get() + bmpHdr.bfOffBits, lpDib, bufSize);
// Do something with your image data ... seriously...
CoTaskMemFree(lpDib);
} // All done!
jeez... so much dis-information. if you're previewing in a directshow graph, then it depends on what you're previewing off of. Capture filters have 1, 2, or 3 pins. If it has 1 pin, it's most likely a "capture" pin (no preview pin). For this, if you want to capture and preview at the same time, you should put in a "Smart Tee" filter, and connect the VMR off of the preview pin, and hook up "something that grabs frames" off the capture pin. since you don't want to fool around with DirectShow's crummy pin start/stop stuff (instead, just simply controlling the entire graph's start/stop state). You don't need to use a SampleGrabber, it's a dead-simple filter and you could write it in a few hours (I should know, I'm the one that wrote it). it's simply a CTransInPlace filter that you can set a forced media type for it to accept, and you can set a callback interface on it to call you back when it receives a sample. It's actually simpler to write a NullRenderer which calls you back when it receives a sample, you could write this quite easily.
If the capture filter has 2 pins, it's most likely a capture pin and a still pin. in this case you still need a Smart Tee connected to the source's capture pin, and need to preview off the smart tee's preview pin, and capture samples off the smart tee's capture pin.
(If you don't know what a SmartTee is, it's a filter that plays allocator tricks and only sends a sample down the preview pin if the capture pin isn't super bogged down. It's job is to provide a path for the VMR to render from which won't botch up the allocators between the capture filter and the filters downstream of the capture filter)
If the capture filter has both capture and preview pins, I think you can figure out what you need to do then.
Anyhow, summary: The SampleGrabber is simply a CTransInPlaceFilter. You could write it as a Null Renderer, too, just make sure to fill out some junk in CheckInputType, and to call your callback back in DoRenderSample.