I'm having a problem using PortAudio and I am not sure whether I don't quite understand how the callbacks work, or I did something wrong. My assumption was that the callbacks should fire continuous "on tick", containing the current samples, but it seems like I'm only receiving a few callbacks when I open and start the stream, and then I won't receive callbacks ever again. My code:
PaStreamParameters inputP, outputP;
inputP.device = DeviceIndex;
inputP.channelCount = CardInfo->maxInputChannels;
inputP.sampleFormat = paFloat32;
inputP.suggestedLatency = CardInfo->defaultLowInputLatency;
inputP.hostApiSpecificStreamInfo = NULL;
outputP.device = DeviceIndex;
outputP.channelCount = CardInfo->maxOutputChannels;
outputP.sampleFormat = paFloat32;
outputP.suggestedLatency = CardInfo->defaultLowOutputLatency;
outputP.hostApiSpecificStreamInfo = NULL;
PaError err = Pa_OpenStream(
&AudioStream,
&inputP,
&outputP,
SAMPLE_RATE,
FRAMES_PER_BUFFER,
paClipOff,
&CAudio::AudioCallback,
this
);
err = Pa_StartStream(AudioStream);
I used the following constants
#define SAMPLE_RATE 44100
#define FRAMES_PER_BUFFER 64
The callback:
int CAudio::AudioCallback(const void* pInputBuffer, void* pOutputBuffer, unsigned long iFramesPerBuffer, const PaStreamCallbackTimeInfo* timeInfo, PaStreamCallbackFlags statusFlags, void* userData)
{
CAudio* AudioInterface = (CAudio*)userData;
const float* buffer = (const float*)pInputBuffer;
printf("Callback: %d frames per buffer %d t %f\n", AudioInterface->CallbackIndex, iFramesPerBuffer, Pa_GetStreamTime(AudioInterface->AudioStream));
AudioInterface->CallbackIndex++;
return paContinue;
}
Now, what happens if I open and start the stream is that I get exactly six callbacks, the following log output
Opening audio stream on 'USB PnP Sound Device: Audio (hw:1,0)'
Opened audio stream, starting it...
Callback: 0 frames per buffer 64 t 21812.485122
Callback: 1 frames per buffer 64 t 21812.497681
Callback: 2 frames per buffer 64 t 21812.514483
Callback: 3 frames per buffer 64 t 21812.525110
Callback: 4 frames per buffer 64 t 21812.626489
Callback: 5 frames per buffer 64 t 21812.635590
Am I doing something wrong, or is my understanding of what happens with the callbacks wrong?
Alright, for now I fixed this by using the blocking API, grabbing the stream and writing it to a buffer in a while loop running in another std::thread, which works kind of good.
Related
I'm trying to visualize a soundwave captured by WASAPI loopback but find that the packets I record do not form a smooth wave when put together.
My understanding of how the WASAPI capture client works is that when I call pCaptureClient->GetBuffer(&pData, &numFramesAvailable, &flags, NULL, NULL) the buffer pData is filled from the front with numFramesAvailable datapoints. Each datapoint is a float and they alternate by channel. Thus to get all available datapoints I should cast pData to a float pointer, and take the first channels * numFramesAvailable values. Once I release the buffer and call GetBuffer again it provides the next packet. I would assume that these packets would follow on from each other but it doesn't seem to be the case.
My guess is that either I'm making an incorrect assumption about the format of the audio data in pData or the capture client is either missing or overlapping frames. But have no idea how to check these.
To make the code below as brief as possible I've removed things like error status checking and cleanup.
Initialization of capture client:
const CLSID CLSID_MMDeviceEnumerator = __uuidof(MMDeviceEnumerator);
const IID IID_IMMDeviceEnumerator = __uuidof(IMMDeviceEnumerator);
const IID IID_IAudioClient = __uuidof(IAudioClient);
const IID IID_IAudioCaptureClient = __uuidof(IAudioCaptureClient);
pAudioClient = NULL;
IMMDeviceEnumerator * pDeviceEnumerator = NULL;
IMMDevice * pDeviceEndpoint = NULL;
IAudioClient *pAudioClient = NULL;
IAudioCaptureClient *pCaptureClient = NULL;
int channels;
// Initialize audio device endpoint
CoInitialize(nullptr);
CoCreateInstance(CLSID_MMDeviceEnumerator, NULL, CLSCTX_ALL, IID_IMMDeviceEnumerator, (void**)&pDeviceEnumerator );
pDeviceEnumerator ->GetDefaultAudioEndpoint(eRender, eConsole, &pDeviceEndpoint );
// init audio client
WAVEFORMATEX *pwfx = NULL;
REFERENCE_TIME hnsRequestedDuration = 10000000;
REFERENCE_TIME hnsActualDuration;
audio_device_endpoint->Activate(IID_IAudioClient, CLSCTX_ALL, NULL, (void**)&pAudioClient);
pAudioClient->GetMixFormat(&pwfx);
pAudioClient->Initialize(AUDCLNT_SHAREMODE_SHARED, AUDCLNT_STREAMFLAGS_LOOPBACK, hnsRequestedDuration, 0, pwfx, NULL);
channels = pwfx->nChannels;
pAudioClient->GetService(IID_IAudioCaptureClient, (void**)&pCaptureClient);
pAudioClient->Start(); // Start recording.
Capture of packets (note that std::mutex packet_buffer_mutex and vector<vector<float>> packet_bufferare already be defined and used by another thread to safely display the data):
UINT32 packetLength = 0;
BYTE *pData = NULL;
UINT32 numFramesAvailable;
DWORD flags;
int max_packets = 8;
std::unique_lock<std::mutex>write_guard(packet_buffer_mutex, std::defer_lock);
while (true) {
pCaptureClient->GetNextPacketSize(&packetLength);
while (packetLength != 0)
{
// Get the available data in the shared buffer.
pData = NULL;
pCaptureClient->GetBuffer(&pData, &numFramesAvailable, &flags, NULL, NULL);
if (flags & AUDCLNT_BUFFERFLAGS_SILENT)
{
pData = NULL; // Tell CopyData to write silence.
}
write_guard.lock();
if (packet_buffer.size() == max_packets) {
packet_buffer.pop_back();
}
if (pData) {
float * pfData = (float*)pData;
packet_buffer.emplace(packet_buffer.begin(), pfData, pfData + channels * numFramesAvailable);
} else {
packet_buffer.emplace(packet_buffer.begin());
}
write_guard.unlock();
hpCaptureClient->ReleaseBuffer(numFramesAvailable);
pCaptureClient->GetNextPacketSize(&packetLength);
}
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
I store the packets in a vector<vector<float>> (where each vector<float> is a packet) removing the last one and inserting the newest at the start so I can iterate over them in order.
Below is the result of a captured sinewave, plotting alternating values so it only represents a single channel. It is clear where the packets are being stitched together.
Something is playing a sine wave to Windows; you're recording the sine wave back in the audio loopback; and the sine wave you're getting back isn't really a sine wave.
You're almost certainly running into glitches. The most likely causes of glitching are:
Whatever is playing the sine wave to Windows isn't getting data to Windows in time, so the buffer is running dry.
Whatever is reading the loopback data out of Windows isn't reading the data in time, so the buffer is filling up.
Something is going wrong in between playing the sine wave to Windows and reading it back.
It is possible that more than one of these are happening.
The IAudioCaptureClient::GetBuffer call will tell you if you read the data too late. In particular it will set *pdwFlags so that the AUDCLNT_BUFFERFLAGS_DATA_DISCONTINUITY bit is set.
Looking at your code, I see you're doing the following things between the GetBuffer and the WriteBuffer:
Waiting on a lock
Sometimes doing something called "pop_back"
Doing something called "emplace"
I quote from the above-linked documentation:
Clients should avoid excessive delays between the GetBuffer call that acquires a packet and the ReleaseBuffer call that releases the packet. The implementation of the audio engine assumes that the GetBuffer call and the corresponding ReleaseBuffer call occur within the same buffer-processing period. Clients that delay releasing a packet for more than one period risk losing sample data.
In particular you should NEVER DO ANY OF THE FOLLOWING between GetBuffer and ReleaseBuffer because eventually they will cause a glitch:
Wait on a lock
Wait on any other operation
Read from or write to a file
Allocate memory
Instead, pre-allocate a bunch of memory before calling IAudioClient::Start. As each buffer arrives, write to this memory. On the side, have a regularly scheduled work item that takes written memory and writes it to disk or whatever you're doing with it.
I have some embedded Devices that have no audio device by default. They communicate with each other via a FPGA. So my question is, how do I capture/play back audio from pjsip in pcm in order to send/receive it with the FPGA?
I know that there is pjmedia_mem_player_create() and pjmedia_mem_capture_create() but I can't seem to find any good info towards using these functions.
I tried the following piece of code, but an assertion failed cause one of the function's parameter is "empty".
Error:
pjmedia_mem_capture_create: Assertion `pool && buffer && size && clock_rate && channel_count && samples_per_frame && bits_per_sample && p_port' failed.
Note: I'm mainly using pjsua2 for everything else like registrations, transports etc. Also the default audio is set to null with ep.audDevManager().setNullDev(); as without this, making/receiving a call would simply fail?!
void MyCall::onCallMediaState(OnCallMediaStateParam &prm){
CallInfo ci = getInfo();
pj_caching_pool_init(&cp, &pj_pool_factory_default_policy, 0);
pj_pool_t *pool = pj_pool_create(&cp.factory, "POOLNAME", 2000, 2000, NULL);
void *buffer;
pjmedia_port *prt;
#define CLOCK_RATE 8000
#define CHANELS 1
#define SAMPLES_PER_FRAME 480
#define BITS_PER_SAMPLE 16
pjmedia_mem_capture_create( pool, //Pool
buffer, //Buffer
2000, //Buffer Size
CLOCK_RATE,
CHANELS,
SAMPLES_PER_FRAME,
BITS_PER_SAMPLE,
0, //Options
&prt); //The return port}
UPDATE
The assertion failed cause the buffer variable doesn't have any memory allocated to it. Allocate with twice the amount of samples per frame to have sufficient memory.
buffer = pj_pool_zalloc(pool, 960);
Also a callback needs to be registered with pjmedia_mem_capture_set_eof_cb2() (The two at the end is necessary for PJSIP 2.10 or later) Apparently from there the buffer can be used. Just that my implementation atm doesn't execute the callback.
Looks like I found the solution, I have modified your code and wrote a simple code in C with pjsua API to dump every frame to file. Sorry for mess, I'm not proficient in C:
pjsua_call_info ci;
pjsua_call_get_info(call_id, &ci);
pjsua_conf_port_info cpi;
pjsua_conf_get_port_info(ci.conf_slot, &cpi);
pj_pool_t *pool = pjsua_pool_create("POOLNAME", 2000, 2000);
pjmedia_port *prt;
uint buf_size = cpi.bits_per_sample*cpi.samples_per_frame/8;
void *buffer = pj_pool_zalloc(pool, buf_size);
pjsua_conf_port_id port_id;
pjmedia_mem_capture_create( pool,
buffer,
buf_size,
cpi.clock_rate,
cpi.channel_count,
cpi.samples_per_frame,
cpi.bits_per_sample,
0,
&prt);
pjmedia_mem_capture_set_eof_cb(prt, buffer, dump_incoming_frames);
pjsua_conf_add_port(pool, prt, &port_id);
pjsua_conf_connect(ci.conf_slot, port_id); //connect port with conference
///////dumping frames///
static pj_status_t dump_incoming_frames(pjmedia_port * port, void * usr_data){
pj_size_t buf_size = pjmedia_mem_capture_get_size(port);
char * data = usr_data;
...
fwrite(data,sizeof(data[0]),buf_size,fptr);
...
}
Documenation says pjmedia_mem_capture_set_eof_cb is deprecated but I couldn't make work pjmedia_mem_capture_set_eof_cb2, buf_size is 0 for every call of dump_incoming_frames so just left with deprecated function. I also succeed the same result with creating custom port.
I hope you can modify it easily to your C++/pjsua2 code
UPD:
I have modified the PJSIP and packed audio in-out streaming into proper PJSUA2/Media classes so it can be called from Python. Full code is here.
I have a continuous audio stream flow from which I'd like to keep only the last 50 seconds.
I do it like so for now but I have issues:
The buffer grows beyond 50 seconds. I tried to use resize on a QByteArray to get closer from a FIFO , but it doesn't seem to care much about it and continue to grow so I guess I misunderstood the doc.
The sound get suddenly ugly when we pass the 15 sec -> I have no clue why the sound suddenly rips (if I load a 5min local sound in the same buffer, everything works great).
Last, but not least of course, the buffer doesn't behave like a FIFO and this is what I'm looking for to keep only the last 50sec
Here is my code: (Windows10 - Qt5)
buffer = new QBuffer;
arr = new QByteArray;
arr->resize(200000);
buffer->setData(*arr); // buffer->setBuffer(*arr);
buffer->open(QIODevice::ReadWrite);
dataStream.setDevice(buffer); `
m_player = new QMediaPlayer(this);
m_player->setMedia(QMediaContent(), buffer);
[...]
connect(m_player, &QMediaPlayer::durationChanged, this, &MainWindow::durationChanged);
connect(m_player, &QMediaPlayer::positionChanged, this, &MainWindow::positionChanged);
I refresh the buffer duration in the latest slot.
So the buffer gets bigger than 50sec of course and the sound get chopped after 15 sec.
Does anyone know how to set up a FIFO buffer with a restricted size with Qt?
---------------------------------------------------------- EDIT ------------------------------------------------------------------
I found this to get closer but I miss something in it :
//.cpp :
#define SAMPLE_RATE 22050
#define CHANNELS 1
#define SAMPLE_SIZE 16
#define SAMPLE_TYPE SignedInt
myAudio::myAudio()
{
formatIn.setSampleRate(SAMPLE_RATE);
formatIn.setChannelCount(CHANNELS);
formatIn.setSampleSize(SAMPLE_SIZE);
[...]
formatIn.setByteOrder(QAudioFormat::LittleEndian);
formatIn.setSampleType(QAudioFormat::SAMPLE_TYPE);
formatOut. //same than formatIn
[...]
//configure device
audioOut = new QAudioOutput(deviceOut,formatOut,0);
audioIn = new QAudioInput (deviceIn, formatIn,0);
buff.resize(0x10000); //create a rx buffer
pbuff=buff.data(); //get the buff address;
RXbuff=0; //set RX buffer pointer
qDebug()<<"File open"<<open(QIODevice::ReadWrite);
qDebug()<<"is device Sequential="<<isSequential();
audioIn->start(this); //start reading device
audioOut->setVolume(0.5); //volume 0 to 1.0
audioOut->start(this); //start writing to device
}
//QIODevice Class (Protected Functions)This function is called by QIODevice.
//send to output(Speaker)
qint64 myAudio::readData(char *data, qint64 len)
{
static quint64 TXbuff=0;
qint64 total = 0;
while (len > total && RXbuff>TXbuff)//write and synchonise buffers
{
//write data to speaker
memcpy(&data[total],&pbuff[TXbuff%0x10000], 2); //copy 2 Bytes
TXbuff+=2; //point to next buffer 16 bit location
total+=2;
}
return total; //the reset interval
}
//audio input (from Microphone)
qint64 myAudio::writeData(const char *data, qint64 len)
{
int total=0;
while (len > total)
{
memcpy(&pbuff[RXbuff%0x10000],&data[total], 2); //write 2Bytes into
circular buffer(64K)
RXbuff+=2; //next 16bit buffer location
total+=2; //next data location
}
return (total); //return total number of bytes received
}
qint64 myAudio::bytesAvailable() const{return 0;}
What I miss is pretty basic... does anyone know When/How are the methods called ?!
I'm working on a project in openGL and it needs to be able to play simple sounds (mp3) from file while not interrupting the draw loop.
I've been playing around with a few different libraries (openAL, portaudio) and eventually settled on mpg123 (to load the mp3) and libao to play the mp3 back.
The current playsound function works but it blocks the openGL draw loop (ie. freezes the game) until the audio has completed playing. I have tried messing around with std::thread but it still blocked the draw loop.
Here is the audio playback function I've been testing with:
void playSound() {
mpg123_handle *mh;
unsigned char *buffer;
size_t buffer_size;
size_t done;
int err;
int driver;
ao_device *dev;
ao_sample_format format;
int channels, encoding;
long rate;
/* initializations */
ao_initialize();
driver = ao_default_driver_id();
mpg123_init();
mh = mpg123_new(NULL, &err);
buffer_size = mpg123_outblock(mh);
buffer = (unsigned char*) malloc(buffer_size * sizeof(unsigned char));
/* open the file and get the decoding format */
mpg123_open(mh, "sounds/door.mp3");
mpg123_getformat(mh, &rate, &channels, &encoding);
/* set the output format and open the output device */
format.bits = mpg123_encsize(encoding) * 8;
format.rate = rate;
format.channels = channels;
format.byte_format = AO_FMT_NATIVE;
format.matrix = 0;
dev = ao_open_live(driver, &format, NULL);
/* decode and play */
while (mpg123_read(mh, buffer, buffer_size, &done) == MPG123_OK)
ao_play(dev, (char*)buffer, done);
/* clean up */
free(buffer);
ao_close(dev);
mpg123_close(mh);
mpg123_delete(mh);
mpg123_exit();
ao_shutdown();
}
How would I go about fixing this so that my game continues to run smoothly and the audio plays in the background?
You should unpack small amount of audio data and feed it to an audio device every frame.
The main trick is to find out how many samples was played by device already. I'm not sure how you can do this with libao, but it pretty simple with OpenAL.
You can check details here Play stream in OpenAL library
Also, you always can use additional thread. It'll be overkill, but very simple to do and can work fine for a small/demo project.
I have four buffers that I am using for audio playback in a synthesizer. I submit two buffers initially, and then in the callback routine I write data into the next buffer and then submit that buffer.
When I generate each buffer I'm just putting a sine wave into it whose period is a multiple of the buffer length.
When I execute I hear brief pauses between each buffer. I've increased the buffer size to 16K samples at 44100 Hz so I can clearly hear that the whole buffer is playing, but there is an interruption between each.
What I think is happening is that the callback function is only called when ALL buffers that have been written are complete. I need the synthesis to stay ahead of the playback so I need a callback when each buffer is completed.
How do people usually solve this problem?
Update: I've been asked to add code. Here's what I have:
First I connect to the WaveOut device:
// Always grab the mapped wav device.
UINT deviceId = WAVE_MAPPER;
// This is an excelent tutorial:
// http://planet-source-code.com/vb/scripts/ShowCode.asp?txtCodeId=4422&lngWId=3
WAVEFORMATEX wfx;
wfx.nSamplesPerSec = 44100;
wfx.wBitsPerSample = 16;
wfx.nChannels = 1;
wfx.cbSize = 0;
wfx.wFormatTag = WAVE_FORMAT_PCM;
wfx.nBlockAlign = (wfx.wBitsPerSample >> 3) * wfx.nChannels;
wfx.nAvgBytesPerSec = wfx.nBlockAlign * wfx.nSamplesPerSec;
_waveChangeEventHandle = CreateMutex(NULL,false,NULL);
MMRESULT res;
res = waveOutOpen(&_wo, deviceId, &wfx, (DWORD_PTR)WavCallback,
(DWORD_PTR)this, CALLBACK_FUNCTION);
I initialize the four frames I'll be using:
for (int i=0; i<_numFrames; ++i)
{
WAVEHDR *header = _outputFrames+i;
ZeroMemory(header, sizeof(WAVEHDR));
// Block size is in bytes. We have 2 bytes per sample.
header->dwBufferLength = _codeSpec->OutputNumSamples*2;
header->lpData = (LPSTR)malloc(2 * _codeSpec->OutputNumSamples);
ZeroMemory(header->lpData, 2*_codeSpec->OutputNumSamples);
res = waveOutPrepareHeader(_wo, header, sizeof(WAVEHDR));
if (res != MMSYSERR_NOERROR)
{
printf("Error preparing header: %d\n", res - MMSYSERR_BASE);
}
}
SubmitBuffer();
SubmitBuffer();
Here is the SubmitBuffer code:
void Vodec::SubmitBuffer()
{
WAVEHDR *header = _outputFrames+_curFrame;
MMRESULT res;
res = waveOutWrite(_wo, header, sizeof(WAVEHDR));
if (res != MMSYSERR_NOERROR)
{
if (res = WAVERR_STILLPLAYING)
{
printf("Cannot write when still playing.");
}
else
{
printf("Error calling waveOutWrite: %d\n", res-WAVERR_BASE);
}
}
_curFrame = (_curFrame+1)&0x3;
if (_pointQueue != NULL)
{
RenderQueue();
_nextFrame = (_nextFrame + 1) & 0x3;
}
}
And here is my callback code:
void CALLBACK Vodec::WavCallback(HWAVEOUT hWaveOut,
UINT uMsg,
DWORD dwInstance,
DWORD dwParam1,
DWORD dwParam2 )
{
// Only listen for end of block messages.
if(uMsg != WOM_DONE) return;
Vodec *instance = (Vodec *)dwInstance;
instance->SubmitBuffer();
}
The RenderQueue code is pretty simple - just copies a piece of a template buffer into the output buffer:
void Vodec::RenderQueue()
{
double white = _pointQueue->White;
white = 10.0; // For now just override with a constant value
int numSamples = _codeSpec->OutputNumSamples;
signed short int *data = (signed short int *)_outputFrames[_nextFrame].lpData;
for (int i=0; i<numSamples; ++i)
{
Sample x = white * _noise->Samples[i];
data[i] = (signed short int)(x);
}
_sampleOffset += numSamples;
if (_sampleOffset >= _pointQueue->DurationInSamples)
{
_sampleOffset = 0;
_pointQueue = _pointQueue->next;
}
}
UPDATE: Mostly solved the issue. I need to increment _nextFrame along with _curFrame (not conditionally). The playback buffer was getting ahead of the writing buffer.
However, when I decrease the playback buffer to 1024 samples, it gets choppy again. At 2048 samples it is clear. This happens for both Debug and Release builds.
1024 samples is just about 23ms of audio data. wav is pretty high level API from Windows Vista onwards. If you want low-latency audio playback, you should use CoreAudio. You can get latencies down to 10 ms in shared mode and 3 ms in exclusive mode. Also, the audio depends upon the processes currently running on your system. In other words, it depends on how frequently your audio thread can run to get data. You should also look at MultiMedia Class Scheduler Service and AvSetMmThreadCharacteristics function.