Set up a QBuffer as a FIFO with a restricted size ? QT - c++

I have a continuous audio stream flow from which I'd like to keep only the last 50 seconds.
I do it like so for now but I have issues:
The buffer grows beyond 50 seconds. I tried to use resize on a QByteArray to get closer from a FIFO , but it doesn't seem to care much about it and continue to grow so I guess I misunderstood the doc.
The sound get suddenly ugly when we pass the 15 sec -> I have no clue why the sound suddenly rips (if I load a 5min local sound in the same buffer, everything works great).
Last, but not least of course, the buffer doesn't behave like a FIFO and this is what I'm looking for to keep only the last 50sec
Here is my code: (Windows10 - Qt5)
buffer = new QBuffer;
arr = new QByteArray;
arr->resize(200000);
buffer->setData(*arr); // buffer->setBuffer(*arr);
buffer->open(QIODevice::ReadWrite);
dataStream.setDevice(buffer); `
m_player = new QMediaPlayer(this);
m_player->setMedia(QMediaContent(), buffer);
[...]
connect(m_player, &QMediaPlayer::durationChanged, this, &MainWindow::durationChanged);
connect(m_player, &QMediaPlayer::positionChanged, this, &MainWindow::positionChanged);
I refresh the buffer duration in the latest slot.
So the buffer gets bigger than 50sec of course and the sound get chopped after 15 sec.
Does anyone know how to set up a FIFO buffer with a restricted size with Qt?
---------------------------------------------------------- EDIT ------------------------------------------------------------------
I found this to get closer but I miss something in it :
//.cpp :
#define SAMPLE_RATE 22050
#define CHANNELS 1
#define SAMPLE_SIZE 16
#define SAMPLE_TYPE SignedInt
myAudio::myAudio()
{
formatIn.setSampleRate(SAMPLE_RATE);
formatIn.setChannelCount(CHANNELS);
formatIn.setSampleSize(SAMPLE_SIZE);
[...]
formatIn.setByteOrder(QAudioFormat::LittleEndian);
formatIn.setSampleType(QAudioFormat::SAMPLE_TYPE);
formatOut. //same than formatIn
[...]
//configure device
audioOut = new QAudioOutput(deviceOut,formatOut,0);
audioIn = new QAudioInput (deviceIn, formatIn,0);
buff.resize(0x10000); //create a rx buffer
pbuff=buff.data(); //get the buff address;
RXbuff=0; //set RX buffer pointer
qDebug()<<"File open"<<open(QIODevice::ReadWrite);
qDebug()<<"is device Sequential="<<isSequential();
audioIn->start(this); //start reading device
audioOut->setVolume(0.5); //volume 0 to 1.0
audioOut->start(this); //start writing to device
}
//QIODevice Class (Protected Functions)This function is called by QIODevice.
//send to output(Speaker)
qint64 myAudio::readData(char *data, qint64 len)
{
static quint64 TXbuff=0;
qint64 total = 0;
while (len > total && RXbuff>TXbuff)//write and synchonise buffers
{
//write data to speaker
memcpy(&data[total],&pbuff[TXbuff%0x10000], 2); //copy 2 Bytes
TXbuff+=2; //point to next buffer 16 bit location
total+=2;
}
return total; //the reset interval
}
//audio input (from Microphone)
qint64 myAudio::writeData(const char *data, qint64 len)
{
int total=0;
while (len > total)
{
memcpy(&pbuff[RXbuff%0x10000],&data[total], 2); //write 2Bytes into
circular buffer(64K)
RXbuff+=2; //next 16bit buffer location
total+=2; //next data location
}
return (total); //return total number of bytes received
}
qint64 myAudio::bytesAvailable() const{return 0;}
What I miss is pretty basic... does anyone know When/How are the methods called ?!

Related

ESP32 i2s_read returns empty buffer after calling this function

I am trying to record audio from an INMP441 which is connected to a ESP32 but returning the buffer containing the bytes the microphone read always leads to something which is NULL.
The code for setting up i2s and the microphone is this:
// i2s config
const i2s_config_t i2s_config = {
.mode = i2s_mode_t(I2S_MODE_MASTER | I2S_MODE_RX), // receive
.sample_rate = SAMPLE_RATE, // 44100 (44,1KHz)
.bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT, // 32 bits per sample
.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT, // use right channel
.communication_format = i2s_comm_format_t(I2S_COMM_FORMAT_I2S | I2S_COMM_FORMAT_I2S_MSB),
.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1, // interrupt level 1
.dma_buf_count = 64, // number of buffers
.dma_buf_len = SAMPLES_PER_BUFFER}; // 512
// pin config
const i2s_pin_config_t pin_config = {
.bck_io_num = gpio_sck, // serial clock, sck (gpio 33)
.ws_io_num = gpio_ws, // word select, ws (gpio 32)
.data_out_num = I2S_PIN_NO_CHANGE, // only used for speakers
.data_in_num = gpio_sd // serial data, sd (gpio 34)
};
// config i2s driver and pins
// fct must be called before any read/write
esp_err_t err = i2s_driver_install(I2S_PORT, &i2s_config, 0, NULL);
if (err != ESP_OK)
{
Serial.printf("Failed installing the driver: %d\n", err);
}
err = i2s_set_pin(I2S_PORT, &pin_config);
if (err != ESP_OK)
{
Serial.printf("Failed setting pin: %d\n", err);
}
Serial.println("I2S driver installed! :-)");
Setting up the i2s stuff is no problem at all. The tricky part for me is reading from the i2s:
// 44KHz * Byte per sample * time in seconds = total size in bytes
const size_t recordSize = (SAMPLE_RATE * I2S_BITS_PER_SAMPLE_32BIT / 8) * recordTime; //recordTime = 5s
// size in bytes
size_t totalReadSize = 0;
// 32 bits per sample set in config * 1024 samples per buffers = total bits per buffer
char *samples = (char *)calloc(totalBitsPerBuffer, sizeof(char));
// number of bytes read
size_t bytesRead;
Serial.println("Start recording...");
// read until wanted size is reached
while (totalReadSize < recordSize)
{
// read to buffer
esp_err_t err = i2s_read(I2S_PORT, (void *)samples, totalBitsPerBuffer, &bytesRead, portMAX_DELAY);
// check if error occurd, if so stop recording
if (err != ESP_OK)
{
Serial.println("Error while recording!");
break;
}
// check if bytes read works → yes
/*
for (int i = 0; i < bytesRead; i++)
{
uint8_t sample = (uint8_t) samples[i];
Serial.print(sample);
} */
// add read size to total read size
totalReadSize += bytesRead;
// Serial.printf("Currently recorded %d%% \n", totalReadSize * 100 / recordSize);
}
// convert bytes to mb
double_t totalReadSizeMB = (double_t)totalReadSize / 1e+6;
Serial.printf("Total read size: %fMb\n", totalReadSizeMB);
Serial.println("Samples deref");
Serial.println(*samples);
Serial.println("Samples");
Serial.println(samples);
return samples;
Using this code leads to the following output:
I2S driver installed! :-)
Start recording...
Total read size: 0.884736Mb
Samples deref
␀
Samples
When I uncomment the part where I iterate over the bytes read part I get something like this:
200224231255255224210022418725525522493000902552550238002241392542552241520020425225508050021624525501286700194120022461104022421711102242271030018010402242510000188970224141930022291022410185022487830021679001127500967200666902241776600246610224895902244757022418353002224802242274302249741022419339009435001223102242432602243322022412120001241402245911022418580084402248325525522461252255044249255224312452552242212372552241272352550342302552241212262552242112212550252216255014621325501682092550112205255224161202255224237198255224235194255224231922552248518725501141832550421812552241951762550144172255018168255034164255224173157255018215525522455152255028148255021014425505214025522487137255014613225522412112825502361252550180120255018011725522451172550252113255224133111255061082550248105255224891042552249910125522439972550138942552242279225503287255224101832552242478125522410178255224231732552244970255224336525501766225501426125502325625522424553255224109492550186[...]
This shows that the microphone is able to record, but I cant return the actual value of the buffer.
While programming this code I looked up at the official doku and some code which seems to work elsewhere.
I am also new to C++ and am not used to work with pointers.
Does anyone know what the problem could be?

PortAudio callbacks not continuous?

I'm having a problem using PortAudio and I am not sure whether I don't quite understand how the callbacks work, or I did something wrong. My assumption was that the callbacks should fire continuous "on tick", containing the current samples, but it seems like I'm only receiving a few callbacks when I open and start the stream, and then I won't receive callbacks ever again. My code:
PaStreamParameters inputP, outputP;
inputP.device = DeviceIndex;
inputP.channelCount = CardInfo->maxInputChannels;
inputP.sampleFormat = paFloat32;
inputP.suggestedLatency = CardInfo->defaultLowInputLatency;
inputP.hostApiSpecificStreamInfo = NULL;
outputP.device = DeviceIndex;
outputP.channelCount = CardInfo->maxOutputChannels;
outputP.sampleFormat = paFloat32;
outputP.suggestedLatency = CardInfo->defaultLowOutputLatency;
outputP.hostApiSpecificStreamInfo = NULL;
PaError err = Pa_OpenStream(
&AudioStream,
&inputP,
&outputP,
SAMPLE_RATE,
FRAMES_PER_BUFFER,
paClipOff,
&CAudio::AudioCallback,
this
);
err = Pa_StartStream(AudioStream);
I used the following constants
#define SAMPLE_RATE 44100
#define FRAMES_PER_BUFFER 64
The callback:
int CAudio::AudioCallback(const void* pInputBuffer, void* pOutputBuffer, unsigned long iFramesPerBuffer, const PaStreamCallbackTimeInfo* timeInfo, PaStreamCallbackFlags statusFlags, void* userData)
{
CAudio* AudioInterface = (CAudio*)userData;
const float* buffer = (const float*)pInputBuffer;
printf("Callback: %d frames per buffer %d t %f\n", AudioInterface->CallbackIndex, iFramesPerBuffer, Pa_GetStreamTime(AudioInterface->AudioStream));
AudioInterface->CallbackIndex++;
return paContinue;
}
Now, what happens if I open and start the stream is that I get exactly six callbacks, the following log output
Opening audio stream on 'USB PnP Sound Device: Audio (hw:1,0)'
Opened audio stream, starting it...
Callback: 0 frames per buffer 64 t 21812.485122
Callback: 1 frames per buffer 64 t 21812.497681
Callback: 2 frames per buffer 64 t 21812.514483
Callback: 3 frames per buffer 64 t 21812.525110
Callback: 4 frames per buffer 64 t 21812.626489
Callback: 5 frames per buffer 64 t 21812.635590
Am I doing something wrong, or is my understanding of what happens with the callbacks wrong?
Alright, for now I fixed this by using the blocking API, grabbing the stream and writing it to a buffer in a while loop running in another std::thread, which works kind of good.

My OpenAL C++ audio streaming buffer gliching

I have for the first time coding sound generating with OpenAL in C++.
What I want to do is to generate endless sinus wave into a double buffering way.
And the problem is that the sound is glittering/lags. I Think it is between the buffering and I don't know why it is like that.
My code:
void _OpenALEngine::play()
{
if(!m_running && !m_threadRunning)
{
ALfloat sourcePos[] = {0,0,0};
ALfloat sourceVel[] = {0,0,0};
ALfloat sourceOri[] = {0,0,0,0,0,0};
alGenSources(1, &FSourceID);
alSourcefv (FSourceID, AL_POSITION, sourcePos);
alSourcefv (FSourceID, AL_VELOCITY, sourceVel);
alSourcefv (FSourceID, AL_DIRECTION, sourceOri);
GetALError();
ALuint FBufferID[2];
alGenBuffers( 2, &FBufferID[0] );
GetALError();
// Gain
ALfloat listenerPos[] = {0,0,0};
ALfloat listenerVel[] = {0,0,0};
ALfloat listenerOri[] = {0,0,0,0,0,0};
alListenerf( AL_GAIN, 1.0 );
alListenerfv(AL_POSITION, listenerPos);
alListenerfv(AL_VELOCITY, listenerVel);
alListenerfv(AL_ORIENTATION, listenerOri);
GetALError();
alSourceQueueBuffers( FSourceID, 2, &FBufferID[0] );
GetALError();
alSourcePlay(FSourceID);
GetALError();
m_running = true;
m_threadRunning = true;
Threading::Thread thread(Threading::ThreadStart(this, &_OpenALEngine::threadPlaying));
thread.Start();
}
}
Void _OpenALEngine::threadPlaying()
{
while(m_running)
{
// Check how much data is processed in OpenAL's internal queue.
ALint Processed;
alGetSourcei( FSourceID, AL_BUFFERS_PROCESSED, &Processed );
GetALError();
// Add more buffers while we need them.
while ( Processed-- )
{
alSourceUnqueueBuffers( FSourceID, 1, &BufID );
runBuffer(); // <--- Generate the sinus wave and submit the Array to the submitBuffer method.
alSourceQueueBuffers( FSourceID, 1, &BufID );
ALint val;
alGetSourcei(FSourceID, AL_SOURCE_STATE, &val);
if(val != AL_PLAYING)
{
alSourcePlay(FSourceID);
}
}
// Don't kill the CPU.
Thread::Sleep(1);
}
m_threadRunning = false;
return Void();
}
void _OpenALEngine::submitBuffer(byte* buffer, int length)
{
// Submit more data to OpenAL
alBufferData( BufID, AL_FORMAT_MONO8, buffer, length * sizeof(byte), 44100 );
}
I generate the sinus wave in the runBuffer() method. And the sinus generator is correct because when I increase the buffer array from 4096 to 40960 the glittering/lags sound with bigger interval. Thank you very much if some one know the problem and will share it :)
Similar Problems are all over the internet and I'm not 100% sure this is the solution to this on. But it probably is, and if not it might at least help others. Most other threads are on different forums and I'm not registering everywhere just to share my knowledge...
The code below is what I came up after 2 days of experimenting. Most solutions I found did not work for me...
(it's not exactly my code, I stripped it of some parts special to my case, so I'm sorry if there are typos or similar that prevent it from being copied verbatim)
My experiments were on an iPhone. Some of the the things I found out, might be iOS-specific.
The problem is that there is no guaranty at what point a processed buffer is marked as such and is available for unqueueing. Trying to build a version that sleeps until a buffer becomes available again I saw that this might be much(I use very small buffers) later than expected. So I realised that the common idea to wait until a buffer is available(which works for most frameworks, but not openAL) is wrong. Instead you should wait until the time you should enqueue another buffer.
With that you have to give up the idea of double-buffering. When the time comes you should check if a buffer exists and unqueue it. But if none is available you need to create a 3rd...
Waiting for when a buffer should be enqueue can be done by calculating times relative to the system-clock, which worked fairly well for me but I decided to go for a version where I rely on a time source that is definitivly in sync with openAL. Best I came up with was wait depending on what s left in the queue. Here, iOS seems not fully in accordance to openAL-spec because AL_SAMPLE_OFFSET should be exact to one sample but I never saw anything but multiples of 2048. That's about 45 microseconds #44100, this is where the 50000 in the code comes from(little more than the smalest unit iOS handles)
Depending on the block-size this can easily be bigger. But with that code I had 3 times that alSourcePlay() was needed again in the last ~hour(compared to up to 10 per minute with other implementations that claimed to be the solution)
uint64 enqueued(0); // keep track of samples in queue
while (bKeepRunning)
{
// check if enough in buffer and wait
ALint off;
alGetSourcei(m_Source, AL_SAMPLE_OFFSET, &off);
uint32 left((enqueued-off)*1000000/SAMPLE_RATE);
if (left > 50000) // at least 50000 mic-secs in buffer
usleep(left - 50000);
// check for available buffer
ALuint buffer;
ALint processed;
alGetSourcei(m_Source, AL_BUFFERS_PROCESSED, &processed);
switch (processed)
{
case 0: // no buffer to unqueue->create new
alGenBuffers(1, &buffer);
break;
case 1: // on buffer to unqueue->use that
alSourceUnqueueBuffers(m_Source, 1, &buffer);
enqueued -= BLOCK_SIZE_SAMPLES;
break;
default: // multiple buffers to unqueue->take one,delete on
{ // could also delete more if processed>2
// but doesn't happen often
// therefore simple implementation(will del. in next loop)
ALuint bufs[2];
alSourceUnqueueBuffers(m_Source, 2, bufs);
alDeleteBuffers(1, bufs);
buffer = bufs[1];
enqueued -= 2*BLOCK_SIZE_SAMPLES;
}
break;
}
// fill block
alBufferData(buffer, AL_FORMAT_STEREO16, pData,
BLOCK_SIZE_SAMPLES*4, SAMPLE_RATE);
alSourceQueueBuffers(m_Source, 1, &buffer);
//check state
ALint state;
alGetSourcei(m_Source, AL_SOURCE_STATE, &state);
if (state != AL_PLAYING)
{
enqueued = BLOCK_SIZE_SAMPLES;
alSourcePlay(m_Source);
}
else
enqueued += BLOCK_SIZE_SAMPLES;
}
I have written OpenAL streaming servers so I know your pain - my instinct is to confirm you have spawned separate threads for the I/O logic which available your streaming audio data - separate from the thread to hold your above OpenAL code ??? If not this will cause your symptoms. Here is a simple launch of each logical chunk into its own thread :
std::thread t1(launch_producer_streaming_io, chosen_file, another_input_parm);
std::this_thread::sleep_for (std::chrono::milliseconds( 100));
std::thread t2(launch_consumer_openal, its_input_parm1, parm2);
// -------------------------
t1.join();
t2.join();
where launch_producer_streaming_io is a method being called with its input parms which services the input/output to continuously supply the audio data ... launch_consumer_openal is a method launched in its own thread where you instantiate your OpenAL class

FFmpeg + OpenAL - playback streaming sound from video won't work

I am decoding an OGG video (theora & vorbis as codecs) and want to show it on the screen (using Ogre 3D) while playing its sound. I can decode the image stream just fine and the video plays perfectly with the correct frame rate, etc.
However, I cannot get the sound to play at all with OpenAL.
Edit: I managed to make the playing sound resemble the actual audio in the video at least somewhat. Updated sample code.
Edit 2: I was able to get "almost" correct sound now. I had to set OpenAL to use AL_FORMAT_STEREO_FLOAT32 (after initializing the extension) instead of just STEREO16. Now the sound is "only" extremely high pitched and stuttering, but at the correct speed.
Here is how I decode audio packets (in a background thread, the equivalent works just fine for the image stream of the video file):
//------------------------------------------------------------------------------
int decodeAudioPacket( AVPacket& p_packet, AVCodecContext* p_audioCodecContext, AVFrame* p_frame,
FFmpegVideoPlayer* p_player, VideoInfo& p_videoInfo)
{
// Decode audio frame
int got_frame = 0;
int decoded = avcodec_decode_audio4(p_audioCodecContext, p_frame, &got_frame, &p_packet);
if (decoded < 0)
{
p_videoInfo.error = "Error decoding audio frame.";
return decoded;
}
// Frame is complete, store it in audio frame queue
if (got_frame)
{
int bufferSize = av_samples_get_buffer_size(NULL, p_audioCodecContext->channels, p_frame->nb_samples,
p_audioCodecContext->sample_fmt, 0);
int64_t duration = p_frame->pkt_duration;
int64_t dts = p_frame->pkt_dts;
if (staticOgreLog)
{
staticOgreLog->logMessage("Audio frame bufferSize / duration / dts: "
+ boost::lexical_cast<std::string>(bufferSize) + " / "
+ boost::lexical_cast<std::string>(duration) + " / "
+ boost::lexical_cast<std::string>(dts), Ogre::LML_NORMAL);
}
// Create the audio frame
AudioFrame* frame = new AudioFrame();
frame->dataSize = bufferSize;
frame->data = new uint8_t[bufferSize];
if (p_frame->channels == 2)
{
memcpy(frame->data, p_frame->data[0], bufferSize >> 1);
memcpy(frame->data + (bufferSize >> 1), p_frame->data[1], bufferSize >> 1);
}
else
{
memcpy(frame->data, p_frame->data, bufferSize);
}
double timeBase = ((double)p_audioCodecContext->time_base.num) / (double)p_audioCodecContext->time_base.den;
frame->lifeTime = duration * timeBase;
p_player->addAudioFrame(frame);
}
return decoded;
}
So, as you can see, I decode the frame, memcpy it to my own struct, AudioFrame. Now, when the sound is played, I use these audio frame like this:
int numBuffers = 4;
ALuint buffers[4];
alGenBuffers(numBuffers, buffers);
ALenum success = alGetError();
if(success != AL_NO_ERROR)
{
CONSOLE_LOG("Error on alGenBuffers : " + Ogre::StringConverter::toString(success) + alGetString(success));
return;
}
// Fill a number of data buffers with audio from the stream
std::vector<AudioFrame*> audioBuffers;
std::vector<unsigned int> audioBufferSizes;
unsigned int numReturned = FFMPEG_PLAYER->getDecodedAudioFrames(numBuffers, audioBuffers, audioBufferSizes);
// Assign the data buffers to the OpenAL buffers
for (unsigned int i = 0; i < numReturned; ++i)
{
alBufferData(buffers[i], _streamingFormat, audioBuffers[i]->data, audioBufferSizes[i], _streamingFrequency);
success = alGetError();
if(success != AL_NO_ERROR)
{
CONSOLE_LOG("Error on alBufferData : " + Ogre::StringConverter::toString(success) + alGetString(success)
+ " size: " + Ogre::StringConverter::toString(audioBufferSizes[i]));
return;
}
}
// Queue the buffers into OpenAL
alSourceQueueBuffers(_source, numReturned, buffers);
success = alGetError();
if(success != AL_NO_ERROR)
{
CONSOLE_LOG("Error queuing streaming buffers: " + Ogre::StringConverter::toString(success) + alGetString(success));
return;
}
}
alSourcePlay(_source);
The format and frequency I give to OpenAL are AL_FORMAT_STEREO_FLOAT32 (it is a stereo sound stream, and I did initialize the FLOAT32 extension) and 48000 (which is the sample rate of the AVCodecContext of the audio stream).
And during playback, I do the following to refill OpenAL's buffers:
ALint numBuffersProcessed;
// Check if OpenAL is done with any of the queued buffers
alGetSourcei(_source, AL_BUFFERS_PROCESSED, &numBuffersProcessed);
if(numBuffersProcessed <= 0)
return;
// Fill a number of data buffers with audio from the stream
std::vector<AudiFrame*> audioBuffers;
std::vector<unsigned int> audioBufferSizes;
unsigned int numFilled = FFMPEG_PLAYER->getDecodedAudioFrames(numBuffersProcessed, audioBuffers, audioBufferSizes);
// Assign the data buffers to the OpenAL buffers
ALuint buffer;
for (unsigned int i = 0; i < numFilled; ++i)
{
// Pop the oldest queued buffer from the source,
// fill it with the new data, then re-queue it
alSourceUnqueueBuffers(_source, 1, &buffer);
ALenum success = alGetError();
if(success != AL_NO_ERROR)
{
CONSOLE_LOG("Error Unqueuing streaming buffers: " + Ogre::StringConverter::toString(success));
return;
}
alBufferData(buffer, _streamingFormat, audioBuffers[i]->data, audioBufferSizes[i], _streamingFrequency);
success = alGetError();
if(success != AL_NO_ERROR)
{
CONSOLE_LOG("Error on re- alBufferData: " + Ogre::StringConverter::toString(success));
return;
}
alSourceQueueBuffers(_source, 1, &buffer);
success = alGetError();
if(success != AL_NO_ERROR)
{
CONSOLE_LOG("Error re-queuing streaming buffers: " + Ogre::StringConverter::toString(success) + " "
+ alGetString(success));
return;
}
}
// Make sure the source is still playing,
// and restart it if needed.
ALint playStatus;
alGetSourcei(_source, AL_SOURCE_STATE, &playStatus);
if(playStatus != AL_PLAYING)
alSourcePlay(_source);
As you can see, I do quite heavy error checking. But I do not get any errors, neither from OpenAL nor from FFmpeg.
Edit: What I hear somewhat resembles the actual audio from the video, but VERY high pitched and stuttering VERY much. Also, it seems to be playing on top of TV noise. Very strange. Plus, it is playing much slower than the correct audio would.
Edit: 2 After using AL_FORMAT_STEREO_FLOAT32, the sound plays at the correct speed, but is still very high pitched and stuttering (though less than before).
The video itself is not broken, it can be played fine on any player. OpenAL can also play *.way files just fine in the same application, so it is also working.
Any ideas what could be wrong here or how to do this correctly?
My only guess is that somehow, FFmpeg's decode function does not produce data OpenGL can read. But this is as far as the FFmpeg decode example goes, so I don't know what's missing. As I understand it, the decode_audio4 function decodes the frame to raw data. And OpenAL should be able to work with RAW data (or rather, doesn't work with anything else).
So, I finally figured out how to do it. Gee, what a mess. It was a hint from a user on the libav-users mailing list that put me on the correct path.
Here are my mistakes:
Using the wrong format in the alBufferData function. I used AL_FORMAT_STEREO16 (as that is what every single streaming example with OpenAL uses). I should have used AL_FORMAT_STEREO_FLOAT32, as the video I stream is Ogg and vorbis is stored in floating points. And using swr_convert to convert from AV_SAMPLE_FMT_FLTP to AV_SAMPLE_FMT_S16 just crashes. No idea why.
Not using swr_convert to convert the decoded audio frame to the target format. After I was trying to use swr_convert to convert from FLTP to S16, and it would simply crash without a reason given, I assumed it was broken. But after figuring out my first mistake, I tried again, converting from FLTP to FLT (non-planar) and then it worked! So OpenAL uses interleaved format, not planar. Good to know.
So here is the decodeAudioPacket function that is working for me with Ogg video, vorbis audio stream:
int decodeAudioPacket( AVPacket& p_packet, AVCodecContext* p_audioCodecContext, AVFrame* p_frame,
SwrContext* p_swrContext, uint8_t** p_destBuffer, int p_destLinesize,
FFmpegVideoPlayer* p_player, VideoInfo& p_videoInfo)
{
// Decode audio frame
int got_frame = 0;
int decoded = avcodec_decode_audio4(p_audioCodecContext, p_frame, &got_frame, &p_packet);
if (decoded < 0)
{
p_videoInfo.error = "Error decoding audio frame.";
return decoded;
}
if(decoded <= p_packet.size)
{
/* Move the unread data to the front and clear the end bits */
int remaining = p_packet.size - decoded;
memmove(p_packet.data, &p_packet.data[decoded], remaining);
av_shrink_packet(&p_packet, remaining);
}
// Frame is complete, store it in audio frame queue
if (got_frame)
{
int outputSamples = swr_convert(p_swrContext,
p_destBuffer, p_destLinesize,
(const uint8_t**)p_frame->extended_data, p_frame->nb_samples);
int bufferSize = av_get_bytes_per_sample(AV_SAMPLE_FMT_FLT) * p_videoInfo.audioNumChannels
* outputSamples;
int64_t duration = p_frame->pkt_duration;
int64_t dts = p_frame->pkt_dts;
if (staticOgreLog)
{
staticOgreLog->logMessage("Audio frame bufferSize / duration / dts: "
+ boost::lexical_cast<std::string>(bufferSize) + " / "
+ boost::lexical_cast<std::string>(duration) + " / "
+ boost::lexical_cast<std::string>(dts), Ogre::LML_NORMAL);
}
// Create the audio frame
AudioFrame* frame = new AudioFrame();
frame->dataSize = bufferSize;
frame->data = new uint8_t[bufferSize];
memcpy(frame->data, p_destBuffer[0], bufferSize);
double timeBase = ((double)p_audioCodecContext->time_base.num) / (double)p_audioCodecContext->time_base.den;
frame->lifeTime = duration * timeBase;
p_player->addAudioFrame(frame);
}
return decoded;
}
And here is how I initialize the context and the destination buffer:
// Initialize SWR context
SwrContext* swrContext = swr_alloc_set_opts(NULL,
audioCodecContext->channel_layout, AV_SAMPLE_FMT_FLT, audioCodecContext->sample_rate,
audioCodecContext->channel_layout, audioCodecContext->sample_fmt, audioCodecContext->sample_rate,
0, NULL);
int result = swr_init(swrContext);
// Create destination sample buffer
uint8_t** destBuffer = NULL;
int destBufferLinesize;
av_samples_alloc_array_and_samples( &destBuffer,
&destBufferLinesize,
videoInfo.audioNumChannels,
2048,
AV_SAMPLE_FMT_FLT,
0);

Is there any memory leak with the QAudioOutput code?

I am playing an audio stream in a QThread like this:
// Setup
QAudioFormat format;
format.setFrequency(44100);
format.setChannels(2);
format.setSampleSize(16);
format.setCodec("audio/pcm");
format.setByteOrder(QAudioFormat::LittleEndian);
format.setSampleType(QAudioFormat::SignedInt);
QAudioDeviceInfo info(QAudioDeviceInfo::defaultOutputDevice());
format = info.nearestFormat(format);
this->m_AudioOutput = new QAudioOutput(format, this);
DECLARE_ALIGNED(16,uint8_t,audio_buffer)[(AVCODEC_MAX_AUDIO_FRAME_SIZE * 3) / 2];
// Playback
QIODevice *iodevice = this->m_AudioOutput->start();
for(;;) {
// Routine that fetches audio data from network
// data_size is length of the buffer
fetch_packet(&audio_buffer, data_size);
qint64 dataRemaining = data_size;
const char *b2 = (const char *)audio_buffer;
while (dataRemaining) {
qint64 bytesWritten = iodevice->write((const char *)b2, dataRemaining);
dataRemaining -= bytesWritten;
b2 = b2 + bytesWritten ;
}
msleep(10);
}
The audio plays just fine but the app's memory consumption seems to increase over time (around 2MB per minute). I was wondering if I have done something wrong. I suppose QAudioOutput should be responsible for deleting the QIODevice's buffer after it has been read and used for playback?
I don't think so, the docs says:
Starting to play an audio stream is simply a matter of calling start() with a QIODevice. QAudioOutput will then fetch the data it needs from the io device.
It it just reading data. The QIODevice should manage the buffer. To be sure, you can check the size of your buffer using QIODevice::size() and see if it is growing.