I am trying to implement streaming audio and I've run into a problem where OpenAL is giving me an error codes seems impossible given the information in the documentation.
int buffersProcessed = 0;
alGetSourcei(m_Source, AL_BUFFERS_PROCESSED, &buffersProcessed);
PrintALError();
int toAddBufferIndex;
// Remove the first buffer from the queue and move it to
//the end after buffering new data.
if (buffersProcessed > 0)
{
ALuint unqueued;
alSourceUnqueueBuffers(m_Source, 1, &unqueued);
/////////////////////////////////
PrintALError(); // Prints AL_INVALID_OPERATION //
/////////////////////////////////
toAddBufferIndex = firstBufferIndex;
}
According to the documentation [PDF], AL_INVALID_OPERATION means: "There is no current context." This seems like it can't be true because OpenAL has been, and continues to play other audio just fine!
Just to be sure, I called ALCcontext* temp = alcGetCurrentContext( ); here and it returned a valid context.
Is there some other error condition that's possible here that's not mentioned in the docs?
More details: The sound source is playing when this code is being called, but the impression I got from reading the spec is you can safely unqueue processed buffers while the source is playing. PrintALError is just a wrapper for alGetError that prints if there is any error.
I am on a Mac (OS 10.8.3), in case it matters.
So far what I've gathered is that it seems this OpenAL implementation incorrectly throws an error if you unqueue a buffer while the source is playing. The spec says that you should be able to unqueue a buffer that has been marked as processing while the source is playing:
Removal of a given queue entry is not possible unless either the source is stopped (in which case then entire queue is considered processed), or if the queue entry has already been processed (AL_PLAYING or AL_PAUSED source).
On that basis I'm gonna say this is probably a bug in my OpenAL implementation. I'm gonna leave the question open in case someone can give a more concrete answer though.
To handle condition for multiple buffers use a loop.
Following works on iOS and linux :
// UN queue used buffers
ALint buffers_processed = 0;
alGetSourcei(streaming_source, AL_BUFFERS_PROCESSED, & buffers_processed); // get source parameter num used buffs
while (buffers_processed > 0) { // we have a consumed buffer so we need to replenish
ALuint unqueued_buffer;
alSourceUnqueueBuffers(streaming_source, 1, & unqueued_buffer);
available_AL_buffer_array_curr_index--;
available_AL_buffer_array[available_AL_buffer_array_curr_index] = unqueued_buffer;
buffers_processed--;
}
Related
I'm building a pipeline in Gstreamer, using a UriDecodeBin. This pipeline is meant to be build and released repeatedly, but I have a hard time to figure out how to properly delete the pipeline.
For now, I'm trying to figure out what to unref and what not to unref to get all resources of a pipeline to be released, with just a UriDecodeBin in the pipeline, but I can't get the full memory to be released somehow.
My sequence of actions is as follows :
Fist, I'm building the pipeline and UriDecodeBin :
_pipeline = (GstPipeline*)gst_pipeline_new("pipeline");
_bus = gst_pipeline_get_bus(_pipeline); //will be manually observed
_decoder = gst_element_factory_make("uridecodebin", "decoder");
gst_bin_add(GST_BIN(_pipeline), _decoder);
gst_element_sync_state_with_parent(_decoder);
std::string uri = std::string("file://some_video_file");
g_object_set(_decoder, "uri", uri.c_str(), nullptr);
g_signal_connect(_decoder, "no-more-pads", G_CALLBACK(GsDecoder::no_more_pads_cb), (gpointer)this);
changePipelineState(GST_STATE_PAUSED);
Then, I wait for the no-more-pads to trigger, and add a blocking probe on each pad :
GstIterator *it = gst_element_iterate_src_pads(_decoder);
GValue padV = G_VALUE_INIT;
while(gst_iterator_next(it, &padV) == GST_ITERATOR_OK){
GstPad* pad = (GstPad*)g_value_get_object(&padV);
GstCaps* caps = gst_pad_get_current_caps(pad);
if(caps_is_audio(caps)){
_decoderAudioSrcs.push_back(pad); //adding pad to a collection
gulong id = gst_pad_add_probe(pad, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, GsDecoder::pad_blocked_cb, (gpointer)this, NULL); //blocking probe
_decoderAudioProbeIds.push_back(id); //remembering id of the probe, for later
} else if (caps_is_video(caps)){
//same with video but file is guaranteed to have a single video track.
_decoderVideoSrc = pad;
_decoderVideoProbeId = gst_pad_add_probe(pad, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, GsDecoder::pad_blocked_cb, (gpointer)this, NULL);
} else {
std::cout << "UNKNOWN PAD FOUND" << std::endl;
}
gst_caps_unref(caps); //releasing caps
g_value_unset(&padV);
}
At this point I wait that all probes block the pads, and then I unblock the pads :
for (unsigned i = 0; i < _decoderAudioSrcs.size(); i++){
gst_pad_remove_probe(_decoderAudioSrcs[i], _decoderAudioProbeIds[i]);
}
gst_pad_remove_probe(_decoderVideoSrc, _decoderVideoProbeId);
play(); //setting the pipeline to play
Finally, after a bit of time, I order the pipeline to delete itself, and there I am a bit confused about what shall be deleted or not (both in this pipeline and in general).
How shall the pipeline be deleted ? From what I understood, I have to set pipeline state to Null, and then unref everything (pads, decoder, bus, pipeline) using g_object_unref, however it still leads to pipeline having few refs pointing to it (in my case, 3), and not releasing its resources, would it be memory nor threads.
Is there a way to tell Gstreamer to fully delete a pipeline and everthing related to it, without having to unref everything by hand ?
Especially, I would be interested in a pointer to some documentation, since I couldn't found any on the subject (even if Gstreamer doc mentions unrefing time to time).
GStreamer objects are reference counted. Whenever the reference drops to 0 the object gets deleted. The GStreamer documentation usually tells you for a function about its transfer behavior. E.g.
gst_pad_get_current_caps()
https://gstreamer.freedesktop.org/documentation/gstreamer/gstpad.html#gst_pad_get_current_caps
Tells you the transfer is full and you are supposed to release it when you make no more use it (you do so in your sample code). If you don't do it, it will never be deleted.
So the typical use case is if-i-don't-need-it-anymore-i-release-it. That does not mean that it gets deleted. For example, if you put an element into a pipeline, the pipeline will add a reference to it. It will take care of the reference and release it once the pipeline's reference count drops to 0. But since you added it to the pipeline and your code itself does not need it anymore you would usually just release it.
So you need to the careful about the references. If you miss one tiny small object which may be part of a big pipeline it may result in many objects to not be released as they depend on each other.
I have an issue when finalizing a video recording into an .mp4 using Media Foundation where the call to IMFSinkWriter->Finalize(); hangs forever. It doesn't always happen, and can happen on almost any machine (seen on Windows server, 7, 8, 10). Flush() is called on the audio and video streams before hand and no new samples are added between Flush and Finalize. Any ideas on what could cause Finalize to hang forever?
Things I've tried:
Logging all HRESULTs to check for any issues (was already checking them before proceeding to the next line of code)
Everything comes back as S_OK, not seeing any issues
Added the IMFSinkWriterCallback on the stream to get callbacks when
the stream process markers (adding markers every 10 samples) and finishes Finalize()
Haven't been able to reproduce since adding this but this would give the best information about what's going on when I get it working.
Searched code samples online to see how others are setting up the
Sink Writer and how Finalize() is used
Didn't find many samples and it looks like my code is similar to the ones that were found
Looked at encoders available and used by each system including version of the encoder dll
Encoders varied between AMD H.264 Hardware MFT Encoder and H264 Encoder MFT on machines that could reproduce the issue. Versions didn't seem to matter and some of the machines were up to date with video drivers.
Here are some code samples without any HRESULT checking (that doubled the amount of code so I took it out)
Building the sink sample:
CComPtr<IMFAttributes> pAttr;
::MFCreateAttributes( &pAttr, 4 );
pAttr->SetGUID( MF_TRANSCODE_CONTAINERTYPE, GetFileContainerType() );
pAttr->SetUINT32( MF_LOW_LATENCY, FALSE ); // Allows better multithreading
pAttr->SetUINT32( MF_SINK_WRITER_DISABLE_THROTTLING, TRUE ); // Does not block
pAttr->SetUINT32( MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, TRUE );
m_pCallback.Attach( new MFSinkWriterCallback() );
pAttr->SetUnknown( MF_SINK_WRITER_ASYNC_CALLBACK, m_pCallback );
::MFCreateSinkWriterFromURL( m_strFilename.c_str(), NULL, pAttr, &m_pSink );
if ( m_pVideoInputType && m_pVideoOutputType )
{
m_pSink->AddStream( m_pVideoOutputType, &m_dwVideoStreamId );
// Attributes for encoding?
CComPtr<IMFAttributes> pAttrVideo;
// Not sure if these are needed
//::MFCreateAttributes( &pAttrVideo, 5 );
m_pSink->SetInputMediaType( m_dwVideoStreamId, m_pVideoInputType, pAttrVideo );
}
if ( m_pAudioInputType && m_pAudioOutputType )
{
m_pSink->AddStream( m_pAudioOutputType, &m_dwAudioStreamId );
// Attributes for encoding?
CComPtr<IMFAttributes> pAttrAudio;
// Not sure if these are needed
//::MFCreateAttributes( &pAttrAudio, 2 );
//pAttrAudio->SetGUID( MF_MT_SUBTYPE, MFAudioFormat_AAC );
//pAttrAudio->SetUINT32( MF_MT_AUDIO_BITS_PER_SAMPLE, 16 );
m_pSink->SetInputMediaType( m_dwAudioStreamId, m_pAudioInputType, pAttrAudio );
}
m_pSink->BeginWriting();
Stopping the recording sample:
if ( m_dwVideoStreamId != (DWORD)-1 )
{
m_sink->Flush( m_dwVideoStreamId );
}
if ( m_dwAudioStreamId != (DWORD)-1 )
{
m_sink->Flush( m_dwAudioStreamId );
}
m_sink->Finalize();
There is a lot of situation where a Media Foundation application can hang :
Calling MFShutDown/CoUninitialize when using Media Foundation objects.
Using a GUI, and doing bad use of windows message pump in multithreaded application.
Bad use of MTA/STA components.
Bad use of critical section/wait for event function.
Forget to call EndXXX() function when BeginXXX() function are used.
Bad use of Callback function.
Forget to call AddRef when necessary, and releasing the object used by another thread.
A bug in Media Foundation (there are some on Windows Seven).
and so on...
When i say a minimal source code, i mean, isolate the source code that do the encoding process, and provide it to Github if it's too large. It's better if we can compile and try the source code, because deadlock are difficult to find.
I'm trying to write some code to read from an Isochronous pipe using LibUsbK in Win32. I have successfully initialised the device into the correct state to send and receive Isochronous data and I can see data being sent over the USB in my hardware USB analyser, but the buffers I am receiving are always unfilled even though the analyser shows that there was data in the packets sent to the PC.
I'm new to LibUsbK and using Isochronous transfers though I'm not new to USB in general but I've been really struggling with this.
The code I'm using to read from the device is something like this...
UsbK_SelectInterface(usbHandle,1,0);
UsbK_SetAltInterface(usbHandle,1,0,1);
IsoK_Init(&isoCtx, ISO_PACKETS_PER_XFER, 0);
IsoK_SetPackets(isoCtx, ISO_PACKET_SIZE); // Size of each individual packet
OvlK_Init(&ovlPool, usbHandle, 4, 0);
OvlK_ResetPipe(usbHandle, 0x83);
OclK_Acquire(&ovlkHandle, ovlPool);
UsbK_IsoReadPipe(usbHandle, 0x83, inBuffer, sizeof(inBuffer), ovlkHandle, isoCtx);
while(!finished)
{
if(OvlK_IsComplete(ovlkHandle)
{
fwrite(inBuffer, sizeof(inBuffer), 1, outFile);
memset(inBuffer,0xcc,sizeof(inBuffer));
OvlK_ReUse(ovlkHandle);
UsbK_IsoReadPipe(usbHandle, 0x83, inBuffer, sizeof(inBuffer), ovlkHandle, isoCtx);
{
}
If I put a breakpoint at the fwrite line then the inBuffer is always full of 0xCC - ie, not having been filled by the iso read.
I've checked all the error return values from the UsbK/OvlK function calls and they are all as they should be. I've checked my buffers are sufficiently big to receive the data.
I use very similar code to write to the ISO out pipe on endpoint 0x02 and that works perfectly, the only difference really between the code above and my write code is that the fwrite/memset commands are replaced with a call to a "fillbuffer" function that populates my outBuffer before calling UsbK_IsoWritePipe function.
I tried looking through any examples I could find in the samples and also online but struggled to understand/get them to work with my particular device.
Any suggestions or help greatly appreciated.
So it appears that the above code did work and I was being mislead by the fact that the debugger was interrupting the flow of things - I keep forgetting that trying to debug real time stuff can introduce it's own issues.
The first issue was that stepping through the code in the debugger was causing issues with the low level libusbk code capturing the usb packets and filling my buffers correctly - once I let it run full speed and found other ways to test the buffers I did actually find there was some data in there.
The second problem I had was that quite often the buffer was starting to be filled part way through only (and not always right from the start) so when I examined the data I was only printing the first part of the buffer to the console and so all I saw was 0xCC and I was therefore assuming it hadn't worked.
Once I realised that there was actually some data later in the buffer I just started looking through the buffer in packet sized chuncks, if the packet was completely contained of 0xCC I would skip it and move on, but if any of it was not 0xCC then I would treat it as a valid packet - this worked perfectly and I was successfully receiving all the data. I'm sure there's a more "proper" way of doing this, but it works for me now.
I need help making a gnuradio OOT module. I am actually trying to extending one code.
I am trying to make 2 TX 1Rx tagged stream block (OOT). For 1Tx 1Rx, it
is working fine. I am trying to extend it. Now the problem is, I am not
being able to configure the send() function. With this code, one transmitter transmits, but other does not work. The subdev specification and frequency and other parameters are allocated correctly. I checked.
If I try make test, it does not show any problem. I checked the every
port of my USRP X310, it is working fine.
Here's the code. I putting a short part which deals with the send and receive buffer.
void
usrp_echotimer_cc_impl::send()
{
// Data to USRP
num_tx_samps = d_tx_stream->send(d_in_send1, total_num_samps,
d_metadata_tx, total_num_samps/(float)d_samp_rate+d_timeout_tx);
num_tx_samps = d_tx_stream->send(d_in_send0, total_num_samps,
d_metadata_tx, total_num_samps/(float)d_samp_rate+d_timeout_tx);
}
int
usrp_echotimer_cc_impl::work (int noutput_items,
gr_vector_int &ninput_items,
gr_vector_const_void_star &input_items,
gr_vector_void_star &output_items)
{
gr_complex *in0 = (gr_complex *) input_items[0];
gr_complex *in1 = (gr_complex *) input_items[1];
gr_complex *out = (gr_complex *) output_items[0];
// Set output items on packet length
noutput_items = ninput_items[0]=ninput_items[1];
// Resize output buffer
if(d_out_buffer.size()!=noutput_items)
d_out_buffer.resize(noutput_items);
// Send thread
d_in_send0 = in0;
d_in_send1 = in1;
d_noutput_items_send = noutput_items;
d_thread_send =
gr::thread::thread(boost::bind(&usrp_echotimer_cc_impl::send, this));
// Receive thread
d_out_recv = &d_out_buffer[0];
d_noutput_items_recv = noutput_items;
d_thread_recv =
gr::thread::thread(boost::bind(&usrp_echotimer_cc_impl::receive, this));
My system config is X310, daughterboard SBX-120 and I am using UHD-3.9. I checked the subdev specification, gain and frequency assignment. Those
are fine.
For completeness:
This has been asked yesterday on the GNU Radio mailing list; Sanjoy has already gotten two responses:
Martin Braun wrote:
Sorry, Sanjoy,
we'll need some more information before we can give you better
feedback. It's not clear exactly what you're trying to achieve, and
what exactly is failing.
Perhaps this helps getting started:
http://gnuradio.org/redmine/projects/gnuradio/wiki/ReportingErrors
Cheers, Martin
And my answer is a bit longish, but it's available here. In excerpts:
Hi Sanjoy,
I am trying to make 2 TX 1Rx tagged stream block(OOT). For 1Tx 1Rx, it was working fine. I am trying to extend it. Now the problem is, I
am not being able to configure the send() function.
Is there a
particular reason you're creating your own block? Is there a feature
missing on the USRP sink and source that come with gr-uhd?
...
your send() function looks a bit strange; what you're doing is
transmitting two things on a single channel after each other. Also,
what you should be doing (from a programming style point of view) is
pass the buffers you want to send as references or something, but not
save them to class properties and then call send(). To send two
buffers to two different channels, you will need to use a vector
containing the two buffers -- have a look at rx_multi_samples; that of
course recv()s rather than sends(), but the semantics are the same.
...
noutput_items is given to you to let your work now how much it may produce, that's why it's a parameter.
...
There's no reason GNU Radio couldn't already call your work function again while usrp_echotimer_cc_impl::send() hasn't even started to transmit samples. Then, your d_send variables will be overwritten.
...
Since a single X310 is inherently coherent and has the same time, you can simply use a USRP sink and a source, use set_start_time(...) on both with the same time spec, and have your flow graph consume and produce samples in different threads, coherently.
I have a problem with Microsoft's WaveOut API:
edit1: Added Link to sample project:
edit2: removed link, its not representative of the issue
After playing some audio, when I want to terminate a given playback stream, I call the function:
waveOutClose(hWaveOut_);
However, even after waveOutClose() is called, sometimes the library will still access memory previously passed to it by waveOutWrite(), causing an invalid memory access.
I then tried to ensure all the buffers are marked as done before freeing the buffer:
PcmPlayback::~PcmPlayback()
{
if(hWaveOut_ == nullptr)
return;
waveOutReset(hWaveOut_); // infinite-loops, never returns
for(auto it = buffers_.begin(); it != buffers_.end(); ++it)
waveOutUnprepareHeader(hWaveOut_, &it->wavehdr_, sizeof(WAVEHDR));
while( buffers_.empty() == false ) // infinite loops
removeCompletedBuffers();
waveOutClose(hWaveOut_);
//Unhandled exception at 0x75629E80 (msvcrt.dll) in app.exe:
// 0xC0000005: Access violation reading location 0xFEEEFEEE.
}
void PcmPlayback::removeCompletedBuffers()
{
for(auto it = buffers_.begin(); it != buffers_.end();)
{
if( it->wavehdr_.dwFlags & WHDR_DONE )
{
waveOutUnprepareHeader(hWaveOut_, &it->wavehdr_, sizeof(WAVEHDR));
it = buffers_.erase(it);
}
else
++it;
}
}
However, this situation never happens - the buffer never becomes empty. There will be 4-5 blocks remaining with wavehdr_.dwFlags == 18 (I believe this means the blocks are still marked as in playback)
How can I resolve this issue?
# Martin Schlott ("Can you provide the loop where you write the buffer to waveOutWrite?")
Its not quite a loop, instead I have a function that is called whenever I receive an audio packet over the network:
void PcmPlayback::addData(const std::vector<short> &rhs)
{
removeCompletedBuffers();
if(rhs.empty())
return;
// add new data
buffers_.push_back(Buffer());
Buffer & buffer = buffers_.back();
buffer.data_ = rhs;
ZeroMemory(&buffers_.back().wavehdr_, sizeof(WAVEHDR));
buffer.wavehdr_.dwBufferLength = buffer.data_.size() * sizeof(short);
buffer.wavehdr_.lpData = (char *)(buffer.data_.data());
waveOutPrepareHeader(hWaveOut_, &buffer.wavehdr_, sizeof(WAVEHDR)); // prepare block for playback
waveOutWrite(hWaveOut_, &buffer.wavehdr_, sizeof(WAVEHDR));
}
The described behavior can happen if you do not call
waveOutUnprepareHeader
to every buffer you used before you use
waveOutClose
The flagfield _dwFlags seems to indicate that the buffers are still enqueued (WHDR_INQUEUE | WHDR_PREPARED) try:
waveOutReset
before unprepare buffers.
After analyses your code, I found two problems/bugs which are not related to waveOut (funny, you use C++11 but the oldest media interface). You use a vector as buffer. During some calling operations, the vector is copied! One bug I found is:
typedef std::function<void(std::vector<short>)> CALLBACK_FN;
instead of:
typedef std::function<void(std::vector<short>&)> CALLBACK_FN;
which forces a copy of the vector.
Try to avoid using vectors if you expect to use it mostly as rawbuffer. Better use std::unique_pointer as buffer pointer.
Your callback in the recorder is not monitored by a mutex, nor does it check if a destructor was already called. The destructing happens during the callback (mostly) which leads to an exception.
For your test program, go back and use raw pointer and static callbacks before blaming waveOut. Your code is not bad, but the first bug already shows, that a small bug will lead to unpredictical errors. As you also organize your buffers in a std::array, I would search for bugs there. I guess, you make a unintentional copy of your whole buffer array, unpreparing the wrong buffers.
I did not have the time to dig deeper, but I guess those are the problems.
I managed to find my problem in the end, it was caused by multiple bugs and a deadlock. I will document what happened here so people can learn from this in the future
I was clued in to what was happening when I fixed the bugs in the sample:
call waveInStop() before waveInClose() in ~Recorder.cpp
wait for all buffers to have the WHDR_DONE flag before calling waveOutClose() in ~PcmPlayback.
After doing this, the sample worked fine and did not display the behavior of the WHDR_DONE flag never being marked.
In my main program, that behavior was caused by a deadlock that occurs in the following situation:
I have a vector of objects representing each peer I am streaming audio with
Each Object owns a Playback class
This vector is protected by a mutex
Recorder callback:
mutex.lock()
send audio packet to each peer.
Remove Peer:
mutex.lock()
~PcmPlayback
wait for WHDR_DONE flags to be marked
A deadlock occurs when I remove a peer, locking the mutex and the recorder callback tries to acquire a lock too.
Note that this will happen often because the playback buffer is usually (~4 * 20ms) while the recorder has a cadence of 20ms.
In ~PcmPlayback, the buffers will never be marked as WHDR_DONE and any calls to the WaveOut API will never return because the WaveOut API is waiting for the Recorder callback to complete, which is in turn waiting on mutex.lock(), causing a deadlock.