Monitoring audio recorded via QMediaCaptureSession
Good afternoon.
When using QMediaCaptureSession, QMediaRecorder to capture audio from a microphone to a file, a problem occurs:
I start recording through the record() method of the QMediaRecorder class. Next, I receive a QMediaRecorder::recorderStateChanged signal with a QMediaRecorder::RecordingState value, and display in the window an inscription that recording has begun. The problem is that the resulting recording starts with some delay, and not at the moment the QMediaRecorder::RecordingState signal is received, i.e. the recording does not include sound or words that were spoken IMMEDIATELY after receiving the QMediaRecorder::RecordingState signal for 1-2 seconds.
However, the Qt documentation has this statement:
https://doc.qt.io/qt-6/qtmultimedia-changes-qt6.html
New features in Qt 6
...
You can now also monitor the audio recorded by a capture session.
...
Questions:
Can I somehow analyze the data that is going to be recorded and understand that the data has not yet begun to arrive and it is not yet necessary to display a message in the program window that the recording has begun?
What does the Qt documentation mean by saying that you can monitor audio recorded via a capture session?
I did not find information on this issue.
Related
After pausing icecast radio player app with just audio and audio service packages from Ryan Heise, I wait a little and tap play again, the stream continues to play from saved position with data in buffers, then suddenly (I think when buffers are empty) changes to live position in stream. It is possible for me to clear buffers on pause/stop in my Audio Handler? Or should I use another approach?
I tried player.seek(null), but it needed more work, so I decided to not use it. We haven't encountered this behaviour anymore since a month or so. It works fine now. Thanks!
I'm planning to write a small program that connects to the Jack Audio Kit on Linux. Whenever there's a new audio available to my program, Jack will call the registered callback. According to the documentation, the callback function should enable "real-time execution".
However, I'd like my program to do some analysis over the data flowing through and displaying this in a window. This is inherently not real-time.
To circumvent this issue, in an exploratory Python script, I used a queue to transfer data into another thread. But although the callback is called, the thread doesn't receive anything. I suspect this is due to the callback being called in another process. The debugger also doesn't stop inside the callback.
I saw, there are several applications for Jack that do other non-real-time things like UI and thelike (e.g., volume meter in Cadence). How is it possible to implement this callback without losing the real-time execution property?
I have my own MediaSink in Windows Media Foundation with one stream. In the OnClockStart method, I instruct the stream to queue (i) MEStreamStarted and (ii) MEStreamSinkRequestSample on itself. For implementing the queue, I use the IMFMediaEventQueue, and using the mtrace tool, I can also see that someone dequeues the event.
The problem is that ProcessSample of my stream is actually never called. This also has the effect that no further samples are requested, because this is done after processing a sample like in https://github.com/Microsoft/Windows-classic-samples/tree/master/Samples/DX11VideoRenderer.
Is the described approach the right way to implement the sink? If not, what would be the right way? If so, where could I search for the problem?
Some background info: The sink is an RTSP sink based on live555. Since the latter is also sink-driven, I thought it would be a good idea queuing a MEStreamSinkRequestSample whenever live555 requests more data from me. This is working as intended.
However, the solution has the problem that new samples are only requested as long as a client is connected to live555. If I now add a tee before the sink, eg to show a local preview, the system gets out of control, because the tee accumulates samples on the output of my sink which are never fetched. I then started playing around with discardable samples (cf. https://social.msdn.microsoft.com/Forums/sharepoint/en-US/5065a7cd-3c63-43e8-8f70-be777c89b38e/mixing-rate-sink-and-rateless-sink-on-a-tee-node?forum=mediafoundationdevelopment), but the problem is either that the stream does not start, queues are growing or the frame rate of the faster sink is artificially limited depending on which side is discardable.
Therefore, the next idea was rewriting my sink such that it always requests a new sample when it has processed the current one and puts all samples in a ring buffer for live555 such that whenever clients are connected, they can retrieve their data from there, and otherwise, the samples are just discarded. This does not work at all. Now, my sink does not get anything even without the tee.
The observation is: if I just request a lot of samples (as in the original approach), at some point, I get data. However, if I request only one (I also tried moderately larger numbers up to 5), ProcessSample is just not called, so no subsequent requests can be generated. I send MeStreamStarted once the clock is started or restarted exactly as described on https://msdn.microsoft.com/en-us/library/windows/desktop/ms701626, and after that, I request the first sample. In my understanding, MEStreamSinkRequestSample should not get lost, so I should get something even on a single request. Is that a misunderstanding? Should I request until I get something?
I had no experience with Controller Area Networks(CAN) or ValueCAN3 prior to this project, and I used an example from Intrepid for my reading of messages. However I am having issues with efficiency and frequency of the updates for my GUI which displays the analog and digital signals I am reading.
My GUI consists of 16 numeric up/down boxes for analog channels and 36 buttons that change to green depending if a digital signal is turned on (1) or off(0). While reading in my CAN messages I then update the GUI controls to display the appropriate feedback. However the digital channels respond almost instantly when I press a button on a CAN joystick that is plugged in, whereas the analog signals don't update that fast with the string pots I am using to vary the signal. Sometimes it takes an analog signal 1 - 2 seconds to respond.
Currently I set the GUI controls, and then I read the values from the GUI controls and send the values out over a socket connection to another application via UDP. I should probably change this to sending the data from the signals I receive directly rather then reading from the GUI controls I am setting, but I don't think that is the issue.
I am using System::Timers::Timer objects to update, read messages, and send out data packets. I need a rate of 50hz - 100hz, preferably closer to 100hz. Using the socket on the other end I can see that my packets are sent frequently enough, but the data doesn't change smoothly or frequently for the analog channels. If anyone has any thoughts on what I might be doing wrong, or how to process the data in a more efficient way please state it.
Here is the segment of code from Intrepid that reads the CAN messages:
// Read the messages every timer event (1000 ms)
if (m_bPortOpen) // only if the port is open
{
// call icsneoGetMessages to read out the messages
lResult = icsneoGetMessages(hObject,stMessages,&lNumberOfMessages,&lNumberOfErrors);
if (lResult != 0)
{
// a successful read
mNumberOfErrorsRead = lNumberOfErrors;
mNumberOfMessagesRead = lNumberOfMessages; // store the number of messages in the current buffer
}
}
My form requests a msg from my CanReader Object using:
msg = can->GetLatestMsg();
and that method grabs the last message received.
public: icsSpyMessage* GetLatestMsg()
{
return &stMessages[mNumberOfMessagesRead - 1];
};
I think this GetLatestMsg() seems like a bad way to implement the retrival of the latest message, but I'm not entirely sure how much this is affecting my program or how else I could do this because the CanReader is seperate from the Form so I'd have to pass an array of messages I think otherwise. I do suspect this might be skipping messages because it only reads the last one grabbed and not the ones leading up to it, which if those were read should make the GUI output appear smoother for transitions.
Another thing to note is that I am reading from 6 different PGNs, the analog signals correspond to 4 of the PGNs and 2 correspond to the digital signals.
UPDATE
After playing with my application and using the string pots on different analog channels I am noticing that some channels are updating more then others. And by checking the PGNs being accessed I find that I am accessing some more frequently then others.
Doesn't a CAN device broadcast the data at relatively the same rates for the different PGNs? and if yes then my GetLatestMsg() method must not be reading the different PGNs effectively. It reads a new msg every 5 milliseconds.
Additionally, does anyone know if I should make seperate reading timers to detect the different PGNs seperately?
If there is additional code I can provide for clarity please let me know.
After lots of testing and debugging I have found a stable solution. I am posting this answer in hopes that someone else may benefit from this explanation.
When reading message on a CANbus the messages are broadcasting across the bus and you have to pick out the messages you want based on the parameter group names (PGNs). Each PGN should be defined for different pieces of data.
For instance engine rpm would correspond to a particular PGN, and when you check the messages in your buffer, you will be looking for the messages with the PGN corresponding to the engine rpm.
In my case, I am reading/wanting 6 different PGNs from the valueCAN3 device I am using. Each of these messages contains different data that corresponds to analog and digital signals. The error with efficiency was related to how I was retreiving the latest message from the buffer.
When you are just streaming all the same data in each packet it makes sense to just grab the latest or most recent message from your buffer. However, in my case since I require 6 different PGNs that means I when I grab the last message off of the buffer it corresponds to just 1 of out the 6 PGNs, and since the messages on my CANbus are sent out at a constant rate and in the same order, I was reading about half of the packets all the time, and missing the other half. By grabbing the 2 most recent packets I found that all my channels are now updating at the speed needed for my application. All my signals change without noticable lag, and are easily meeting my 50hz - 100hz range that I required.
However, I think the optimal solution would be to read the latest message for each PGN when I pull from the buffer, rather then just the latest two messages, which could correspond to any of my 6 PGNs because I read more often then I receive the packets.
Does anybody know any reasons why IMemAllocator:GetBuffer (Directshow) hangs, apart from all samples being in use?
I have a directshow application which uses GMFBridge by Geraint Davies to connect two graphs. The GMFBridge is used to be able to switch inputs, but I am not switching in this situation. The application captures audio and video, and should do that non-stop. But after about 10 hours it stops. I found out both audio and video got stuck in a call to IMemAllocator:GetBuffer:
/* _COM_SMARTPTR_TYPEDEF(IMemAllocator, IID_IMemAllocator); */
/* IMemAllocatorPtr m_pCopyAllocator; */
hr = m_pCopyAllocator->GetBuffer(&pOut, NULL, NULL, 0);
If all samples are in use, this function can block, but I am pretty sure this is not the case. There are two threads calling this function, one for the video and one for the audio samples. The Audio thread blocks first, and after the GetBuffer has returned a buffer for almost 60 video samples, the video thread blocks too. (this is about 2 seconds later)
After almost 8 hours both threads continue for a small period, first the audio thread, and after 45 buffers for audio samples have been returned, the video thread unblocks too.
So because both threads do not block at the same time, it looks to me there is not a problem with all samples being in use.
The stacktrace shows a function inside quartz.dll is being called at that moment.
UPDATE
It looks like there was a memoryleak caused by decoder filters already installed on the pc. The graph included decoding of mpeg, for example the audio decoding used a cyberlink decoder. After installing ffdshow, the ffdshow audio + video decoder was used instead, and the problem seems to be disappeared. Lesson learned, do not depend automatically on existing filters.
Not sure that I can debug this from the info given. Can you create a log file (create an empty file c:\gmfbridge.txt, run until it hangs, then zip the file and email it). Also, if you set up your symbols with _NT_SYMBOL_PATH, you could look at the stack trace to see where in quartz.dll the various threads are.
G