My pipeline splits in the middle to be sent over an unreliable connection. This results in some buffers having bit errors that break the pipeline if I do not account for them. To solve this, I have an appsink that parses buffers for their critical information (timestamps, duration, data, and data size), serializes them, and then sends that over the unreliable channel with a CRC. If the receiving pipeline reads a buffer from the unreliable channel and detect a bit error with the CRC, the buffer is dropped. Most decoders are able to recover fine from a dropped buffer, aside from some temporary visual artifacts.
Is there a GStreamer plugin that does this automatically? I looked into the GDPPay and GDPDepay plugins which appeared to meet my needs due to there serialization of buffers and inclusion of CRC's for their header and payload, however the plugin assumes that the data is being sent over a reliable channel (why this assumption and the inclusion of CRCs, I do not know).
I am tempted to take the time to make a plugin/make a pull request to the GDP plugins that just drop bad buffers instead of pausing the pipeline with a GST_FLOW_ERROR.
Any suggestions would be greatly appreciated. Ideally it would also be tolerant to either pipeline crashing/restarting. (The plugin also expects the Caps filter information to be the first buffer sent, which in my case I do not need to send as I have a fixed purpose and can hard-code both ends to know what to expect. This is only a problem if the receiver restarts and the sender is already sending data, but the receiver will not get the data because it is waiting for the Caps data that the sender already sent.)
When faced with similar issue (but for GstEvents), I used GstProbe. You'll probably need to install it for GST_PAD_PROBE_TYPE_BUFFER and return GST_PAD_PROBE_DROP for the buffers that doesn't satisfy your conditions. It is easier than defining a plugin and definitely it is easier to modify (GstProbe can be created and handled from the code, so changing the dropping logic is easier). Caveat: I haven't done it for the buffers, but it should be doable.
Let me know if it worked!
Related
I have tried to use [transcirbe_steaming_infinite.py] module with multiple mics. The first one is equipped on my pc(mac book pro) and the other one is external one (Jabra EVOLVE 20). Through Audio MIDI setup I made an aggregate device option (Jabra for channel #1, mac for #2).
To use these mics I modified the codes like ResumableMicrophoneStream._num_channels as 2 and added two extra lines after RecognitionConfig audio_channel_count=2 and enable_seperate_recognition_per_channel=True. And the language in ja-JP
When I tried to use these codes at least work (they are able to recognize each channels) but the problem is that in a certain case, responses comes too late.
The case is when I switch the mic from one of each to the other one. For example, when I try to use mic on channel #1(Jabra) right after using the mic on channel #2, I cannot get the response in time but about 15000ms later.
When I checked the mics on Audio MIDI setup those two's sample rate was different(16kHz, 44.1kHz per each), so I thought up with a possibility it has affected on the library processing audio input streams like PyAudio and finally it has caused late request and response as well. It will be dummiest hypothesis XD.
So I want to know, as the title this problem(late response) can be fixed with good mics setup or just there is another good problem solving way for this case.
A common cause of latency is the API not detecting the end of the audio and therefore it will continue to listen and process audio until either the stream is closed directly, or the stream's limit length has been exceeded. You can discard it using single_utterance which indicates whether the request should automatically end after speech is no longer detected. Also if you are using noise filtering, that should be removed so that Cloud sees the raw audio and can properly detect isFinal.
If the latency issues only occurs when mics are changed and you are following the best practices, you can reach the STT team through the public issue tracker
I have my own MediaSink in Windows Media Foundation with one stream. In the OnClockStart method, I instruct the stream to queue (i) MEStreamStarted and (ii) MEStreamSinkRequestSample on itself. For implementing the queue, I use the IMFMediaEventQueue, and using the mtrace tool, I can also see that someone dequeues the event.
The problem is that ProcessSample of my stream is actually never called. This also has the effect that no further samples are requested, because this is done after processing a sample like in https://github.com/Microsoft/Windows-classic-samples/tree/master/Samples/DX11VideoRenderer.
Is the described approach the right way to implement the sink? If not, what would be the right way? If so, where could I search for the problem?
Some background info: The sink is an RTSP sink based on live555. Since the latter is also sink-driven, I thought it would be a good idea queuing a MEStreamSinkRequestSample whenever live555 requests more data from me. This is working as intended.
However, the solution has the problem that new samples are only requested as long as a client is connected to live555. If I now add a tee before the sink, eg to show a local preview, the system gets out of control, because the tee accumulates samples on the output of my sink which are never fetched. I then started playing around with discardable samples (cf. https://social.msdn.microsoft.com/Forums/sharepoint/en-US/5065a7cd-3c63-43e8-8f70-be777c89b38e/mixing-rate-sink-and-rateless-sink-on-a-tee-node?forum=mediafoundationdevelopment), but the problem is either that the stream does not start, queues are growing or the frame rate of the faster sink is artificially limited depending on which side is discardable.
Therefore, the next idea was rewriting my sink such that it always requests a new sample when it has processed the current one and puts all samples in a ring buffer for live555 such that whenever clients are connected, they can retrieve their data from there, and otherwise, the samples are just discarded. This does not work at all. Now, my sink does not get anything even without the tee.
The observation is: if I just request a lot of samples (as in the original approach), at some point, I get data. However, if I request only one (I also tried moderately larger numbers up to 5), ProcessSample is just not called, so no subsequent requests can be generated. I send MeStreamStarted once the clock is started or restarted exactly as described on https://msdn.microsoft.com/en-us/library/windows/desktop/ms701626, and after that, I request the first sample. In my understanding, MEStreamSinkRequestSample should not get lost, so I should get something even on a single request. Is that a misunderstanding? Should I request until I get something?
We're working with a C++ webrtc data channels library and in our test application, upon sending a few small packets that would totally amount to about 256kB, the usrsctp_sendv() call returns -1 (with errno as EWOULDBLOCK/EAGAIN which means "Resource is temporarily unavailable"). We believe this is because we're hitting the usrsctp's send buffer limit, which is 256 kB by default. We've tried adding several sleep delays in between each send call hoping it clears that buffer, but nothing works.
The receiving side, (a JS web page) does indeed receive all the bytes that we've sent up until it errors out. It's also worth noting that this only happens when we try to send data from the C++ application to the JS and not the other way around. We tried looking around mozilla's datachannels implementation, but can't seem to draw any conclusions on what the issue could be about.
It is hard to answer such question straight away. I would start looking into wireshark traces in order to see if your remote side (JS page) actually acknowledges data you send (e.i. if SACK chunks are sent back) and what is the value of received buffer (a_rwnd) reported in these SACKs. It might be possible that it is not an issue on your side, but you are getting EWOULDBLOCKS just because sending side SCTP cannot flush the data from buffers because it is still awaiting for delivery confirmation from remote end.
Please provide more details about your case, also if this is possible provide sample code for your JS page.
I'm working for a few days now on a pipeline with the following configuration :
- 2 live input streams (RTMP)
- going into one compositor
- outputing to another RTMP stream
With some converter, queue, etc. in between, it works pretty well.
But my problem is that one of the RTMP input may not be available at start time, so the pipeline can't start, crashing with the followings errors:
- error: Failed to read any data from stream
- error: Internal data flow error
What would be the proper way to make this work, that is, to start the stream with the first input, even if the second one is not ready yet ?
I tried several ways : dynamically changing the pipeline, playing with pad probes, listening to error message, .. but so far I can't make it work.
Thanks,
PL
As you didnt post any code I guess you are ok with conceptual answer..
There are few options for rtspsrc with which you can control when it will fail - reagarding timeout exceeded or number of trials exceeded maximum. Those are (not sure if all):
retry - this may be not very useful if it deals only with ports ..
timeout - if you want to try with UDP some longer time you can enlarge this one
tcp-timeout - this is important, try to play with it - make it much larger
connection-speed - maybe it will help to make smaller this one
protocols - I have experience that for bad streams TCP was much better for me
The actual concept (I am not an expert, take it as another view to the problem):
You can create two bins - one for each stream. I would use rtspsrc and decodebin and block the output pads of decodebin untill I have all the pads, then I would connect to the compositor.
When you recieve any error (it should be during the phase of waiting for all pads) then you would put the bin to NULL state (I mean GStreamer state called NULL) and PLAYING/PAUSED again..
Well you have to use the pad probles properly (no idea what that is :D) .. can you post your code regarding this?
Maybe try to discard the error message to not disintegrate the pipe..
Also, do you have only video inputs?
I guess no, you can use audiomixer for audio.. also the compositor has nice OpenGL version which is much faster its called glvideomixer.. but it may introduce another OpenGL related problems.. if you have intel GPUs then you are probably safe.
I’m working on an embedded application, where i receive some sensor values over UDP. The board I’m using runs the 2.4 kernel on an ARM processor. The problem is the following: once my internal socket buffer is full only the newest value gets replaced. So the internal buffer is not implemented as a circular buffer, which it should be, as i found out studying some articles. Can i somehow change the behaviour of the internal receive buffer?
I already found out that there is no way to "flush" that buffer from the application side. The best idea I’ve got is checking whether the receive buffer is full, before receiving any packets and if so fist read out all the old packets manually. Is there any better approach?
I hope it's somehow clear what I mean, any help is appreciated.
The best idea I’ve got is checking whether the receive buffer is full,
before receiving any packets and if so fist read out all the old
packets manually.
I'd not bother checking whether the receive buffer is full, rather always read all packets until no more are there and use the last received, which contains the newest value.