I have a requirement in which I have to maintain 5 independent pipelines simultaneously.
These pipelines are similar and look like this:
udpsrc -> rtppcmadepay -> alawdec -> audioconvert -> audioresample -> voamrwbenc -> rtpamrpay -> udpsink
I have a few questions regarding that:
1, When I am sending one stream (initiate one pipeline), The audio from udpsink is very clear after encoding and decoding. But, as soon as 2 streams are originated (initiated 2 streams), a lot of noise and Jitter is introduced.
Question: Are these 5 pipelines accessing the same udpsrc or other elements (even though I have created them each time for every pipeline), and thus introduction the jitter or noise?
2, In such a encoding-decoding call, may you please tell me the amount of CPU that this is suppose to take? In my case, it varies from 4-7% in one call (input streams) and nearly 50-80% for 5 calls (input streams).
Is this behavior normal?
Shouldn't the increase in CPU be linear as each pipeline is expected to take the same amount of CPU provided? The elements in each of them are same and the work for them is also same.
Thanks for the help in advance and pardon for grammar.
Related
I have a GStreamer pipeline I'm instantiating on two time-synchronized computers (running GStreamer 1.4.5) as follows:
On the data-generating computer:
gst-launch-1.0 audiotestsrc samplersperbuffer=<samps_per_buf> ! alwaenc ! inserttimecode consumer=0 ! udpsink port=5002 host=<ip address of data-consuming computer>
On the data-consuming computer:
gst-launch-1.0 udpsrc caps="audio/x-alaw,rate=(int)44100,channels=1" port=5002 ! inserttimecode consumer=1 ! alawdec ! alasasink
inserttimecode is an instance of GstInsertTimeCode, a custom plugin subclass of GstTransform that either
consumer=0: copies the incoming data and then adds 20 bytes (custom marker, current time in nanoseconds, and a 1-up sequence number) to the packet and then sends the data downstream
OR
consumer=1: Strips out the 20 bytes and then sends the rest of the data downstream. Reports jumps in sequence number and also reports latency using the consumed 20 bytes and the current clock time.
I would have expected inserttimecode to add the timestamp, pass it to udpsink, then udpsrc would receive it and pass it on directly to inserttimecode. In my mind, there should be very little delaying this transfer: just network delay plus a small processing delay. However, when I run this code, my latency numbers are larger than expected. I have found empirically that the unexpected offset grows over time but levels out (after just a handful of timestamps) at a value that can be calculated as
offset_seconds = samps_per_buf / sample_rate
In other words, the larger I set samps_per_buf in the audiotestsrc call, the larger the computed latency. Why would this be?
On the receiving side, is it back pressure from alsasink?
On the sending side, is inserttimecode clocked to send out packets after each block is sent from audiotestsrc?
SIDE NOTE: I suppose I'm not super surprised by this. In the parlance of signal processing and filtering, I would call this "filter delay". I am used to seeing a delay induced by a filter (often by half the length of the filter) but if I just use that paradigm, I think the delay is half as large as I would expect.
I want to understand the origin of offset_seconds.
It seems like it could be a function of clocking of the audiotestsrc. I'm still trying to understand how that works (see Different scheduling modes). I assume the pads of GstInsertTimeCode are operating in push mode (means it's waiting for the upstream elements to push data to it). I am unsure if this default changes if my transform is not "in-place" (it isn't since I'm adding extra data to the stream). I'm not sure this has anything to do with it since I would expect the offset to be constant instead of ramping up to a steady-state value. But if it is, how do I change a GstTransform object to go to pull mode? Or would setting pull mode in udpsink do the trick? Does the sync property with udpsink have anything to do with it?
I've also looked at the possibility that this is latency reported to the elements during setup. However, I have found (using GST_DEBUG=6 on the command line) that this latency is not variant with samps_per_buf and is always 0.2 seconds. (For more information see Clocking)
Since the value is ramping up, it feels like buffering happening somewhere in the pipeline but I cannot figure out where.
Why is the delay in my GStreamer pipeline dependent on the blocksize of audiotestsrc?
Are there certain debug outputs I could search for to shed light on the delay?
Where would I look to understand more about delays induced by elements in the pipeline?
I have a two GStreamer pipelines, one is like a "source" pipeline streaming a live camera feed into an external channel, and the second pipeline is like a "sink" pipeline that reads from the other end of that channel and outputs the live video to some form of sink.
[videotestsrc] -> [appsink] ----- Serial Channel ------> [appsrc] -> [autovideosink]
First Pipeline Second Pipeline
The first pipeline starts from a videotestsrc, encodes the video and wraps it in gdppay payload, and then sinks the pipeline into a serial channel (but for the sake of the question, any sink that can be read from to start another pipeline like a filesink writing to serial port or udpsink), where it is read by the source of the next pipeline and shown via a autovideosrc:
"Source" Pipeline
gst-launch-1.0 -v videotestsrc ! videoconvert ! video/x-raw,format=I420 ! x265enc ! gdppay ! udpsink host=127.0.0.1 port=5004
"Sink" pipeline
gst-launch-1.0 -v udpsrc uri=udp://127.0.0.1:5004 ! gdpdepay ! h265parse ! avdec_h265 ! autovideosink
Note: Given the latency induced using a udpsink/udpsrc, that pipeline complains about timestamp issues. If you replace the udpsrc/udpsink with a filesrc/filesink to a serial port you can see the problem that I am about to describe.
Problem:
Now that I have described the pipelines, here is the problem:
If I start both pipelines, everything works as expected. However, if after 30s, I stop the "source" pipeline, and restart the pipeline, the Running Time gets reset back to zero, causing the timestamps of all buffers to be sent to be considered old buffers by the sink pipeline because it has already received buffers for timestamps 0 through 30s, so the playback on the other end won't resume until after 30s:
Source Pipeline: [28][29][30][0 ][1 ][2 ][3 ]...[29][30][31]
Sink Pipeline: [28][29][30][30][30][30][30]...[30][30][31]
________________________^
Source pipeline restarted
^^^^^^^^^^^^^^^^...^^^^^^^^
Sink pipeline will continue
to only show the "frame"
received at 30s until a
"newer" frame is sent, when
in reality each sent frame
is newer and should be shown
immediately.
Solution
I have found that adding sync=false to the autovideosink does solve the problem, however I was hoping to find a solution where the source would send its timestamps (DTS and PTS) based on the Clock time as seen in the image on that page.
I have seen this post and experimented with is-live and do-timestamp on my video source, but they do not seem to do what I want. I also tried to manually set the timestamps (DTS, PTS) in the buffers based on system time, however to no avail.
Any suggestions?
I think you should just restart the receiver pipeline as well. You could add the -e switch to the sender pipeline and when you stop the pipeline it should correctly propagate EOS via the GDP element to the receiver pipeline. Else I guess you can send a new segment or discontinuity to the receiver. Some event has to be signaled though to make the pipeline aware of that change, else it is somewhat bogus data. I'd say restarting the receiver is the simplest way.
I am currently playing around with an AudioOverIP Project and wondered if you could help me out.
I have a LAN, with an Audio Source (Dante/AES67-RTP-Stream) which I would like to distribute to multiple receivers (SBC (e.g. RaspberryPi) with an Audio Output (e.g. Headphone jack):
PC-->Audio-USB-Dongle-->AES67/RTP-Multicast-Stream-->LAN-Network-Switch-->RPI (Gstreamer --> AudioJack)
I currently use Gstreamer for the Pipeline:
gst-launch-1.0 -v udpsrc uri=udp://239.69.xxx.xx:5004 caps="application/x-rtp,channels=(int)2,format=(string)S16LE,media=(string)audio,payload=(int)96,clock-rate=(int)48000,encoding-name=(string)L24" ! rtpL24depay ! audioconvert ! alsasink device=hw:0,0
It all works fine, but if I watch a video on the PC and listen to the Audio from the RPI, I have some latency (~200-300ms), therefore my questions:
Do I miss something in my Gstreamer Pipeline to be able to reduce latency?
What is the minimal Latency to be expected with RTP-Streams, is <50ms achievable?
Would the latency occur due to the network or due to the speed of the RPi?
Since my audio-input is not a Gstreamer input, I assume rtpjitterbuffer or similar would not help to decrease latency?
How do I put unrelated audio into any generated video stream in a way that keeps them in sync in gstreamer?
Context:
I want to stream audio from icecast into a Kinesis Video stream, and then view it with Amazon's player. The player only works if there is video as well as audio, so I generate video with testvideosrc.
The video and audio need to be in sync in terms of timestamps, or the Kinesis sink 'kvssink' throws an error. But because they are two separate sources, they are not in sink.
I am using gst-launch-1.0 to run my pipeline.
My basic attempt was like this:
gst-launch-1.0 -v \
videotestsrc pattern=red ! video/x-raw,framerate=25/1 ! videoconvert ! x264enc ! h264parse ! video/x-h264,stream-format=avc,alignment=au ! \
queue ! kvssink name=sink stream-name="NAME" access-key="KEY" secret-key="S_KEY" \
uridecodebin uri=http://ice-the.musicradio.com/LBCLondon ! audioconvert ! voaacenc ! aacparse ! queue ! sink.
The error message I get translates to:
STATUS_MAX_FRAME_TIMESTAMP_DELTA_BETWEEN_TRACKS_EXCEEDED
This indicates that the audio and video timestamps are too different, so I want to force them to match, maybe by throwing away the video timestamps?
There are different meanings of "sync". Let us ignore lip sync for a moment (where audio and video match to each other).
There is sync in terms of timestamps - e.g. do they carry similar timestamps in their representation. And sync in terms of when in real time do these samples with timestamps actually arrive at the sink (latency).
Hard to tell by the error which one exactly the sink is complaining about.
Maybe try x264enc tune=zerolatency for a start, as without that options the encoder produces a two second latency which may cause issues for certain requirements.
Then again the audio stream will have some latency too. It may not be easy to tune these two to match. The sink should actually do the buffering ans synchronization.
I have this pipeline :
gst-launch -v filesrc location=video.mkv ! matroskademux name=d \
d. ! queue ! ffdec_h264 ! subtitleoverlay name=overlay ! ffmpegcolorspace ! x264enc ! mux. \
d. ! queue ! aacparse ! mux. \
filesrc location=fr.srt ! subparse ! overlay. \
matroskamux name=mux ! filesink location=vid.mkv
I'm trying to burn the subtitles to the video. I have succdeded to read the file with the subtitles but the above pipeline stuck and I have this message :
queue_dataflow gstqueue.c:1243:gst_queue_loop:<queue0> queue is empty
What's wrong with my pipeline? What the queue element do? I haven't really understood what it said in the doc.
The queue element adds a thread boundary to the pipeline and support for buffering. The input side will put buffers into a queue, which is then emptied on the output side from another thread. Via properties on the queue element you can set the size of the queue and some other things.
I don't see anything specifically wrong with your pipeline, but the message there tells you that at some point one of the queues is empty. Which might be a problem or not. It might become fuller again later.
You'll have to check the GStreamer debug logs to see if there's anything in there that hints at the actual problem. My best guess here would be that the audio queue running full because of the encoder latency of x264enc. Try making the audio queue larger, or set tune=zerolatency on x264enc.
Also I see that you're using GStreamer 0.10. It is no longer maintained since more than two years and for new applications you should really consider upgrading to the 1.x versions.
A queue is the thread boundary element through which you can force the use of threads. It does so by using a classic provider/consumer model as learned in threading classes at universities all around the world. By doing this, it acts both as a means to make data throughput between threads threadsafe, and it can also act as a buffer. Queues have several GObject properties to be configured for specific uses. For example, you can set lower and upper thresholds for the element. If there's less data than the lower threshold (default: disabled), it will block output. If there's more data than the upper threshold, it will block input or (if configured to do so) drop data.
To use a queue (and therefore force the use of two distinct threads in the pipeline), one can simply create a “queue” element and put this in as part of the pipeline. GStreamer will take care of all threading details internally.