Use "Clock Time" instead of Running Time for GStreamer Pipeline - gstreamer

I have a two GStreamer pipelines, one is like a "source" pipeline streaming a live camera feed into an external channel, and the second pipeline is like a "sink" pipeline that reads from the other end of that channel and outputs the live video to some form of sink.
[videotestsrc] -> [appsink] ----- Serial Channel ------> [appsrc] -> [autovideosink]
First Pipeline Second Pipeline
The first pipeline starts from a videotestsrc, encodes the video and wraps it in gdppay payload, and then sinks the pipeline into a serial channel (but for the sake of the question, any sink that can be read from to start another pipeline like a filesink writing to serial port or udpsink), where it is read by the source of the next pipeline and shown via a autovideosrc:
"Source" Pipeline
gst-launch-1.0 -v videotestsrc ! videoconvert ! video/x-raw,format=I420 ! x265enc ! gdppay ! udpsink host=127.0.0.1 port=5004
"Sink" pipeline
gst-launch-1.0 -v udpsrc uri=udp://127.0.0.1:5004 ! gdpdepay ! h265parse ! avdec_h265 ! autovideosink
Note: Given the latency induced using a udpsink/udpsrc, that pipeline complains about timestamp issues. If you replace the udpsrc/udpsink with a filesrc/filesink to a serial port you can see the problem that I am about to describe.
Problem:
Now that I have described the pipelines, here is the problem:
If I start both pipelines, everything works as expected. However, if after 30s, I stop the "source" pipeline, and restart the pipeline, the Running Time gets reset back to zero, causing the timestamps of all buffers to be sent to be considered old buffers by the sink pipeline because it has already received buffers for timestamps 0 through 30s, so the playback on the other end won't resume until after 30s:
Source Pipeline: [28][29][30][0 ][1 ][2 ][3 ]...[29][30][31]
Sink Pipeline: [28][29][30][30][30][30][30]...[30][30][31]
________________________^
Source pipeline restarted
^^^^^^^^^^^^^^^^...^^^^^^^^
Sink pipeline will continue
to only show the "frame"
received at 30s until a
"newer" frame is sent, when
in reality each sent frame
is newer and should be shown
immediately.
Solution
I have found that adding sync=false to the autovideosink does solve the problem, however I was hoping to find a solution where the source would send its timestamps (DTS and PTS) based on the Clock time as seen in the image on that page.
I have seen this post and experimented with is-live and do-timestamp on my video source, but they do not seem to do what I want. I also tried to manually set the timestamps (DTS, PTS) in the buffers based on system time, however to no avail.
Any suggestions?

I think you should just restart the receiver pipeline as well. You could add the -e switch to the sender pipeline and when you stop the pipeline it should correctly propagate EOS via the GDP element to the receiver pipeline. Else I guess you can send a new segment or discontinuity to the receiver. Some event has to be signaled though to make the pipeline aware of that change, else it is somewhat bogus data. I'd say restarting the receiver is the simplest way.

Related

play multiple sources at once, each from a different position

I would like to use gstreamer to play multiple sources (for instance two video files) simultaneously using a single pipeline but each video starting from a different position, for instance first video from the beginning and the second from the middle. Could someone guide me on how to achieve it?
Simplifying, my pipeline is an equivalent of:
gst-launch-1.0 \
uridecodebin uri=file:///Users/tmikolaj/Downloads/videoalpha_video_dancer1.webm ! videoconvert ! autovideosink \
uridecodebin uri=file:///Users/tmikolaj/Downloads/videoalpha_video_dancer1.webm ! videoconvert ! autovideosink
, but created programmatically.
Obviously, simple seeking the pipeline seeks two files at once.
I was trying to register a probe of the GST_PAD_PROBE_TYPE_EVENT_UPSTREAM type from inside the pad-added signal callback of the uridecodebin element. Inside the probe I wanted to catch the GST_EVENT_SEEK event and drop it for the first video. However, it seems that dropping the SEEK message leaves pipeline in a PAUSED state and even an explicit state change to PLAYING does nothing.
Does anybody has some hints on how to solve that problem?

How can I save RTCP time information in video output

Is it possible to extract the NTP timestamps from the RTCP SR to be synchronized later?
I am currently recording my video and audio feeds separately using the following launch strings:
matroskamux name=mux ! filesink location=/tmp/video.mkv udpsrc port=26770 caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,payload=(int)101,encoding-name=(string)VP8,ssrc=(uint)608893168" ! .recv_rtp_sink rtpsession name=session .recv_rtp_src ! rtpvp8depay ! queue ! mux.video_0 udpsrc address=127.0.0.1 port=24694 ! session.recv_rtcp_sink
and
matroskamux name=mux ! filesink location=/tmp/audio.mkv udpsrc port=26540 caps="application/x-rtp,media=(string)audio,clock-rate=(int)48000,payload=(int)100,encoding-name=(string)OPUS,ssrc=(uint)632722900" ! .recv_rtp_sink rtpsession name=session .recv_rtp_src ! rtpopusdepay ! queue ! mux.audio_0 udpsrc address=127.0.0.1 port=23815 ! session.recv_rtcp_sink
I am recording them separately because the video or audio may start and stop multiple times throughout the session (for example, the user begins with just audio, then starts streaming video, then stops audio, etc etc etc). I figured that it would be easier to just start/stop separate pipelines when the user starts/stops a stream rather than trying to stop an audio-only pipeline and switch to an audio+video pipeline (and potentially have a gap in the audio between the stop and the start).
As a proof-of-concept, I used the created timestamp of the resulting mkv file (using gst-discoverer-1.0 and looking at the datetime field) and it mostly worked but seemed to be just a little bit off. I'm wondering if there's a way I can use the RTCP SR packets to encode the "real" timestamp for the start of the stream. It would be great if I could encode it into the mkv somehow but even if I could just access it in code with a signal then I could store the information elsewhere. I looked through the signals on rtpsession but nothing was jumping out at me as a possible solution.

GStreamer recording a stream to a wav file with wrong duration

I'm recording a wav file using GStreamer receiving G711 flow through a UDP port.
Any wav player can play the file, but shows a wrong duration and cannot fast forward.
I believe that GStreamer writes the header at the beginning with empty data.
This pipeline can reproduce the issue:
gst-launch-1.0 udpsrc port=3000 caps="application/x-rtp,media=(string)audio, payload=0,clock-rate=(int)8000" ! rtpjitterbuffer ! rtppcmudepay ! mulawdec ! wavenc ! filesink append=true location=c:/recordings/audio-zz.wav
Florian Zwoch suggested to use -e and the file will be closed properly.
Indeed it works perfectly.
I'm using this pipeline inside a Java program with the gst1-java-core library.
Seems that I'm missing something closing the pipeline.
My program has the same behaviour as gst-launch without -e parameter.
Before stopping the pipeline I send an EOS Event.
pipeline.sendEvent(new EOSEvent());
How can I fix it?
The append parameter of filesink element does not allow rewriting the header.
Thank you.
How do you stop the pipeline? If you interrupt the pipeline with ctrl-c it may indeed be that the header finalization is skipped. Run your pipeline with the -e option so that on ctrl-c your pipeline gets stopped gracefully.

How to make rtpjitterbuffer work on a stream without timestamps?

I am sending an H.264 bytestream over RTP using gstreamer.
# sender
gst-launch-1.0 filesrc location=my_stream.h264 ! h264parse disable-passthrough=true ! rtph264pay config-interval=10 pt=96 ! udpsink host=localhost port=5004
Then I am receiving the frames, decoding and displaying in other gstreamer instance.
# receiver
gst-launch-1.0 udpsrc port=5004 ! application/x-rtp,payload=96,media="video",encoding-name="H264",clock-rate="90000" ! rtph264depay ! h264parse ! decodebin ! xvimagesink
This works as is, but I want to try adding an rtpjitterbuffer in order to perfectly smooth out playback.
# receiver
gst-launch-1.0 udpsrc port=5004 ! application/x-rtp,payload=96,media="video",encoding-name="H264",clock-rate="90000" ! rtpjitterbuffer ! rtph264depay ! h264parse ! decodebin ! xvimagesink
However, as soon as I do, the receiver only displays a single frame and freezes.
If I replace the .h264 file with an MP4 file, the playback works great.
I assume that my h264 stream does not have the required timestamps to enable the jitter buffer to function.
I made slight progress by adding identity datarate=1000000. This allows the jitterbuffer to play, however this screws with my framerate because P frames have less data than I frames. Clearly the identity element adds the correct timestamps, but just with the wrong numbers.
Is it possible to automatically generate timestamps on the sender by specifying the "framerate" caps correctly somewhere? So far my attempts have not worked.
You've partially answered the problem already:
If I replace the .h264 file with an MP4 file, the playback works great.
I assume that my h264 stream does not have the required timestamps to enable the jitter buffer to function.
Your sender pipeline has no negotiated frame rate because you're using a raw h264 stream, while you should really be using a container format (e.g., MP4) which has this information. Without timestamps udpsink cannot synchronise against clock to throttle, so the sender is spitting out packets as fast as pipeline can process them. It's not a live sink.
However adding a rtpjitterbuffer makes your receiver act as live source. It freezes because it's trying its best to cope with the barrage of packets of malformed timestamps. RTP doesn't transmit "missing" timestamps to best of my knowledge, so all packets will probably have the same timestamp. Thus it probably reconstructs the first frame and drops the rest as duplicates.
I must agree with user1998586 in the sense that it ought to be better for the pipeline to crash with a good error message in this case rather trying its best.
Is it possible to automatically generate timestamps on the sender by specifying the "framerate" caps correctly somewhere? So far my attempts have not worked.
No. You should really use a container.
In theory, however, an au aligned H264 raw stream could be timestamped by just knowing the frame rate, but there are no gstreamer elements (I know of) that do this and just specifying caps won't do it.
I had the same problem, and the best solution I found was to add timestamps to the stream on the sender side, by adding do-timestamp=1 to the source.
Without timestamps I couldn't get rtpjitterbuffer to pass more than one frame, no matter what options I gave it.
(The case I was dealing with was streaming from raspvid via fdsrc, I presume filesrc behaves similarly).
It does kinda suck that gstreamer so easily sends streams that gstreamer itself (and other tools) doesn't process correctly: if not having timestamps is valid, then rtpjitterbuffer should cope with it; if not having timestamps is invalid, then rtph264pay should refuse to send without timestamps. I guess it was never intended as a user interface...
You should try to set the rtpjitterbuffer mode to another value than the default one:
mode : Control the buffering algorithm in use
flags: readable, writable
Enum "RTPJitterBufferMode" Default: 1, "slave"
(0): none - Only use RTP timestamps
(1): slave - Slave receiver to sender clock
(2): buffer - Do low/high watermark buffering
(4): synced - Synchronized sender and receiver clocks
Like that:
... ! rtpjittrbuffer mode=0 ! ...

How to get Pipeline created by playbin in textual format in Gstreamer?

I'm playing a transport stream file (*.ts) using the following pipeline:
gst-launch-0.10 playbin2 uri=file:///c:/bbb.ts
But I need to convert that into a pipeline myself. I'm not sure how to achieve this.
So far I have tried: (works fine)
gst-launch-0.10 -v filesrc location=c:/bbb.ts ! tsdemux ! audio/x-ac3 ! fakesink
But if i replace fakesink with autoaudiosink it fails with a not-linked error.
And even the fakesink doesn't work for video:
gst-launch-0.10 -v filesrc location=c:/bbb.ts ! tsdemux ! video/x-mpeg2 ! fakesink
So I have two questions:
How to find out pipeline created by playbin element.
How to play mpeg2-ts file using gstreamer pipeline.
Answer to question 1 -
There is a way to get the graphs of the pipeline created mentioned in documentation of basic tutorial-11.
A brief from the page
Getting pipeline graphs
For those cases where your pipeline starts to grow too large and you
lose track of what is connected with what, GStreamer has the
capability to output graph files. These are .dot files, readable with
free programs like GraphViz, that describe the topology of your
pipeline, along with the caps negotiated in each link.
This is also very handy when using all-in-one elements like playbin2
or uridecodebin, which instantiate several elements inside them.
I hope this resolves what you want