This is the pipeline:
webrtcbin name=webrtcbin bundle-policy=max-bundle! rtph264depay ! h264parse ! qtmuxfragment-duration=1000 ! filesink location=/home/ubuntu/recordings/abcde/test.mkv
Modern browsers vary resolution and bitrate during the startup phase of a session. It is not uncommon to see 640x360 for a while, followed by 960x540 up to maybe 1280x720 later in a H.264 stream.
Trying to record this to disk requires the use of either mp4mux or qtmux. Both did break up the recording on the first resolution change in versions prior GStreamer 1.20. Practically this was the end of the recording when the first change from 640x360 to 960x540 happened.
Meanwhile (after 1.20) the qtmux survives the renegotiation with upstream, but the downstream does not contain the required (and updated) SPS/PPS. Just the IDR frame (initiated by the browser) is sent downstream. In the end this means, that after the first resolution change there is only garbage in the output file.
Any idea, how this could be changed?
Related
I have this existing program that uses gst-plugin-1.0 and it passes this:
-e udpsrc port=3003 buffer-size=200000 ! h264parse ! queue ! http://mux.video_0 alsasrc device=plughw:1,0 ! "audio/x-raw,channels=1,depth=16,width=16,rate=44100" ! voaacenc bitrate=128000 ! aacparse ! queue ! http://mux.audio_0 qtmux name=mux ! filesink location="$RECPATH/record-`date +%Y%m%d%-H%M%S`.mp4" sync=true
This takes the video from an udp source which is in x264 and the audio directly from the microphone. It works but since it doesn't encode the video and the audio at the same time I have a bit of delay on the audio when the video stream has latency (due to higher quality settings).
So as a quick-fix I was thinking about adding a delay on the audio recording to compensate. I would calculate that delay by hand depending on the video quality.
Constraint: gst-launch-1.0 version 1.10.4 (on a raspberry pi, debian stretch), use-driver-timestamps doesn't seem to be accessible, I get the error 'WARNING: erroneous pipeline: no property "use-driver-timestamps" in element "alsasrc0"'.
So my question is: is there an easy way to add delay to the audio?
the queue element had the min-threshold-time property, which lets you hold on to data for n amount of time.
https://gstreamer.freedesktop.org/documentation/coreelements/queue.html?gi-language=c#queue:min-threshold-time
Alternatively I found this too, might be useful for your case pipeline Gstremer video streaming with delay
Try ! autoaudiosink ts-offset=100000000
ts-offset is documented here.
You can also experiment pipelines with latency compensation;
https://gstreamer.freedesktop.org/documentation/additional/design/latency.html#latency-compensation
I want to read the feed from a webcam and host a RTSP stream without encoding the feed. I have access to high bandwidth network but the CPUs are very low end and have other tasks to full fill due to which I want to skip the encoding/decoding steps to save up on CPU usage. Before jumping on to RTSP I tried a simple MJPG stream and tried to skip the jpegenc (JPG encoding) as it can be done directly with a simple gst pipeline:
gst-launch-1.0 -v autovideosrc ! videoconvert ! videoscale ! video/x-raw,format=I420,width=800,height=600,framerate=25/1 ! rtpjpegpay ! udpsink host=10.0.1.10 port=5000
However, I got a warning:
WARNING: erroneous pipeline: could not link videoscale0 to
rtpjpegpay0, rtpjpegpay0 can't handle caps video/x-raw,
format=(string)I420, width=(int)800, height=(int)600,
framerate=(fraction)25/1
I'm new to Gstreamer and not sure if this is possible and how to move forward next. The same command above works if I include the jpg encoding. Any suggestions would be appreciated.
rtpjpegpay is an element that takes in a Motion JPEG stream and translates it to RTP. The input you're giving it however isvideo/x-raw, which means it is unencoded, rather than encoded with Motion JPEG. If you want to use this element, you'll first have to encode it to Motion JPEG, using something like jpegenc.
Like #vermaete already mentions: if you really, really don't want to encode your video, you can use someting like rtpvrawpay, which will translate your raw video into RTP packets. However: sending raw, unencoded video over the network is not really advisable (and not even workable if you have a bad connection, or even impossible if you plan on sending it over the Internet). You might also end up using a lot of resources on your CPU just to get everything payloaded properly, and gettign it sent to your network card.
I am trying to capture and store a webcam stream. The requirements are 1920x1080#30fps. And it must be done by a single-board-computer (Raspberry).
The duration to capture is 10 minutes. (For the moment I only capture 10 seconds for testing)
In general the camera (usbfhd01m from ELP) is able to provide an MJPEG stream in 1920x1080#30fps. I am just not able to store it. And I don't know why. I tried it with the following pipeline:
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 do-timestamp=true ! image/jpeg,width=1920,height=1080,framerate=30/1 ! queue ! avimux ! filesink location=test.avi
The result is a video file which is far away from being fluent. What is missing in my pipeline?
When I use the same pipeline, but decode the stream and save it in a raw file like this:
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 do-timestamp=true ! image/jpeg,width=1920,height=1080,framerate=30/1 ! queue ! jpegdec ! filesink location=test.yuv
then the raw video is absolutely fluent. Therefore, I think the pipeline and the device is able to record in 1920x1080#30fps, but there seems to be something wrong for saving the stream.
Storing the stream into matroska fileformat does not change my problem. And for transcoding on the fly to H264 the Raspberry Pi 3 doesn't seem to be powerful enough. (Even by using omxh264enc)
What happens when you remove the do-timestamp=true? This options applies current pipeline timestamps to the sample buffers - overwriting those coming out from the device. You probably want to store the original timestamps instead of overwriting them as they can carry some pipeline jitter.
In your second pipeline you save the stream as raw. Basically removing all timestamp information that you have (also the jitter timestamps). So when you play back the raw stream it assumes a constant framerate instead.
I am sending an H.264 bytestream over RTP using gstreamer.
# sender
gst-launch-1.0 filesrc location=my_stream.h264 ! h264parse disable-passthrough=true ! rtph264pay config-interval=10 pt=96 ! udpsink host=localhost port=5004
Then I am receiving the frames, decoding and displaying in other gstreamer instance.
# receiver
gst-launch-1.0 udpsrc port=5004 ! application/x-rtp,payload=96,media="video",encoding-name="H264",clock-rate="90000" ! rtph264depay ! h264parse ! decodebin ! xvimagesink
This works as is, but I want to try adding an rtpjitterbuffer in order to perfectly smooth out playback.
# receiver
gst-launch-1.0 udpsrc port=5004 ! application/x-rtp,payload=96,media="video",encoding-name="H264",clock-rate="90000" ! rtpjitterbuffer ! rtph264depay ! h264parse ! decodebin ! xvimagesink
However, as soon as I do, the receiver only displays a single frame and freezes.
If I replace the .h264 file with an MP4 file, the playback works great.
I assume that my h264 stream does not have the required timestamps to enable the jitter buffer to function.
I made slight progress by adding identity datarate=1000000. This allows the jitterbuffer to play, however this screws with my framerate because P frames have less data than I frames. Clearly the identity element adds the correct timestamps, but just with the wrong numbers.
Is it possible to automatically generate timestamps on the sender by specifying the "framerate" caps correctly somewhere? So far my attempts have not worked.
You've partially answered the problem already:
If I replace the .h264 file with an MP4 file, the playback works great.
I assume that my h264 stream does not have the required timestamps to enable the jitter buffer to function.
Your sender pipeline has no negotiated frame rate because you're using a raw h264 stream, while you should really be using a container format (e.g., MP4) which has this information. Without timestamps udpsink cannot synchronise against clock to throttle, so the sender is spitting out packets as fast as pipeline can process them. It's not a live sink.
However adding a rtpjitterbuffer makes your receiver act as live source. It freezes because it's trying its best to cope with the barrage of packets of malformed timestamps. RTP doesn't transmit "missing" timestamps to best of my knowledge, so all packets will probably have the same timestamp. Thus it probably reconstructs the first frame and drops the rest as duplicates.
I must agree with user1998586 in the sense that it ought to be better for the pipeline to crash with a good error message in this case rather trying its best.
Is it possible to automatically generate timestamps on the sender by specifying the "framerate" caps correctly somewhere? So far my attempts have not worked.
No. You should really use a container.
In theory, however, an au aligned H264 raw stream could be timestamped by just knowing the frame rate, but there are no gstreamer elements (I know of) that do this and just specifying caps won't do it.
I had the same problem, and the best solution I found was to add timestamps to the stream on the sender side, by adding do-timestamp=1 to the source.
Without timestamps I couldn't get rtpjitterbuffer to pass more than one frame, no matter what options I gave it.
(The case I was dealing with was streaming from raspvid via fdsrc, I presume filesrc behaves similarly).
It does kinda suck that gstreamer so easily sends streams that gstreamer itself (and other tools) doesn't process correctly: if not having timestamps is valid, then rtpjitterbuffer should cope with it; if not having timestamps is invalid, then rtph264pay should refuse to send without timestamps. I guess it was never intended as a user interface...
You should try to set the rtpjitterbuffer mode to another value than the default one:
mode : Control the buffering algorithm in use
flags: readable, writable
Enum "RTPJitterBufferMode" Default: 1, "slave"
(0): none - Only use RTP timestamps
(1): slave - Slave receiver to sender clock
(2): buffer - Do low/high watermark buffering
(4): synced - Synchronized sender and receiver clocks
Like that:
... ! rtpjittrbuffer mode=0 ! ...
I am running a 9 sec video on my NVIDIA Tegra jetson TK1 board using gstreamer as:
gst-launch-0.10 playbin uri=file:///home/ubuntu/widescreen.avi
I notice this drops a lot of frames and gstreamer prints these messages:
WARNING: from element /GstPlayBin:playbin0/GstBin:vbin/GstAutoVideoSink:videosink/GstXvImageSink:videosink-actual-sink-xvimage: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2875): gst_base_sink_is_too_late (): /GstPlayBin:playbin0/GstBin:vbin/GstAutoVideoSink:videosink/GstXvImageSink:videosink-actual-sink-xvimage:
There may be a timestamping problem, or this computer is too slow.
I ran top while executing this and indeed gstreamer is taking 95% of the CPU.
Now, when i play this video through the default media player, it plays completely fine and without any lag. I was wondering if anyone knows what may be the reason that gstreamer is unable to play it properly. I am new to gstreamer and wondering if I can do something to alleviate this.
Bins are many elements combined into one element type structure(i.e with appropriate number of sinks and sources). playbin is a standard bin with an "autovideosink" element which automatically detects video sinks.
Firstly I would suggest you to upgrade to gstreamer-1.0 in which many bugs were fixed.
Secondly the problem seems to be with your xvimagesink, hence try using ximagesink by defining explicitly, your autovideosink is selecting xvimagesink by default.
try this pipeine:-
(for hardware decoding)
gst-launch-1.0 filesrc location="location of h264 video/file.avi" ! avidemux ! h264parse ! omxh264dec ! videoconvert ! ximagesink
or for cpu decoding use avdec_h264 instead of omxh264dec
The default media player is likely using video hardware decoding if the media file is H.264, which allows it decode with less CPU resources. Try creating a gstreamer pipeline that explicitly uses the omx H.264 decoding element as discussed here (http://elinux.org/Jetson/H264_Codec).
eg. gst-launch-0.10 filesrc location=/home/ubuntu/widescreen.avi ! avidemux ! h264parse ! nv_omx_h264dec ! autovideosink