I am receiving h264 frames over a serial link, trying to play them with gstreamer. I set the caps to gst_caps_from_string("video/x-h264"), and it seems to accept them (if I use other caps, e.g. application/x-rtp, then the gstreamer log output complains about incompatible caps).
More specifically, I use the following elements: appsrc ! h264parse ! rtph264pay, and it seems like h264parse is the one not being happy.
When I pass (through appsrc) a frame that I get as a byte[] of length 8018, I get the following log output:
WARN h264parse gsth264parse.c:1496:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 6 SEI, Size: 12 will be dropped
WARN h264parse gsth264parse.c:1496:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 6 SEI, Size: 12 will be dropped
WARN h264parse gsth264parse.c:1496:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 7986 will be dropped
Note how those 3 WARN lines are similar, with the first two dropping size 12, and the last one dropping size <my byte[] size> - 32.
This is consistent over all the frames I send: it always complains twice about dropping 12, and then about dropping the length of the frame I pass minus 32.
Why could that be? Could it be related to my pipeline not receiving SPS/PPS data? I have been dumping the frames (appsrc ! identity dump=true ! ...), but I can't seem to find any frame that starts with 00 00 00 01 67 or 00 00 00 01 68, though I'm not sure if that could create those warnings or not.
From the source code, this message comes from:
if (!gst_h264_parse_process_nal (h264parse, &nalu)) {
GST_WARNING_OBJECT (h264parse,
"broken/invalid nal Type: %d %s, Size: %u will be dropped",
nalu.type, _nal_name (nalu.type), nalu.size);
[...]
}
Where gst_h264_parse_process_nal(), for SEI frames, returns FALSE when:
case GST_H264_NAL_SEI:
/* expected state: got-sps */
if (!GST_H264_PARSE_STATE_VALID (h264parse, GST_H264_PARSE_STATE_GOT_SPS))
return FALSE;
With GST_H264_PARSE_STATE_VALID defined as:
#define GST_H264_PARSE_STATE_VALID(parse, expected_state) \
(((parse)->state & (expected_state)) == (expected_state))
It seems that this error comes when the state does not have GST_H264_PARSE_STATE_GOT_SPS set.
As a conclusion, the frames in the question above are dropped because the SPS data has not been received at this point.
Related
I am trying to convert a webcam on a raspberry pi to x264, but keep running into an error about an " Unsupported profile constrained-baseline".
GST_DEBUG=3 /home/pi/gst-rtsp-server/examples/test-launch "( v4l2src device=/dev/video0 ! videoconvert ! omxh264enc ! h264parse ! rtph264pay name=pay0 )"
stream ready at rtsp://127.0.0.1:8554/test
0:00:03.043939441 10314 0x75c08350 WARN v4l2src gstv4l2src.c:692:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
0:00:03.044207251 10314 0x7491de30 FIXME default gstutils.c:3981:gst_pad_create_stream_id_internal:<appsrc1:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:03.044211053 10314 0x7491de60 FIXME default gstutils.c:3981:gst_pad_create_stream_id_internal:<appsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:03.087901354 10314 0x7491de90 ERROR omxh264enc gstomxh264enc.c:706:gst_omx_h264_enc_set_format:<omxh264enc-omxh264enc0> Unsupported profile constrained-baseline
0:00:03.087992083 10314 0x7491de90 ERROR omxvideoenc gstomxvideoenc.c:2239:gst_omx_video_enc_set_format:<omxh264enc-omxh264enc0> Subclass failed to set the new format
0:00:03.088080988 10314 0x7491de90 WARN videoencoder gstvideoencoder.c:678:gst_video_encoder_setcaps:<omxh264enc-omxh264enc0> rejected caps video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)5/1, pixel-aspect-ratio=(fraction)1/1, format=(string)I420, interlace-mode=(string)progressive
0:00:03.092800068 10314 0x7491de90 ERROR omxh264enc gstomxh264enc.c:706:gst_omx_h264_enc_set_format:<omxh264enc-omxh264enc0> Unsupported profile constrained-baseline
0:00:03.092867411 10314 0x7491de90 ERROR omxvideoenc gstomxvideoenc.c:2239:gst_omx_video_enc_set_format:<omxh264enc-omxh264enc0> Subclass failed to set the new format
0:00:03.092942775 10314 0x7491de90 WARN videoencoder gstvideoencoder.c:678:gst_video_encoder_setcaps:<omxh264enc-omxh264enc0> rejected caps video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)5/1, pixel-aspect-ratio=(fraction)1/1, format=(string)I420, interlace-mode=(string)progressive
0:00:03.092983764 10314 0x7491de90 WARN GST_PADS gstpad.c:4226:gst_pad_peer_query:<videoscale0:src> could not send sticky events
0:00:04.001816134 10314 0x7491de90 ERROR omxh264enc gstomxh264enc.c:706:gst_omx_h264_enc_set_format:<omxh264enc-omxh264enc0> Unsupported profile constrained-baseline
0:00:04.001956914 10314 0x7491de90 ERROR omxvideoenc gstomxvideoenc.c:2239:gst_omx_video_enc_set_format:<omxh264enc-omxh264enc0> Subclass failed to set the new format
0:00:04.002098111 10314 0x7491de90 WARN videoencoder gstvideoencoder.c:678:gst_video_encoder_setcaps:<omxh264enc-omxh264enc0> rejected caps video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)5/1, pixel-aspect-ratio=(fraction)1/1, format=(string)I420, interlace-mode=(string)progressive
0:00:04.037642275 10314 0x7491de90 ERROR omxh264enc gstomxh264enc.c:706:gst_omx_h264_enc_set_format:<omxh264enc-omxh264enc0> Unsupported profile constrained-baseline
0:00:04.037781284 10314 0x7491de90 ERROR omxvideoenc gstomxvideoenc.c:2239:gst_omx_video_enc_set_format:<omxh264enc-omxh264enc0> Subclass failed to set the new format
0:00:04.037959303 10314 0x7491de90 WARN videoencoder gstvideoencoder.c:678:gst_video_encoder_setcaps:<omxh264enc-omxh264enc0> rejected caps video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)5/1, pixel-aspect-ratio=(fraction)1/1, format=(string)I420, interlace-mode=(string)progressive
0:00:04.038154562 10314 0x7491de90 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop:<v4l2src0> error: Internal data stream error.
0:00:04.038222947 10314 0x7491de90 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop:<v4l2src0> error: streaming stopped, reason not-negotiated (-4)
0:00:04.038854870 10314 0x75c08350 WARN rtspmedia rtsp-media.c:1834:default_handle_message: 0x75c4e120: got error Internal data stream error. (gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:media-pipeline/GstBin:bin0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4))
0:00:04.039143357 10314 0x20b2490 WARN rtspmedia rtsp-media.c:2127:wait_preroll: failed to preroll pipeline
0:00:04.039219346 10314 0x20b2490 WARN rtspmedia rtsp-media.c:2384:gst_rtsp_media_prepare: failed to preroll pipeline
0:00:04.053306794 10314 0x7491de90 ERROR omxh264enc gstomxh264enc.c:706:gst_omx_h264_enc_set_format:<omxh264enc-omxh264enc0> Unsupported profile constrained-baseline
0:00:04.053442678 10314 0x7491de90 ERROR omxvideoenc gstomxvideoenc.c:2239:gst_omx_video_enc_set_format:<omxh264enc-omxh264enc0> Subclass failed to set the new format
0:00:04.053582781 10314 0x7491de90 WARN videoencoder gstvideoencoder.c:678:gst_video_encoder_setcaps:<omxh264enc-omxh264enc0> rejected caps video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)5/1, pixel-aspect-ratio=(fraction)1/1, format=(string)I420, interlace-mode=(string)progressive
0:00:04.079457589 10314 0x20b2490 ERROR rtspclient rtsp-client.c:678:find_media: client 0x2010610: can't prepare media
0:00:04.081144660 10314 0x20b2490 ERROR rtspclient rtsp-client.c:2210:handle_describe_request: client 0x2010610: no media
There isn't anything in the docs for omxh264enc about profiles that I can find:
gst-inspect-1.0 omxh264enc
Factory Details:
Rank primary + 1 (257)
Long-name OpenMAX H.264 Video Encoder
Klass Codec/Encoder/Video
Description Encode H.264 video streams
Author Sebastian Dröge <sebastian.droege#collabora.co.uk>
Plugin Details:
Name omx
Description GStreamer OpenMAX Plug-ins
Filename /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstomx.so
Version 1.14.4
License LGPL
Source module gst-omx
Source release date 2018-10-02
Binary package GStreamer OpenMAX Plug-ins source release
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstVideoEncoder
+----GstOMXVideoEnc
+----GstOMXH264Enc
+----GstOMXH264Enc-omxh264enc
Implemented Interfaces:
GstPreset
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
width: [ 16, 4096 ]
height: [ 16, 4096 ]
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "omxh264enc-omxh264enc0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events from downstream
flags: readable, writable
Boolean. Default: false
control-rate : Bitrate control method
flags: readable, writable, changeable only in NULL or READY state
Enum "GstOMXVideoEncControlRate" Default: -1, "default"
(0): disable - Disable
(1): variable - Variable
(2): constant - Constant
(3): variable-skip-frames - Variable Skip Frames
(4): constant-skip-frames - Constant Skip Frames
(-1): default - Component Default
target-bitrate : Target bitrate in bits per second (0xffffffff=component default)
flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
quant-i-frames : Quantization parameter for I-frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
quant-p-frames : Quantization parameter for P-frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
quant-b-frames : Quantization parameter for B-frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
inline-header : Inline SPS/PPS header before IDR
flags: readable, writable, changeable only in NULL or READY state
Boolean. Default: true
periodicity-idr : Periodicity of IDR frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
periodicty-idr : Periodicity of IDR frames (0xffffffff=component default) DEPRECATED - only for backwards compat
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
interval-intraframes: Interval of coding Intra frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
b-frames : Number of B-frames between two consecutive I-frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
entropy-mode : Entropy mode for encoding process
flags: readable, writable, changeable only in NULL or READY state
Enum "GstOMXH264EncEntropyMode" Default: -1, "default"
(0): CAVLC - CAVLC entropy mode
(1): CABAC - CABAC entropy mode
(-1): default - Component Default
constrained-intra-prediction: If enabled, prediction only uses residual data and decoded samples from neighbouring coding blocks coded using intra prediction modes
flags: readable, writable, changeable only in NULL or READY state
Boolean. Default: false
loop-filter-mode : Enable or disable the deblocking filter (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Enum "GstOMXH264EncLoopFilter" Default: -1, "default"
(0): enable - Enable deblocking filter
(1): disable - Disable deblocking filter
(2): disable-slice-boundary - Disables deblocking filter on slice boundary
(-1): default - Component Default
I am new to gstreamer so I could be reading this all wrong, would appreciate any ideas or feedback! Thanks!
You can change that through the SRC cap, try e.g.:
v4l2src device=/dev/video0 ! videoconvert ! omxh264enc ! video/x-h264,profile=baseline ! h264parse ! rtph264pay name=pay0
gstreamer raises an error when trying to stream side-by-side video from a stereoscopic UVC camera.
I have a stereoscopic camera attached via USB to an ARM board, but on the highest resolution setting that the camera allows gstreamer is raising an Invalid Dimension 0x0 error.
v4l2-ctl --list-formats-ext -d /dev/video2
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: 'MJPG' (compressed)
Name : Motion-JPEG
Size: Discrete 2560x960
Interval: Discrete 0.017s (60.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 2560x720
Interval: Discrete 0.017s (60.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1280x480
Interval: Discrete 0.017s (60.000 fps)
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 640x240
Interval: Discrete 0.017s (60.000 fps)
Interval: Discrete 0.033s (30.000 fps)
$ gst-launch-1.0 -v v4l2src device=/dev/video2 ! "image/jpeg, width=2560, height=960, framerate=60/1" ! progressreport ! rtpjpegpay ! udpsink host=127.0.0.1 port=5000
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = image/jpeg, width=(int)2560, height=(int)960, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = image/jpeg, width=(int)2560, height=(int)960, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstProgressReport:progressreport0.GstPad:src: caps = image/jpeg, width=(int)2560, height=(int)960, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstRtpJPEGPay:rtpjpegpay0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)JPEG, a-framerate=(string)60.000000, x-dimensions=(string)"2560\,960", payload=(int)26, ssrc=(uint)1656850644, timestamp-offset=(uint)2590317031, seqnum-offset=(uint)18356
/GstPipeline:pipeline0/GstUDPSink:udpsink0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)JPEG, a-framerate=(string)60.000000, x-dimensions=(string)"2560\,960", payload=(int)26, ssrc=(uint)1656850644, timestamp-offset=(uint)2590317031, seqnum-offset=(uint)18356
/GstPipeline:pipeline0/GstRtpJPEGPay:rtpjpegpay0.GstPad:sink: caps = image/jpeg, width=(int)2560, height=(int)960, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstProgressReport:progressreport0.GstPad:sink: caps = image/jpeg, width=(int)2560, height=(int)960, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = image/jpeg, width=(int)2560, height=(int)960, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
0:00:00.163107430 7652 0x55d47a920a30 WARN v4l2bufferpool gstv4l2bufferpool.c:790:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold
/GstPipeline:pipeline0/GstRtpJPEGPay:rtpjpegpay0: timestamp = 2590320450
/GstPipeline:pipeline0/GstRtpJPEGPay:rtpjpegpay0: seqnum = 18356
progressreport0 (00:00:05): 4 seconds
progressreport0 (00:00:10): 9 seconds
progressreport0 (00:00:15): 14 seconds
0:00:15.770955137 7652 0x55d47a920a30 WARN v4l2src gstv4l2src.c:968:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:15.622372937
progressreport0 (00:00:20): 19 seconds
progressreport0 (00:00:25): 24 seconds
progressreport0 (00:00:30): 29 seconds
progressreport0 (00:00:35): 34 seconds
Then on the viewing machine (currently just using localhost on the same laptop):
$ gst-launch-1.0 -e -v udpsrc port=5000 ! application/x-rtp, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = application/x-rtp, encoding-name=(string)JPEG, payload=(int)26, media=(string)video, clock-rate=(int)90000
/GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0.GstPad:sink: caps = application/x-rtp, encoding-name=(string)JPEG, payload=(int)26, media=(string)video, clock-rate=(int)90000
WARNING: from element /GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0: Invalid Dimension 0x0.
Additional debug info:
gstrtpjpegdepay.c(741): gst_rtp_jpeg_depay_process (): /GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0
WARNING: from element /GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0: Invalid Dimension 0x0.
The two lowest resolution modes work with this configuration, but the 720p side-by-side mode throws the foregoing errors.
What am I doing wrong here? And does this have anything to do with gst-launch-1.0 not supporting full screen mode perhaps?
Thanks in advance
See RFC 2435 https://www.rfc-editor.org/rfc/rfc2435 on width and height:
3.1.5. Width: 8 bits
This field encodes the width of the image in 8-pixel multiples (e.g.,
a width of 40 denotes an image 320 pixels wide). The maximum width
is 2040 pixels.
3.1.6. Height: 8 bits
This field encodes the height of the image in 8-pixel multiples
(e.g., a height of 30 denotes an image 240 pixels tall). When
encoding interlaced video, this is the height of a video field, since
fields are individually JPEG encoded. The maximum height is 2040
pixels.
Here you can see that the limit is 2040 pixels in both dimensions. This is a protocol limitation.
Check the GStreamer source code https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/blob/master/gst/rtp/gstrtpjpegdepay.c:
/* allow frame dimensions > 2040, passed in SDP session or media attributes
* from gstrtspsrc.c (gst_rtspsrc_sdp_attributes_to_caps), or in caps */
if (!width)
width = rtpjpegdepay->media_width;
if (!height)
height = rtpjpegdepay->media_height;
Here you can see the GStreamer devs offer you a solution for this protocol limitation. When receiving images over 2040 pixels in either dimension you probably will have to add width and height information to the caps.
I cannot create a pipeline with gstreamer and I don't know how I can debug it further.
gst-launch-1.0 --gst-debug=GST_CAPS:4 -v -e customsrc num-buffers=1000 ! video/x-h264,width=600,height=600,framerate=1/12,stream-format=byte-stream ! mpegtsmux ! udpsink host=10.92.7.2 port=5000
WARNING: erroneous pipeline: could not link customsrc0 to mpegtsmux0
The capabilities of customsrc and mpegtsmux are matching. But obviously something is missing.
customsrc
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 1/2147483647, 2147483647/1 ]
stream-format: avc
alignment: au
mpegtsmux
Pad Templates:
SINK template: 'sink_%d'
Availability: On request
Has request_new_pad() function: 0x76beca8c
Capabilities:
video/x-h264
stream-format: byte-stream
alignment: { au, nal }
What else can I do to figure out the mismatch?
Caps are for filtering and defining how the pipeline runs--they don't cause any transformation by themselves. For example, if you have two elements with these caps on their source and sink pads:
video/x-h264
stream-format: byte-stream
alignment: { au, nal }
video/x-h264
stream-format: byte-stream
alignment: { au, nal }
And you place this caps filter between them:
video/x-h264,alignment=nal
You'll cause the pipeline to use nal alignment there. If your elements have these caps on their pads:
video/x-h264
stream-format: avc
alignment: { au, nal }
video/x-h264
stream-format: byte-stream
alignment: { au, nal }
You need to add an element that will convert video/x-h264,stream-format=avc into video/x-h264,stream-format=byte-stream. h264parse will do this because it takes any video/x-h264 content on its sink pad, and outputs whatever stream-format and alignments are needed for its downstream source:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
parsed: true
stream-format: { avc, avc3, byte-stream }
alignment: { au, nal }
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-h264
I have seen this kind of pipeline-running commands in gstreamer:
e.g.,
gst-launch-1.0 videotestsrc ! video/x-raw, format=I420, framerate=25/1, width=640, height=360 ! xvimagesink
And I have read in some pages that video/x-raw, format=I420, framerate=25/1, width=640, height=360 specifies the media-type. But I am not able to understand what effect would it make - is it transforming the input to the specified framerate/format/width/height etc.. or is it just like specifying the input is in this framerate/width/ht already? And what effect it would have on the pipeline if it is just specifying the input is in this framerate etc... instead of transforming.
And is it really necessary or we can ignore it?
They call this video/x-raw, format=I420, framerate=25/1, width=640, height=360 thing "capabilities" or "caps" for short.
Sometimes it is required to specify explicitly what type of data flows between elements in a gstreamer pipe. And caps is the way to do this.
Why this is required sometimes? Because some gstreamer elements can accept (or produce)
several types of media. You can see this with gst-inspect command:
$ gst-inspect videotestsrc
...
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw-yuv
format: YUY2
color-matrix: { sdtv, hdtv }
chroma-site: { mpeg2, jpeg }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw-yuv
format: UYVY
color-matrix: { sdtv, hdtv }
chroma-site: { mpeg2, jpeg }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw-yuv
format: YVYU
color-matrix: { sdtv, hdtv }
...
This means that videotestsrc has pad src which can produce outputs in various formats (YUY2, UYVY, YVYU, etc), sizes, framerates, etc.
The same is for xvimagesink:
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw-rgb
framerate: [ 0/1, 2147483647/1 ]
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
video/x-raw-yuv
framerate: [ 0/1, 2147483647/1 ]
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
It is able to view streams of data in various formats.
GStreamer uses the process called caps negotiation to decide which concrete data format to use between two elements. It tries to select "best fitting" format if no capabilities are provided by the user. That is why it is possible to drop capabilities from your pipeline and it will still work:
$ gst-launch-1.0 -v videotestsrc ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)320, height=(int)240, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)320, height=(int)240, framerate=(fraction)30/1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
I added -v flag to see what caps gstreamser actually decided to use.
But sometimes there are cases when gstreamser fails to negotiate data format or you have different preferences.
E.g. in the case of pipeline that reads stream from socket it is impossible guess data format for certain and you need to provide correct capabilities.
You can see that specifying caps makes difference by executing these two pipelines:
$ gst-launch -v videotestsrc ! 'video/x-raw-yuv, width=600, height=600' ! xvimagesink
$ gst-launch -v videotestsrc ! 'video/x-raw-yuv, width=60, height=60' ! xvimagesink
It is important to understand that capabilities does not convert data but specify which format elements will produce or consume.
I wish to build a single gstreamer pipeline that does both rtp audio send and receive.
Based on the examples (few as they are) that I've found, here is my almost working code.
(the program is written in Rexx, but it's pretty obvious what is happening, I think. Here, it looks a lot like bash!). Line catenation char is comma. The "", bits just insert blank lines for readability.
rtp_recv_port = 8554
rtp_send_port = 8555
pipeline = "gst-launch -e",
"",
"gstrtpbin",
" name=rtpbin",
"",
"udpsrc port="rtp_recv_port, -- do-timestamp=true
' ! "application/x-rtp,media=audio,payload=8,clock-rate=8000,encoding-name=PCMA,channels=1" ',
" ! rtpbin.recv_rtp_sink_0",
"",
"rtpbin. ",
" ! rtppcmadepay",
" ! decodebin ",
' ! "audio/x-raw-int, width=16, depth=16, rate=8000, channels=1" ',
" ! volume volume=5.0 ",
" ! autoaudiosink sync=false",
"",
"autoaudiosrc ",
" ! audioconvert ",
' ! "audio/x-raw-int,width=16,depth=16,rate=8000,channels=1" ',
" ! alawenc ",
" ! rtppcmapay perfect-rtptime=true mtu=2000",
" ! rtpbin.send_rtp_sink_1",
"",
"rtpbin.send_rtp_src_1 ",
" ! audioconvert",
" ! audioresample",
" ! udpsink port="rtp_send_port "host="ipaddr
pipeline "> pipe.out"
If I comment out the lines after
" ! autoaudiosink sync=false",
The receive-only portion works just fine. However, if I leave those lines in place I get this error:
ERROR: from element /GstPipeline:pipeline0/GstUDPSrc:udpsrc0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2582): gst_base_src_loop (): /GstPipeline:pipeline0/GstUDPSrc:udpsrc0:
streaming task paused, reason not-linked (-1)
So what's suddenly become unlinked? I'd understand if the error was in the autoaudiosrc portion, but suddenly showing up in the udpsrc section?
Suggestion of help, anyone?
(FWIW) After I get this part working I will go back in and add the rtcp parts or the pipeline.
Here is a pipeline that will send and receive audio (full duplex). I manually set the sources so that it is expandable(you can put video on this as well and I have a sample pipeline for you if you want to do both). I set the jitter buffer mode to BUFFER because mine is implemented on a network with a TON of jitter. Now, within this sample pipe, you could add all your variable changes (volume, your audio source, encoding and decoding etc.).
sudo gst-launch gstrtpbin \
name=rtpbin audiotestsrc ! queue ! audioconvert ! alawenc ! \
rtppcmapay pt=8 ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! \
multiudpsink clients="127.0.0.1:5002" sync=false async=false \
udpsrc port=5004 caps="application/x-rtp, media=audio, payload=8, clock-rate=8000, \
encoding-name=PCMA" ! queue ! rtpbin.recv_rtp_sink_0 \
rtpbin. buffer-mode=RTP_JITTER_BUFFER_MODE_BUFFER ! rtppcmadepay ! alawdec ! alsasink
I have had issues with the Control(RTCP) packets. I have found that a loop back test is not sufficient if you are utilizing RTCP. You will have to test on two computers talking to each other.
Let me know if this works for you as I have tested on 4 different machines and all have worked.