Gstreamer RTSP Decoding Frame Timestamps - c++

We are decoding RTSP stream frames using Gstreamer in C++. We need to read the frame NTP timestamps, which we think that resides in RTCP packets. After some documentation digging, we found an element called GstRTPBaseDepayload, which has a property called "stats", which has a field "timestamp", explained as the "last seen RTP timestamp".
Our original pipeline:
gst-launch-1.0 rtspsrc port-range=5000-5100 location="rtsp://.." latency=300 is-live=true ! queue ! rtph265depay name=depayer! video/x-h265 , stream-format=byte-stream, alignment=au ! h265parse ! video/x-h265 , stream-format=byte-stream, alignment=au ! appsink name=mysink sync=true
I named the depay element as rtph265depay name=dp, then:
depayer_=gst_bin_get_by_name(GST_BIN(pipeline_), "dp");
GstStructure * stat;
g_object_get((GstRTPBaseDepayload*)depayer_,"stats",stat);
GType type = gst_structure_get_field_type(stat,"timestamp");
It gave an error saying that the stat structure does not have a field, in fact, it did not have any fields. I did not find any example usage of GstRTPBaseDepayload, and the documentation is lacking as always. I would appreciate any guidance regarding the frame timestamps.
Edit:
I also tried to check if depayer_ has a null value:
depayer_=gst_bin_get_by_name(GST_BIN(pipeline_), "dp");
if(depayer_!=nullptr){
GstStructure * stat;
// GstRTPBaseDepayload* depayload;
g_object_get(depayer_,"stats",stat,NULL);
if(gst_structure_has_field(stat,"timestamp")){ //this line causes segfault
guint timestamp;
gst_structure_get_uint(stat,"timestamp",&timestamp);
}
}
Neither depayer nor stat object is null, however gst_structure_has_field(stat,"timestamp") causes a segfault. Any help is much appreciated.

I guess you can try
GstStructure *stat;
// GstRTPBaseDepayload* depayload;
g_object_get(depayer_,"stats",&stat,NULL);
notice I used &stat in g_object_get, not stat.
https://docs.gtk.org/gobject/method.Object.get.html
I found an example:
https://github.com/pexip/gst-rtsp-server/blob/master/examples/test-mp4.c#L38-L42

Related

Gstreamer buffer pts

I have a simple mjpeg pipeline and I want to access the buffer on the sink to get the pts to calculate the latency.
Pipeline:
souphttpsrc -> jpegparse -> imxvpudec -> imxipusink
What is the best way to do this? Some code examples would be great.
The time things in gstreamer confusing me a little bit.
I'd add an identity element in your pipeline where you want to analyze the PTS:
souphttpsrc ! jpegparse ! identity ! imxvpudec ! imxipusink
Then connect to the "handoff" signal:
static void pts_analysis_cb(GstElement *identity,
GstBuffer *buffer,
gpointer user_data) {
GstClockTime pts = GST_BUFFER_PTS(buffer);
//analysis
}
g_signal_connect_data(identity, "handoff",
G_CALLBACK(pts_analysis_cb),
NULL, NULL, GConnectFlags());
If you're seeing MJPEG related latency though you may just need to have sync=false on your tail element or set flags to drop buffers if it's falling behind.

Storing AAC Audio and Retrieving

I would like to store a file which has AAC audio frames,
For that i used the below pipeline,
gst-launch-1.0 filesrc location=Test_44100Hz_2ch_s16le.wav ! "audio/x-raw,rate=44100,format=s16le,channels=2" ! audioparse format=raw raw-format=s16le rate=44100 channels=2 ! faac ! aacparse ! queue ! filesink location=a1
While reading that file again to pulsesink using below pipeline,
gst-launch-1.0 filesrc location=a1 ! aacparse ! faad ! audioconvert ! audioresample ! pulsesink
I am Receiving below error, I used GST_DEBUG=3, but i am not able find the solution.
0:00:00.031924804 3379 0x2231d60 WARN basesrc gstbasesrc.c:3483:gst_base_src_start_complete:<filesrc0> pad not activated yet
Pipeline is PREROLLING ...
0:00:00.033044700 3379 0x2231050 WARN baseparse gstbaseparse.c:3255:gst_base_parse_loop:<aacparse0> error: No valid frames found before end of stream
ERROR: from element /GstPipeline:pipeline0/GstAacParse:aacparse0: No valid frames found before end of stream
Additional debug info:
gstbaseparse.c(3255): gst_base_parse_loop (): /GstPipeline:pipeline0/GstAacParse:aacparse0
ERROR: pipeline doesn't want to preroll.
Can anybody help me, To solve this? I need to store AAC audio frames and need to stream that file as AAC audio stream.
This is it, tested working:
gst-launch-1.0 filesrc location=WAV_44_16bit.wav ! decodebin ! audioconvert ! queue ! voaacenc ! aacparse ! queue ! mp4mux ! filesink location=aac.mp4
gst-launch-1.0 filesrc location=aac.mp4 ! decodebin ! audioconvert ! audioresample ! alsasink
In container there are metadata information stored.. without them the decoder does not know how to process the data.
AAC Audio streams require a container in order to be useful within gstreamer
For decoder initialization it is necessary to know sampling frequency and Audio Object. In gstreamer we are unable to pass this metadata directly to the parser or the decoder. The parser collects this data instead from the mp4 header then the encoder inherits the frame structure/size and sample rate. So this is a deficiency in either aacparse(parser) or avdec_aac/faad(decoder), none of which have exposed parameters to specify frame size of a raw file, the afore mentioned metadata. That being said, I haven't found a compelling reason why anyone would need to do this. I found myself trying to do it before I discovered the aac simply needed to be muxed into an MP4(mp4mux) or another container to work and be portable. The container/framing only adds a small amount of data to the stream.

Gstreamer: Could not swtich codebooks: rtpvorbisdepay

I am trying to stream audio with the following GStreamer pipeline:
Server:
gst-launch-1.0 -v audiotestsrc ! audioconvert ! vorbisenc ! rtpvorbispay ! udpsink host=127.0.0.1 port=5000
Client:
gst-launch-1.0 udpsrc port=5000 ! "application/x-rtp, media=audio, clock-rate=44100, encoding-name=VORBIS, encoding-params=1, payload=96" ! rtpvorbisdepay ! vorbisdec ! audioconvert ! autoaudiosink
I get the following message from GStreamer:
WARNING: from element /GstPipeline:pipeline0/GstRtpVorbisDepay:rtpvorbisdepay0: Could not decode stream.
Additional debug info: gstrtpvorbisdepay.c(614): gst_rtp_vorbis_depay_process (): /GstPipeline:pipeline 0/GstRtpVorbisDepay:rtpvorbisdepay0: Could not switch codebooks
And I don't get any sound on the client. Can anyone help?
[EDIT:]
When I copy-paste the caps from the server side... It works! But among those caps there is a configuration parameter which looks really ugly (link here). I noticed that if I just delete this parameter it doesn't work anymore. Moreover I used gst-inspect on udpsrc and rtpvorbisdepay elements and there is nothing about this parameter. Can someone explain me what this parameter corresponds to? Is there a way to avoid it?
I think this is Theora Vorbis thing.. those are some configuration parameters for initialization of decoder if I understand that properly..
Theora makes the same controversial design decision that Vorbis made to
include the entire probability model for the DCT coecients and all the quan-
tization parameters in the bitstream headers. This is often several hundred
elds. It is therefore impossible to decode any frame in the stream without
having previously fetched the codec info and codec setup headers.
~ from here
some similar question

How to check type of new added pad?

My pipeline scheme(dynamic link):
videotestsrc OR audiotestsrc ! decodebin ! queue ! autovideosink OR
autoaudiosink
I trying to use this advice to check which type of data I got (video/audio), but if I use decodebin like demuxer, then I get just "src_0" instead of "audio" or "video". How I can check my pad type for linking right element for playback? May be I can use one universal element for audio playback and video playback, like playsink(but it does not work for video)?
You can get the caps of the newly added pad and check if it contains audio or video caps (or something else).
Try with:
gst_pad_get_current_caps (pad);
or:
gst_pad_get_allowed_caps (pad);
If you are using gstreamer 0.10 (which is 3+ years obsolete an unmantained), you have:
gst_pad_get_caps_reffed (pad);
Then just check the returned caps if it is audio or video by getting the structure from the caps and checking if its name starts with video or audio.
/* There might be multiple structures depending on how you do it,
* but usually checking one in this case is enough */
structure = gst_caps_get_structure (caps, 0);
name = gst_structure_get_name (structure);
if (g_str_has_prefix (name, "video/")) {
...
} else if (g_str_has_prefix (name, "audio/")) {
...
}

Gstreamer, rtspsrc and payload type

I'm having difficulties in retrieving rtsp stream from a specific camera, because the rtp payload type the camera is providing is 35 (unassigned) and payload types accepted by the rtph264depay plugin are in range [96-127]. The result is that gstreamer displays ann error like:
<udpsrc0> error: Internal data flow error.
<udpsrc0> error: streaming task paused, reason not-linked (-1)
Other cameras that I have tested are working because they define a good payload type.
FFmpeg, MPlayer and other tools play the stream, although they may display a warning for the unknown type, for instance in Mplayer:
rtsp_session: unsupported RTSP server. Server type is 'unknown'
Is there any way in gstreamer to fake the payload type, ignore the mismatching property, force linking between the plugins or otherwise create a workaroud to my problem?
Pipeline I am using is:
gst-launcg-0.10 rtspsrc location="..." ! rtph264depay ! capsfilter caps="video/x-h264,width=1920,height=1080,framerate=(fraction)25/1" ! h264parse ! matroskamux ! filesink location="test.mkv"
I figured it out and got it working. Posting an answer here in hope that it might benefit someone. There are multiple similar questions out there, but they lack proper answers.
Following does the trick:
GstElement* depay = gst_element_factory_make("rtph264depay", "video_demux");
assert(depay);
GstPad* depay_sink = gst_element_get_static_pad(depay, "sink");
GstCaps* depay_sink_caps = gst_caps_new_simple("application/x-rtp",
"media", G_TYPE_STRING, "video",
"encoding-name", G_TYPE_STRING, "H264",
NULL);
gst_pad_use_fixed_caps(depay_sink);
gst_pad_set_caps(depay_sink, depay_sink_caps);
gst_object_unref(depay_sink);
it overrides the rtph264depay plugin's sink pad caps to be less restrictive, now it accepts any payload type (and any clock-rate) as long as it is rtp and has H.264 encoding.
I don't think this is possible with gst-launch.
There is a select-stream signal in rtspsrc module documented here http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-rtspsrc.html#GstRTSPSrc-select-stream
it's a callback where you check the stream and if you return true, gstreamer will SETUP and PLAY the stream, if you return false it will ignore it, this should let you ignore the unsupported stream, in my case I'm having trouble with ONVIF metadata stream, it always tries to play it and there is no parser for it, I really wish gstreamer will just ignore the streams that can't play and work with what it has or at least a flag to toggle that behaviour.