I'm looking for the correct technique, if one exists, for dynamically replacing an element in a running gstreamer pipeline. I have a gstreamer based c++ app and the pipeline it creates looks like this (using gst-launch syntax) :
souphttpsrc location="http://localhost/local.ts" ! mpegtsdemux name=d ! queue ! mpeg2dec ! xvimagesink d. ! queue ! a52dec ! pulsesink
During the middle of playback (i.e. GST_STATE_PLAYING is the pipeline state and the user is happily watching video), I need to remove souphttpsrc from the pipeline and create a new souphttpsrc, or even a new neonhttpsource, and then immediately add that back into the pipeline and continue playback of the same uri source stream at the same time position where playback was before we performed this operation. The user might see a small delay and that is fine.
We've barely figured out how to remove and replace the source, and we need more understanding. Here's our best attempt thus far:
gst_element_unlink(source, demuxer);
gst_element_set_state(source, GST_STATE_NULL);
gst_bin_remove(GST_BIN(pipeline), source);
source = gst_element_factory_make("souphttpsrc", "src");
g_object_set(G_OBJECT(source), "location", url, NULL);
gst_bin_add(GST_BIN(pipeline), source);
gst_element_link(source, demuxer);
gst_element_sync_state_with_parent(source);
This doesn't work perfectly because the source is playing back from the beginning and the rest of the pipeline is waiting for the correct timestamped buffers (I assume) because after several seconds, playback picks back up. I tried seeking the source in multiple ways but nothing has worked.
I need to know the correct way to do this. It would be nice to know a general technique, if one exists, as well, in case we wanted to dynamically replace the decoder or some other element.
thanks
I think this may be what you are looking for:
http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-block.txt
(starting at line 115)
Related
I'm using appsrc to generate an HLS stream, this is my successful pipeline:
appsrc->videoconvert->openh264enc->h264parse->mpegtsmux->hlssink
However, I'd like to generate some audio via audiotestsrc before mpegtsmux which would look like the following:
audiotestsrc->lamemp3enc->mpegtsmux
Audiotestsrc and lame have 'always' pads, so I link the two just like my other video elements.
When it comes to linking lame's "always" "src", to mpegtsmux's "request" "sink_%d", the result says that there's no issue:
//Returns 0
gst_pad_link(h264ParsePad, mpegtsmuxSinkPad);
//Returns 0
gst_pad_link(audioEncPad, mpegtsmuxSinkPad);
//Returns 0
gst_pad_link(mpegtsmuxSrcPad, hlssinkPad);
But running the app results in pipeline failure with
"Internal data stream error."
Removing the audioEncPad linking just makes the stream work like normal but of course without audio. How should I go about doing this?
Few things needed to be done:
Use aacparse
Clean the solution
Link voaacenc with aacparse
#2 caused me a lot of torment since everything theoretically should've worked. D'oh.
The goal here is to be able to send some metadata (timestamps, objects found per frame) with a stream within a single pipeline or multiple pipelines over network (e.g. RTP UDP).
Within The pipeline:
This is straight forward by defining a new GstMeta API and register and implement it. Then add it to GstBuffers via buffer probes or another element solely designed for this purpose.
Multiple pipelines over Network:
The only solution I have heard about is to add one or more RTP header extension to RTP packets coming out of a payloader. Actually some people decided to transform their Custom GstMeta into those header extensions. But Is This really a good use-case for it?
There's more ....
Another idea is to create your own custom media type video/x-mytype and for that you will need to write a typefinding function and an autoplugger on top of it, Not to mention a couple of elements to deal with this new type and convert it to other exiting media types. Now that new type should actually have a place where it can handle the meta data I was talking about.
That summarizes my research, Are there any other methods that I am not aware of?
I would definitely appreciate your input on this!
Thanks in advance!
I decided to implement a new media type inside gstreamer (e.g. video/x-h265-with-meta). This new media type shall have inherent metadata handling capabilities. You can design how it encode/decode the metadata as you see fit in this new type. You would also need write 2 more gstreamer elements. one to convert regular video/x-h265 to video/x-h265-with-meta (say muxmymeta) and the other to reverse such conversion (say demuxmymeta). So essentially muxmymeta will read the metadata injected in the GstBuffer* via GstMeta API implementation and encodes in the outgoing stream. demuxmymeta will decode such metadata from the stream and add it back to GstBuffer*. To ensure that for every encoded h265 frame will have a corresponding meta, muxmymeta must have its src caps in this format:
video/x-h265
stream-format: byte-stream
alignment: au
The byte-stream is mandatory for network transmission and the au alignment forces a whole encoded frame as input for muxmymeta.
I also used rtpgstpay and rtpgstdepay elements to use this new media type with RTP/RTSP. You will NOT need to write a typefinding function at this point since rtpgstpay/rtpgstdepay will be able to figure that out from the caps. This works well out of the box for h265 streams in gstreamer v1.6.2 but I had to make a fix inside gstreamer itself to make it work with h264 as well. Finally, to make autopluggers such as decodebin/uridecodebin to decode your stream, you must set the klass for demuxmymeta to Codec/Decoder/Demuxer and set a priority for it other than none, this basically tells gstreamer to use this element for decoding.
Here are 2 pipelines for illustration:
Sending H265 RTSP with metadata pipeline (set as the launch line for GstRTSPMediaFactory)
( rtspsrc location=rtsp://localhost:8554/test latency=0 ! rtph265depay ! h265parse ! muxmymeta ! rtpgstpay name=pay0 pt=96 )
Here, the rtsp://localhost:8554/test stream is a regular h265 stream video/x-h265. yet assume that the rtsp-server will produce rtsp://localhost:8553/test-meta which has the type video/x-h265-with-meta
Receiver
rtspsrc location=rtsp://localhost:8553/test-meta ! rtpgstdepay ! demuxmymeta ! h265parse ! avdec_h265 ! videoconvert ! autovideosink
With an Autoplugger it will look as neat as:
uridecodebin uri=rtsp://localhost:8553/test-meta ! autovideosink
Though this might seem like a longshot solution, it provides a way to transport self-synchronizing metadata between pipelines over the network.
UPDATE::
I want to stream video data (H264) through RTSP in Gstreamer.
gst_rtsp_media_factory_set_launch (factory, "videotestsrc ! x264enc ! rtph264pay name=pay0 pt=96 ");
I want "videotestsrc ! x264enc ! rtph264pay name=pay0 pt=96" this pipeline would also be in C programming in place of direct command.
Actually I have custom pipeline, i want to pass this pipeline to GstRTSPMediaFactory.
With launch i am not able to pass my pipline.
source = gst_element_factory_make("videotestsrc", "test-source");
parse = gst_element_factory_make("x264enc", "parse");
sink = gst_element_factory_make("rtph264pay", "sink");
gst_bin_add_many(GST_BIN(pipeline), source, parse, sink, NULL);
gst_element_link_many(source, parse, sink, NULL);
Now, I want to stream this pipeline using RTSP. I can stream with gst_rtsp_media_factory_set_launch,
But i want to pass only pipeline variable, and has to stream the video.
Can it possible, if so How?
I Modified the rtsp-media-factory.c as follows,
Added GstElement *pipeline in struct _GstRTSPMediaFactoryPrivate.
And the Added two more functions get_pipeline & set pipeline
void
gst_rtsp_media_factory_set_launch_pipeline (GstRTSPMediaFactory * factory, GstElement *pipeline)
{
g_print("PRASANTH :: SET LAUNCH PIPELINE\n");
GstRTSPMediaFactoryPrivate *priv;
g_return_if_fail (GST_IS_RTSP_MEDIA_FACTORY (factory));
g_return_if_fail (pipeline != NULL);
priv = factory->priv;
GST_RTSP_MEDIA_FACTORY_LOCK (factory);
// g_free (priv->launch);
priv->pipeline = pipeline;
Bin = priv->pipeline;
GST_RTSP_MEDIA_FACTORY_UNLOCK (factory);
}
In the Same way get also.
And at last in place of gst_parse_launch in function default_create_element,
added this line
element = priv->pipeline; // priv is of type GstRTSPMediaFactoryPrivate
return element;
but I am not able to receive the data.
When i put pay0 for rtpmp2pay it is working.
But it is working for once only. If Client stops and again starts its not working. To work it, again i am restarting the server.
What is the problem?
** (rtsp_server:4292): CRITICAL **: gst_rtsp_media_new: assertion 'GST_IS_ELEMENT (element)' failed
To have some answer here.
It solves the main problem according to comments discussion, but there is still problem with requesting another stream (when stopping and starting client).
The solution was to add proper name for payloader element as stated in docs:
The pipeline description should contain elements named payN, one for each
stream (ex. pay0, pay1, ...). Also, for increased compatibility each stream
should have a different payload type which can be configured on the payloader.
So this has to be changed to:
sink = gst_element_factory_make("rtph264pay", "pay0");
notice the change in name of element from sink -> pay0.
For the stopping client issue I would check if this works for parse version.
If yes then check if the parse pipeline string (in original source code of rtsp server) is saved anywhere and reused after restart.. you need to debug this.
I am designing a pipeline to Encode a video frame from a opencv application (got from a web cam) to video/x-h264 format, send it via network and decode it on another device of different type (probably a raspberry pi ) to a proper RGB stream for my project.
For this I am supposed to use a hardware accelerated Encoder and Decoder.
Since , the whole scenario is huge , the current development is performed on a Intel machine using the gstreamer VAAPI plugins(vaapiencode_h264 & vaapidecode ) . Ánd also, the fact that we need to NOT use any of the networking plugins like TCPServer or UDPServer
For this I have used the below pipeline for my purpose :
On the Encoder End:
appsrc name=applicationSource ! videoconvert ! video/x-raw, format=I420, width=640, height=480,framerate=30/1, pixel-aspect-ratio=1/1,interlace-mode=progressive ! vaapiencode_h264 bitrate=600 tune=high-compression ! h264parse config-interval=1 ! appsink name=applicationSink sync=false
The Appsrc part works perfectly well while the appsink part is having some issue with it.
The appsink part of this pipeline has been set with the below caps:
"video/x-h264, format=(string){avc,avc3,byte-stream },alignment=(string){au,nal};video/mpeg, mpegversion=(int)2, profile=(string)simple"
The code for the data extraction of my appsink is
bool HWEncoder::grabData()
{
// initial checks..
if (!cameraPipeline)
{
GST_ERROR("ERROR AS TO NO PIPE FOUND ... Stopping FRAME GRAB HERE !! ");
return false;
}
if (gst_app_sink_is_eos (GST_APP_SINK(applicationSink)))
{
GST_WARNING("APP SINK GAVE US AN EOS! BAILING OUT ");
return false;
}
if (sample)
{
cout << "sample available ... unrefing it ! "<< endl;
gst_sample_unref(sample);
}
sample = gst_app_sink_pull_sample (GST_APP_SINK(applicationSink));
if (!sample)
{
GST_WARNING("No valid sample");
return false; // no valid sample pulled !
}
sink_buffer = gst_sample_get_buffer(sample);
if (!sink_buffer)
{
GST_ERROR("No Valid Buffer ");return false;
}
return true;
}
After bringing up the pipeline and checking for the buffer filling up in my appsink, I am getting stuck at the below said lines ofmy code indefinitely:
sample = gst_app_sink_pull_sample (GST_APP_SINK(applicationSink));
I have the following questions :
1) Is my Caps for appsink correct ? If not How can I determine the caps for them ?
2) Is there something wrong in my pipeline above ?
How can I fix this issue with Appsink ??
Any kind of help would be useful!
Thanks !!
Just a guess (I had similar problems) the problem having appsink and appsrc in same pipeline may be that when you fill/empty one of them it will block the other(more on that below).
appsink and appsrc would block when they are full/empty - this is normal desired behaviour. There is option drop for appsink or for appsrc there is option block - but using these it may be just workaround and you will get glitches in your stream. Proper solution is to handle the synchronisation between appsrc and appsink in a better way.
You can react on appsrc signals enough-data and need-data - this is our way. Also we fiddled with properties of appsrc: is-live, do-timestamp and buffer size (this may or may not help you):
g_object_set(src->appsrc,
"stream-type", GST_APP_STREAM_TYPE_STREAM,
"format", GST_FORMAT_TIME,
"do-timestamp", TRUE,
"is-live", TRUE,
"block", TRUE,
NULL);
Why do they block each other?
Because (I guess) you process appsink and at the same time appsrc in main application thread. When one of the appsink/appsrc block the thread there is no one that would handle the processing for the other one. So when appsink is blocked because it does not have any data there is noone that can feed appsrc with new data - thus endless deadlock.
We also implemented noblock version of appsink *pull_sample method but it was just a workaround and resulted in more problems than solutions.
If you want to debug what is happening you can add GST_DEBUG entry for appsrc/appsink (I do not remember what they were), you can add callback on mentioned enough-data and need-data signals or you may add queues and enable GST_DEBUG=queue_dataflow:5 to see which queue is filled first etc.. this is always helpful when debugging the "data-deadlock".
I'm constructing a gstreamer pipeline that receives two RTP streams from an networked source:
ILBC Audio stream + corresponding RTCP stream
H263 Video stream + corresponding RTCP stream
Everything is put into one gstreamer pipeline so it will use the RTCP from both streams to synchronize audio/video. So far I've come up with this (using gst-launch for prototyping):
gst-launch -vvv gstrtpbin name=rtpbin
udpsrc caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H263-2000" port=40000 ! rtpbin.recv_rtp_sink_0
rtpbin. ! rtph263pdepay ! ffdec_h263 ! xvimagesink
udpsrc port=40001 ! rtpbin.recv_rtcp_sink_0
rtpbin.send_rtcp_src_0 ! udpsink port=40002 sync=false async=false
udpsrc caps="application/x-rtp,media=(string)audio,clock-rate=(int)8000,encoding-name=(string)PCMU,encoding-params=(string)1,octet-align=(string)1" port=60000 rtpbin.recv_rtp_sink_1
rtpbin. ! rtppcmudepay ! autoaudiosink
udpsrc port=60001 ! rtpbin.recv_rtcp_sink_1
rtpbin.send_rtcp_src_1 ! udpsink port=60002 sync=false async=false
This pipeline works well if the networked source starts out with sending both video and audio. If the videostream is paused later on, gstreamer will still playback audio and even will start playing back the video when the networked source resumes the video stream.
My problem is however that if the networked source starts out with only an audio stream (video might be added later on), the pipeline seems to pause/freeze until the video stream starts as well.
Since video is optional (and can be added/removed at will by the user) in my application, is there any way I can hook up for instance an 'videotestsrc' that will provide some kind of fallback video data to keep the pipeline running when there is no networked video data?
I've tried experimenting with 'videotestsrc' and a thing called 'videomixer' but I think that mixer still requires both streams to be alive. Any feedback is greatly appreciated!
I present a simple function for pause resume by changing bins. In the following example I provide the logic to change destination bin on the fly dynamically. This shall not completely stop the pipeline which is what you seek I believe. A similar logic could be used for src bins. Here you may remove your network source bin and related decoder/demux bins and add videotestsrc bins.
private static void dynamic_bin_replacement(Pipeline pipe, Element src_bin, Element dst_bin_new, Element dst_bin_old) {
pipe.pause();
src_bin.unlink(dst_bin_old);
pipe.remove(dst_bin_old);
pipe.add(dst_bin_new);
dst_bin_new.syncStateWithParent();
src_bin.link(dst_bin_new);
pipe.ready();
pipe.play();
}
The other logic you may want to try is "PADLOCKING". Please take a look at the following posts
http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-block.txt
and
http://web.archiveorange.com/archive/v/8yxpz7FmOlGqxVYtkPb4
and
Adding and removing audio sources to/from GStreamer pipeline on-the-go
UPDATE
Try output-selector and input-selector bins as they seem to be better alternative. I found them most reliable and have had immense luck with them. I use fakesink or fakesrc respectively as the other end of the selector.
valve bin is another alternative that I found doesn't even need fakesink or fakesrc bins. It is also extremely reliable.
Also the correct state transition order for media file source
NULL -> READY -> PAUSED -> PLAYING (Upwards)
PLAYING -> PAUSED -> READY -> NULL (Downwards)
My order in the above example should be corrected where ready() should come before pause(). Also I would tend to think un-linking should be performed after null() state and not after pause(). I haven't tried these changes but theoretically they should work.
See the following link for detailed info
http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-states.txt?h=BRANCH-RELEASE-0_10_19