GStreamer c++ - seeking before starting any playback - c++

I added an start-offset option to my C++ application using the GStreamer library. It basically tells the application to start mid-video instead of playing from the beginning.
The pipeline is created like this: filesrc location="file_path" ! qtdemux ! h264parse ! mppvideodec ! waylandsink. This is what gives me best results on my hardware, and I'm running under Weston (wayland)
I do the seeking with a simple gst_element_seek on the pipeline. First I tried waiting until the pipeline changes state to PLAYING or PAUSED in the bus, and then seeking once. This worked, but on some larger videos it produced an effect that after launching the app, the first video frame is displayed for a glimpse of a second and then the video actually jumps to the desired position. I don't want that, I'd like to jump right to where I want without displaying anything else earlier.
So I changed my seeking method to happen before I even set the pipeline to PLAYING state like this:
// seek if needed
if(startTime > 0) {
// First set pipeline to PAUSED and wait for async state change
gst_element_set_state(m_pipeline, GST_STATE_PAUSED);
gst_element_get_state(m_pipeline, nullptr, nullptr, GST_CLOCK_TIME_NONE);
if (!gst_element_seek (m_pipeline, 1.0, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH, GST_SEEK_TYPE_SET, startTime, GST_SEEK_TYPE_NONE, -1)) {
g_print ("Seek failed!\n");
}
gst_element_get_state(m_pipeline, nullptr, nullptr, GST_CLOCK_TIME_NONE); // wait for seek to finish and then set to PLAYING
}
m_ret = gst_element_set_state(m_pipeline, GST_STATE_PLAYING);
if (m_ret == GST_STATE_CHANGE_FAILURE) {
g_printerr("Unable to set the pipeline to the playing state.\n");
gst_object_unref(m_pipeline);
exit(0);
}
And while this also works, it unfortunately produces the exact same effect as before...the first frame is still being displayed even though the seek happens before the pipeline is started.
How can I ensure that nothing is displayed until the seek is actually done?
Update
I couldn't get any method of seeking to work like I wanted, but I achieved the effect by setting the show-preroll-frame on the GstVideoSink before constructing the pipeline, like that:
m_sink = gst_element_factory_make("waylandsink", "sink");
g_object_set(m_sink, "show-preroll-frame", FALSE, NULL); // prevent preroll frame when seeking
And it works! I'll use it for now if there's no better way. The seek is still done in PAUSED state, but video is displayed directly from the seeking point.

As the documentation of the gst_element_seek_simple method says:
Some elements allow for seeking in the READY state...
This is true if the pipeline is completely prerolled before the seek.
Then you can get the result you expect, showing only the frames you expect.
gst_element_set_state(m_pipeline, GST_STATE_READY);
//make sure all the pipeline is prerolled (dynamic pads linked and other stuff)
if (!gst_element_seek (m_pipeline, 1.0, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH, GST_SEEK_TYPE_SET, startTime, GST_SEEK_TYPE_NONE, -1)) {
g_print ("Seek failed!\n");
}
gst_element_get_state(m_pipeline, nullptr, nullptr, GST_CLOCK_TIME_NONE);

Related

Gstreamer : proper way to release resources of a pipeline?

I'm building a pipeline in Gstreamer, using a UriDecodeBin. This pipeline is meant to be build and released repeatedly, but I have a hard time to figure out how to properly delete the pipeline.
For now, I'm trying to figure out what to unref and what not to unref to get all resources of a pipeline to be released, with just a UriDecodeBin in the pipeline, but I can't get the full memory to be released somehow.
My sequence of actions is as follows :
Fist, I'm building the pipeline and UriDecodeBin :
_pipeline = (GstPipeline*)gst_pipeline_new("pipeline");
_bus = gst_pipeline_get_bus(_pipeline); //will be manually observed
_decoder = gst_element_factory_make("uridecodebin", "decoder");
gst_bin_add(GST_BIN(_pipeline), _decoder);
gst_element_sync_state_with_parent(_decoder);
std::string uri = std::string("file://some_video_file");
g_object_set(_decoder, "uri", uri.c_str(), nullptr);
g_signal_connect(_decoder, "no-more-pads", G_CALLBACK(GsDecoder::no_more_pads_cb), (gpointer)this);
changePipelineState(GST_STATE_PAUSED);
Then, I wait for the no-more-pads to trigger, and add a blocking probe on each pad :
GstIterator *it = gst_element_iterate_src_pads(_decoder);
GValue padV = G_VALUE_INIT;
while(gst_iterator_next(it, &padV) == GST_ITERATOR_OK){
GstPad* pad = (GstPad*)g_value_get_object(&padV);
GstCaps* caps = gst_pad_get_current_caps(pad);
if(caps_is_audio(caps)){
_decoderAudioSrcs.push_back(pad); //adding pad to a collection
gulong id = gst_pad_add_probe(pad, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, GsDecoder::pad_blocked_cb, (gpointer)this, NULL); //blocking probe
_decoderAudioProbeIds.push_back(id); //remembering id of the probe, for later
} else if (caps_is_video(caps)){
//same with video but file is guaranteed to have a single video track.
_decoderVideoSrc = pad;
_decoderVideoProbeId = gst_pad_add_probe(pad, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, GsDecoder::pad_blocked_cb, (gpointer)this, NULL);
} else {
std::cout << "UNKNOWN PAD FOUND" << std::endl;
}
gst_caps_unref(caps); //releasing caps
g_value_unset(&padV);
}
At this point I wait that all probes block the pads, and then I unblock the pads :
for (unsigned i = 0; i < _decoderAudioSrcs.size(); i++){
gst_pad_remove_probe(_decoderAudioSrcs[i], _decoderAudioProbeIds[i]);
}
gst_pad_remove_probe(_decoderVideoSrc, _decoderVideoProbeId);
play(); //setting the pipeline to play
Finally, after a bit of time, I order the pipeline to delete itself, and there I am a bit confused about what shall be deleted or not (both in this pipeline and in general).
How shall the pipeline be deleted ? From what I understood, I have to set pipeline state to Null, and then unref everything (pads, decoder, bus, pipeline) using g_object_unref, however it still leads to pipeline having few refs pointing to it (in my case, 3), and not releasing its resources, would it be memory nor threads.
Is there a way to tell Gstreamer to fully delete a pipeline and everthing related to it, without having to unref everything by hand ?
Especially, I would be interested in a pointer to some documentation, since I couldn't found any on the subject (even if Gstreamer doc mentions unrefing time to time).
GStreamer objects are reference counted. Whenever the reference drops to 0 the object gets deleted. The GStreamer documentation usually tells you for a function about its transfer behavior. E.g.
gst_pad_get_current_caps()
https://gstreamer.freedesktop.org/documentation/gstreamer/gstpad.html#gst_pad_get_current_caps
Tells you the transfer is full and you are supposed to release it when you make no more use it (you do so in your sample code). If you don't do it, it will never be deleted.
So the typical use case is if-i-don't-need-it-anymore-i-release-it. That does not mean that it gets deleted. For example, if you put an element into a pipeline, the pipeline will add a reference to it. It will take care of the reference and release it once the pipeline's reference count drops to 0. But since you added it to the pipeline and your code itself does not need it anymore you would usually just release it.
So you need to the careful about the references. If you miss one tiny small object which may be part of a big pipeline it may result in many objects to not be released as they depend on each other.

Memory not freed after gstreamer pipeline set to GST_STATE_NULL

My application requires gstreamer pipeline to be restarted multiple times.
But after setting the pipeline to GST_STATE_NULL and calling unref on the pipeline, memory appears to be not freed.
After every restart, the memory associated with the process keeps increasing.
I was able to reproduce the problem with just videotestsrc-fakesink elements as given below :
//g++ -Wall testpage_Simple.cpp -o testpage_Simple $(pkg-config --cflags --libs gstreamer-1.0)
#include <gst/gst.h>
GstElement *pipeline;
GstElement *src;
GstElement *sink;
void clearPipeline () {
// g_print ("clearPipeline ");
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (pipeline);
}
void createPipeline () {
pipeline = gst_pipeline_new ("pipelinePlay");
src = gst_element_factory_make ("videotestsrc", "source");
sink = gst_element_factory_make ("fakesink", "sink");
gst_bin_add_many (GST_BIN (pipeline), src, sink, NULL);
if (gst_element_link (src, sink)!= TRUE) {
g_printerr ("src, sink could not be linked.\n");
}
gst_element_set_state (pipeline, GST_STATE_PLAYING);
}
gint main (gint argc, gchar * argv[])
{
gst_init (NULL, NULL);
system("gst-launch-1.0 --gst-version");
g_print ("Start Test - ");
// for (int i=1; i<=10; i++) {
system ("top -b -n 1 | grep testpage | awk '{print $6}'");
createPipeline();
clearPipeline();
// }
g_print ("End of test !! ");
system ("top -b -n 1 | grep testpage | awk '{print $6}'");
gst_deinit();
return 0;
}
Sample output on Ubuntu 19.04 (showing only RES column value from top command for this process):
GStreamer Core Library version 1.16.1
Start Test - 7140
End of test !! 8504
We observed similar memory trend on Ubuntu 18.04 with gstreamer 1.12.4 as well.
Is this the expected behavior or did I miss something in cleaning up the pipeline ??
I checked the below link. It appears to be a similar problem, but this question is unanswered
GStreamer memory leak after the pipeline restart
I tried the disk cache suggestion in the below link, with a similar problem. But that did not help as well.
http://gstreamer-devel.966125.n4.nabble.com/Properly-freeing-resources-td4658631.html
Just a guess, maybe the problem is related to possible state transitions for elements
according
https://gstreamer.freedesktop.org/documentation/additional/design/states.html?gi-language=c#
the following state changes are possible:
NULL -> READY:
The element must check if the resources it needs are available. Device sinks and sources typically try to probe the device to constrain their caps.
The element opens the device, this is needed if the previous step requires the device to be opened.
READY -> PAUSED:
The element pads are activated in order to receive data in PAUSED. Streaming threads are started.
Some elements might need to return ASYNC and complete the state change when they have enough information. It is a requirement for sinks to return ASYNC and complete the state change when they receive the first buffer or EOS event (preroll). Sinks also block the dataflow when in PAUSED.
A pipeline resets the running_time to 0.
Live sources return NO_PREROLL and don't generate data.
PAUSED -> PLAYING:
Most elements ignore this state change.
The pipeline selects a clock and distributes this to all the children before setting them to PLAYING. This means that it is only allowed to synchronize on the clock in the PLAYING state.
The pipeline uses the clock and the running_time to calculate the base_time. This base_time is distributed to all children when performing the state change.
Sink elements stop blocking on the preroll buffer or event and start rendering the data.
Sinks can post the EOS message in the PLAYING state. It is not allowed to post EOS when not in the PLAYING state.
While streaming in PAUSED or PLAYING elements can create and remove sometimes pads.
Live sources start generating data and return SUCCESS.
PLAYING -> PAUSED:
Most elements ignore this state change.
The pipeline calculates the running_time based on the last selected clock and the base_time. It stores this information to continue playback when going back to the PLAYING state.
Sinks unblock any clock wait calls.
When a sink does not have a pending buffer to play, it returns ASYNC from this state change and completes the state change when it receives a new buffer or an EOS event.
Any queued EOS messages are removed since they will be reposted when going back to the PLAYING state. The EOS messages are queued in GstBins.
Live sources stop generating data and return NO_PREROLL.
PAUSED -> READY:
Sinks unblock any waits in the preroll.
Elements unblock any waits on devices
Chain or get_range() functions return FLUSHING.
The element pads are deactivated so that streaming becomes impossible and all streaming threads are stopped.
The sink forgets all negotiated formats
Elements remove all sometimes pads
READY -> NULL:
Elements close devices
Elements reset any internal state.
if the current state is playing then st. like
gst_element_set_state (pipeline, GST_STATE_PAUSED);
gst_element_set_state (pipeline, GST_STATE_READY);
gst_element_set_state (pipeline, GST_STATE_NULL);
might help
Checking the return value of gst_element_set_state(...) might not be a bad idea either :)

How to set a GstPlayer pipeline?

I have constructed a custom GStreamer pipeline that I will use to play RTSP streams. At the same time I'd like to create a new GstPlayer to use this pipeline. The problem is that there isn't a way that I can see to set a GstPlayer's pipeline (the only related method is gst_player_get_pipeline(). I don't understand how there is no way to customize a pipeline for a GstPlayer. This seems like basic functionality, so I must be missing something.
My pipeline:
GstElement *pipeline, *source, *filter, *sink;
// Create pipeline elements
pipeline = gst_pipeline_new ("vdi-pipeline");
source = gst_element_factory_make ("rtspsrc", "vdi-source");
filter = gst_element_factory_make ("decodebin", "vdi-filter");
sink = gst_element_factory_make ("appsink", "vdi-sink");
if (!source || !filter || !sink)
{
__android_log_print (ANDROID_LOG_ERROR, "Error", "A GstElement could not be created. Exiting.");
return;
}
// Add elements to pipeline
gst_bin_add_many (GST_BIN (pipeline), source, filter, sink, NULL);
// Link elements together
if (!gst_element_link_many (source, filter, sink, NULL)) {
__android_log_print (ANDROID_LOG_ERROR, "Warning", "Failed to link elements!");
}
But you can play rtsp via GstPlayer out of the box.. why do you want custom pipeline?
The player is using playbin which accept any kind of url.. and it will create pipeline dynamically according to what is being played..
What about patching the player itself, if you really cannot use playbin? I dont think it is intended for custom pipelines.. but you can hack it here.
You will then have hook the newpads and other callback on the rtspsrc instead of playbin.. and other stuff - I guess you do not want this.
The other way is - when the playbin constructs pipeline it uses rtspsrc inside - you can get this element from pipeline object and change some parameters.. but be carefull as changing parameters during playback is very tricky..
UPDATE:
Hm I think I overlook the appsink somehow.. well I think you can set playbin property audio-sink or video-sink to override it to use appsink.
But still you will have to somehow get the playbin element out of GstPlayer or set the playbin parameter upon initialization (I dont know how) - in this case I would ask on IRC (freenode, #gstreamer) if you are going the right direction.
Maybe better way would be to create your own application using decodebin and or even playbin and pass there the appsink element.. why do you want to use GstPlayer if you are not playing but processing buffers?
HTH

Gstreamer Appsink not getting Data from the Pipeline

I am designing a pipeline to Encode a video frame from a opencv application (got from a web cam) to video/x-h264 format, send it via network and decode it on another device of different type (probably a raspberry pi ) to a proper RGB stream for my project.
For this I am supposed to use a hardware accelerated Encoder and Decoder.
Since , the whole scenario is huge , the current development is performed on a Intel machine using the gstreamer VAAPI plugins(vaapiencode_h264 & vaapidecode ) . Ánd also, the fact that we need to NOT use any of the networking plugins like TCPServer or UDPServer
For this I have used the below pipeline for my purpose :
On the Encoder End:
appsrc name=applicationSource ! videoconvert ! video/x-raw, format=I420, width=640, height=480,framerate=30/1, pixel-aspect-ratio=1/1,interlace-mode=progressive ! vaapiencode_h264 bitrate=600 tune=high-compression ! h264parse config-interval=1 ! appsink name=applicationSink sync=false
The Appsrc part works perfectly well while the appsink part is having some issue with it.
The appsink part of this pipeline has been set with the below caps:
"video/x-h264, format=(string){avc,avc3,byte-stream },alignment=(string){au,nal};video/mpeg, mpegversion=(int)2, profile=(string)simple"
The code for the data extraction of my appsink is
bool HWEncoder::grabData()
{
// initial checks..
if (!cameraPipeline)
{
GST_ERROR("ERROR AS TO NO PIPE FOUND ... Stopping FRAME GRAB HERE !! ");
return false;
}
if (gst_app_sink_is_eos (GST_APP_SINK(applicationSink)))
{
GST_WARNING("APP SINK GAVE US AN EOS! BAILING OUT ");
return false;
}
if (sample)
{
cout << "sample available ... unrefing it ! "<< endl;
gst_sample_unref(sample);
}
sample = gst_app_sink_pull_sample (GST_APP_SINK(applicationSink));
if (!sample)
{
GST_WARNING("No valid sample");
return false; // no valid sample pulled !
}
sink_buffer = gst_sample_get_buffer(sample);
if (!sink_buffer)
{
GST_ERROR("No Valid Buffer ");return false;
}
return true;
}
After bringing up the pipeline and checking for the buffer filling up in my appsink, I am getting stuck at the below said lines ofmy code indefinitely:
sample = gst_app_sink_pull_sample (GST_APP_SINK(applicationSink));
I have the following questions :
1) Is my Caps for appsink correct ? If not How can I determine the caps for them ?
2) Is there something wrong in my pipeline above ?
How can I fix this issue with Appsink ??
Any kind of help would be useful!
Thanks !!
Just a guess (I had similar problems) the problem having appsink and appsrc in same pipeline may be that when you fill/empty one of them it will block the other(more on that below).
appsink and appsrc would block when they are full/empty - this is normal desired behaviour. There is option drop for appsink or for appsrc there is option block - but using these it may be just workaround and you will get glitches in your stream. Proper solution is to handle the synchronisation between appsrc and appsink in a better way.
You can react on appsrc signals enough-data and need-data - this is our way. Also we fiddled with properties of appsrc: is-live, do-timestamp and buffer size (this may or may not help you):
g_object_set(src->appsrc,
"stream-type", GST_APP_STREAM_TYPE_STREAM,
"format", GST_FORMAT_TIME,
"do-timestamp", TRUE,
"is-live", TRUE,
"block", TRUE,
NULL);
Why do they block each other?
Because (I guess) you process appsink and at the same time appsrc in main application thread. When one of the appsink/appsrc block the thread there is no one that would handle the processing for the other one. So when appsink is blocked because it does not have any data there is noone that can feed appsrc with new data - thus endless deadlock.
We also implemented noblock version of appsink *pull_sample method but it was just a workaround and resulted in more problems than solutions.
If you want to debug what is happening you can add GST_DEBUG entry for appsrc/appsink (I do not remember what they were), you can add callback on mentioned enough-data and need-data signals or you may add queues and enable GST_DEBUG=queue_dataflow:5 to see which queue is filled first etc.. this is always helpful when debugging the "data-deadlock".

dynamically replacing elements in a playing gstreamer pipeline

I'm looking for the correct technique, if one exists, for dynamically replacing an element in a running gstreamer pipeline. I have a gstreamer based c++ app and the pipeline it creates looks like this (using gst-launch syntax) :
souphttpsrc location="http://localhost/local.ts" ! mpegtsdemux name=d ! queue ! mpeg2dec ! xvimagesink d. ! queue ! a52dec ! pulsesink
During the middle of playback (i.e. GST_STATE_PLAYING is the pipeline state and the user is happily watching video), I need to remove souphttpsrc from the pipeline and create a new souphttpsrc, or even a new neonhttpsource, and then immediately add that back into the pipeline and continue playback of the same uri source stream at the same time position where playback was before we performed this operation. The user might see a small delay and that is fine.
We've barely figured out how to remove and replace the source, and we need more understanding. Here's our best attempt thus far:
gst_element_unlink(source, demuxer);
gst_element_set_state(source, GST_STATE_NULL);
gst_bin_remove(GST_BIN(pipeline), source);
source = gst_element_factory_make("souphttpsrc", "src");
g_object_set(G_OBJECT(source), "location", url, NULL);
gst_bin_add(GST_BIN(pipeline), source);
gst_element_link(source, demuxer);
gst_element_sync_state_with_parent(source);
This doesn't work perfectly because the source is playing back from the beginning and the rest of the pipeline is waiting for the correct timestamped buffers (I assume) because after several seconds, playback picks back up. I tried seeking the source in multiple ways but nothing has worked.
I need to know the correct way to do this. It would be nice to know a general technique, if one exists, as well, in case we wanted to dynamically replace the decoder or some other element.
thanks
I think this may be what you are looking for:
http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-block.txt
(starting at line 115)