Gstreamer. Write appsink to filesink - c++

I have written a code for appsrc to appsink and it works. I see the actual buffer. It's encoded in H264(vpuenc=avc). Now I want to save it in a file(filesink). How I approach it?
app:
int main(int argc, char *argv[]) {
gst_init (NULL, NULL);
GstElement *pipeline, *sink;
gchar *descr;
GError *error = NULL;
GstAppSink *appsink;
descr = g_strdup_printf (
"mfw_v4lsrc device=/dev/video1 capture_mode=0 ! " // grab from mipi camera
"ffmpegcolorspace ! vpuenc codec=avc ! "
"appsink name=sink"
);
pipeline = gst_parse_launch (descr, &error);
if (error != NULL) {
g_print ("could not construct pipeline: %s\n", error->message);
g_error_free (error);
exit (-1);
}
gst_element_set_state(pipeline, GST_STATE_PAUSED);
sink = gst_bin_get_by_name (GST_BIN (pipeline), "sink");
appsink = (GstAppSink *) sink;
gst_app_sink_set_max_buffers ( appsink, 2); // limit number of buffers queued
gst_app_sink_set_drop( appsink, true ); // drop old buffers in queue when full
gst_element_set_state (pipeline, GST_STATE_PLAYING);
int i = 0;
while( !gst_app_sink_is_eos(appsink) )
{
GstBuffer *buffer = gst_app_sink_pull_buffer(appsink);
uint8_t* data = (uint8_t*)GST_BUFFER_DATA(buffer);
uint32_t size = GST_BUFFER_SIZE(buffer);
gst_buffer_unref(buffer);
}
return 0; }

If as mentioned in the comments, what you actually want to know is how to do a network video stream in GStreamer, you should probably close this question because you're on the wrong path. You don't need to use an appsink or filesink for that. What you'll want to investigate are the GStreamer elements related to RTP, RTSP, RTMP, MPEGTS, or even MJPEGs (if your image size is small enough).
Here are two basic send/receive video stream pipelines:
gst-launch-0.10 v4l2src ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=640,height=480 ! vpuenc ! h264parse ! rtph264pay ! udpsink host=localhost port=5555
gst-launch-0.10 udpsrc port=5555 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! ffdec_h264 ! videoconvert ! ximagesink

In this situation you don't write your own while loop. You register callbacks and wait for buffers (GStreamer 0.10) to arrive. If you're using GStreamer 1.0, you use samples instead of buffers. Samples are a huge pain in the ass compared to buffers but oh well.
Register the callback:
GstAppSinkCallbacks* appsink_callbacks = (GstAppSinkCallbacks*)malloc(sizeof(GstAppSinkCallbacks));
appsink_callbacks->eos = NULL;
appsink_callbacks->new_preroll = NULL;
appsink_callbacks->new_sample = app_sink_new_sample;
gst_app_sink_set_callbacks(GST_APP_SINK(appsink), appsink_callbacks, (gpointer)pointer_to_data_passed_to_the_callback, free);
And your callback:
GstFlowReturn app_sink_new_sample(GstAppSink *sink, gpointer user_data) {
prog_data* pd = (prog_data*)user_data;
GstSample* sample = gst_app_sink_pull_sample(sink);
if(sample == NULL) {
return GST_FLOW_ERROR;
}
GstBuffer* buffer = gst_sample_get_buffer(src);
GstMemory* memory = gst_buffer_get_all_memory(buffer);
GstMapInfo map_info;
if(! gst_memory_map(memory, &map_info, GST_MAP_READ)) {
gst_memory_unref(memory);
gst_sample_unref(sample);
return GST_FLOW_ERROR;
}
//render using map_info.data
gst_memory_unmap(memory, &map_info);
gst_memory_unref(memory);
gst_sample_unref(sample);
return GST_FLOW_OK;
}
You can keep your while loop as it is--using gst_app_sink_is_eos()--but make sure to put a sleep in it. Most of the time I use something like the following instead:
GMainLoop* loop = g_main_loop_new(NULL, FALSE);
g_main_loop_run(loop);
g_main_loop_unref(loop);
Note: Unless you need to do something special with the data you can use the "filesink" element directly.

Simpler option would be write to the file directly in the appsink itself ie when you get a callback when the buffer is done write to the file and make sure you close it on eos.
Hope that helps.

Related

GStreamer Appsrc: how do I get the negotiated caps?

I am currently using an appsrc element in a program I am writing.
I let the user provide a launch string such as this example:
appsrc name=mysource format=3 is-live=1 \
! video/x-raw(memory:NVMM), width=5400, height=3400, framerate=30/1, format=NV12 \
! nvv4l2h265enc bitrate=8000000 control-rate=0 iframeinterval=2 \
! h265parse \
! matroskamux \
! filesink location=myfile.mkv
The launch string could be anything, provided it has an appsrc called mysource.
In my program, I locate mysource and I would like to know the format property that was provided by the user (to create the right kind of data buffer). I query the src pad of my appsrc element:
GstPad * pad = gst_element_get_static_pad(m_appsrc, "src");
if (!pad) fprintf(stderr, "pad is null\n");
GstCaps * caps = gst_pad_get_current_caps(pad);
if (!caps) fprintf(stderr, "caps is null\n");
for (guint i = 0; i < gst_caps_get_size (caps); i++) {
GstStructure *structure = gst_caps_get_structure (caps, i);
g_print ("%s%s\n", " ", gst_structure_get_name (structure));
gst_structure_foreach (structure, print_field, (gpointer) " ");
}
pad is non-null, but caps is always returned as NULL.
Looking at the .dot graph created by GST_DEBUG_BIN_TO_DOT, I'm seeing that the caps indicated on the output of the appsrc are "ANY".
Do I have to do something special like traversing the pipeline to get the final negotiated caps?
Looks like if I specify a caps property in the appsrc launch string, caps is not NULL and can be traversed to describe the user-specified capabilities. This is good enough for me:
appsrc name=mysource format=3 is-live=1 caps="video/x-raw(memory:NVMM), width=5400, height=3400, framerate=30/1, format=NV12" \
! nvv4l2h265enc bitrate=8000000 control-rate=0 iframeinterval=2 \
! h265parse \
! matroskamux \
! filesink location=myfile.mkv

How can I write Open CV frames to an RTSP stream with Gstreamer and C++?

I'm trying to take a video frame into OpenCV, do some processing on it (to be exact, aruco detection) and then package the resultant frame into a RTSP stream with GStreamer.
I've seen a Python solution to this problem, but I'm having trouble translating it to C++.
Here's my attempt at recreating the SensorFactory class:
#include <glib-object.h>
#include <iostream>
#include "SensorFactory.h"
SensorFactory::SensorFactory(std::string launch) {
launchString = launch;
cap = cv::VideoCapture(0);
// should be incremented once on each frame for timestamping
numberFrames = 0;
// simple struct with only the cap (int*), lastFrame (cv::Mat*) and numberFrames (int* again) fields
CVData cvData;
cvData.cap = ∩
cvData.lastFrame = &lastFrame;
cvData.numberFrames = &numberFrames;
}
GstFlowReturn SensorFactory::on_need_data(GstElement *src, CVData *datum) {
if (datum->cap->isOpened()) {
if (datum->cap->read(*(datum->lastFrame))) {
std::string data = std::string(reinterpret_cast<char * > (datum->lastFrame->data));
GstBuffer *buf = gst_buffer_new_allocate(nullptr, data.max_size(), nullptr);
gst_buffer_fill(buf, 0, &data, data.max_size());
buf->duration = static_cast<GstClockTime>(duration);
GstClockTimeDiff timestamp = *(datum->numberFrames) * duration;
buf->pts = buf->dts = static_cast<GstClockTime>(timestamp);
buf->offset = static_cast<guint64>(timestamp);
int *numf = datum->numberFrames;
*numf += 1;
g_signal_emit_by_name(src, "push-buffer", buf);
gst_buffer_unref(buf);
return GST_FLOW_OK;
}
}
// never reached
return GST_FLOW_NOT_LINKED;
}
GstElement *SensorFactory::create_element(const GstRTSPUrl *url) { return gst_parse_launch(launchString.c_str(), nullptr); }
void SensorFactory::configure(GstRTSPMedia *rtspMedia) {
numberFrames = 0;
GstElement *appsrc;
appsrc = gst_rtsp_media_get_element(rtspMedia);
g_signal_connect(appsrc, "need-data", (GCallback) on_need_data, &cvData);
}
The header for SensorFactory is nothing special:
#include <gst/rtsp-server/rtsp-media-factory.h>
#include <gst/rtsp-server/rtsp-media.h>
#include <gst/app/gstappsrc.h>
#include <opencv2/videoio.hpp>
class SensorFactory : public GstRTSPMediaFactory {
public:
typedef struct _CVData {
cv::VideoCapture *cap;
cv::Mat *lastFrame;
int *numberFrames;
} CVData;
CVData cvData;
std::string launchString;
cv::VideoCapture cap;
cv::Mat lastFrame;
int numberFrames = 0;
const static int framerate = 30;
const static GstClockTimeDiff duration = 1 / framerate * GST_SECOND;
explicit SensorFactory(std::string launch);
static GstFlowReturn on_need_data(GstElement *src, CVData *datum);
GstElement *create_element(const GstRTSPUrl *url);
void configure(GstRTSPMedia *media);
};
And then main.cpp looks like so:
#include <gst/gst.h>
#include "src/SensorFactory.h"
int main() {
gst_init(nullptr, nullptr);
GstRTSPServer *server;
server = gst_rtsp_server_new();
SensorFactory sensorFactory("appsrc name=source is-live=true block=true format=GST_FORMAT_TIME"
"caps=video/x-raw,format=BGR ! "
"videoconvert ! video/x-raw,format=I420 ! "
"x264enc speed-preset=ultrafast tune=zerolatency ! rtph264pay name=pay0");
g_print("setting shared\n");
gst_rtsp_media_factory_set_shared(&sensorFactory, true);
g_print("set shared\n");
GstRTSPMountPoints *mounts;
mounts = gst_rtsp_server_get_mount_points(server);
gst_rtsp_mount_points_add_factory(mounts, "/test", &sensorFactory);
GMainLoop *loop;
loop = g_main_loop_new(nullptr, false);
g_main_loop_run(loop);
}
The program compiles fine, and will even start running, but segfaults on gst_rtsp_media_factory_set_shared(&sensorFactory, true);. There isn't any other hacky memory management in this program.
You can try the steps below to write the stream as RTMP.
if (platform is "Windows") {
// if the platform is windows, then add the head data of the video
// otherwise it will not work on the HTML flash player
headData = " ! video/x-h264,profile=high";
}
// to rtmp (media server e.g: NGINX)
rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! rtmpsink location=rtmp://192.168.1.25/mylive/test";
// using UDP broadcast to all 1~255 IPs
rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! udpsink host=192.168.1.255 port=5000";
// using UDP broadcast specific IP
rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! udpsink host=192.168.1.25 port=5000";
// give the FPS and the size of the video
VideoWriter writer = new VideoWriter(rtmpUrl, Videoio.CAP_GSTREAMER, FOURCC, currentFps, new Size(width, height));
// then you can write the video using writer
NOTE: Make sure you build OpenCV with GStreamer.
Here is an alternative approach.
Seperate your SensorFactory from the rtsp code for now.
Start your SensorFactory with the pipeline.
appsrc name=source is-live=true block=true format=GST_FORMAT_TIME caps=video/x-raw,format=BGR,width=640,height=480,framerate=30/1 ! videoconvert ! video/x-raw,format=I420 ! x264enc speed-preset=ultrafast tune=zerolatency ! udpsink port=5050
We end that pipeline by piping the h264 over a udpsink on port 5050.
Then compile the gstreamer rtsp server example here
And launch that with pipeline
./test-launch "( udpsrc port=5050 ! rtph264pay name=pay0 pt=96 )"
Assuming your SensorFactory works as you intend, this should get you an RTSP Stream serving at rtsp://localhost:8554/test

How to get the correct video frame number or time position from appsink buffer

Thanks in advance.
I want to record a video from an rtsp video camera and at the same time
process the video frame obtained from appsink throught new-sample signal.
Then, in a separate application I read the recorded video and show the information related to the frames processed.
Documentation say that buffer->offset have the video frame number, but doesnt work for me, it allways have the same number.
I have this pipeline:
rtspsrc location=rtsp://10.0.0.1:554/video.sdp latency=100 ! rtph264depay ! tee name=t
! queue ! vaapidecodebin ! vaapipostproc format=rgba ! appsink name=appsink t.
! queue ! h264parse ! mp4mux ! filesink sync=false name=filer location=/home/VideoDB/2017-09-04_16:33:46.mp4
Code example:
GstFlowReturn GstVideoSourcePrivate::newSample(GstAppSink* sink, gpointer user_data)
{
....
GstSample* sinkSample = gst_app_sink_pull_sample(GST_APP_SINK(sink));
if (sinkSample) {
GstBuffer* buffer = gst_sample_get_buffer(sinkSample);
// I need this position to be the same as the recorded video
// or get the frame video sequence number, so that we
GstClockTime pos;
gst_element_query_position(self->pipeline(), GST_FORMAT_TIME, &pos);
...
}
...
}
Thanks for your answer.
I did what you told me, but I can not get the expected result.
Then I discovered that when a videorate element is inserted into the pipeline, buffer->offset begins to display the correct sequence of video frame. But again, I can not get a good sync for a few milliseconds.
So, I read the doc one more time and I made this code to get a better result. It seems that there are few latencies that need to be compensated.
https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html
https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/clock.html
...
int64_t timestam = GST_BUFFER_TIMESTAMP(buffer);
GstSegment* segment = gst_sample_get_segment(sinkSample);
gint64 pos = gst_segment_to_stream_time(segment, GST_FORMAT_TIME, timestam);
GstQuery*q = gst_query_new_latency();
if (gst_element_query (self->m_pipeline, q)) {
gboolean live;
GstClockTime minlat, maxlat;
gst_query_parse_latency (q, &live, &minlat, &maxlat);
pos+= minlat;
}
gst_query_unref (q);
...

GStreamer: Caps negotiation failure

I have a problem with linking of a two elements: avdec_h264 and avenc_mpeg4. I think that somehow these elements can't negotiate a capabilities of a data.
I've tested my pipeline with gst-launch:
gst-launch-1.0 rtspsrc location="rtsp://camera" ! rtph264depay ! h264parse ! avdec_h264 ! avenc_mpeg4 ! fakesink
It have worked fine.
When I use my application where the pipeline is implemented:
pipeline_ = gst_pipeline_new("default");
if (!pipeline_)
{
return false;
}
receiver_ = gst_element_factory_make("rtspsrc", "receiver");
demuxer_ = gst_element_factory_make("rtph264depay", "demuxer");
parser_ = gst_element_factory_make("h264parse", "parser");
decoder_ = gst_element_factory_make("avdec_h264", "decoder");
encoder_ = gst_element_factory_make("avenc_mpeg4, "encoder");
output_ = gst_element_factory_make("fakesink", "output");
if (!receiver_ || !demuxer_ || !parser_ ||
!decoder_ || !encoder_ || !output_)
{
return false;
}
g_object_set(GST_OBJECT(receiver_), "location", "rtsp://camera", nullptr);
// On this signal source pad of the receiver is being connected to
// the sink pad of the demuxer.
g_signal_connect(receiver_, "pad-added", G_CALLBACK(on_pad_added), this);
gst_bin_add_many(GST_BIN(pipeline_), receiver_, demuxer_, parser_,
decoder_, encoder_, output, nullptr);
if (!gst_element_link_many(demuxer_, parser_, decoder_,
encoder_, output_, nullptr))
{
return false;
}
Everything links successfully. All elements change their state to PLAYING, but I get nothing: I do not get GST_MESSAGE_STREAM_START on the pipeline's bus.
Here is the graphs from gst-launch and my application:
If I change avenc_mpeg4 to, videoconvert element, which is not an encoder, everything will works well. If I put an other encoder, I will still have current problem.
Probably I don't know about some particular things on how to work with the encoder. But I could not find solution.
Thank you.
A few points:
The code you posted above should listed for pad-added messages of decodebin. I am surprised if the code would work as is (maybe put the full code on a gist and link from here). See https://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-pads.html
Insert a videoconvert between the decoder and the encoder.
Where are you linking reciever to demuxer ? that is needed as i understand.

mux klv data with h264 by mpegtsmux

I need to mux klv metadata into the h264 stream. I have created application. But the stream is playing only as long as klv-data is being inserted. When i stop pushing klv-data the whole stream stops. What is the right method to mux asynchronous klv data by mpegtsmux?
Klv-data need to be inserted into the following working pipeline:
v4l2src input-src=Camera ! videorate drop-only=true ! 'video/x-raw, format=(string)NV12, width=1920, height=1088, framerate=25/1' ! ce_h264enc target-bitrate=6000000 idrinterval=25 intraframe-interval=60 ! queue ! mpegtsmux alignment=7 ! udpsink host=192.168.0.1 port=3000 -v
This pipeline is collected in the application. To insert klv-metedata appsrc is created:
appSrc = gst_element_factory_make("appsrc", nullptr);
gst_app_src_set_caps (GST_APP_SRC (appSrc), gst_caps_new_simple("meta/x-klv", "parsed", G_TYPE_BOOLEAN, TRUE, "sparse", G_TYPE_BOOLEAN, TRUE, nullptr));
g_object_set(appSrc, "format", GST_FORMAT_TIME, nullptr);
Then appsrc is linked to the pipeline:
gst_bin_add(GST_BIN(pipeline), appSrc);
gst_element_link(appSrc, mpegtsmux);
Here is push function:
void AppSrc::pushData(const std::string &data)
{
GstBuffer *buffer = gst_buffer_new_allocate(nullptr, data.size(), nullptr);
GstMapInfo map;
GstClock *clock;
GstClockTime abs_time, base_time;
gst_buffer_map (buffer, &map, GST_MAP_WRITE);
memcpy(map.data, data.data(), data.size());
gst_buffer_unmap (buffer, &map);
GST_OBJECT_LOCK (element);
clock = GST_ELEMENT_CLOCK (element);
base_time = GST_ELEMENT (element)->base_time;
gst_object_ref (clock);
GST_OBJECT_UNLOCK (element);
abs_time = gst_clock_get_time (clock);
gst_object_unref (clock);
GST_BUFFER_PTS (buffer) = abs_time - base_time;
GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale_int (1, GST_SECOND, 1);
gst_app_src_push_buffer(GST_APP_SRC(element), buffer);
}
Gstreamer version is 1.6.1.
What can be wrong with my code? I'd appreciate your help.
I can push dummy klv-packets to maintain video stream. But i don't want to pollute upcomming stream and i am sure there should be more delicate solution.
I have found that i can send event with GST_STREAM_FLAG_SPARSE, which should be appropriate for subtitles. But as a result i have no output at all.
GstEvent* stream_start = gst_event_new_stream_start("klv-04");
gst_event_set_stream_flags(stream_start, GST_STREAM_FLAG_SPARSE);
GstPad* pad = gst_element_get_static_pad(GST_ELEMENT(element), "src");
gst_pad_push_event (pad, stream_start);
While debugging i have found that after applying the following patch to the gstreamer and using GST_STREAM_FLAG_SPARSE, the stream doesn't stop when the appsrc stops pushing packets.
diff --git a/libs/gst/base/gstcollectpads.c b/libs/gst/base/gstcollectpads.c
index 8edfe41..14f9926 100644
--- a/libs/gst/base/gstcollectpads.c
+++ b/libs/gst/base/gstcollectpads.c
## -1440,7 +1440,8 ## gst_collect_pads_recalculate_waiting (GstCollectPads * pads)
if (!GST_COLLECT_PADS_STATE_IS_SET (data, GST_COLLECT_PADS_STATE_WAITING)) {
/* start waiting */
gst_collect_pads_set_waiting (pads, data, TRUE);
- result = TRUE;
+ if (!GST_COLLECT_PADS_STATE_IS_SET (data, GST_COLLECT_PADS_STATE_LOCKED))
+ result = TRUE;
}
}
}
Anyway, the receiver stops updating screen 10 seconds after the last klv packet.
This is a bit of an old thread but,
In my experience though, if there is no queue between the appsrc and the muxer, you will get this behavior. I would change your:
gst_element_link(appSrc, mpegtsmux);
To this:
gst_element_link(appSrc, appSrcQueue);
gst_element_link(appSrcQueue, mpegtsmux);
And I'm not sure if the mpegtsmux has the capability for it or not but the muxer that we have used has a property called do-timestamping and when that was set to TRUE we had a better experience.
Another tip I would give is to use the gst-inspect tool to see what options each elements have.