Decode H264 stream over Network OpenH264 - c++

I am facing a problem with OpenH264 library https://github.com/cisco/openh264
I would like to decode a stream sent by my raspberry pi camera over network on my C++/Qt Program which will display the image.
I'm using gstreamer on Raspberry Side to send the stream with this command line :
raspivid -n -t 0 -w 1280 -h 720 -fps 25 -b 2500000 -o - | gst-launch-1.0 fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=my_ip port=5000
On desktop side when I execute :
gst-launch-1.0 -v tcpclientsrc host=raspberry_ip port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false
I'm able to see the stream of the camera correctly.
Okay, so now. I would like to make my own decoder using QT/C++ & OpenH264 decoder.
Here is my code :
void Manager::initDecoder(int width, int height) {
long ret = WelsCreateDecoder(&(this->decoder));
if (ret == 0) {
this->decodingParam = { 0 };
this->decodingParam.sVideoProperty.eVideoBsType = VIDEO_BITSTREAM_DEFAULT;
this->decoder->Initialize(&decodingParam);
this->bufferInfo = { 0 };
this->yuvData = new uint8_t*[3];
this->yuvData[0] = new uint8_t[width * height];
this->yuvData[1] = new uint8_t[width * height / 4];
this->yuvData[2] = new uint8_t[width * height / 4];
this->tcpSocket->connectToHost("ip_raspberry", 5000);
}
}
bool Manager::decodeStream(const unsigned char *rawEncodedData, const int rawEncodedDataLength, uint8_t **yuvData) {
DECODING_STATE err = decoder->DecodeFrame2(rawEncodedData, rawEncodedDataLength, yuvData, &bufferInfo);
if (err != 0) {
qDebug() << "H264 decoding failed. Error code: " << err << ".";
return false;
}
qDebug() << "----------- Succeedeeed --------------";
return true;
}
When I get a new data I call decodeStream but this function is returning an error :
dsBitstreamError = 0x04, ///< error bitstreams(maybe broken internal frame) the decoder cared
dsNoParamSets = 0x10, ///< no parameter set NALs involved
I don't know what I'm doing wrong... Should I change some parameters on the raspberry sending ?
Thanks for helping.

Related

gstreamer appsink, cpu load is high when drop = true

I'm using gstreamer's appsink and I find that its cpu load is very high when I set drop=true
the following pipeline works:
gst-launch-1.0 -v rtspsrc location="rtsp://somelink" latency=300 ! rtph265depay ! h265parse ! avdec_h265 ! videoconvert ! appsink caps="video/x-raw,format=BGR"
Please note that appsink is used.
But if I add drop=true to appsink, the cpu load turn to be very high (as high as 100% for 1 cpu core)
I checked the appsink's source code: https://git.launchpad.net/ubuntu/+source/gst-plugins-base1.0/tree/gst-libs/gst/app/gstappsink.c
if (priv->drop) {
GstMiniObject *old;
/* we need to drop the oldest buffer/list and try again */
if ((old = dequeue_buffer (appsink))) {
GST_DEBUG_OBJECT (appsink, "dropping old buffer/list %p", old);
gst_mini_object_unref (old);
}
}
then the source code of dequeue_buffer:
static GstMiniObject *
dequeue_buffer (GstAppSink * appsink)
{
GstMiniObject *obj;
do {
obj = dequeue_object (appsink);
if (GST_IS_BUFFER (obj) || GST_IS_BUFFER_LIST (obj)) {
break;
}
gst_mini_object_unref (obj);
} while (TRUE);
return obj;
}
Please note that there is a while (TRUE) in the function and I really suspect that it causes the high cpu usage, but I didn't find a way to debug it.
Any one has the same issue and what's the workaround?

GStreamer Appsrc: how do I get the negotiated caps?

I am currently using an appsrc element in a program I am writing.
I let the user provide a launch string such as this example:
appsrc name=mysource format=3 is-live=1 \
! video/x-raw(memory:NVMM), width=5400, height=3400, framerate=30/1, format=NV12 \
! nvv4l2h265enc bitrate=8000000 control-rate=0 iframeinterval=2 \
! h265parse \
! matroskamux \
! filesink location=myfile.mkv
The launch string could be anything, provided it has an appsrc called mysource.
In my program, I locate mysource and I would like to know the format property that was provided by the user (to create the right kind of data buffer). I query the src pad of my appsrc element:
GstPad * pad = gst_element_get_static_pad(m_appsrc, "src");
if (!pad) fprintf(stderr, "pad is null\n");
GstCaps * caps = gst_pad_get_current_caps(pad);
if (!caps) fprintf(stderr, "caps is null\n");
for (guint i = 0; i < gst_caps_get_size (caps); i++) {
GstStructure *structure = gst_caps_get_structure (caps, i);
g_print ("%s%s\n", " ", gst_structure_get_name (structure));
gst_structure_foreach (structure, print_field, (gpointer) " ");
}
pad is non-null, but caps is always returned as NULL.
Looking at the .dot graph created by GST_DEBUG_BIN_TO_DOT, I'm seeing that the caps indicated on the output of the appsrc are "ANY".
Do I have to do something special like traversing the pipeline to get the final negotiated caps?
Looks like if I specify a caps property in the appsrc launch string, caps is not NULL and can be traversed to describe the user-specified capabilities. This is good enough for me:
appsrc name=mysource format=3 is-live=1 caps="video/x-raw(memory:NVMM), width=5400, height=3400, framerate=30/1, format=NV12" \
! nvv4l2h265enc bitrate=8000000 control-rate=0 iframeinterval=2 \
! h265parse \
! matroskamux \
! filesink location=myfile.mkv

How can I write Open CV frames to an RTSP stream with Gstreamer and C++?

I'm trying to take a video frame into OpenCV, do some processing on it (to be exact, aruco detection) and then package the resultant frame into a RTSP stream with GStreamer.
I've seen a Python solution to this problem, but I'm having trouble translating it to C++.
Here's my attempt at recreating the SensorFactory class:
#include <glib-object.h>
#include <iostream>
#include "SensorFactory.h"
SensorFactory::SensorFactory(std::string launch) {
launchString = launch;
cap = cv::VideoCapture(0);
// should be incremented once on each frame for timestamping
numberFrames = 0;
// simple struct with only the cap (int*), lastFrame (cv::Mat*) and numberFrames (int* again) fields
CVData cvData;
cvData.cap = ∩
cvData.lastFrame = &lastFrame;
cvData.numberFrames = &numberFrames;
}
GstFlowReturn SensorFactory::on_need_data(GstElement *src, CVData *datum) {
if (datum->cap->isOpened()) {
if (datum->cap->read(*(datum->lastFrame))) {
std::string data = std::string(reinterpret_cast<char * > (datum->lastFrame->data));
GstBuffer *buf = gst_buffer_new_allocate(nullptr, data.max_size(), nullptr);
gst_buffer_fill(buf, 0, &data, data.max_size());
buf->duration = static_cast<GstClockTime>(duration);
GstClockTimeDiff timestamp = *(datum->numberFrames) * duration;
buf->pts = buf->dts = static_cast<GstClockTime>(timestamp);
buf->offset = static_cast<guint64>(timestamp);
int *numf = datum->numberFrames;
*numf += 1;
g_signal_emit_by_name(src, "push-buffer", buf);
gst_buffer_unref(buf);
return GST_FLOW_OK;
}
}
// never reached
return GST_FLOW_NOT_LINKED;
}
GstElement *SensorFactory::create_element(const GstRTSPUrl *url) { return gst_parse_launch(launchString.c_str(), nullptr); }
void SensorFactory::configure(GstRTSPMedia *rtspMedia) {
numberFrames = 0;
GstElement *appsrc;
appsrc = gst_rtsp_media_get_element(rtspMedia);
g_signal_connect(appsrc, "need-data", (GCallback) on_need_data, &cvData);
}
The header for SensorFactory is nothing special:
#include <gst/rtsp-server/rtsp-media-factory.h>
#include <gst/rtsp-server/rtsp-media.h>
#include <gst/app/gstappsrc.h>
#include <opencv2/videoio.hpp>
class SensorFactory : public GstRTSPMediaFactory {
public:
typedef struct _CVData {
cv::VideoCapture *cap;
cv::Mat *lastFrame;
int *numberFrames;
} CVData;
CVData cvData;
std::string launchString;
cv::VideoCapture cap;
cv::Mat lastFrame;
int numberFrames = 0;
const static int framerate = 30;
const static GstClockTimeDiff duration = 1 / framerate * GST_SECOND;
explicit SensorFactory(std::string launch);
static GstFlowReturn on_need_data(GstElement *src, CVData *datum);
GstElement *create_element(const GstRTSPUrl *url);
void configure(GstRTSPMedia *media);
};
And then main.cpp looks like so:
#include <gst/gst.h>
#include "src/SensorFactory.h"
int main() {
gst_init(nullptr, nullptr);
GstRTSPServer *server;
server = gst_rtsp_server_new();
SensorFactory sensorFactory("appsrc name=source is-live=true block=true format=GST_FORMAT_TIME"
"caps=video/x-raw,format=BGR ! "
"videoconvert ! video/x-raw,format=I420 ! "
"x264enc speed-preset=ultrafast tune=zerolatency ! rtph264pay name=pay0");
g_print("setting shared\n");
gst_rtsp_media_factory_set_shared(&sensorFactory, true);
g_print("set shared\n");
GstRTSPMountPoints *mounts;
mounts = gst_rtsp_server_get_mount_points(server);
gst_rtsp_mount_points_add_factory(mounts, "/test", &sensorFactory);
GMainLoop *loop;
loop = g_main_loop_new(nullptr, false);
g_main_loop_run(loop);
}
The program compiles fine, and will even start running, but segfaults on gst_rtsp_media_factory_set_shared(&sensorFactory, true);. There isn't any other hacky memory management in this program.
You can try the steps below to write the stream as RTMP.
if (platform is "Windows") {
// if the platform is windows, then add the head data of the video
// otherwise it will not work on the HTML flash player
headData = " ! video/x-h264,profile=high";
}
// to rtmp (media server e.g: NGINX)
rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! rtmpsink location=rtmp://192.168.1.25/mylive/test";
// using UDP broadcast to all 1~255 IPs
rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! udpsink host=192.168.1.255 port=5000";
// using UDP broadcast specific IP
rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! udpsink host=192.168.1.25 port=5000";
// give the FPS and the size of the video
VideoWriter writer = new VideoWriter(rtmpUrl, Videoio.CAP_GSTREAMER, FOURCC, currentFps, new Size(width, height));
// then you can write the video using writer
NOTE: Make sure you build OpenCV with GStreamer.
Here is an alternative approach.
Seperate your SensorFactory from the rtsp code for now.
Start your SensorFactory with the pipeline.
appsrc name=source is-live=true block=true format=GST_FORMAT_TIME caps=video/x-raw,format=BGR,width=640,height=480,framerate=30/1 ! videoconvert ! video/x-raw,format=I420 ! x264enc speed-preset=ultrafast tune=zerolatency ! udpsink port=5050
We end that pipeline by piping the h264 over a udpsink on port 5050.
Then compile the gstreamer rtsp server example here
And launch that with pipeline
./test-launch "( udpsrc port=5050 ! rtph264pay name=pay0 pt=96 )"
Assuming your SensorFactory works as you intend, this should get you an RTSP Stream serving at rtsp://localhost:8554/test

Gstreamer. Write appsink to filesink

I have written a code for appsrc to appsink and it works. I see the actual buffer. It's encoded in H264(vpuenc=avc). Now I want to save it in a file(filesink). How I approach it?
app:
int main(int argc, char *argv[]) {
gst_init (NULL, NULL);
GstElement *pipeline, *sink;
gchar *descr;
GError *error = NULL;
GstAppSink *appsink;
descr = g_strdup_printf (
"mfw_v4lsrc device=/dev/video1 capture_mode=0 ! " // grab from mipi camera
"ffmpegcolorspace ! vpuenc codec=avc ! "
"appsink name=sink"
);
pipeline = gst_parse_launch (descr, &error);
if (error != NULL) {
g_print ("could not construct pipeline: %s\n", error->message);
g_error_free (error);
exit (-1);
}
gst_element_set_state(pipeline, GST_STATE_PAUSED);
sink = gst_bin_get_by_name (GST_BIN (pipeline), "sink");
appsink = (GstAppSink *) sink;
gst_app_sink_set_max_buffers ( appsink, 2); // limit number of buffers queued
gst_app_sink_set_drop( appsink, true ); // drop old buffers in queue when full
gst_element_set_state (pipeline, GST_STATE_PLAYING);
int i = 0;
while( !gst_app_sink_is_eos(appsink) )
{
GstBuffer *buffer = gst_app_sink_pull_buffer(appsink);
uint8_t* data = (uint8_t*)GST_BUFFER_DATA(buffer);
uint32_t size = GST_BUFFER_SIZE(buffer);
gst_buffer_unref(buffer);
}
return 0; }
If as mentioned in the comments, what you actually want to know is how to do a network video stream in GStreamer, you should probably close this question because you're on the wrong path. You don't need to use an appsink or filesink for that. What you'll want to investigate are the GStreamer elements related to RTP, RTSP, RTMP, MPEGTS, or even MJPEGs (if your image size is small enough).
Here are two basic send/receive video stream pipelines:
gst-launch-0.10 v4l2src ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=640,height=480 ! vpuenc ! h264parse ! rtph264pay ! udpsink host=localhost port=5555
gst-launch-0.10 udpsrc port=5555 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! ffdec_h264 ! videoconvert ! ximagesink
In this situation you don't write your own while loop. You register callbacks and wait for buffers (GStreamer 0.10) to arrive. If you're using GStreamer 1.0, you use samples instead of buffers. Samples are a huge pain in the ass compared to buffers but oh well.
Register the callback:
GstAppSinkCallbacks* appsink_callbacks = (GstAppSinkCallbacks*)malloc(sizeof(GstAppSinkCallbacks));
appsink_callbacks->eos = NULL;
appsink_callbacks->new_preroll = NULL;
appsink_callbacks->new_sample = app_sink_new_sample;
gst_app_sink_set_callbacks(GST_APP_SINK(appsink), appsink_callbacks, (gpointer)pointer_to_data_passed_to_the_callback, free);
And your callback:
GstFlowReturn app_sink_new_sample(GstAppSink *sink, gpointer user_data) {
prog_data* pd = (prog_data*)user_data;
GstSample* sample = gst_app_sink_pull_sample(sink);
if(sample == NULL) {
return GST_FLOW_ERROR;
}
GstBuffer* buffer = gst_sample_get_buffer(src);
GstMemory* memory = gst_buffer_get_all_memory(buffer);
GstMapInfo map_info;
if(! gst_memory_map(memory, &map_info, GST_MAP_READ)) {
gst_memory_unref(memory);
gst_sample_unref(sample);
return GST_FLOW_ERROR;
}
//render using map_info.data
gst_memory_unmap(memory, &map_info);
gst_memory_unref(memory);
gst_sample_unref(sample);
return GST_FLOW_OK;
}
You can keep your while loop as it is--using gst_app_sink_is_eos()--but make sure to put a sleep in it. Most of the time I use something like the following instead:
GMainLoop* loop = g_main_loop_new(NULL, FALSE);
g_main_loop_run(loop);
g_main_loop_unref(loop);
Note: Unless you need to do something special with the data you can use the "filesink" element directly.
Simpler option would be write to the file directly in the appsink itself ie when you get a callback when the buffer is done write to the file and make sure you close it on eos.
Hope that helps.

Gstreamer Elements not linking

I am new to Gstreamer and I have a question about why my elements will not link together. Here is my code:
CustomData data;
data.videosource = gst_element_factory_make("uridecodebin", "source");
cout << "Created source element " << data.videosource << endl;
data.demuxer = gst_element_factory_make("qtdemux", "demuxer");
cout << "Created demux element " << data.demuxer << endl;
data.decoder = gst_element_factory_make("ffdec_h264", "video-decoder");
cout << "Went to the video path " << data.decoder << endl;
data.videoconvert = gst_element_factory_make("ffmpegcolorspace", "convert");
cout << "Created convert element " << data.videoconvert << endl;
data.videosink = gst_element_factory_make("autovideosink", "sink");
cout << "Created sink element " << data.videosink << endl;
if (!data.videosource ||!data.demuxer || !data.decoder || !data.videoconvert || !data.videosink)
{
g_printerr ("Not all elements could be created.\n");
system("PAUSE");
return;
}
//Creating the pipeline
data.pipeline = gst_pipeline_new("video-pipeline");
if (!data.pipeline)
{
g_printerr ("Pipeline could not be created.");
}
//Setting up the object
g_object_set(data.videosource, "uri", videoFileName[camID] , NULL);
//videoFileName[camID] is a char** with the content uri=file:///C://videofiles/...mp4
//Adding elements to the pipeline
gst_bin_add_many(GST_BIN (data.pipeline), data.videosource, data.demuxer, data.decoder, data.videoconvert, data.videosink, NULL);
//This is where the issue occurs
if(!gst_element_link(data.videosource, data.demuxer)){
g_printerr("Elements could not be linked. \n");
system("PAUSE");
return;
}
What I am trying to do is to break down a mp4 file and display only the video content but for some reason when I try to link source and demuxer, it comes out as false.
Thank you guys so much!
Let's have a look at the pipeline you're using (I'll use gst-launch here for its brevity, but the same goes for any GStreamer pipelines):
gst-launch uridecodebin uri=file:///path/to/movie.avi \
! qtdemux ! ffdec_h264 ! ffmpegcolorspace \
! autovideosink
gst-inspect uridecodebin states:
Autoplug and decode an URI to raw media
So uridecodebin takes any audio/video source and decodes it by internally using some of GStreamer's other elements.
Its output is something like video/x-raw-rgb or audio/x-raw-int (raw audio/video)
qtdemux on the other hand takes a QuickTime stream (still encoded) and demuxes it.
But what it gets in your example is the already decoded raw video (which is why it won't link).
So, you've basically got two options:
just use uridecodebin
gst-launch uridecodebin uri=file:///path/to/movie.avi \
! autovideosink
which will allow your pipeline to decode pretty much any video file
just use the qtdemux ! ffdec_h264 ! ffmpegcolorspace elements:
gst-launch filesrc=/path/to/movie.avi \
! qtdemux ! ffdec_h264 ! ffmpegcolorspace
! autovideosink
Keep in mind however that your pipeline doesn't play audio.
To get that as well do one of the following:
Simply use playbin2
gst-launch playbin2 uri=file:///path/to/movie.avi
Connect your decodebin to an audio sink as well
gst-launch uridecodebin name=d uri=... ! autovideosink d. ! autoaudiosink