I would like to implement a gstreamer pipeline for video streaming without using a v4l2 driver in Linux. The thing is that the video frames I have them already in the RAM(the vdma core which is configured by a different OS on a different core takes care of that) . And also I had difficulties debugging some DMA slave errors which appeared always after a dma completion callback.
Therefore I would be happy if I would not have to use v4l2 driver in order to have gstreamer on top.
I have found this plugin from Bosch that fits my case:
https://github.com/igel-oss/v4l-gst
My question would be if somebody has experience with this approach and if is a feasible one?
Other question would be how to configure the source in the gstreamer pipeline as it is not a device /dev/videoxxx but rather a memory location or even a bmp file.
Thanks, Mihaita
You could use appsrc and repeatedly call gst_app_src_push_buffer (). Your application will have all freedom to read the video data from anywhere it likes - memory, files etc. See also the relevant section of the GStreamer Application Development Manual.
If you want more flexibility, like using the video source in several applications, you should consider implementing your own custom GStreamer element.
Related
Hello everyone,
I am trying to implement low-latency video streaming using WebRTC. I write my code in C++ (websocket etc.), use only webrtc signalling server which is written in Python (ref1).
When I use a webcam, I do not have any problem streaming video to the client, however, I try to use the FLIR camera, I get a lot of problems while implementation.
There are a few questions in my mind to clear. I hope you guys give me some recommendations.
Is there any specific data-type that I should do pipeline to webrtc as a source? I just would like to know what kind of data I should send as a source in webrtc?
I try to send an image to check whether my WebRTC implementation works properly (except webcam), it gives me the error "Pipeline is empty". What can cause this problem? This is actually the main problem why I would like to know data type etc. to understand what exactly I should pipe into webrtc.
ref1: https://github.com/centricular/gstwebrtc-demos/tree/master/signalling
P.S.:
Client and Jetson Nano in the network
Server for signals is running on Jetson Nano
By running gst-inspect-1.0 webrtcbin you will find that both source and sink capability for this plugin is just application/x-rtp.
Therefore, if you want webrtcbin to work as a source pad, you will need to pipe it to some sort of RTP depayloader such as rtph264depay for video and rtpopusdepay for audio.
The project I'm currently working on, requires an addition to the already existing VoIP capabilities. The core for speech processing is in C, the remainder is in C++ with Qt - the audio is handled via portaudio. The connection between users is currently established via UDP, which I think has to be changed for the planned video connection. Developing platform is Windows on VS2012 - however, the system is cross-platform.
In a nutshell, what I want to do is: Grab the video signal from a webcam, synchronize audio coming from C core and video from webcam and use a library and codecs for (de-)coding/muxing the signals on the respective sides and sending via RTP. The system should be capable of multicast transmission.
I did some research for possible libraries and stumbled upon ffmpeg and libVLC. For the codec I thought about using x264. And if I'm correct, ffmpeg and libVLC should both be capable of what I'm looking for?
However I'm not sure which one to pick, and from their documentations I really can't extract, which library is the better fit. Has anybody had similar problems and can help me out - I'm quite a newbie, when it comes to video processing and encoding.
Extra question: Do you have any hints or approaches on syncing the video and audio signals?
If anyone is interested, this is what I ended up doing:
I am using the WebM container format, VP8 with Vorbis currently (but going to change to VP9 with Opus soon if out of beta), handled by ffmpeg/libav libraries for encoding/decoding/muxing etc. and SDL for displaying and threading. ffmpeg/libav was cross-compiled on Unix with LGPL support to keep our project closed source.
My application displays video and audio and I want to add a recording feature.
I've considered FFmpeg, but I have to compile my application with VS so I can't use it. So I'm trying to do it with GStreamer, but I'm not finding any example or guide on how to create a video. Any help?
(I can also consider using any other alternatives, but they must be cross-platform).
Application Development Manual explains very well how to use gstreamer from your code. Try to read it first.
Than you can experiment with gst-launch tool, build pipeline and execute it from your application using gst-parse-launch function.
You can expose more details of your problem if you want more helpful answer.
I wanted to get some ideas one how some of you would approach this problem.
I've got a robot, that is running linux and uses a webcam (with a v4l2 driver) as one of its sensors. I've written a control panel with gtkmm. Both the server and client are written in C++. The server is the robot, client is the "control panel". The image analysis is happening on the robot, and I'd like to stream back the video from the camera to the control panel for two reasons:
A) for fun
B) to overlay image analysis results
So my question is, what are some good ways to stream video from the webcam to the control panel as well as giving priority to the robot code to process it? I'm not interested it writing my own video compression scheme and putting it through the existing networking port, a new network port (dedicated to video data) would be best I think. The second part of the problem is how do I display video in gtkmm? The video data arrives asynchronously and I don't have control over main() in gtkmm so I think that would be tricky.
I'm open to using things like vlc, gstreamer or any other general compression libraries I don't know about.
thanks!
EDIT:
The robot has a 1GHz processor, running a desktop like version of linux, but no X11.
Gstreamer solves nearly all of this for you, with very little effort, and also integrates nicely with the Glib event system. GStreamer includes V4L source plugins, gtk+ output widgets, various filters to resize / encode / decode the video, and best of all, network sink and sources to move the data between machines.
For prototype, you can use the 'gst-launch' tool to assemble video pipelines and test them, then it's fairly simply to create pipelines programatically in your code. Search for 'GStreamer network streaming' to see examples of people doing this with webcams and the like.
I'm not sure about the actual technologies used, but this can end up being a huge synchronization ***** if you want to avoid dropped frames. I was streaming a video to a file and network at the same time. What I eventually ended up doing was using a big circular buffer with three pointers: one write and two read. There were three control threads (and some additional encoding threads): one writing to the buffer which would pause if it reached a point in the buffer not read by both of the others, and two reader threads that would read from the buffer and write to the file/network (and pause if they got ahead of the producer). Since everything was written and read as frames, sync overhead could be kept to a minimum.
My producer was a transcoder (from another file source), but in your case, you may want the camera to produce whole frames in whatever format it normally does and only do the transcoding (with something like ffmpeg) for the server, while the robot processes the image.
Your problem is a bit more complex, though, since the robot needs real-time feedback so can't pause and wait for the streaming server to catch up. So you might want to get frames to the control system as fast as possible and buffer some up in a circular buffer separately for streaming to the "control panel". Certain codecs handle dropped frames better than others, so if the network gets behind you can start overwriting frames at the end of the buffer (taking care they're not being read).
When you say 'a new video port' and then start talking about vlc/gstreaming i'm finding it hard to work out what you want. Obviously these software packages will assist in streaming and compressing via a number of protocols but clearly you'll need a 'network port' not a 'video port' to send the stream.
If what you really mean is sending display output via wireless video/tv feed that's another matter, however you'll need advice from hardware experts rather than software experts on that.
Moving on. I've done plenty of streaming over MMS/UDP protocols and vlc handles it very well (as server and client). However it's designed for desktops and may not be as lightweight as you want. Something like gstreamer, mencoder or ffmpeg on the over hand is going to be better I think. What kind of CPU does the robot have? You'll need a bit of grunt if you're planning real-time compression.
On the client side I think you'll find a number of widgets to handle video in GTK. I would look into that before worrying about interface details.
I want to create a Qt widget that can play incoming RTP streams where the video is encoded as H264 and contains no audio.
My basic plan for implementation is this:
Create a Phonon MediaSource object (Stream type).
Connect it with a QIODevice subclass that provides the data
Obtain the video data using either:
The JRTPLIB client library
The GStreamer gstrtpbin plugin. This plugin takes care depayloading the packages and decoding the video. Maybe this improves the chances that Phonon will recognize the data.
My environment:
Ubuntu 9.10
Qt 4.6
My questions:
Is my approach a good one? Perhaps I'm overlooking a more obvious or simple solution?
I'm currently experiencing this issue: when trying to play the video stream the state of the MediaObject turns to ErrorState with errorType FatalError. Can anyone tell me what I'm doing wrong?
Edit
One solution I found is using libVLC in combination with Qt, which I learned about in this thread. Here's a code sample for the interested.
I'm still looking for a Phonon-based solution.
Ideally I would only need to provide an SDP file and job is done.
I was able to get it to work using the libVLC solution. I can't garantuee that this is the best solution though as I simply stopped looking after that.
Here's a link to the libVLC sample.
The way I understand Phonon works at least in Windows is that QT provides a phonon backend plugin for DirectShow (\plugins\phonon_backend\phonon_ds94.dll) and GStreamer in your case. Then you would either obtain or write your own DirectShow filter which can accept RTP streams as a source. DirectShow takes care of the decoding, and Phonon will take care of the rendering.
So if the backend works, the application code is as simple as:
Phonon::MediaObject *media = new Phonon::MediaObject();
Phonon::VideoWidget *video = new Phonon::VideoWidget();
Phonon::createPath(media, video);
media->setCurrentSource(source);
media->play();
Seems that the problem lies with the GStreamer backend accepting RTP as a source. Can you playback that source in standalone GStreamer without any problems?