I manged to run WebRTC peerconnection example, but it is not running on the browser.
I'm trying to find a way to stream both video and audio from browser to my native program.
Is there any way?
It can be done. WebRTC is designed to work in a peer-to-peer manner between two WebRTC agents (typically a Web Browser). Your native program needs to become the second peer.
If you need to rely on open source components a good starting point is:
OpenSSL for the DTLS key exchange.
libsrtp to encrypt the RTP packets.
ffmpeg to decode the PCM audio from the browser (libvpx if you need to do video).
You'll also need to handle the ICE negotiation which requires processing STUN messages. Also extract the media payloads from the RTP packets. All these steps are also after you've determined a signalling method to exchange the SDP offer and answer between you app and the browser.
As you've probably realised starting from scratch it's a major task. There are probably some commercial libraries that will do the job and save you a lot of pain.
If that doesn't scare you and you do still want to make an attempt using open source components this example "may" help. The sample is doing the reverse of what you've asked and is sending a video stream to Chrome rather than receiving an audio stream. The useful aspect is the connection negotiation. The sample program is able to get RTP packets flowing which is often the main problem.
The example is also using Windows Media Foundation which is Windows specific. It also has lots of shortcuts particularly with the RTP and STUN packet processing.
Related
I want to build a live streaming app.
My thought process:
Get the Video/Audio data from the
navigator.mediaDevices.getUserMedia(constraints); [client-streamer]
create rooms using sockets(Socket.IO or WebSockets from flask) [backend]
Send the data in 1 to the room members using sockets.
display the media on the client-side.
Is that correct? How should I do it?
how do I broadcast data to specific room members and not to everyone? (flask)
How to consistently send data from the streamer -> server -> room members. the stream is given from 1 is an object, where is the data?
any other better ideas will be great! thanks.
I need to implement the server-side by myself without help from libraries that will do the work for me.
Implementing a streaming platform is not trivial. Unfortunately, it is not as simple as emitting chunks received from the MediaRecorder with onndatavailable and forwarding them to users using a WebSocket server - this is not scalable nor efficient nor reliable.
Below are some strategies you can try for different types of scenarios:
P2P: If you want to have simple peer-to-peer streaming, you can use WebRTC to achieve that with a simple socket.io server for signaling purposes.
Conference: Here things start to get more complicated. You will need a media server if you want to be somewhat scalable. One approach is to route your stream to the users using an SFU or MCU. This will take care of forwarding/processing media to different peers efficiently.
Broadcast: Here things are also non-trivial. Common WebRTC-based architectures include ingesting the WebRTC stream and forward that to an HLS server which will let your stream chunks available for clients through a CDN, or perform RTP forwarding of the WebRTC stream, convert it to RTMP using something like FFmpeg and deliver it through Youtube Live or Twitch to leverage from their infrastructure.
Be aware that the last 2 items are resource-intensive and will certainly not be cheap to maintain.
Below are some open source projects that could help you along the way:
Janus
MediaSoup
AntMedia
Jitsi
Good luck!
Explaining all this is far beyond the scope of a Stack Overflow answer.
Here are a few hints:
You need to use the MediaRecorder API to capture compressed data from your gUM (getUserMedia) stream. MediaRecorder support is inconsistent between makes and models of browser. though.
It kicks a Blob into its onndatavailable handler every so often.
They're compressed as a webm data stream.
You can push those Blobs to a server with socket.io, and the server can turn around and push them to whatever clients you want to.
Playing the webm on the clients is tricky. You may, on some makes and models of browsers, be able to feed the webm stream to the Media Source API using appendBuffer(). But some browsers cannot consume the webm streams.
These webm streams are useless to a player without all their Blob data in order. You can't just start sending a new client the Blobs of the stream when they sign in; you have to restart the MediaRecorder.
(You may be able to make it work without a MediaRecorder restart if you send the first few k bytes of the stream to each new client before sending the current Blob. Extracting those bytes is an intricate programming job involving the ebml package to parse the webm stream and extract the prologue. I have not proven this concept.)
Because getting all this to work -- originator -- server -- viewer is such a pain in the xxx neck, you may want to investigate using something like mediasoup instead. It uses WebRTC transport rather than socket.io, and works cross-platform.
Summary: How do I stream high quality video using WebRTC native?
I have an h264 stream that's 1920x1080 at about 30fps. I can currently stream this from a server on localhost to a native client on localhost just fine.
I wrote a WebRTC server using Google's WebRTC native library. I've written a VideoEncoder and VideoEncoderFactory that takes frames consisting of already encoded data and and broadcasts it over a video track. Using this I can send my h264 stream to the WebRTC server over a pipe and I can see the video stream in a browser.
However, any time something moves the video gets corrupted. It continues to play but is full of artifacts. Eventually I discovered that WebRTC is dropping some of my frames. When I attach a sequentially increasing ID to each frame before I pass it to rtc::AdaptedVideoTrackSource::OnFrame and I log this same ID in webrtc::VideoEncoder::Encode I can see that some of my frames simply disappear.
This kind of makes sense, I'm trying to stream high quality video over something meant for video chat and lowing my framerate fixes the corruption. However, I'm not asking the WebRTC library to do a lot, it's just forwarding already encoded data to a client on localhost. I have a native app that does this fine and I've seen one browser WebRTC client that can do this. Is there a field in the SDP or some configuration change that will allow me to stream my video?
This was the solution How to control bandwidth in WebRTC video call? .
I had heard about changing the offer sdp but dismissed it because I was told that the browser will accept unlimited bandwidth by default and that you'd only need to to this if you want to limit bandwidth. However, adding "b=AS:high number" has fixed all of my problems.
So I have on one hand an embedded device with a camera running openCV and on the other hand a C++ (Qt) GUI. I would like to connect both i.e.:
"stream" all the output image frames/video from openCV to my remote C++ gui
send commands from my C++ gui to the embedded device
How can I do this, what possibilities do I have? I was thinking about sockets, but I don't know whether that is the easiest solution to stream the image frames from openCV to my Qt gui.
Thank you
You should give us more details about what you're trying to achieve.
You say "stream [...] to my remote C++ GUI": do you mean sending the data over a cabled connection? over a LAN network? over the Internet?
Depending on the answer this changes your system's architecture quite a bit. Especially in case you want to stream the data over the Internet. If your use case implies a LAN network, you can easily setup a peer-to-peer connection between the embedded device and the C++ app to send data. However, it's much more complicated if you want to send data over the Internet, because it is difficult to create a peer-to-peer connection if you don't have static IPs (which I'm assuming you do not have). You will need a server (which can be written with Qt as well) to work as a relay for sending data from the device to your C++ app.
Do you need actual video streaming (at 25fps), or is a low refresh rate (1-0.5fps) sufficient ?
(I'm making the assumption you want to send data over a network)
Because if a low image rate is sufficient, using WebSockets to send images on a regular basis might just do the trick.
Otherwise, you'll need to setup a UDP connection with a video buffer.
Hope this helps!
D
I have an audio processing server, and I'd like to be able to connect to it via WebRTC.
The native library from Google seems suitable for that (from looking at the peerconnection example): https://webrtc.org/native-code/native-apis/
But the library relies too much on the audio devices: it opens them behind the scenes. I've managed to grab the incoming audio by appending my own AudioTrackSinkInterface to the stream, but haven't yet found how to inject the audio into the outbound stream. And these hacks don't avoid opening the devices anyway.
How to do it cleanly?
I am implementing a client/server application where video streaming occurs between two computers (in one direction). I would like to have the server publish an SDP file when it starts streaming. The client would then be able to download this SDP file and use it to get the stream. In order to implement this it seems I need to include a RTSP server in my server application.
I am planning to use either libVLC or GStreamer for the client. Both are able to get incoming video streams using the info from an SDP file.
Server-side I don't really know where to start. Can anyone recommend a good C++ library that would allow me to create a small RTSP server?
Use Live555 LGPL library or for fun, read the RFC and implement :-)
Libcurl's library offers a simple example that can be usefull for the server side..
Take a look at: https://curl.haxx.se/libcurl/c/rtsp.html