I wrote two command line applications in C++ which use WebRTC:
Client creates a PeerConnection and opens an AudioStream
Server receives and plays the AudioStream
The basic implementation works: They exchange SDP-Offer and -Answer, find their external IPs using ICE, a PeerConnection and a PeerConnectionFactory with the corresponding constraints are created, etc. I added a hook on the server side to RtpReceiverImpl::IncomingRtpPacket which writes the received payload to a file. The file contains valid PCM audio. Therefore, I assume the client streams data successfully through the network to the server application.
On the server side, my callback PeerConnectionObserver::OnAddStream is called and receives a MediaStreamInterface. Furthermore, i can iterate with my DeviceManagerInterface::GetAudioOutputDevices through my audio devices. So basically, everything is fine.
What is missing: I need some kind of glue to tell WebRTC to play my AudioStream on the corresponding device. I have seen that I can get an AudioSink, AudioRenderer and AudioTrack objects. Again: Unfortunatly, I do not see an interface to pass them to the audio device. Can anyone help me with that?
One important note: I want to avoid to debug real hardware. Therefore, I added -DWEBRTC_DUMMY_FILE_DEVICES when building my WebRTC modules. It should write audio to an output file but the file just contains 0x00. The input file is read successfully because as I mentioned earlier, audio is send via RTP.
Finally, I found the solution: First, I have to say that my Code uses a WebRTC from 2017. So, the following things may have been changed and/or are fixed already:
After debugging my code and the WebRTC library I saw: When a remote stream is added, playback should start automatically. There is no need on the developer side to call playout() in the VoiceEngine or something comparable. When the library recognizes a remote audio stream, playback is paused, the new source is added to the mixer, and playback is resumed. The only APIs to control playback are provided by webrtc::MediaStreamInterface which is passed via the PeerConnectionObserver::OnAddStream. Examples are SetVolume() or set_enabled().
So, what went wrong in my case? I used the FileAudioDevice class which should write raw audio data to an output file instead of speakers. My implementation contains two bugs:
FileAudioDevice::Playing() returned true in any case. Due to this, the library added the remote stream, wanted to resume playout, called FileAudioDevice::Playing() which returned true and aborted because it thought the AudioDevice was already in playout mode.
There seems to be a bug in the FileWrapper class. The final output is written in FileAudioDevice::PlayThreadProcess() via _outputFile.Write(_playoutBuffer, kPlayoutBufferSize) onto disk. However, this does not work. Luckily, a plain C fwrite() as hacky bugfix does work.
Related
With QT 6.4.x (Windows), how can I capture microphone audio and repackage it and forward the repackaged audio to a QUdpSocket.
The repackaging will involve changing the captured audio format from its typical 16 bit little endian format and converting to 24 bit big endian format where each packet will have a constant size potentially different size payload to that from the microphone. I am not sure but somehow I think I need to replace the QAudioSink with a QAudioDecoder as the description indicates:
The QAudioDecoder class is a high level class for decoding audio media files. It is similar to the QMediaPlayer class except that audio is provided back through this API rather than routed directly to audio hardware.
I have a partially working example that contains a mixture of sending synthesized audio directly to the speaker. This functionality is based off the 'Audio Output Example' that ships with Qt 6 (my modified example sends a sine wave generated tone to the speakers).
Also in this RtpWorker thread, using the 'Audio Source Example' for inspiration, I was also able to capture and intercept audio packets from the microphone, but I do not know how to send these packets (repackaged per the above) to a UDP socket in a fixed size datagrams, instead I just log the captured packets. I think I need an intermediate circular buffer (the write part of which fills it with captured microphone audio while the read part gets called by a QAudioSink or QAudioDecoder in pull mode).
Per my comment above I think I might need to send them to a QAudioDevice so I can handle the packaging and sending over the network myself.
My code is contained in 2 attachment in the following QTBUG-108383.
It would be great if someone could point to some useful examples that try to do something similar.
try to run Mac OS or Linux its seems Windows bug
First of all, I hope you'll understand my poor english skill. I'll do my best to write question about it.
I'm currently developing some application. That application will captures desktop and encodes to MP4 video and send mp4 video data/chunks to another clients (all this process should run in memory not file). so another client should can play mp4 video/data/chunks that received from server.
so I used avio = avio_open(...) and reassigned write function as like avio->write_packet = some_function. but this way seems not "generate" mp4 file/chunk correctly. and It seems the original write_packet callback having actual logic that write packet to a file. but I don't want to write to a file. so I was thinking about it for a week but I did't getting any of idea about it.
so my question is: How can I intercept "encoded" video data that should processed by avio->write_packet?
Don't hijack avio_open-provided aviocontext, that's bound to cause you troubles. If you want to provide custom AVIOContext, do so - use avio_alloc_context and provide your callbacks there. Use flag AVFMT_FLAG_CUSTOM_IO to indicate that you did so (so this AVIOcontext won't be destroyed automatically when muxer closes).
I need to find out if it is possible and how (I do not care about the language C/C++, Lua, Python ...) to make a VLC plugin which purpose will be to be called by the VLC player and at specific times of the video stream will do some action.
The action that I need to do is to open a UDP socket and send some data read from a file that comes along with the video currently played.
I need to make something like a subtitle reader that on it's best can initialize UDP socket and send the read data to the server.
I am not sure that creation of UDP socket is possible in Lua maybe the better option will be a binary C/C++ plugin but can't find any example.
In general at the best my requirements are:
Load settings file at VLC launch
Need to be triggered by the player at specific times of the video stream
Get the file name of the source video stream
Open the file (script) with the same name but different extension
Open a UDP socket
Compose the message
Send the message
Continue the loop until the end of the video stream
Any information, example or site, link is greatly appreciated.
Looks like you would like to create a control interface module. Those are written in C/C++ within the VLC context and in turn need to be (re-) compiled for each platform you would like to target.
Have a look at the audioscrobbler module to see how to interact with the current input stream and how to retrieve metadata such as file name, etc. Since those modules are in C, opening sockets and transmitting data is not a big deal.
The biggest caveat is probably that you need a complex compilation environment if you would like to target the Windows platform. Have a look at the compilation HOWTO's on the wiki http://wiki.videolan.org/Compile_VLC/ since this is probably what you would like to try prior to doing any coding.
Thinking about it, you can probably achieve a similarly featured extension in lua, which is easier to develop (since you don't need to compile VLC yourself and it will cross-platform). Opening UDP sockets might be problematic though. TCP will just work. This page could be a nice starting point: http://www.coderholic.com/extending-vlc-with-lua/
In my application we will present the video stream from a traffic camera to a client viewer. (And eventually several client viewers.) The client should have the ability to watch the live video or rewind the video and watch earlier footage including video that occurred prior to connecting with the video stream. We intend to use wxWidgets to view the video and within that we will probably use the wxMediaCtrl.
Now, from the above statements some of you might be thinking "Hey, he doesn't know what he's talking about." And you would be correct! I'm new to these concepts and I confused by the surplus of information. Are the statements above reasonable? Can anyone recommend a basic server/client architecture for this? We will definitely be using C++ wxWidgets for the GUI, but perhaps wxMediaCtrl is not what I want... should I be directly using something like the ffmpeg libraries?
Our current method seems less than optimal. The server extracts a bitmap from each video frame and then waits for the single client to send a "next frame" message, at which point the server sends the bitmap. Effectively we've recreated our own awkward, non-standard, inefficient, and low-functionality video streaming protocol and viewer. There has to be something better!
You should check out this C++ RTMP Server: http://www.rtmpd.com/. I quickly downloaded, compiled and successfully tested it without any real problems (on Ubuntu Maverick). The documentation is pretty good if a little all over the place. I suspect that once you have a streaming media server capable of supporting the typical protocols (which rtmpd seems to do), then writing a client should fall into place naturally, especially if you're using wxWidgets as the interface api. Of course, it's easy to write that here, from the comfort of my living room, it'll be a different story when you're knee deep in code :)
you can modify your software such that:
The server connect, server grabs an image, passes it to ffmpeg establishing stream, then copy the encoded data from ffmpeg stream and send to client via network, if the connection drops, close the ffmpeg stream.
Maybe you can use the following to your own advantage:
http://www.kirsle.net/blog.html?u=kirsle&id=63
There is a player called VLC. It has a library for c++ and you can use it to embed the player in your GUI application. It supports a very wide range of protocols. So you should leave connecting, retrieving and playing jobs to VLC and take care of the starting and stopping jobs only. It would be easy and probably a better solution than doing it yourself.
For media playing facility, both music and audio, you can a look on GStream. And talking about the server, I think Twisted (A network library in Python) should be good option. The famous live video social website justin.tv is based on Twisted. Here you can read the story from here. Also, I built a group of server for streaming audio on Twisted, too. They can serve thousands of listener on line in same time.
I have an application that grabs video from multiple webcams, does some image processing, and displays the result on the screen. I'd like to be able to stream the video output on to the web - preferably to some kind of distribution service rather than connecting to clients directly myself.
So my questions are:
Do such streaming distribution services exist? I'm thinking of something like ShoutCAST relays, but for video. I'm aware of ustream.tv, but I think they just take a direct webcam connection rather than allow you to send any stream.
If so, is there a standard protocol for doing this?
If so, is there a free library implementation of this protocol for Win32?
Ideally I'd just like to throw a frame of video in DIB format at a SendToServer(bitmap) function, and have it compress, send, and distribute it for me ;)
Take a look at video LAN client (or VLC for short) as a means for streaming video.
As for distribution sites, I don't know how well it works with ustream.tv and similar new services.
ustream.tv works by using Adobe Flash's support for reading input from a webcam. To fake it out, you need a fake webcam driver. Looking on the ustream.tv site, they point to an application called WebCamMax that allows effects and splicing in video. It works by creating a pseudo-webcam that mixes video from one or more cameras along with other sources. Since that app can do it, your own code could do that too, although you'll probably need to write a Windows driver to get it all working correctly.