In webRTC no frame transmitted to browser from C++ code after calling onFrame in the capturer - c++

Question

I am trying to make my app up to date with webrtc. This is basically a desktop streaming application with a C++ application video streaming to a browser using webRTC.
My implementation used to use a bunch of deprecated stuff like SignalFrameCaptured and cricket::CapturedFrame.
Looking at webRTC right now it seems those classes/signals are not available.
Here is my capturer:
class Capturer
: public cricket::VideoCapturer,
public webrtc::DesktopCapturer::Callback
{
public:
sigslot::signal1<Capturer*> SignalDestroyed;
Capturer();
~Capturer();
void ResetSupportedFormats(const std::vector<cricket::VideoFormat>& formats);
bool CaptureFrame();
cricket::CaptureState Start(const cricket::VideoFormat& format) override;
void Stop() override;
bool IsRunning() override;
bool IsScreencast() const override;
bool GetPreferredFourccs(std::vector<uint32_t>* fourccs) override;
virtual void OnCaptureResult(webrtc::DesktopCapturer::Result result,
std::unique_ptr<webrtc::DesktopFrame> desktop_frame) override;
private:
bool running_;
int64_t initial_timestamp_;
int64_t next_timestamp_;
std::unique_ptr<webrtc::DesktopCapturer> _capturer;
};
Capturer::CaptureFrame() is called periodically from another thread and Capturer::OnCaptureResult is called as expected with a DesktopFrame as parameter.
Now looking at the implementation of OnCaptureResult:
void Capturer::OnCaptureResult(webrtc::DesktopCapturer::Result result,
std::unique_ptr<webrtc::DesktopFrame> desktopFrame)
{
if (result != webrtc::DesktopCapturer::Result::SUCCESS)
{
return; // Never called. Which leads me to conclude thedesktop capturer works
}
int width = desktopFrame->size().width();
int height = desktopFrame->size().height();
rtc::scoped_refptr<webrtc::I420Buffer> res_i420_frame = webrtc::I420Buffer::Create(width, height);
webrtc::ConvertToI420(webrtc::VideoType::kABGR,
desktopFrame->data(),
0, 0,
width, height,
0,
webrtc::kVideoRotation_0,
res_i420_frame);
webrtc::VideoFrame frame = webrtc::VideoFrame(res_i420_frame, webrtc::kVideoRotation_0, next_timestamp_ / rtc::kNumNanosecsPerMicrosec);
this->OnFrame(frame, width, height);
}
No frame are ever transmitted to the connected browser, as seen from chrome::webrtc-internals.
Back to the code, here is how I create the peerConnection:
void Conductor::connectToPeer() {
this->_peerConnectionFactory = webrtc::CreatePeerConnectionFactory();
if (!this->_peerConnectionFactory.get())
{
std::cerr << "Failed to initialize PeerConnectionFactory" << std::endl;
throw new std::runtime_error("Cannot initialize PeerConnectionFactory");
}
webrtc::PeerConnectionInterface::RTCConfiguration config;
webrtc::PeerConnectionInterface::IceServer server;
server.uri = "stun:stun.l.google.com:19302";
config.servers.push_back(server);
webrtc::FakeConstraints constraints;constraints.AddOptional(webrtc::MediaConstraintsInterface::kEnableDtlsSrtp, "true");
this->_peerConnection = this->_peerConnectionFactory->CreatePeerConnection(config, &constraints, NULL, NULL, this);
if (!this->_peerConnection.get())
{
std::cerr << "Failed to initialize PeerConnection" << std::endl;
throw new std::runtime_error("Cannot initialize PeerConnection");
}
auto capturer = new Capturer();
CapturerThread *capturerThread = new CapturerThread(capturer); // This thread sole functionis to call CaptureFrameperiodically
rtc::scoped_refptr<webrtc::VideoTrackSourceInterface> videoSource = this->_peerConnectionFactory->CreateVideoSource(capturer, NULL);
rtc::scoped_refptr<webrtc::VideoTrackInterface> videoTrack(this->_peerConnectionFactory->CreateVideoTrack("video_label", videoSource));
rtc::scoped_refptr<webrtc::MediaStreamInterface> stream = this->_peerConnectionFactory->CreateLocalMediaStream("stream_label");
stream->AddTrack(videoTrack);
if (!this->_peerConnection->AddStream(stream))
{
std::cerr << "Adding stream to PeerConnection failed" << std::endl;
throw new std::runtime_error("Cannot add stream");
}
typedef std::pair<std::string,rtc::scoped_refptr<webrtc::MediaStreamInterface>> MediaStreamPair;
this->_activeStreams.insert(MediaStreamPair(stream->label(), stream));
webrtc::SdpParseError error;
webrtc::SessionDescriptionInterface* sessionDescription(webrtc::CreateSessionDescription("offer", this->_offer, &error));
if (!sessionDescription)
{
std::cerr << "Cannot initialize session description." << std::endl;
throw new std::runtime_error("Cannot set session description");
}
this->_peerConnection->SetRemoteDescription(DummySetSessionDescriptionObserver::Create(), sessionDescription);
this->_peerConnection->CreateAnswer(this, NULL);
}
My problem is no video frames are even transmitted to the browser client even though both the capturer and the peer connection are in place as expected. Is there something I'm missing ?

Answer 1

I got to the bottom of this. The local description was simply not set after creating the answer.
Calling createAnswer will hopefully trigger the OnSuccess callback in the CreateSessionDescriptionObserver (in my case Conductor).
There I failed to save the answer as the local description. Here is how the OnSuccess method should be implemented:
void Conductor::OnSuccess(webrtc::SessionDescriptionInterface *desc)
{
this->_peerConnection->SetLocalDescription(DummySetSessionDescriptionObserver::Create(), desc);
}
Doing this will ultimately trigger OnIceGatheringChange with kIceGatheringComplete as parameter, meaning both sides are ready.

Answer 2

According with google group discuss-webrtc thread
you should implement VideoTrackSourceInterface. In my case, I used the base class AdaptedVideoTrackSource and I created a method FrameCaptured it's is called from my thread. In the method FrameCaptured I call the base method OnFrame. It's work fine !!!
class StreamSource : public rtc::AdaptedVideoTrackSource
{
void OnFrameCaptured(const webrtc::VideoFrame& frame);
}
void StreamSource::OnFrameCaptured(const webrtc::VideoFrame& frame)
{
OnFrame(frame);
}