Multi-threading with QT + OpenCV - c++

Question

I'm trying to code a simple program that reads three video files (actually, 3 cameras that are in the same room), using 3 different threads. The code I'm using is the following:
mainwindow.cpp
void MainWindow::init()
{
numCams = 3;
// Resize the video for displaying to the size of the widget
int WidgetHeight = ui->CVWidget1->height();
int WidgetWidth = ui->CVWidget1->width();
for (int i = 0; i < numCams; i++){
// Create threads
threads[i] = new QThread;
// Create workers
string Path = "/Users/alex/Desktop/PruebasHilos/Videos/" + to_string(i+1) + ".m2v";
workers[i] = new Worker(QString::fromStdString(Path), i, WidgetHeight, WidgetWidth);
workers[i]->moveToThread(threads[i]);
connectSignals2Slots(threads[i], workers[i]);
threads[i]->start();
qDebug() << "Thread from camera " << (i+1) << " started";
}
}
void MainWindow::connectSignals2Slots(QThread *thread, Worker *worker)
{
connect(thread, SIGNAL(started()), worker, SLOT(readVideo()));
connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
connect(worker, SIGNAL(frameFinished(Mat, int)), this, SLOT(displayFrame(Mat,int)));
connect(worker, SIGNAL(finished(int)), thread, SLOT(quit()));
connect(worker, SIGNAL(finished(int)), worker, SLOT(deleteLater()));
}
void MainWindow::displayFrame(Mat frame, int index)
{
if (index == 0) {
// Camera 1
ui->CVWidget1->showImage(frame);
}
else if (index == 1) {
// Camera 2
ui->CVWidget2->showImage(frame);
}
else if (index == 2) {
// Camera 3
ui->CVWidget3->showImage(frame);
}
}
worker.cpp
Worker::Worker(QString path, int id, int WidgetHeight, int WidgetWidth) : filepath(path), index(id), WidgetHeight(WidgetHeight), WidgetWidth(WidgetWidth) {
}
Worker::~Worker(){
}
void Worker::readVideo()
{
VideoCapture cap(filepath.toStdString());
if (! cap.isOpened()) {
qDebug() << "Can't open video file " << filepath;
emit finished(index);
return;
}
Mat ActualFrame;
while (true) {
cap >> ActualFrame;
if (ActualFrame.empty()) {
// Empty frame to display when the video has finished
ActualFrame = Mat(Size(720, 576), CV_8UC3, Scalar(192, 0, 0));
emit frameFinished(ActualFrame, index);
qDebug() << "Video finished";
break;
}
// Background Subtraction
BackgroundSubtraction(ActualFrame, BackgroundMask);
emit frameFinished(ActualFrame.clone(), index);
QThread::msleep(35);
}
emit finished(index);
}
void Worker::BackgroundSubtraction(Mat ActualFrame, Mat &BackgroundMask)
{
pMOG2->apply(ActualFrame, BackgroundMask);
}
Just reading the frames from VideoCapture and displaying them into the UI by another different class that uses QWidgets works well.
However, when I include the BackgroundSubstraction method, the UI does not display the same frame number for the three cameras, maybe Camera1 is computing frame 100 and Camera2 and Camera3 are in frame 110.
This is because some frames are calculated faster than other and this leads to syntonization problems.
I'm quite new using threads in QT so i would like to make some synconization between threads so I know when the three different frames have been process in order to call the displayFrame method, and so, that the three same frames are displayed at the exact same time.
EDIT:
I assume that the easiest way to do this is using Barriers.
http://www.boost.org/doc/libs/1_55_0/doc/html/thread/synchronization.html#thread.synchronization.barriers . But I have no clue how to do this.
EDIT 2:
I have implemented this Syncronizacion using barriers and now the code looks like this:
barrier.h
#ifndef BARRIER_H
#define BARRIER_H
#include <QMutex>
#include <QWaitCondition>
#include <QSharedPointer>
// Data "pimpl" class (not to be used directly)
class BarrierData
{
public:
BarrierData(int count) : count(count) {}
void wait() {
mutex.lock();
--count;
if (count > 0)
condition.wait(&mutex);
else
condition.wakeAll();
mutex.unlock();
}
private:
Q_DISABLE_COPY(BarrierData)
int count;
QMutex mutex;
QWaitCondition condition;
};
class Barrier {
public:
// Create a barrier that will wait for count threads
Barrier(int count) : d(new BarrierData(count)) {}
void wait() {
d->wait();
}
private:
QSharedPointer<BarrierData> d;
};
#endif // BARRIER_H
updated worker.cpp
void Worker::readVideo()
{
VideoCapture cap(filepath.toStdString());
int framenumber = 0;
if (! cap.isOpened()) {
qDebug() << "Can't open video file " << filepath;
emit finished(index);
return;
}
Mat ActualFrame;
while (true) {
cap >> ActualFrame;
if (ActualFrame.empty()) {
// Empty frame to display when the video has finished
ActualFrame = Mat(Size(720, 576), CV_8UC3, Scalar(192, 0, 0));
emit frameFinished(ActualFrame, index);
qDebug() << "Video finished";
break;
}
// Background Subtraction
BackgroundSubtraction(ActualFrame, BackgroundMask);
QThread::msleep(5);
barrier.wait();
qDebug() << "Thread " << index << " processing frame " << framenumber ;
emit frameFinished(ActualFrame.clone(), index);
framenumber++;
}
emit finished(index);
}
void Worker::BackgroundSubtraction(Mat ActualFrame, Mat &BackgroundMask)
{
pMOG2->apply(ActualFrame, BackgroundMask);
}
It seems to work perfectly, however the output of the program is the following:
Thread 1 processing frame 0
Thread 0 processing frame 0
Thread 2 processing frame 0
Thread 2 processing frame 1
Thread 1 processing frame 1
Thread 0 processing frame 1
Thread 2 processing frame 2
Thread 1 processing frame 2
Thread 0 processing frame 2
Thread 2 processing frame 3
Thread 1 processing frame 3
Thread 0 processing frame 3
Thread 2 processing frame 4
Thread 1 processing frame 4
Thread 0 processing frame 4
Thread 2 processing frame 5
Thread 0 processing frame 5
Thread 1 processing frame 5
Thread 2 processing frame 6
Thread 1 processing frame 6
Thread 2 processing frame 7
Thread 0 processing frame 6
Thread 1 processing frame 7
Thread 2 processing frame 8
Thread 0 processing frame 7
Thread 1 processing frame 8
Thread 2 processing frame 9
Thread 0 processing frame 8
Thread 1 processing frame 9
Thread 1 processing frame 10
Thread 2 processing frame 10
Thread 0 processing frame 9
Thread 1 processing frame 11
Thread 2 processing frame 11
Thread 0 processing frame 10
Thread 1 processing frame 12
At the beginning the syncronization is perfectly working, but then it seems that the barrier is not working and threads are not waiting to each other...
EDIT 3: SOLVED
It seems that changing the value of
QThread::msleep(5);
to
QThread::msleep(35);
solves the problem of the syncronization although I do not really understand the reason.

Answer 1

even without the background subtraction you'd need some synchronization to be sure that the same frame number is processed by each thread.
In Qt the easiest (and imho the right) way to do it is to remove the infinite loop and instead call a slot of each thread to compute the next image, after all the threads emitted their signal frameFinished.
You could further use some buffering to precompute images in your threads and just load them from that buffer. In that scenario you could do the following:
each of your threads fills his buffer in an endless loop as long as there is free buffer space available. If the buffer is full, the thread waits until buffer space got freed.
when your gui has displayed and has waited some time, it sends a signal that is connected to each thread's slot like sendMeANewImage.
each thread sends the next available image from its buffer, or waits (infinite loop or conditional wait) for an image, if the buffer is empty. Then emits a frameFinished signal and frees the used buffer-space.
when each thread has emitted the signal, display all the images, wait some time and emit sendMeANewImage again.
This isn't threadsafe yet, you'll have the critical sections in reading and writing from the buffer. For each buffer, create a QMutex and call mutex.lock() whenever reading or writing or asking size etc. from that buffer. Call mutex.unlock() immediately afterwards.
When a mutex is locked and another thread (or even the same thread) tries to lock it again, the thread will wait there, until the other thread has unlocked the mutex. That way, only a single thread can enter a critical section.