using opencv waitKey() in a multithreading application - c++

Question

I have a multi-threading application written in C++ with Qt5.7 and OpenNI. It has a main thread that starts a second thread which capture frame from a .oni recording file (asus xtion pro live) does some processing and through the Qt signal-slot mechanism pass the frame to the main thread, which display it using imshow().
What I want to do is to implement a pause key, so pressing for example 'p' the processing pause. I am thinking of something like this:
void Camera::run(){
while(!cameraStop && this->device.isValid())
{
try {
if (!buttonPause) {
getFrame();
process();
emit sigFrameImageReady(frame);
if (cv::waitKey(1)==112){
setButtonPause(!(getButtonPause()));
}
}
}
catch(std::exception &ex) {
std::cerr << "getFrame()" << ex.what() << std::endl;
}
}
}
In this way it doesn't work, I think that's because the frame is displayed by another thread (the main one), the waitKey() here simply blocks the entire process, but if I put it in the main thread, just after imshow() in this way:
void Process::FrameImageReady(cv::Mat FrameImage)
{
if (modedebug)
cv::imshow("bgr", FrameImage);
if (cv::waitKey(1)==112){
cam->setButtonPause(!(getButtonPause()));
}
}
waitkey seems to be ignored (image displaying works fine).. any idea?
EDIT
The GUI part is only for debugging purpose.

Answer 1

You should implement thread safe FIFO bufer or circular buffer in your displaying thread. Signal from the camera thread would be pushing images to this buffer and the displaying thread would be taking them out and display them in a separate loop. Only that way you separate the camera event loop from the display thread.

Answer 2

Maybe it's a late answer but I would like to present what I have done in my recent project, in case anyone is in the similar situation and want to use the same techniques.
Basically my project is a command line program and uses QT for multi threading & inter-thread communications, while we also want some minimal UI to display images, and handle user key events, so OpenCV Highgui module is pretty much enough for me. And here is what I did to make them work together:
(Important) you have to build OpenCV with QT support to make the
following code work, that is, check WITH_QT option in cmake-gui when building OpenCV. Or if you use vcpkg, use:
vcpkg install opencv[qt]:x64-windows
First create the class for working thread, I use this thread to constantly retrieve image frames from the camera buffer and notify the main thread every time the image is ready:
class MyThread : public QThread
{
Q_OBJECT
public:
MyThread() : mAbort(false)
{
}
~MyThread()
{
mMutex.lock();
mAbort = true; // make the thread out of the loop
mMutex.unlock();
wait(); // wait until the run() function returns
cout << "thread terminated" << endl;
}
protected:
void run() override
{
while(!mAbort) {
cv::Mat frame;
if (myCamera->grab(frame) && !frame.empty()) {
emit imageReady(frame);
}
else {
cout << "failed to grab" << endl;
}
}
}
signals:
void imageReady(const cv::Mat& frame);
private:
bool mAbort;
QMutex mMutex;
};
Then create the controller class to handle the signal from working thread, I use it to display the frame. Note it is running in the main thread:
class MyController : public QObject
{
Q_OBJECT
public:
MyController(MyThread* pThread) : mThread(pThread)
{
// this is needed as the argument is passed through queued connection
qRegisterMetaType<cv::Mat>("CvMat");
connect(mThread, &MyThread::imageReady, this, &MyController::onImageReady);
connect(mThread, &MyThread::finished, mThread, &QObject::deleteLater);
mThread->start();
}
public slots:
void onImageReady(const cv::Mat& frame) {
cv::imshow("camera live view", frame);
}
private:
MyThread* mThread;
};
Now in the main function, we can start the thread, and handle key events using cv::waitKey(), note it will also start the QT main event loop internally.
Q_DECLARE_METATYPE(cv::Mat)
...
int main(int argc, char* argv[])
{
...
// start working thread
MyThread thread = new MyThread();
MyController controller(thread);
// handle key events
while (true) {
int key = cv::waitKey();
cout << "key = " << key << endl;
// press "ESC" to exit
if (key == 27) {
break;
}
}
// clean up
...
delete thread;
delete myCamera;
return 0;
}