As the title, i'm trying to do the following things:
First, a videoCapture object that read frame from external source (like video, webcam).
Second, for each frame captured, the processing thread(s) processes it and imshow to the screen.
I've tried this:
#define FRAME_FPS 10
void frameCaptureThread(cv::VideoCapture* cap, cv::Mat* ref){
while (cap->read(*ref)) {
cv::waitKey(1000/FRAME_FPS);
}
}
void frameProcessThread(cv::Mat* ref){
while(true){
if(!ref->empty()){
/* Processing ... */
cv::imshow("frameWindow", *ref);
cv::waitKey(1000/FRAME_FPS);
}
}
}
int main() {
cv::VideoCapture videoCapture("videoPath");
cv::Mat sharedMat;
std::thread t1(frameCaptureThread, &videoCapture, &sharedMat);
std::thread t2(frameProcessThread, &sharedMat);
t1.join();
t2.join();
return 0;
}
This looks wrong right, because i think the frameProcessingThread is put in a while(True) loop thats run forerver and beside that, it also has to check if the *ref is empty or not.
Is there a way to make the Processing Thread only do the process if:
the new frame is coming AND it just finished it recent task.
And if i want to create another thread to do other processing task, what should i do ?
I've look around for some async concepts but have no idea what is the optimal way for this situation.
Related
I am trying to mess around with multi-threading and OpenCV so I wrote the following program.
void performSearch(Mat& original, Mat& grey, Mat& result)
{
for(size_t index = 0; index < container.size(); ++index)
{
// do stuff
imshow("Results", result);
waitKey();
}
}
int main()
{
thread(performSearch, ref(image), ref(image_grey), ref(answer)).join();
thread(performSearch, ref(image2), ref(image2_grey), ref(answer2)).join();
}
This program will throw an NSInternalInconsistencyException exception because waitKey() needs to be run on the main thread. My question is, how do I attach it to the main thread?
I have an iOS background and with grand central dispatch, I'd simply have to do something along the lines of:
DispatchQueue.main.async // this gets me the main queue asynchronously
{
// update UI
}
Is it possible to achieve this in C++? If yes, how?
Heads Up:
I have looked at this question and they're using QT for their GUI. I'm not.
I have a multi-threading application written in C++ with Qt5.7 and OpenNI. It has a main thread that starts a second thread which capture frame from a .oni recording file (asus xtion pro live) does some processing and through the Qt signal-slot mechanism pass the frame to the main thread, which display it using imshow().
What I want to do is to implement a pause key, so pressing for example 'p' the processing pause. I am thinking of something like this:
void Camera::run(){
while(!cameraStop && this->device.isValid())
{
try {
if (!buttonPause) {
getFrame();
process();
emit sigFrameImageReady(frame);
if (cv::waitKey(1)==112){
setButtonPause(!(getButtonPause()));
}
}
}
catch(std::exception &ex) {
std::cerr << "getFrame()" << ex.what() << std::endl;
}
}
}
In this way it doesn't work, I think that's because the frame is displayed by another thread (the main one), the waitKey() here simply blocks the entire process, but if I put it in the main thread, just after imshow() in this way:
void Process::FrameImageReady(cv::Mat FrameImage)
{
if (modedebug)
cv::imshow("bgr", FrameImage);
if (cv::waitKey(1)==112){
cam->setButtonPause(!(getButtonPause()));
}
}
waitkey seems to be ignored (image displaying works fine).. any idea?
EDIT
The GUI part is only for debugging purpose.
You should implement thread safe FIFO bufer or circular buffer in your displaying thread. Signal from the camera thread would be pushing images to this buffer and the displaying thread would be taking them out and display them in a separate loop. Only that way you separate the camera event loop from the display thread.
Maybe it's a late answer but I would like to present what I have done in my recent project, in case anyone is in the similar situation and want to use the same techniques.
Basically my project is a command line program and uses QT for multi threading & inter-thread communications, while we also want some minimal UI to display images, and handle user key events, so OpenCV Highgui module is pretty much enough for me. And here is what I did to make them work together:
(Important) you have to build OpenCV with QT support to make the
following code work, that is, check WITH_QT option in cmake-gui when building OpenCV. Or if you use vcpkg, use:
vcpkg install opencv[qt]:x64-windows
First create the class for working thread, I use this thread to constantly retrieve image frames from the camera buffer and notify the main thread every time the image is ready:
class MyThread : public QThread
{
Q_OBJECT
public:
MyThread() : mAbort(false)
{
}
~MyThread()
{
mMutex.lock();
mAbort = true; // make the thread out of the loop
mMutex.unlock();
wait(); // wait until the run() function returns
cout << "thread terminated" << endl;
}
protected:
void run() override
{
while(!mAbort) {
cv::Mat frame;
if (myCamera->grab(frame) && !frame.empty()) {
emit imageReady(frame);
}
else {
cout << "failed to grab" << endl;
}
}
}
signals:
void imageReady(const cv::Mat& frame);
private:
bool mAbort;
QMutex mMutex;
};
Then create the controller class to handle the signal from working thread, I use it to display the frame. Note it is running in the main thread:
class MyController : public QObject
{
Q_OBJECT
public:
MyController(MyThread* pThread) : mThread(pThread)
{
// this is needed as the argument is passed through queued connection
qRegisterMetaType<cv::Mat>("CvMat");
connect(mThread, &MyThread::imageReady, this, &MyController::onImageReady);
connect(mThread, &MyThread::finished, mThread, &QObject::deleteLater);
mThread->start();
}
public slots:
void onImageReady(const cv::Mat& frame) {
cv::imshow("camera live view", frame);
}
private:
MyThread* mThread;
};
Now in the main function, we can start the thread, and handle key events using cv::waitKey(), note it will also start the QT main event loop internally.
Q_DECLARE_METATYPE(cv::Mat)
...
int main(int argc, char* argv[])
{
...
// start working thread
MyThread thread = new MyThread();
MyController controller(thread);
// handle key events
while (true) {
int key = cv::waitKey();
cout << "key = " << key << endl;
// press "ESC" to exit
if (key == 27) {
break;
}
}
// clean up
...
delete thread;
delete myCamera;
return 0;
}
I want to run boost::asio::io_service.run() in a background thread. So when I need it post() func into.
This is main func:
int main(int /*argc*/, char** /*argv*/)
{
std::string message = "hello";
logg = new logger_client(filename,ip,13666);
logg->start();
while (true)
logg->add_string(message);
return 0;
}
And some relevant funcs from logger_client:
std::auto_ptr<boost::asio::io_service::work> work;
logger_client::logger_client(std::string& filename,std::string& ip, uint16_t port) : work(new boost::asio::io_service::work(io_service))
{
}
void logger_client::start()
{
ios_thread = new boost::thread(boost::bind(&io_service.run,&io_service));
}
void print_nothing()
{
printf("%s\n","lie");
}
void logger_client::add_string(std::string& message)
{
io_service.post(boost::bind(print_nothing));
//io_service.post(strand->wrap(boost::bind(&logger_client::add_string_imp,this,message)));
//io_service.run();
}
When i run this, my program eats 2Gb less than a minute. If i remove endless work and change to this:
void logger_client::add_string(std::string& message)
{
io_service.post(boost::bind(print_nothing));
//io_service.post(strand->wrap(boost::bind(&logger_client::add_string_imp,this,message)));
io_service.run();
}
Program works just fine. But I don't want to invoke async operations on this (main) thread. What am i doing wrong?
UPDATE
I added sleep(1sec) in while(true) loop and memory is no longer growing. But this is not a solution. Because if I call run() after post() (i.e. use main thread for processing handles) and even add five more threads with while(true) loops memory is not growing. So why main thread is so much better than newly created? I also tried thread pool for io_service::run - did not help.
io_service.run will exit unless there are pending operations.
Therefore, your ios_thread will exit immediately.
The solution is to use io_service::work.
In addition, endless loop spam like this
while (true)
logg->add_string(message);
is not a good idea, maybe add some sleep(), to slow it down a bit and keep it under control.
So my task is this - I have a GUI thread with sliders of HSV values (among other things), and a worker thread that does all the OpenCV work and sends processed video images back to GUI thread.
Like it usually is, the OpenCV work is inside of an endless loop. The thing is, half the work is transforming the current video frame according to HSV values sent from GUI sliders. If sent before the loop starts, it works. But not while it's going on, and I need it to work on the fly.
Is there any good way to communicate with that thread and change the HSV values the OpenCV loop is using, or is it a fool's errand? I can think of two solutions, one of which is probably highly inefficient (involves saving values to a file). I'm fairly new to Qt, and I could've easly missed something in the documentation and tutorials.
edit:
Here's how my app works - in GUI thread, user picks a file. A signal with an url is sent to the worker thread, which starts working away. When the user changes HSV values, a signal is sent to change the values from another thread. If the loop hasn't been started, they're received and QDebug shows me that.
edit2:
I might've been thinking about it all wrong. Is there a way for the thread to pull values from the other one? Instead of waiting for them to be sent?
edit3:
kalibracja.cpp, for Micka.
int hueMin=0;
int hueMax=180;
int satMin=0;
int satMax=255;
int valMin=15;
int valMax=255;
int satMinDua=133; //tests
HSV::HSV(QObject * parent) : QObject(parent)
{
hsvThread = new QThread;
hsvThread ->start();
moveToThread( hsvThread );
}
HSV::~HSV() //destruktor
{
hsvThread ->exit(0);
hsvThread ->wait();
delete hsvThread ;
}
void HSV::processFrames(QString kalibracja) {
while(1) {
cv::VideoCapture kalibrowanyPlik;
kalibrowanyPlik.open(kalibracja.toStdString());
int maxFrames = kalibrowanyPlik.get(CV_CAP_PROP_FRAME_COUNT);
for(int i=0; i<maxFrames; i++)
{
cv::Mat frame;
cv::Mat gray;
//satMin=kontenerHsv->satMin;
qDebug() << "kalibracja satMin - " << satMin;
qDebug() << "fdfdf - " << satMinDua;
kalibrowanyPlik.read(frame);
cv::cvtColor(frame, gray, CV_BGR2GRAY);
QImage image(cvMatToQImage(frame));
QImage processedImage(cvMatToQImage(gray));
emit progressChanged(image, processedImage);
QThread::msleep(750); //so I can read qDebug() messages easly
}
}
}
void MainWindow::onProgressChagned(QImage image, QImage processedImage) {
QPixmap processed = QPixmap::fromImage(processedImage);
processed = processed.scaledToHeight(379);
ui->labelHsv->clear();
ui->labelHsv->setPixmap(processed);
QPixmap original = QPixmap::fromImage(image);
original = original.scaledToHeight(379);
ui->labelKalibracja->clear();
ui->labelKalibracja->setPixmap(original);
}
void HSV::updateHsv(QString hmin, QString hmax, QString smin, QString smax, QString vmin, QString vmax){
satMinDua=smin.toInt();
}
mainwindow.cpp connection
HSV *hsv = new HSV;
(.... all kinds of things ....)
void MainWindow::updateHsvValues() {
QMetaObject::invokeMethod(hsv, "updateHsv", Qt::QueuedConnection,
Q_ARG(QString, hmin),
Q_ARG(QString, hmax),
Q_ARG(QString, smin),
Q_ARG(QString, smax),
Q_ARG(QString, vmin),
Q_ARG(QString, vmax));
}
It is certainly possible, but you need to be careful.
One of the ways to achieve this would be:
Create an object that stores the "current" HSV values to be used
Give a reference (or pointer) to this object to both the GUI thread and the OpenCV thread
When the GUI wants to "tell" the processing thread to use new values, it published them to that object
When the processing thread is ready to move on the the next frame (start of loop body), it fetches the values from that object.
You only need to make sure that the set and get methods on that shared object are synchronized, using a mutex for example, to prevent the processing thread from reading half-written values (data races lead to undefined behavior in C++).
If you use QThread in the "wrong" way (by subclassing QThread and using ::run , compare to https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/ ), signal-slot parameter change works in endless loops too:
This is a small sample thread for testing:
void MyThread::run()
{
// start an infinite loop and test whether the sliderchange changes my used parameters
std::cout << "start infinite loop" << std::endl;
while(true)
{
unsigned long long bigVal = 0;
int lastVal = mValue;
std::cout << "start internal processing loop " << std::endl;
for(unsigned long long i=0; i<1000000000; ++i)
{
bigVal += mValue;
if(lastVal != mValue)
{
std::cout << "changed value: " << mValue << std::endl;
lastVal = mValue;
}
}
std::cout << "end internal processing loop: " << bigVal << std::endl;
}
std::cout << "stop infinite loop" << std::endl;
}
with this SLOT, which is connected to the main window slider SIGNAL
void MyThread::changeValue(int newVal)
{
// change a paramter. This is a slot which will be called by a signal.
// TODO: make this call thread-safe, e.g. by atomic operations, mutual exclusions, RW-Lock, producer-consumer etc...
std::cout << "change value: " << newVal << std::endl;
mValue = newVal;
}
giving me this result after using the slider:
this is how the slot was connected:
QObject::connect(mSlider, SIGNAL(valueChanged(int)), mTestThread, SLOT(changeValue(int)) );
if the infinite loop is performed as some kind of workerObject method which was moved to the thread with moveToThread, you can either change the way how the slot is called:
QObject::connect(mSlider, SIGNAL(valueChanged(int)), mTestThread, SLOT(changeValue(int)), Qt::DirectConnection );
Never used, but I guess the same should work for invoke:
QMetaObject::invokeMethod(hsv, "updateHsv", Qt::DirectConnection, ...
(the main thread will call changeValue then so the worker thread doesnt need to stop processing to change values => value access should be thread safe!
or you have to process the event queue of that thread:
while(true)
{
[processing]
QApplication::processEvents();
}
I think the simplest solution here may be to take advantage of the fact that Qt Signals/Slots work across threads.
Setup the appropriate slots in the processing thread and then signal them from the GUI thread.
There are all sorts of interesting questions about whether you signal for every user input, or whether you batch up changes for a moment on the GUI side...
There is some ideas for thread sync in the docs: http://doc.qt.io/qt-5/threads-synchronizing.html
I want to render via openGL 2camera in the same time. I want to refresh each frame as soon as possible. To update those frame I am using a thread in an infinite loop.
My actual problem is that my program crash if I put the resolution of those camera to high.
With 160:120 there's no problem. But if i put the max resolution (1920:1080) there's just 5 or 6 update of the image before crashing.
note: the number of update is not always the same before crashing
I suppose that If the resolution is low enough, frame_L and frame_R are changed quick enough that there's no colision beetween the main loop and the thread.
So I suppose that my mutex isnt doing what it should. How should I do?
(I'm not an expert in thread and variable safety)
My code:
#include <boost/thread/mutex.hpp>
#include <boost/thread/thread.hpp>
#include <opencv2/opencv.hpp>
boost::mutex m; //for an other thread
boost::mutex n;
cv::VideoCapture capture_L(0);
cv::VideoCapture capture_R(1);
cv::Mat frame_L;
cv::Mat frame_R;
void MyThreadFunction()
{
while (1)
{
{
boost::mutex::scoped_lock lk(n);
if (capture_L.grab()){
capture_L.retrieve(frame_L);
cv::transpose(frame_L, frame_L);
}
if (capture_R.grab()){
capture_R.retrieve(frame_R);
cv::transpose(frame_R, frame_R);
}
}
}
}
int main()
{
capture_L.set(CV_CAP_PROP_FRAME_WIDTH, 160);
capture_L.set(CV_CAP_PROP_FRAME_HEIGHT, 120);
capture_R.set(CV_CAP_PROP_FRAME_WIDTH, 160);
capture_R.set(CV_CAP_PROP_FRAME_HEIGHT, 120);
boost::thread thrd(&MyThreadFunction);
while(1)
{
[ use frame_L and frame_R ]
}
}
This is the code that I use for my threaded camera grab. Its part of a camera Object (eg; Each camera has it's own object and own thread for the capturing.
void Camera::setCapture(cv::VideoCapture cap)
{
pthread_mutex_lock(&latestFrameMutex);
videoCapture = cap;
videoCapture.read(latestFrame);
pthread_mutex_unlock(&latestFrameMutex);
int iret = pthread_create(&cameraGrabThread,NULL,&Camera::exec,this);
}
void *Camera::exec(void* thr)
{
reinterpret_cast<Camera *> (thr)->grabFrame();
}
This ensures that a new thread is started when a capture is set for that camera. The following code is run by the exec part of the thread to actually grab the frame.
void *Camera::grabFrame()
{
while(videoCapture.isOpened())
{
pthread_mutex_lock(&latestFrameMutex);
if(!videoCapture.read(latestFrame))
std::cout << "Unable to read frame" << std::endl;
pthread_mutex_unlock(&latestFrameMutex);
usleep(8000); // Sleep 8ms
}
}
And finally, the camera needs to be able to return the latest frame;
cv::Mat Camera::getLatestFrame()
{
while (getMilisecSinceLastCapture() < 35) //Enforce min time of 35 ms between frame requests.
{
usleep(5000);
}
pthread_mutex_lock(&latestFrameMutex);
cv::Mat result = latestFrame.clone();
pthread_mutex_unlock(&latestFrameMutex);
return result.clone();
}
Changing the size can be done by using
void Camera::setImageSize(const cv::Size& size)
{
pthread_mutex_lock(&latestFrameMutex);
videoCapture.set(CV_CAP_PROP_FRAME_HEIGHT, size.height);
videoCapture.set(CV_CAP_PROP_FRAME_WIDTH, size.width);
pthread_mutex_unlock(&latestFrameMutex);
}