I have a loop to take in images from a high speed framegrabbger at 250fps.
/** Loop processes 250 video frames per second **/
while(1){
AcquireFrame();
DoProcessing();
TakeAction();
}
At the same time, I would like the user to be able to monitor what is going on. The user only needs to see images at around 30 fps (or less). How do I set up a second thread that displays the current frame every so often?
Thread(){
cvShowImage();
Wait(30); /** Wait for 30 ms **/
}
I am on Windows on a quad core Intel machine using MinGW, gcc and OpenCV 1.1. The main criteria is that the display thread must take as little time away from my main processing loop as possible. Every millisecond counts.
I have tried using CreateThread() to create a new thread with cvShowImage() and cvWaitKey() but apparently those functions are not threadsafe.
I am considering using OpenMP, but some people report problems with OpenMP and OpenCV. I also am considering trying to use DirectX directDraw because apparently it is very fast. but it looks complicated and evidentally there are problems using Windows DLL's with MinGw.
Which of these avenues would be the best place to start?
Ok. So embarrassingly my question is also its own answer.
Using CreateThread(), CvShowImage() and CvWaitKey() as described in my question actually works-- contrary to some postings on the web which suggest otherwise.
In any event, I implemented something like this:
/** Global Variables **/
bool DispThreadHasFinished;
bool MainThreadHasFinished;
iplImage* myImg;
/** Main Loop that loops at >100fps **/
main() {
DispThreadHasFinished = FALSE;
MainThreadHasFinished = FALSE;
CreateThread(..,..,Thread,..);
while( IsTheUserDone() ) {
myImg=AcquireFrame();
DoProcessing();
TakeAction();
}
MainThreadHasFinished = TRUE;
while ( !DisplayThreadHasFinished ) {
CvWaitKey(100);
}
return;
}
/** Thread that displays image at ~30fps **/
Thread() {
while ( !MainThreadHasFinished ) {
cvShowImage(myImg);
cvWaitKey(30);
}
DispThreadHasFinished=TRUE;
return;
}
When I originally posted this question, my code was failing for unrelated reasons. I hope this helps!
Since the frame grabbing doesn't need to use the UI, I'd set up a secondary thread to handle the frame grabbing, and have the original thread that handles the UI display the sample frames. If you tried to display the frame currently be grabbed, you'd have to lock the data (which is generally fairly slow). To avoid that, I'd display a frame one (or possibly two) "behind" the one currently being grabbed, so there's no contention between grabbing and displaying the data. You'll still have to ensure that incrementing the current frame number is thread-safe, but that's pretty simple -- use InterlockedIncrement in the capture thread.
I'm sorry I can't give you a better answer right now, but it seems that your question is not about the structure of your program but rather about the tool you should use to implement multithreading. For this I would recommend Qt. I have been using Qt for a while but I'm just now getting into multithreading.
It seems to me that your best bet might be a QReadWriteLock. This allows you to read from an image but the reader thread will give up its lock when the writer thread comes along. In this case you could keep a copy of the image you last displayed and display it if the image is locked for writing.
Sorry again that I can't be more detailed but, like I said, I'm just getting into this as well. I'm basically trying to do the same thing that you are, but not nearly as fast :). Good luck!
I'm not sure why this happens, but I've added a cvWaitKey after every cvShowImage and the picture was displayed properly.
cvShowImage(myImage);
cvWaitKey(1);
Related
I am using Qt 5.7 for a GUI application with a QProgressBar. I suspect that there might be a memory leak since memory usage increases during runtime for about 50MB/s. I could narrow down the problem to one line of code.
QProgressBar *pbarQuality;
...
int curQuality = data.getQuality();
if (curQuality < 0) {
curQuality = 0;
qWarning("Value set to 0. ");
}
if (curQuality > 100) {
curQuality = 100;
qWarning("Value set to 100. ");
}
ui.pbarQuality->setValue(curQuality); //The memory problem doesn't occur when this single line is commented out
The value of the QProgressBar(pbarQuality) is only for displaying. It isn't used anywhere else.
I find this a very strange behaviour. Am I missing something?
Here is the auto generated code by the Qt Designer:
pbarQuality = new QProgressBar(frame_5);
pbarQuality->setObjectName(QStringLiteral("pbarQuality"));
pbarQuality->setGeometry(QRect(10, 50, 130, 23));
pbarQuality->setValue(24);
Try replacing setValue by pbarQuality.update(); QCoreApplication::processEvents(); and see if that reproduces the problem. If it does, you're leveraging the nested event loop to keep the GUI responsive while your blocking code runs, and that's a bad thing. setValue calls processEvents as a naive way to work around broken user code. IMHO it's a dangerous favor. The only fix then is to unbreak your code and return control to the main event loop instead of blocking.
This answer shows how to avoid the effects of an image storm by leveraging the QImage's RAII behavior, and links to another answer that demonstrates free image scaling by leveraging OpenGL.
My application runs another thread besides the GUI thread which periodically(up to 60 times per second) sends information(images) to the GUI thread. I am doing some minor image editing (resizing) within the GUI thread. Turns out this takes too long in order to keep up with the data posted by the other thread. Consequentially the event queue gets bigger and bigger and so does the used RAM too.
Lesson learned: Be aware of the processing speed of the thread if the data gets posted periodically. Data processing needs to be done before new data is available.
Thanks to #KubaOber for giving me the hint.
If you've ever used XNA game studio 4 you are familiar with the update method. By default the code within is processed at 60 times per second. I have been struggling to recreate such an effect in c++.
I would like to create a method where it will only process the code x amount of times per second. Every way I've tried it processes all at once, as loops do. I've tried for loops, while, goto, and everything processes all at once.
If anyone could please tell me how and if I can achieve such a thing in c++ it would be much appreciated.
With your current level of knowledge this is as specific as I can get:
You can't do what you want with loops, fors, ifs and gotos, because we are no longer in the MS-DOS era.
You also can't have code running at precisely 60 frames per second.
On Windows a system application runs within something called an "event loop".
Typically, from within the event loop, most GUI frameworks call the "onIdle" event, which happens when an application is doing nothing.
You call update from within the onIdle event.
Your onIdle() function will look like this:
void onIdle(){
currentFrameTime = getCurrentFrameTime();
if ((currentFrameTime - lastFrameTime) < minUpdateDelay){
sleepForSmallAmountOfTime();//using Sleep or anything.
//Delay should be much smaller than minUPdateDelay.
//Doing this will reduce CPU load.
return;
}
update(currentFrameTime - lastFrameTime);
lastFrameTime = currentFrameTime;
}
You will need to write your own update function, your update function should take amount of time passed since last frame, and you need to write a getFrameTime() function using either GetTickCount, QueryPerformanceCounter, or some similar function.
Alternatively you could use system timers, but that is a bad idea compared to onIdle() event - if your app runs too slowly.
In short, there's a long road ahead of you.
You need to learn some (preferably cross-platform) GUI framework, learn how to create a window, the concept of an event loop (can't do anything without it today), and then write your own "update()" and get a basic idea of multithreading programming and system events.
Good luck.
As you are familiar with XNA then i assume you also are familiar with "input" and "draw". What you could do is assign independant threads to these 3 functions and have a timer to see if its time to run a thread.
Eg the input would probably trigger draw, and both draw and input would trigger the update method.
-Another way to handle this is my messages events. If youre using Windows then look into Windows messages loop. This will make your input, update and draw event easier by executing on events triggered by the OS.
Got really stuck, need some advise or real examples.
1) I have boost::thread vector producer thread (data arrives fast ~ 100 samples per second)
2) I want QMathGL to paint data as it arrives
3) I don't want my Qt gui freeze
I tried to move QMathGL::update() to separate thread - Qt argues that QPixmap not allowed in separate thread.
What should i try, Without modifying QMathGL?
Only thing comes in mind to repaint on timer (fps?), but i don't like this solution, please tell me if i am wrong.
I would strongly advise to go with a timer. Repaint operations are costly and I would assume that no user could realistically process more then 10 printed vectors a second. So I can't see a real benefit for the end user, apart from maybe that the display is updated more "smoothly" and entry for entry. But you could achieve these effects far easier with animations ;)
When repainting with every data change, you get the annoying behaviour you describe. Working around that is (imho) not worth the trouble.
I´ve also come along a similar problem sometimes.
The usual resolution i used is to buffer the data and repainting on a timer. This goes along the line of this (Pseudo Code) :
void Widget::OnNewData(void *dataSample)
{
this->threadSafebuffer->appendData(dataSample);
}
void Widget::OnTimeout()
{
DataBuffer renderBatch = this->threadSafebuffer->interlockedExchange();
/* Do UI updates according to renderBatch */
}
This assumes that OnNewData is called on a background thread. The OnTimeout is called from a QTimer on the UI-EventLoop. To prevent contention it justs does an interlocked exchange of the current buffer pointer with a second buffer. So no heavy synchronization (e.g. Mutext/Semaphore) is needed.
This will only work if the amount of work to do for rendering a renderBatch is less than the timeout.
Am I correct in thinking that the QGraphics* classes are not thread-safe? I am porting an old app to Qt and in the process attempting to make it multi-threaded too. I looked at the update code and I see no locks whatsoever.
I started off right, all my processing is done in a set of worker threads so that the GUI does not have to block. But as soon as I come to display my visual representation the whole thing falls down like a pack of cards as the update code attempts to read from the buffer the other thread is writing to.
Here is my test case:
Create a bunch of ellipse objects
Create a thread and pass it the scene pointer
In a loop modify any setting on any object in the scene.
Test function:
bool CBasicDocument::Update( float fTimeStep )
{
const QList<QGraphicsItem*> tObjects = items();
for( QList<QGraphicsItem*>::const_iterator tIter = tObjects.constBegin();
tIter != tObjects.constEnd();
++tIter )
{
QGraphicsEllipseItem* pElipse = (QGraphicsEllipseItem*)(*tIter);
if( pElipse )
{
pElipse->setPen( QPen( QColor( (int)(255.0f * sinf( fTimeStep )), (int)(255.0f * cosf( fTimeStep )), (int)(255.0f * sinf( fTimeStep )) ) ) );
}
}
return true;
}
I have been thinking about ways I can fix this and none of them are particularly pretty.
Ideally what I want to happen is when I change a setting on an object it is buffered until the next render call, but for the time being I'll settle with it not crashing!
At the moment I have four options:
Double buffer the whole scene maintaining two scene graphs in lockstep (one rendering, one updating). This is how our multithreaded game engine works. This is horrible here though because it will require double the CPU time and double the memory. Not to mention the logistics of maintaining both scene graphs.
Modify QGraphics* to be thread safe as above. This is probably the most practical approach but it will be a lot of work to get it done.
Push modifications to the scene into a queue and process them from the main thread.
Throw multithreading to the wind for the time being and just let my app stall when the document updates. (Not pretty given the data size for some documents)
None of them are particularly appealing and all require a massive amount of work.
Does anybody have any ideas or attempted multithreading QGraphicsScene before?
Cheers
I've always read it's a better idea to have all the GUI work happen in the main thread, no matter you're using GTK or Qt or else.
In your case I would go for option 2.
Modifying Qt's code to put locks here and there is the worst thing to do ihmo as you won't be able to sync your modified Qt with upstream without a significant amount of work.
I notice in your test function that you are already casting (using c-style casts, at that) the QGraphicsItem to a QGraphicsEllipseItem. If you only use a certain number of item types, it might be easier to subclass those types, and provide the thread-safe functions in those subclasses. Then cast down to those subclasses, and call what you want to from the other thread. They can handle buffering the changes, or mutex locking, or whatever you want to do for those changes.
An alternative would be to have one mutex, and each thread locks that when processing. This would likely slow you down a lot as you wait for the mutex to become available. You could also maintain a map of items->mutexes somewhere that both threads can get to, and each thread locks the mutex for the item it is working on while it is working with it. This would be simpler to layer in if you think the subclassing option would be too complex.
What is the best way to exit out of a loop as close to 30ms as possible in C++. Polling boost:microsec_clock ? Polling QTime ? Something else?
Something like:
A = now;
for (blah; blah; blah) {
Blah();
if (now - A > 30000)
break;
}
It should work on Linux, OS X, and Windows.
The calculations in the loop are for updating a simulation. Every 30ms, I'd like to update the viewport.
The calculations in the loop are for
updating a simulation. Every 30ms, I'd
like to update the viewport.
Have you considered using threads? What you describe seems the perfect example of why you should use threads instead of timers.
The main process thread keeps taking care of the UI, and have a QTimer set to 30ms to update it. It locks a QMutex to have access to the data, performs the update, and releases the mutex.
The second thread (see QThread) does the simulation. For each cycle, it locks the QMutex, does the calculations and releases the mutex when the data is in a stable state (suitable for the UI update).
With the increasing trend on multi-core processors, you should think more and more on using threads than on using timers. Your applications automatically benefits from the increased power (multiple cores) of new processors.
While this does not answer the question, it might give another look at the solution. What about placing the simulation code and user interface in different threads? If you use Qt, periodic update can be realized using a timer or even QThread::msleep(). You can adapt the threaded Mandelbrot example to suit your need.
The code snippet example in this link pretty much does what you want:
http://www.cplusplus.com/reference/clibrary/ctime/clock/
Adapted from their example:
void runwait ( int seconds )
{
clock_t endwait;
endwait = clock () + seconds * CLOCKS_PER_SEC ;
while (clock() < endwait)
{
/* Do stuff while waiting */
}
}
If you need to do work until a certain time has elapsed, then docflabby's answer is spot-on. However, if you just need to wait, doing nothing, until a specified time has elapsed, then you should use usleep()
Short answer is: you can't in general, but you can if you are running on the right OS or on the right hardware.
You can get CLOSE to 30ms on all the OS's using an assembly call on Intel systems and something else on other architectures. I'll dig up the reference and edit the answer to include the code when I find it.
The problem is the time-slicing algorithm and how close to the end of your time slice you are on a multi-tasking OS.
On some real-time OS's, there's a system call in a system library you can make, but I'm not sure what that call would be.
edit: LOL! Someone already posted a similiar snippet on SO: Timer function to provide time in nano seconds using C++
VonC has got the comment with the CPU timer assembly code in it.
According to your question, every 30ms you'd like to update the viewport. I wrote a similar app once that probed hardware every 500ms for similar stuff. While this doesn't directly answer your question, I have the following followups:
Are you sure that Blah(), for updating the viewport, can execute in less than 30ms in every instance?
Seems more like running Blah() would be done better by a timer callback.
It's very hard to find a library timer object that will push on a 30ms interval to do updates in a graphical framework. On Windows XP I found that the standard Win32 API timer that pushes window messages upon timer interval expiration, even on a 2GHz P4, couldn't do updates any faster than a 300ms interval, no matter how low I set the timing interval to on the timer. While there were high performance timers available in the Win32 API, they have many restrictions, namely, that you can't do any IPC (like update UI widgets) in a loop like the one you cited above.
Basically, the upshot is you have to plan very carefully how you want to have updates occur. You may need to use threads, and look at how you want to update the viewport.
Just some things to think about. They caught me by surprise when I worked on my project. If you've thought these things through already, please disregard my answer :0).
You might consider just updating the viewport every N simulation steps rather than every K milliseconds. If this is (say) a serious commercial app, then you're probably going to want to go the multi-thread route suggested elsewhere, but if (say) it's for personal or limited-audience use and what you're really interested in is the details of whatever it is you're simulating, then every-N-steps is simple, portable and may well be good enough to be getting on with.
See QueryPerformanceCounter and QueryPerformanceFrequency
If you are using Qt, here is a simple way to do this:
QTimer* t = new QTimer( parent ) ;
t->setInterval( 30 ) ; // in msec
t->setSingleShot( false ) ;
connect( t, SIGNAL( timeout() ), viewPort, SLOT( redraw() ) ) ;
You'll need to specify viewPort and redraw(). Then start the timer with t->start().