I have a QTimer for executing OpenCV code and changing an image in a QLabel every 20 milliseconds, but I want to run this OpenCV code more naturally and not depend on the timer.
Instead, I want to have one main thread that deals with user input and another thread that process images with OpenCV, what I can't find is a thread safe way to change the QLabel image (pixmap) in one thread from another thread, could someone describe this process, maybe give some code examples? I also want to know the pros and cons of using QThread, since it's plataform free, it sounds to be user level thread and not a system level which usually runs smoother.
You can only instantiate and work with QPixmap on the main (GUI) thread of your application (e.g. what is returned by QApplication::instance()->thread())
That's not to say you can't work with a QPainter and graphics objects on other threads. Most things work, with exceptions guided by constraints imposed by the OS.
Successive versions of Qt have managed to find ways to support things that previously didn't work. For instance:
Qt 4.0 added rendering QImages from separate threads
Qt 4.4 added the ability to render text into images
Qt 4.8 added the ability to use QPainter in a separate thread to render to a QGLWidget, QGLPixelBuffer and QGLFrameBufferObject
Where Qt 4.4 introduced QFontDatabase::supportsThreadedFontRendering to check to see if font rendering was supported outside the GUI thread, in Qt5 this is considered obsolete and always returns true
Note: you shouldn't hold your breath for the day that Qt adds support to work with QPixmap on non-GUI threads. The reason they exist is to engage the graphics layer directly; and finding a way to work around it just so you could use something named QPixmap wouldn't do any good, as at that point you'd just be using something equivalent to what already exists as QBitmap.
So all that has to do with the ability to instantiate graphics objects like QFont and QImage on another thread, and use them in QPainter calls to generate a graphical image. But that image can't be tied directly to an active part of the display...you'll always be drawing into some off-screen buffer. It will have to be safely passed back to the main thread...which is the gateway that all the direct-to-widget drawing and mouse events and such must pass through.
A well known example of how one might do this is Qt's Mandelbrot Sample; and I'd suggest getting started with that... making sure you understand it completely.
Note: For a different spin on technique, you might be interested to look at my Thinker-Qt re-imagining of that same sample.
Related
I am developing a 3D app using Qt and OpenGL. The app is composed of a QMainWindow with a QOpenGLWidget as central widget and a QML UI as a dock widget. I realized that the user inputs and the UI depends on the rendering performance: if my app runs with low fps, the user inputs are not all caught and it gets difficult to use the UI.
So I was thinking about doing the rendering in a separate thread. I tried several techniques, like using a QTimer or a QThread, but I always get problems sharing the OpenGL context, resizing or using a QPainter.
I am wondering if doing the rendering in another thread is a good approach.
Any suggestions, advices ?
Thanks.
Typical GUI frameworks are not designed to be used from multiple threads directly, and QT is not an exception from. Trying to do GUI stuff from different threads typically results in problems of some kind.
Those frameworks normally have an internal event queue where events are placed in and then processed one after another, which, if the framework is used correctly, assures that the GUI related stuff is accessed from one single thread only. But they allow to add additional events into the queue.
And here we are at the way to go: Keep the entire GUI in one single thread and do user input processing in the other thread. As soon as user data is processed, feed your GUI with appropriately.
Ways to do so offered by Qt are e. g. invoke function or the event system.
Just don't use QOpenGLWidget. Use a single QML window for everything.
Render your OpenGL things in pre-render or post-render function of the QML by using the QQuickWindow::beforeRendering() or QQuickWindow::afterRendering() signals.
That will be using the rendering thread of the QML, so you won't need to create it. And the use cases and synchronization are explained in the qt docs:
http://doc.qt.io/qt-5/qtquick-scenegraph-openglunderqml-example.html
I am working on a project that will use OpenCL to render graphics for display in a QOpenGLWidget. The recommended way to do this seems to be creating a second QOpenGLContext beside the one already present in the QOpenGLWidget, then create a thread where this secondary context can live together with the OpenCL code.
This way Qt can go about it's day as usual with the eventloop running in the main thread. And whenever the QOpenGLWidget decides to paint it will simply fetch data from a buffer prepared in the second thread by the secondary context and the OpenCL inter-op set up there.
This all sounds great on paper, but I am having some problems getting this to work. My question is about how to make the secondary QOpenGLContext "current" in the thread. Because QOpenGLContext::makeCurrent() takes a mandatory QSurface as parameter, and the only surface I have is the one that is available from my QOpenGLWidget, but using that in the secondary thread does not work. I get the following error:
Cannot make QOpenGLContext current in a different thread
So what surface should I use? Or, is there something I missed, or should do differently?
You can create and use a QOffscreenSurface for this purpose.
I am experimenting with Qt for a new layout for an instrument simulation program at work. Our current sim is running everything in a single window (we've used both glut (old) and fltk), it uses glViewport(...) and glScissor(...) to split instrument readouts into their own views, and then it uses some form of "ortho2D" calls to create their own virtual pixel space. The simulator currently updates the instruments and then draws each in their own viewport one by one, all in the same thread.
We want to find a better approach, and we settled on Qt. I am working under a few big constraints:
Each of the instrument panels still need to be in their OpenGL viewport. There are a lot of buttons and a lot of instruments. My tentative solution is to use a QOpenGLWidget for each. I have made progress on this.
The sim is not just a pretty readout, but also simulates many of the instruments as feedback for the instrument designers, so it sometimes has a hefty CPU load. It isn't a full hardware emulator, but it does simulate the logic. I don't think that it's feasible to tell the instruments to update themselves at the beginning of its associated widget's paintEvent(...) method, so I want simulation updates to run in a separate thread.
Our customers may have old computers and thus more recent versions of OpenGL have been ruled out. We are still using glBegin() and glEnd() and everything in between, and the instruments draw a crap ton of variable symbols, so drawing is takes a lot of time and I want to split drawing off into it's own thread. I don't yet know if OpenGL 3 is on the table, which will be necessary (I think) for rendering to off-screen buffers.
Problem: QOpenGLWidget does not have on overrideable "update" method, and it only draws during the widgets' paintEvent(...) and paintGL(...) calls.
Tentative Solution: Split the simulator into three threads:
GUI: Runs user input, paintEvent(...), and paintGL(...).
Simulator: Runs all instrument logic and updates values for symbology.
Drawing: Renders latest symbology to an offscreen buffer (will use a frame buffer object (FBO)).
In this design, cross-thread talking is cyclic and one-way, with the GUI thread providing input, the simulator thread taking that input into account on its next loop, the drawing thread reading the latest symbology and rendering it to the FBO and setting a "next frame available" flag to true (or maybe emitting a signal), and then the paintGL(...) method will take that FBO and spit it out to the widget, thus keeping event processing down and GUI responsiveness up. Continue this cycle.
Bottom line question: I've read here that GUI operations cannot be done in a separate thread, so is my approach even feasible?
If feasible, any other caution or suggestions would be appreciated.
Each OpenGL widget has its own OpenGL context, and these contexts are QObjects and thus can be moved to other threads. As with any otherwise non-threadsafe object, you should only access them from their thread().
Additionally - and this is also portable to QML - you could use worker functors to compute display lists that are then submitted to the render thread to be converted into draw calls. The render thread doesn't do any logic and doesn't compute anything: it takes data (vertex arrays, etc.) and submits it for drawing. The worker functors would be submitted for execution on a thread pool using QtConcurrent::run.
You can thus have a main thread, a render thread (perhaps one per widget, but not necessarily), and functors that run your simulation steps.
In any case, convoluting logic and rendering is a very bad idea. Whether you're doing drawing using QPainter on a raster widget, or using QPainter on an QOpenGLWidget, or using direct OpenGL calls, the thread that does the drawing should not have to compute what's to be drawn.
If you don't want to mess with OpenGL calls, and you can represent most of your work as array-based QPainter calls (e.g. drawRects, drawPolygons), these translate almost directly into OpenGL draw calls and the OpenGL backend will render them just as quickly as if you hand-coded the draw calls. QPainter does all this for you if you use it on a QOpenGLWidget!
I've got a main GUI class and another Worker class: the first copes with GUI things (drawing a QPixmap into a QGraphicsScene), the second with computations thing (drawing QLines and QPoints onto that QPixmap).
The two classes run in two different threads.
When I create the Worker thread, I pass the address of the GUI's QPixmap to the Worker class, so they share the same object.
The QPixmap is modified in the Worker class, and drawn in the GUI class. Even if I didn't have any problem, I decided to use a QMutex to ensure my program wouldn't try to access the QPixmap while it was being drawn. Now, in order to do this, I have a QMutex shared between GUI class and Worker class (Worker class has again a pointer to the GUI's QMutex). Whenever I read or modify the QPixmap I lock the QMutex.
Is this a valid approach? I never got errors so far, but I wonder if it is logically correct and whether Qt provides a better way to accomplish this.
Thank you in advance.
According to the Qt5 thread-safety page:
QPainter can be used in a thread to paint onto QImage, QPrinter, and
QPicture paint devices. Painting onto QPixmaps and QWidgets is not
supported.
So the official line is no, you should not be modifying a QPixmap outside of the main thread. You may be "getting lucky" in that it happens to work on your current platform under your current use case, but Qt doesn't guarantee that it will work.
A safer approach might be to have your worker thread draw into a QImage object instead, and then when the GUI thread wants to update the GUI, it can grab and draw the latest version of the QImage object (using mutexes or some other mechanism to make sure that the worker thread isn't simultaneously updating the QImage).
I agree according to the documentation, it's not allowed to use QPixmap in a worker thread. However, according to the code.
The constructor checks if it's in the main thread. If it's not in the main thread, it checks for a feature called ThreadedPixmap. If enabled, it will continue without problem. As far as I can see, ThreadedPixmap is supported on all platforms, so it seems possible to use QPixmaps on other threads.
OpenCV recently upgraded its display window, when it is used in Qt.
It looks very good, however I did not find any possibility for it to be embedded into an existing Qt GUI window. The only possibility seems to be the creation of a cvNamedWindow or cv::namedWindow, but it creates a free-floating independent window.
Is there any possibility to create that OpenCV window inside an existing GUI? All I could find on the OpenCV forums is an unanswered question, somewhat similar to my own.
There is a straight-forward possibility to show an OpenCV image in Qt, but it has two major problems:
it involves copying the image pixel by pixel, and it's quite slow. It has function calls for every pixel! (in my test application, if I create a video out of the images, and display it in a cvNamedWindow, it runs very smoothly even for multiple videos the same time, but if I go through the IplImage --> QImage --> QPixmap --> QLabel route, it has severe lag even for one single video)
I can't use those nice new controls of the cvNamedWindow with it.
First of all, the image conversion is not as inefficient as you think. The 'function calls' per pixel at least in my code (one of the answers to the question you referenced) are inlined by optimized compilation.
Second, the code in highgui/imshow does the same. You have to get from the matrix to an ARGB image either way. The conversion QImage -> QPixmap is essentially nothing else than moving the data from main memory to GPU memory. That's also the reason why you cannot access the QPixmap data directly and have to go through QImage.
Third, it is several times faster if you use a QGLWidget to draw the image, and I assume you have QT_OPENGL enabled in your OpenCV build. I use QPainter to draw the QPixmap within a QGLWidget, and speed is no issue. Here is example code:
http://sourceforge.net/p/gerbil/svn/19/tree/gerbil-gui/scaledview.h
http://sourceforge.net/p/gerbil/svn/19/tree/gerbil-gui/scaledview.cpp
Now to your original question: Your current option is to take the code from OpenCV, include into your project under a different namespace, and alter it to fit your needs. Apart from that you have no alternative right now. OpenCV's highgui uses its own event loop, connection to the X server etc. and it is nothing you can intercept.
My first guess is to want to say this: I'm sure that if you dig into the code for namedWindow, you will find that they use some sort of standard, albeit not oft-referenced, object for painting said window (that's in the openCV code). If you were ambitious enough, you could extend this class yourself, to interface directly to a frame or custom widget in Qt. There might even be a way to take the entire window and embed it, using a similar method of a Qt frame, or an extension of the (general) widget class. This is a very interesting question and relates rather directly to work I've been doing of late, so I'll continue to think about and research it and see if I can't come up with something else more helpful.
[EDIT] What specific new controls are you so interested in? It might be more efficient on the part of the programmer to extend a Qt control to emulate that, as opposed to my first suggestion.[/EDIT]
simply check out the opencv highgui implementation. as i recall it uses qt.