I'm working on a vision application and I need to have a "Live View" from the camera displayed on the screen using a QPixmap object. We will be updating the screen at 30frames/second on a continuous basis.
My problem is that this application has to run on some 3-5 year old computers that, by todays standards, are slow. So what I would like to do is to be able to directly write to the display byte array inside of QPixmap. After going through the program code, almost option for changing the contents of a Pixmap results in a new QPixmap being created. This is the overhead I'm trying to get ride of.
Additionally, I would like to prevent all the new/deletes from occurring just to keep memory fragmentation under control.
Any suggestions?
First of all, the most important piece of information regarding the "picture" classes in Qt:
QImage is designed and optimized for I/O, and for direct pixel access and manipulation, while QPixmap is designed and optimized for showing images on screen.
What this means is that QPixmap is a generic representation of your platform's native image format: Pixmap on Unix, HBITMAP on Windows, CGImageRef on the Mac. QImage is a "pixel array with operations" type of class.
I'm assuming the following:
You are reading raw camera frames in a specific pixel format
You really are having memory fragmentation issues (as opposed to emotionally having them)
My advice is to use QImage instead of QPixmap. Specifically, there is a constructor that accepts a raw byte array and uses it directly as the pixel buffer:
QImage::QImage(uchar *data, int width, int height, int bytesPerLine, Format format)
Having constructed a QImage, use a QPainter to draw it to a widget at the desired frequency. Be warned however that:
If you are reading raw camera frames, format conversion may still be necessary. Twice, in the worst case: Camera ➔ Qimage ➔ Platform Bitmap.
You cannot avoid memory allocation from the free store when using QPixmap and QImage: they are implicitly shared classes and necessarily allocate memory from the free store. (On the other hand, that means you should not new/delete them explicitly.)
Our team managed to display fullscreen compressed video smoothly on Atom-powered computers using only Qt (albeit at a lower framerate). If this does not solve your problem, however, I'd bypass Qt and use the native drawing API. If you absolutely need platform independence, then OpenGL or SDL may be good solutions.
I have found that QImages are faster for Direct I/O operations.
Could you provide more detail as to what you are getting and trying to do with the QPixmap?
Related
I need to do some quite intensive drawing processes in my appliation - I'm recording a camera and a desktop. I'm using ffmpeg to capture and encode/decode my videos. Those processes are quite fast and they working in a separate thread, but when it comes to render - perfomance drops drastically. I'm using QLabel to display captured frames via setPixmap() method and I see that Qt not using any GPU power even if a matrix operation, such as scaling an image to fit the window is clearly implied. Is there any way to paint images in Qt using hardware acceleration? Or is there another ways to speed up painting process?
Software rendering is not the only possible performance killer.
I also used to use QWidget for similar kinds of jobs, and it seems to be OK.
I assume the result of FFmpeg processing is an uncompressed byte array with YUV or RGB color scheme.
You already have some memory allocated for your FFmpeg image frame.
I believe you're creating your pixmaps with QPixmap::fromImage(...), which implies copying.
But first, you need to construct QImage, which may imply copying.
So we have one or two full copies every frame.
Try one of the QImage constructors that uses an existing memory buffer (see Qt docs).
Or, ideally, you should have a once allocated QImage and use its memory buffer with FFmpeg (FFmpeg directly writes to QImage)
Sublass QWidget, reimplement paintEvent() and paint your QImage (not QPixmap) there.
Ideally, it should be the same QImage from the previous step.
ffmpeg
{
mutex.lock();
write(image);
mutex.unlock();
}
QWidet::paintEvent
{
mutex.lock();
paint(image);
mutex.unlock();
}
You definitely can use QOpenGLWidget for GPU drawing, which, in my opinion, wouldn't help you much.
QOpenGLWidget uses buffered rendering (i.e. copying), and it will take some cycles to upload your image from the CPU side to the GPU side.
Transform operations will become faster, though.
I'm porting an old 4.8 application to 5.2.1 and back in that time, I used QImage to render some raw data on the screen, in a QLabel.
I am grabbing images from a camera, so i want to display those images in real-time. Until now, with QImage, i achieve over 20FPS (the camera is able to grab 30 FPS).
I'm wondering if rendering this data on OpenGL (maybe in a QML Quick / Qt Widgets new application) would be faster than the current developed method?
With next assumptions in mind :
your implementation in OpenGL is using HW acceleration
your implementation is using optimal texture parameters, to display the image (i.e. the driver is not doing some conversion)
you may achieve better performances using OpenGL. QImage still has to hold data at both the memory and GPU, meaning at least one additional copy is needed when updating QImage. With OpenGL, you can copy data directly to GPU memory and you do not need to store the data somewhere in memory.
However, what may be optimal on one GPU, doesn't have to be optimal on another. So, if you are implementing something that needs to run on various hardware, I would advise to go for QImage.
But as said, the only way is to implement and measure.
I'm currently writing a 3D renderer (for fun and research), so I need a way to draw my framebuffer to a window. Since I'm doing all of my calculations on CPU, the drawing needs to be as fast as possible.
One of my goals is to use no existing graphics library (OpenGL/DirectX) so the drawing to the screen is pure Win32. In my research I've found a couple of ways to create and draw bitmaps and now I'm looking for the best one.
My current implementation uses a bitmap created with CreateDIBSection(), which is drawn to my window DC using BitBlt().
CreateDIBSection() give me a pointer to my bitmap bytes so I can manipulate it without copying. Using this method I achieve an update rate of about 260 FPS (without any rendering done).
This seems a bit slow, so I'm looking for optimizations.
I've read something about that if you don't create a bitmap with the same palette as the system palette, some slow color conversions are done.
How can I make sure my DIB bitmap and window are compatible?
Are there methods of drawing an bitmap which are faster than my current implementation?
I've also read something about DrawDibDraw(), can anyone confirm that this is faster?
I've read something about that if you don't create a bitmap with the same palette as the system palette, some slow color conversions are done.
Very few systems run in a palette mode any more, so it seems unlikely this is an issue for you.
Aside from palettes, some GDI functions also cause a color matching conversion to be applied if the source bitmap and the destination have different gamuts. BitBlt, however, does not do this type of color matching, so you're not paying a price for that.
How can I make sure my DIB bitmap and window are compatible?
You don't. You can use DIBs (which are Device-Independent Bitmaps) or compatible (device-dependent) bitmaps. It's possible that your DIB bitmap matches the current mode of your device. For example, if you're using a 32 bpp DIB, and your display is in that same mode, then no conversion is necessary. If you want a bitmap that's guaranteed to be in the same mode as your device, then you can't use a DIB and all the nice properties it provides for predictable pixel layout and format.
Are there methods of drawing an bitmap which are faster than my current implementation?
The limitation is most likely in getting the data from system memory to graphics adapter memory. To get around that limitation, you need a faster graphics bus, or you need to render directly into graphic memory, which means you'd need to do your computation on the GPU rather than the CPU.
If you're rendering a 1920 x 1080 pixel image at 24 bits per pixel, that's close to 6 MB for your frame buffer. That's an awful lot of data. If you're doing that 260 times per second, that's actually pretty impressive.
I've also read something about DrawDibDraw(), can anyone confirm that this is faster?
It's conceivable, but the only way to know would be to measure it. And the results might vary from machine to machine because of differences in the graphics adapter (and which bus they use).
I realize this question might be closed with the "not enough research". However I did spent like 2 days googling for it and didn't find a conclusive answer.
Well I have an application that spawns a window, not written in c++. This application can have a c-interface with dlls. Now I wish to use the power of OpenCV, so I started on a dll to extend. Ss passing image data from/to the application is near impossible (only capable of passing c-strings & double values directly - using the hard drive for drawing is slowing down too much for real time image manipulation).
I am looking into letting opencv draw the image data directly - onto the window. I can gain the window handle easily, so would it then be possible to let openCV draw their data "over" the other window - or better into the other window?
Is this even possible with any library (FFMPEG, or something else)?
Yes, it's possible, but it's far from ideal. You can use GDI to draw on top of the other window (just convert IplImage to HBITMAP). Another technique is to do such drawing in a borderless layered window.
An easier approach is, since you own both applications, to write a function that passes an IplImage between them using standard C data types, after all, IplImage is nothing but a data type that is built from these standard types.
Here is how you will disassemble IplImage into 5 standard parameters:
The size (int, int) of the image (width/height);
The (int) bit depth of the image;
The number (int) of channels;
And the (unsigned char*) pixels of the image;
After receiving these parameters on the other side, you may wonder: how do I assemble a IplImage from scratch? Call cvCreateImageHeader() followed by cvSetData().
I'm currently in the process of designing and developing GUI's for some audio applications made in C++ (using the Juce framework).
So far I've been playing with using bitmap graphics to create custom sliders and dials, by using 'film strip' style images to animate the components (meaning when the user interacts with a slider it triggers a method that changes the offset of a film-strip image to change the components appearance). Depending on the size of the original image and the number of 'frames', the CPU usage level changes quite dramatically.
Firstly, what would be the most efficient bitmap file format to use in terms of CPU consumption? At the moment I'm using PNG images.
Secondly, would it be more efficient to use vector graphics for these kind of graphical components? I understand the main differences between bitmap and vector graphics, but I haven't found any information regarding their CPU usage levels with regard to GUI interaction.
Or would CPU usage be down to the particular methods/functions/libraries/frameworks being used?
Thanks!
Or would CPU consumption be down to the particular methods/functions/libraries/frameworks being used?
Any of these things could influence it.
Pixel based images might take a while to read off of disk the bigger they are. Compressed types might take more time to uncompress. Vector might take more time to render when are loaded.
That being said, I would definitely not expect that your choice of image type to have any impact on its performance. Since you didn't provide a code example it is hard to speculate beyond that.
In general, you would expect that the run-time costs of the images to happen when they are loaded. So whenever you create an image object. If you create an images all over the place, then maybe its expensive. It is possible that your film strip is recreating the images instead of loading them once and caching them.
Before choosing bitmap vs. vector graphics, investigate if your graphics processor supports vector or bitmap graphics. Some things take a long time to draw as vectors.
Have you tried double-bufferring?
This is where you write to a buffer in memory while the display (graphics processor) is loading another.
Load your bitmaps from the resource once. Store them as memory snapshots to avoid the additional cost of translating them from a format.
Does your graphic processor support "blitting"?
Blitting is where the graphics processor can copy a rectangular area in memory (bitmap) and display it along with apply optional operations before displaying (such as XOR with existing bits).
Summary:
To improve your rendering speed, only convert images from the file into a bitmap form once. Store this somewhere. Refer to this converted bitmap as needed. Next, investigate and implement double buffering. Lastly, investigate and use bit-blitting or blitting.
Other optimization rules apply here too, such as reviewing the design, removing requirements, loop unrolling, passing images via pointer vs. copying them, and reduce "if" statements by using boolean logic and Karnaugh (sp?) maps.
In general, calculations for rendering vector graphics are going to take longer than blitting a rectangular region of a bitmap to the screen. But for basic UI stuff, neither should be particularly intensive.
You probably should do some profiling. Perhaps you're redrawing much more frequently than necessary. Or perhaps the PNG is being decoded each time you try to draw from it. (I'm not familiar with Juce.)
For a straight Windows app, I'd probably render vector graphics into a device-dependent bitmap once on startup and then just blit from the bitmap to the screen. Using vector gives you DPI independence, and blitting from a device-dependent bitmap is about the fastest way to paint a block of pixels. I believe the color matching is done when you render to the device-dependent bitmap, so you don't even have the ICM overhead on the screen drawing.
Vector graphics was ditched long ago - bitmap graphics are more performant. The thing is that you can send a bitmap to the GPU once and then render it forever more by a simple copy.
Secondly, the GPU uses it's own texture compression. DirectX is DXT5, I believe, but when the GPU sees the texture, it doesn't care what you loaded it from.
However, a modern CPU even with a crappy integrated GPU should have absolutely no problem with simple GUI rendering. If you're struggling, then it's time to look again at the technique you're using. Perhaps your framework is slow or your use of it is suboptimal.