I have very a simply application written in QT in which I want to display a movie by using QMediaPlayer, but before I will display any frame I would like to detect on it some objects and mark them by drawing a rectangle over it.
I've read in http://doc.qt.io/qt-5/videooverview.html that I can access each frame by subclassing QAbstractVideoSurface and so I dit it.
class VideoSurface : public QAbstractVideoSurface {
Q_OBJECT
bool present(const QVideoFrame &frame) override {
if (surfaceFormat().pixelFormat() != frame.pixelFormat()
|| surfaceFormat().frameSize() != frame.size()) {
setError(IncorrectFormatError);
stop();
return false;
} else {
currentFrame = frame;
return true;
}
}
...
}
Now, I am receiving in this member function frames that I want to modify by drawing on it rectangles in places where I detected objects and then I would like to display them on the screen (preferably on some widget).
How can I do this?
Should my VideoSurface class contain QWidget as a member? or should I subclass QWidget which will contain VideoSurface?
In both cases, how can I display this frame? Should I first convert it to QImage and then display (it would be convinien for me, because my detection system is working with QImage, but would it be efficient)? I know that I can't paint from outside a paint event, so I can't paint in present function, so where exactly should be this painting function and how I can call it?
Where should I detect those object and modify frame? In present function, or in drawing function?
This is up to you and depends on how you prefer to structure your classes. I would prefer to have a separate widget that holds a pointer to your VideoSurface and draws the data that is returned by some member function of VideoSurface (depends on your solution in 2.)
a) QImage is efficient enough for some purposes and if you are already using it in your detection code, then you already have everything in memory and can work on that. As with all performance-related worries: Test and see if the performance is good enough for you. If it is not, you probably also have to consider doing the detection in a different way. I have worked on a project where we processed QImages converted from a similar VideoSurface on a camera stream on mobile devices (for relatively low-resolution images), and the performance was good enough that we haven't yet bothered to use other techniques. The source code for the VideoSurface class in that project (Neuronify) is hosted here.
b) Your present() function could emit a signal that you can connect to from other objects that fetch the latest data from the VideoSurface and keeps it until their paint-function is called. Or you could apply the data directly to some widget that accepts image data. See Use of QAbstractVideoSurface for an example of that.
Again, that is up to you :) However, if you need to improve performance at some point you might want to do that work on a different thread to keep your GUI from locking up while you are processing the data. And if you do that, you need to decide whether you have to process every frame or if some frames can skip processing to improve the FPS of the playback. In the latter case, you should probably not do it in the present() function, as that is likely going to keep the media player from being able to feed you more frames while you are processing old frames.
Related
I am writing a video display software capable of displaying multiple video streams. For this I have a GridView holding VideoOutputs in QML connected to a QAbstractListModel derived class in c++ which provides instances of an object with a QAbstractVideoSurface Q_PROPERTY. It's working quite beautifully so far.
The video frames I am displaying come with metadata, however, containing data for axis aligned bounding boxes. I don't know beforehand how many boxes there are, the number could even change on a frame by frame basis, their position and size is also not set.
Ultimately, it should look something like this:
As I need to be able to display a few video streams at once, and preferably at 30+ fps, I need a fast method of drawing these boxes. Using QPainter on the QImage on which the QVideoFrame is based is rather slow so I was considering a few other approaches:
Using the QML object Rectangle in a Repeater with a c++ provided model (Was hoping to simply provide a QVariantList::fromVector() ): Could work, however I would need a lot of models which in turn I need to provide to QML with a model, and I would likely need to call begin/endResetModel every frame that the boxes change to cause QML to update - this is also very slow.
Using a Shader to draw the boxes: This is a rather difficult approach. I'm no stranger to shaders, but in Qt/Qml I don't know how to provide the shader with the information necessary.
Using OpenGL directly to draw the boxes: Again, I have no clue how to do this, but I think I could work it out if I googled.
My question: Which one, if any, of these approaches is the best? If none of these, which other approach could I use?
Thank you so much for taking the time to read my rather long question!
I'm using Qt and I can encapsulate the video stream from my Logitech webcam in a QCamera, subclass QVideoAbstractSurface and from there feed it to a video widget (specifically a QCameraViewfinder) without problem. Afaik the video frames never enter into system memory and that's what I want. However I further want to manipulate this video (add some overlays). To do this, for each frame I need to get the handle (QAbstractVideoBuffer::handle()) and use this with e.g. the OpenGL API to add my overlays.
bool MyVideoSurface::present(const QVideoFrame& frame)
{
QVariant h = frame.handle();
// ... Manipulate h using OpenGL
// ... Send frame to a video widget
}
The problem is that when I do this, h is invalid, no matter how I specify the QVideoSurfaceFormat. E.g. earlier at initialization I have
QVideoSurfaceFormat qsvf(surfaceSize, pixelFormat, QAbstractVideoBuffer::GLTextureHandle);
mVideoSurface->start(qsvf); // MyVideoSurface instance
(Relevant docs here and here.) It doesn't matter what handle type I input, none of them are valid (not surprisingly in some cases).
Is there any way for me to access my webcam video frames without mapping? I'm happy with solutions that break any specific assumptions I have about which Qt classes are the relevant ones, but there need to be overlays and it needs to be done on the GPU. Is the ability to access them hardware dependent, i.e. I need to choose a device in advance that will support this?
I'm trying to time an application to see how long it takes to load up some information, and paint a graph. My function loads up the data first, then draws the graph.
The timing is fairly simple, it calls an external function that gets msecs since some date.
The problem is even if I set t1 in the beginning and t2 right after I call the draw function, t2 will return before the QGraphicsView is actually updated. (I know, it makes sense why this should be asynchronous)
For instance when I load a large file, it will return with 700 msecs after I subtract the two values, but the actual rendering doesn't finish until a few seconds later.
I've looked all over the web and scoured the Qt documentation. I can find tons of information on updating widgets yourself, but nothing on any kind of signal or event that is fired off after rendering finishes.
Even the QGraphicsScene::changed signal appears to only be fired off when the scene changes underneath, not when rendering is done and the user can SEE the changes.
Any help on how to do this?
Does a signal or event exist for when a QGraphicsView or QWidget is done being painted or rendered?
As far as I know, it does not exist. (looked for something similar)
user can SEE the changes
As far as I know, Qt uses double buffering, so if painting is finished, it doesn't mean that user can see the changes.
Any help on how to do this?
If you want to know when painting has finished, then...
You can subclass QGraphicsScene and implement your own drawItems, drawBackground or drawForeground. This is NOT simple (because item painting algorithm is complicated), but you'll be able to tell when every item has finished painted.
You can fire/emit signals from within paintEvent (QWidget-based classes) or paint() (QGraphicsItem/QGraphicsObject-based classes). You'll need to use your own classes, obviously, and you'll have to subclass either QGraphicsView, or items you're drawing within view, or QGraphicsScene.
You could also create proxy QPainter class, and this way you'll be able to know what exactly is being paitned and when.
Having said that I suspect you're approaching your problem incorrectly.
If you're only trying to draw a graph, then there's no reason for you to know when painting is finished.
If painting is finished, it doesn't mean user can see the result.
Paint events might be called more than once.
Recommended approach:
Receive/read the data (you're drawing in your graph) from external source using threads or timer events (you'll need to read it in small chunks if you're using timer events, obviously), then update the graph from time to time, and let Qt handle repainting.
How exactly does this allow me to detect the amount of time it takes from when I choose to open a file to when all the data is loaded and the graph is drawn and is visible?
You can detect when paintEvent has finished painting by subclassing whatever widget you're using to paint Graph, overriding paintEvent and firing signal from within paintEvent, calling a subroutine or doing whatever you want.
There is no warranty that paintEvent will be called only once.
To measure how slow individual routine is and locate bottlenecks, use profilers. VerySleepy, AQTime, and so on.
Instead of measuring how long it takes to load AND display data, it will make much more sense to measure separately loading time and display time. This is a GUI application, not a game engine, so you do not control precisely when something is being drawn.
I have not testet it, but I think by subclassing QGraphicsScene and reimplementing the render method you can measure the render time.
#include <QGraphicsScene>
#include <QTime>
class MyGraphicsScene : public QGraphicsScene
{
Q_OBJECT
public:
void render ( QPainter * painter, const QRectF & target = QRectF(), const QRectF & source = QRectF(), Qt::AspectRatioMode aspectRatioMode = Qt::KeepAspectRatio ) {
QTime t;
t.start();
QGraphicsScene::render (painter, target, source, aspectRatioMode);
qDebug("render time %d msec", t.elapsed());
}
};
I'm creating a image visualizer that open large images(2gb+) in Qt.
I'm doing this by breaking the large image into several tiles of 512X512. I then load a QGraphicsScene of the original image size and use addPixmap to add each tile onto the QGraphic Scene. So ultimately it looks like a huge image to the end user when in fact it is a continuous array of smaller images stuck together on the scene.First of is this a good approach?
Trying to load all the tiles onto the scene takes up a lot of memory. So I'm thinking of only loading the tiles that are visible in the view. I've already managed to subclass QGraphicsScene and override its drag event thus enabling me to know which tiles need to be loaded next based on movement. My problem is tracking movement on the scrollbars. Is there any way I can create an event that get called every time the scrollbar moves. Subclassing QGraphicsView in not an option.
QGraphicsScene is smart enough not to render what isn't visible, so here's what you need to do:
Instead of loading and adding pixmaps, add classes that wrap the pixmap, and only load it when they are first rendered. (Computer scientists like to call this a "proxy pattern"). You could then unload the pixmap based on a timer. (They would be transparently re-loaded if unloaded too soon.) You could even notify this proxy path of the current zoom level, so that it loads lower resolution images when they will be rendered smaller.
Edit: here's some code to get you started. Note that everything that QGraphicsScene draws is a QGraphicsItem, (if you call ::addPixmap, it's converted to a ...GraphicsItem behind the scenes), so that's what you want to subclass:
(I haven't even compiled this, so "caveat lector", but it's doing the right thing ;)
class MyPixmap: public QGraphicsItem{
public:
// make sure to set `item` to nullptr in the constructor
MyPixmap()
: QGraphicsItem(...), item(nullptr){
}
// you will need to add a destructor
// (and probably a copy constructor and assignment operator)
QRectF boundingRect() const{
// return the size
return QRectF( ... );
}
void paint(QPainter *painter, const QStyleOptionGraphicsItem *option,
QWidget *widget){
if(nullptr == item){
// load item:
item = new QGraphicsPixmapItem( ... );
}
item->paint(painter, option, widget);
}
private:
// you'll probably want to store information about where you're
// going to load the pixmap from, too
QGraphicsPixmapItem *item;
};
then you can add your pixmaps to the QGraphicsScene using QGraphicsScene::addItem(...)
Although an answer has already been chosen, I'd like to express my opinion.
I don't like the selected answer, especially because of that usage of timers. A timer to unload the pixmaps? Say that the user actually wants to take a good look at the image, and after a couple of seconds - bam, the image is unloaded, he will have to do something in order the image to reappear. Or may be you will put another timer, that loads the pixmaps after another couple of seconds? Or you will check among your thousand of items if they are visible? Not only is this very very irritating and wrong, but that means that your program will be using resources all the time. Say the user minimizes you program and plays a movie, he will wonder why on earth my movie is freezing every couple of seconds...
Well, if I misunderstood the proposed idea of using timers, execuse me.
Actually the idea that mmutz suggested is better. It reminded me of the Mandelbrot example. Take a look at it. Instead of calculating what to draw you can rewrite this part to loading that part of the image that you need to show.
In conclusion I will propose another solution using QGraphicsView in a much simpler way:
1) check the size of the image without loading the image (use QImageReader)
2) make your scene's size equal to that of the image
3) instead of using pixmap items reimplement the DrawBackground() function. One of the parameters will give you the new exposed rectangle - meaning that if the user scrolls just a little bit, you will load and draw only this new part(to load only part of an image use setClipRect() and then read() methods of the QImageReader class). If there are some transformations you can get them from the other parameter(which is QPainter) and apply them to the image before you draw it.
In my opinion the best solution will be to combine my solution with the threading shown in the Mandelbrot example.
The only problem that I can think of now is if the user zooms out with a big scale factor. Then you will need a lot of resources for some time to load and scale a huge image. Well I see now that there is some function of the QImageReader that I haven't tried yet - setScaledSize(), which maybe do just what we need - if you set a scale size and then load the image maybe it won't load first the entire image – try it. Another way is just to limit the scale factor, a thing that you should do anyway if you stick to the method with the pixmap items.
Hope this helps.
Unless you absolutely need the view to be a QGraphicsView (e.g. because you place other objects on top of the large background pixmap), I'd really recommend just subclassing QAbstractScrollArea and reimplementing scrollContentsBy() and paintEvent().
Add in a LRU cache of pixmaps (see QPixmapCache for inspiration, though that one is global), and make the paintEvent() pull used pixmaps to the front, and be set.
If this sounds like more work than the QGraphicsItem, believe me, it's not :)
What is the fastest way to display images to a Qt widget? I have decoded the video using libavformat and libavcodec, so I already have raw RGB or YCbCr 4:2:0 frames. I am currently using a QGraphicsView with a QGraphicsScene object containing a QGraphicsPixmapItem. I am currently getting the frame data into a QPixmap by using the QImage constructor from a memory buffer and converting it to QPixmap using QPixmap::fromImage().
I like the results of this and it seems relatively fast, but I can't help but think that there must be a more efficient way. I've also heard that the QImage to QPixmap conversion is expensive. I have implemented a solution that uses an SDL overlay on a widget, but I'd like to stay with just Qt since I am able to easily capture clicks and other user interaction with the video display using the QGraphicsView.
I am doing any required video scaling or colorspace conversions with libswscale so I would just like to know if anyone has a more efficient way to display the image data after all processing has been performed.
Thanks.
Thanks for the answers, but I finally revisited this problem and came up with a rather simple solution that gives good performance. It involves deriving from QGLWidget and overriding the paintEvent() function. Inside the paintEvent() function, you can call QPainter::drawImage(...) and it will perform the scaling to a specified rectangle for you using hardware if available. So it looks something like this:
class QGLCanvas : public QGLWidget
{
public:
QGLCanvas(QWidget* parent = NULL);
void setImage(const QImage& image);
protected:
void paintEvent(QPaintEvent*);
private:
QImage img;
};
QGLCanvas::QGLCanvas(QWidget* parent)
: QGLWidget(parent)
{
}
void QGLCanvas::setImage(const QImage& image)
{
img = image;
}
void QGLCanvas::paintEvent(QPaintEvent*)
{
QPainter p(this);
//Set the painter to use a smooth scaling algorithm.
p.setRenderHint(QPainter::SmoothPixmapTransform, 1);
p.drawImage(this->rect(), img);
}
With this, I still have to convert the YUV 420P to RGB32, but ffmpeg has a very fast implementation of that conversion in libswscale. The major gains come from two things:
No need for software scaling. Scaling is done on the video card (if available)
Conversion from QImage to QPixmap, which is happening in the QPainter::drawImage() function is performed at the original image resolution as opposed to the upscaled fullscreen resolution.
I was pegging my processor on just the display (decoding was being done in another thread) with my previous method. Now my display thread only uses about 8-9% of a core for fullscreen 1920x1200 30fps playback. I'm sure it could probably get even better if I could send the YUV data straight to the video card, but this is plenty good enough for now.
I have the same problem with gtkmm (gtk+ C++ wrapping). The best solution besides using a SDL overlay was to update directly the image buffer of the widget then ask for a redraw. But I don't know if it is feasible with Qt ...
my 2 cents
Depending on your OpenGL/shading skills you could try to copy the videos frames to a texture, map the texture to a rectangle (or anything else..fun!) and display it in a OpenGL scene. Not the most straight approach, but fast, because you're writing directly into the graphics memory (like SDL). I would also recoomend to use YCbCR only since this format is compressed (color, Y=full Cb,Cr are 1/4 of the frame) so less memory + less copying is needed to display a frame. I'm not using Qts GL directly but indirectly using GL in Qt (vis OSG) and can display about 7-11 full HD (1440 x 1080) videos in realtime.