Callback on each QML Video frame - c++

Question

I have a QML QuickControls 2 application with a Video component/control. I want to create a C++ callback to handle each frame in the video. The C++ callback function will process each frame, ie, find the edges in the image/frame and return that edge image for the UI to display.
How can I hook all this up? Ie, somehow tell the QML to call a c++ callback on each frame?
Video {
id: video
fillMode: VideoOutput.PreserveAspectFit
anchors.fill : parent
source: "file:///D:/cards.mp4"
muted: true
focus: true
Keys.onSpacePressed: video.playbackState == MediaPlayer.PlayingState ? video.pause() : video.play()
Keys.onLeftPressed: video.seek(video.position - 5000)
Keys.onRightPressed: video.seek(video.position + 5000)
}
My callback class, not sure if correct:
class ImageProcessor : public QObject
{
Q_OBJECT
public:
explicit ImageProcessor(QObject *parent = nullptr);
Q_INVOKABLE void processImage(QString va);
signals:
public slots:
};

Answer 1

You can create a VideoOutput from the video:
Rectangle {
width: 800
height: 600
color: "black"
MediaPlayer {
id: player
source: "file://video.webm"
autoPlay: true
}
VideoOutput {
id: videoOutput
source: player
anchors.fill: parent
}
}
You can add filters to the VideoOutput. For example here a faceRecognitionFilter:
VideoOutput {
...
filters: [ faceRecognitionFilter ]
}
In the C++ implemtation of the filter you can reach the frame:
QVideoFrame FaceRecogFilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)
{
// Convert the input into a suitable OpenCV image format, then run e.g. cv::CascadeClassifier,
// and finally store the list of rectangles into a QObject exposing a 'rects' property.
...
return *input;
}
You can collect some info here:
http://doc.qt.io/qt-5/qml-qtmultimedia-videooutput.html
http://blog.qt.io/blog/2015/03/20/introducing-video-filters-in-qt-multimedia/