I am struggling with writing a video player which uses OpenCV to read a frame from video and display it in a QWidget.
This is my code:
// video caputre is opened here
...
void VideoPlayer::run()
{
int sleep = 1000 / static_cast<unsigned long>(video_capture_.get(CV_CAP_PROP_FPS));
forever
{
QScopedPointer<cv::Mat> frame(new cv::Mat);
if(!video_capture_.read(*frame))
break;
cv::resize(*frame, *frame, cv::Size(640, 360), 0, 0, cv::INTER_CUBIC);
cv::cvtColor(*frame, *frame, CV_BGR2RGB);
QImage image(frame->data, frame->cols, frame->rows, QImage::Format_RGB888);
emit signalFrame(image); // notifying QWidget to draw an image
msleep(sleep); // wait before we read another frame
}
}
and on the QWidget side, I am just using this image and drawing it in paintEvent.
It just looks to me that parameter sleep doesn't play important role here. As much as I decrease it (to get more FPS) the video is just not smooth.
The only thing here left for me is that I gave up on that approach because it doesn't work, but I wanted to ask here one more time, just to be sure - am I doing something wrong here?
Related
I am using QT to build a custom component for QML. I want this component to be able to play a WEBM video that contains an alpha channel. However, all of my attempts have resulted in the transparent pixels of the video getting replaced with black ones. This is my code currently:
MyClass::MyClass()
{
m_pixmap = new QPixmap(1920, 1080); // Create a canvas to draw on:
// Create something that can be drawn:
m_painter = new QPainter(m_pixmap);
m_rect = new QRect(0, 0, 1920, 1080);
// Create an area to present on:
m_label = new QLabel();
m_label->setPixmap(*m_pixmap);
m_label->show();
// Play video:
m_videoSink = new QVideoSink();
m_mediaPlayer = new QMediaPlayer();
m_mediaPlayer->setVideoSink(m_videoSink);
m_mediaPlayer->setSource(QUrl::fromLocalFile("path..."));
m_mediaPlayer->setLoops(QMediaPlayer::Infinite);
m_mediaPlayer->play();
// Add a an event for when the video frame changes:
connect(m_videoSink, SIGNAL(videoFrameChanged(QVideoFrame)), this, SLOT(SetFrame(QVideoFrame)));
qDebug() << "Constructed";
}
void MyClass::SetFrame(QVideoFrame frame)
{
frame.paint(m_painter, *m_rect, m_options); //Stores the frame in the m_pixmap
m_label->setPixmap(*m_pixmap);
}
In this example I attempt to use a QMediaPlayer to play the video, then a QVideoSink to extract the currently playing QVideoFrame and paint that to a QPixmap that is finally being displayed in a QLabel.
I have also tried to have the QMediaPlayer directly hooked up to a QVideoWidget.
I know that my WEBM video works as it is displaying as expected when imported in to other programs.
Here is the code piece that I wrote for streaming the camera captures:
VideoCapture cap;
VideoStreamer::VideoStreamer()
{
connect(&tUpdate,&QTimer::timeout,this,&VideoStreamer::streamVideo);
}
VideoStreamer::~VideoStreamer()
{
cap.release();
tUpdate.stop();
}
void VideoStreamer::streamVideo()
{
//Capture frame by frame
cap>>frame;
QImage imgOriginal = QImage(frame.data,frame.cols,frame.rows,QImage::Format_RGB32).rgbSwapped();
}
void VideoStreamer::openVideoCamera()
{
int deviceID = 1;
int apiID = cv::CAP_ANY;
cap.open(deviceID, apiID);
double fps = cap.get(cv::CAP_PROP_FPS);
tUpdate.start(0);
}
and the output:
Problems:
Capture should not be made of the grids like these.
Capture should not be gray.
Notes:
I was using the same codes for another camera and it was working very well. I changed the camera and this happened.
I am using Linux - Ubuntu (18.04) and I think I installed all the necessary drivers. Camera is working fine in the app (Tcam-Capture)
Update: I looked the details on the Tcam app and noticed that it uses GRBG format. So I think I should format the QImage to GRBG, instead of RGB32. But there is no such thing as Format_Bayer or sth. So how can I fix this problem?
I am working on the QT Creater.
Camera Model: DFM 27UR0135-ML - USB 3.0 color board camera
I'm writing a QT GUI application in wich a live stream of a connected camera is shown on a QGraphicsview. Therefore an openCV image is first converted to a QImage and than to a QPixmap. This is added to the QGraphicsScene of the QGraphicsView.
The bandwidth is not a problem, the cameras are connected via ethernet or USB.
I am testing the performance with the Analyze Toole build in Visual Studio 2012 and it shows that the conversion to the QPixmap is very slow and takes 60% of the computation time (of displaying the image), so that I end up with 1 FPS or so. The images are 2560 by 1920 or even bigger. Scaling the cv::Ptr stream_image befor converting it to a QImage improves the performance significantly but I need all the image detail in the image.
EDIT
Here is some code how I do the conversion:
cv::Ptr<IplImage> color_image;
// stream_image is a cv::Ptr<IplImage> and holds the current image from the camera
if (stream_image->nChannels != 3) {
color_image = cvCreateImage(cvGetSize(stream_image), IPL_DEPTH_8U, 3);
cv::Mat gr(stream_image);
cv::Mat col(color_image);
cv::cvtColor(gr, col, CV_GRAY2BGR);
}
else {
color_image = stream_image;
}
QImage *tmp = new QImage(color_image->width, color_image->height, QImage::Format_RGB888);
memcpy(tmp->bits(), color_image->imageData, color_image->width * color_image->height * 3);
// update Scene
m_pixmap = QPixmap::fromImage(*tmp); // this line takes the most time!!!
m_scene->clear();
QGraphicsPixmapItem *item = m_scene->addPixmap(m_pixmap);
m_scene->setSceneRect(0,0, m_pixmap.width(), m_pixmap.height());
delete tmp;
m_ui->graphicsView->fitInView(m_scene.sceneRect(),Qt::KeepAspectRatio);
m_ui->graphicsView->update();
EDIT 2
I tested the method from from Thomas answer, but it is as slow as my method.
QPixmap m_pixmap = QPixmap::fromImage(QImage(reinterpret_cast<uchar const*>(color_image->imageData),
color_image->width,
color_image->height,
QImage::Format_RGB888));
EDIT 3
I tried to incorporate Thomas second suggestion:
color_image = cvCreateImage(cvGetSize(resized_image), IPL_DEPTH_32F, 3);
//[...]
QPixmap m_pixmap = QPixmap::fromImage(QImage(
reinterpret_cast<uchar const*>( color_image->imageData),
color_image->width,
color_image->height,
QImage::Format_RGB32));
But that crashes when the drawEvent of the Widget is called.
Q: Is there a way to display the image stream in a QGraphicsView without converting it to a QPixmap first or any other fast/performant way? The QGraphicsView is importent since I want to add overlays to the image.
I have figured out a solution that works for me but also tested a little with different methods and how they perform:
Method one is performant even in debug mode and takes only 23.7 % of the execution time of the drawing procedure (using ANALYZE in VS2012):
color_image = cvCreateImage(cvGetSize(stream_image), IPL_DEPTH_8U, 4);
cv::Mat gr(stream_image);
cv::Mat col(color_image);
cv::cvtColor(gr, col, CV_GRAY2RGBA,4);
QPixmap m_pixmap = QPixmap::fromImage(QImage(reinterpret_cast<uchar const*>( color_image->imageData),
color_image->width,
color_image->height,
QImage::Format_ARGB32));
Method two is still performant in debug mode taking 42,1% of the execution time. when the following enum is used in QPixmap::fromeImage instead
QImage::Format_RGBA8888
Method three is the one I showed in my question and it is very slow in debug builds being responsible for 68,3% of the drawing workload.
However, when I compile in release all three methods are seamingly equally performant.
This is what I usually do. Use one of the constructors for QImage that uses an existing buffer and then use QPixmap::fromImage for the rest. The format of the buffer should be compatible with the display, such as QImage::Format_RGB32. In this example a vector serves as the storage for the image.
std::vector<QRgb> image( 2560 * 1920 );
QPixmap pixmap = QPixmap::fromImage( QImage(
reinterpret_cast<uchar const*>( image.data() ),
2560,
1920,
QImage::Format_RGB32 ) );
Note the alignment constraint. If the alignemnt is not 32-bit aligned, you can use one of the constructors that takes a bytesPerLine argument.
Edit:
If your image is 32bit, then you can write.
QPixmap pixmap = QPixmap::fromImage( QImage(
reinterpret_cast<uchar const*>( color_image->imageData ),
color_image->width,
color_image->height,
QImage::Format_RGB32 ) );
In my program, I'm reading from a webcam or a video file, via OpenCV and displaying it via Qt.
I get the fps from the video properties and set the timer accordingly.
I have no problem reading the videos, the fps calibration is good (since the webcam show 0fps, I set it to 30)
But when I record the images, I set the output video's fps to the same as the original video, yet, when I read it in VLC or even Windows Media Player the video is accelerated.
The most curious thing is when I play the recorded video in my program, the fps is good, and the video isn't accelerated.
Here's how I do it :
Constructor()
{
// Initializing the video resources
cv::VideoCapture capture;
cv::VideoWriter writer;
cv::Mat frame;
int fps;
if (webcam)
{
capture.open(0);
fps = 30;
}
else
{
capture.open(inputFilePath);
fps = this->capture.get(CV_CAP_PROP_FPS);
}
// .avi
writer = cv::VideoWriter(outputFilePath, CV_FOURCC('M', 'J', 'P', 'G'), fps, frame.size());
// Defining the refresh timer;
QTimer timer = new QTimer();
connect(timer, SIGNAL(timeout()), this, SLOT(updateFeed()));
this->timer->start(1000 / fps);
}
updateFeed()
{
// ... display the image in a QLabel ... at a reasonnable speed.
writer.write(frame);
}
Now is there anything I do wrong ?
Thanks
I am working on a project with OpenFrameworks using ofxCV, ofxOpencv and ofxColorQuantizer. Technically, the project is analyzing live video captured via webcam and analysis's the image in real time to gather and output the most prominent color in the current frame. When generating the most prominent color I am using the pixel difference between the current frame and the previous frame to generate the what colors have updated and use the updated or moving areas of the video frame to figure out the most prominent colors.
The reason for using the pixel difference's to generate the color pallet is because I want to solve for the case of a user walks into the video frame, I want try and gather the color pallet of the person, for instance what they are wearing. For example red shirt, blue pants will be in the pallet and the white background will be excluded.
I have a strong background in Javascript and canvas but am fairly new to OpenFrameworks and C++ which is why I think I am running into a roadblock with this problem I described above.
Along with OpenFrameworks I am using ofxCV, ofxOpencv and ofxColorQuantizer as tools for this installation. I am taking a webcam image than making it a cv:Mat than using pyrdown on the webcam image twice followed by a absdiff of the mat which I am than trying to pass the mat into the ofxColorQuantizer. This is where I think I am running into problems — I don't think the ofxColorQuantizer likes the mat format of the image I am trying to use. I've tried looking for the different image format to try and convert the image to to solve this issue but I haven't been able to come to solution.
For efficiencies I am hoping to to the color difference and color prominence calculations on the smaller image (after I pyrdown' the image) and display the full image on the screen and the generated color palette is displayed at the bottom left like in the ofxColorQuantizer example.
I think there may be other ways to speed up the code but at the moment I am trying to get this portion of the app working first.
I have my main.cpp set up as follows:
#include "ofMain.h"
#include "ofApp.h"
#include "ofAppGlutWindow.h"
//========================================================================
int main( ){
ofAppGlutWindow window;
ofSetupOpenGL(&window, 1024,768, OF_WINDOW); // <-------- setup the GL context
// ofSetupOpenGL(1024,768,OF_WINDOW); // <-------- setup the GL context
// this kicks off the running of my app
// can be OF_WINDOW or OF_FULLSCREEN
// pass in width and height too:
ofRunApp(new ofApp());
}
My ofApp.h file is as follows:
#pragma once
#include "ofMain.h"
#include "ofxOpenCv.h"
#include "ofxCv.h"
#include "ofxColorQuantizer.h"
class ofApp : public ofBaseApp{
public:
void setup();
void update();
void draw();
ofVideoGrabber cam;
ofPixels previous;
ofImage diff;
void kMeansTest();
ofImage image;
ofImage img;
cv::Mat matA, matB;
ofImage diffCopy;
ofImage outputImage;
ofxCv::RunningBackground background;
ofxColorQuantizer colorQuantizer;
// a scalar is like an ofVec4f but normally used for storing color information
cv::Scalar diffMean;
};
And finally my ofApp.cpp is below:
#include "ofApp.h"
using namespace ofxCv;
using namespace cv;
//--------------------------------------------------------------
void ofApp::setup(){
ofSetVerticalSync(true);
cam.initGrabber(320, 240);
// get our colors
colorQuantizer.setNumColors(3);
// resize the window to match the image
// ofSetWindowShape(image.getWidth(), image.getHeight());
ofSetWindowShape(800, 600);
// imitate() will set up previous and diff
// so they have the same size and type as cam
imitate(previous, cam);
imitate(diff, cam);
imitate(previous, outputImage);
imitate(diff, outputImage);
}
//--------------------------------------------------------------
void ofApp::update(){
cam.update();
if(cam.isFrameNew()) {
matA = ofxCv::toCv(cam.getPixelsRef());
ofxCv::pyrDown(matA, matB);
ofxCv::pyrDown(matB, matA);
ofxCv::medianBlur(matA, 3);
ofxCv::toOf(matA, outputImage);
// take the absolute difference of prev and cam and save it inside diff
absdiff(previous, outputImage, diff);
}
}
//--------------------------------------------------------------
void ofApp::draw(){
// If the image is ready to draw, then draw it
if(outputImage.isAllocated()) {
outputImage.update();
outputImage.draw(0, 0, ofGetWidth(), ofGetHeight());
}
ofBackground(100,100,100);
ofSetColor(255);
ofImage diffCopy;
diffCopy = diff;
diffCopy.resize(diffCopy.getWidth()/2, diffCopy.getHeight()/2);
// there is some sort of bug / issue going on here...
// prevent the app from compiling
// comment out to run and see blank page
colorQuantizer.quantize(diffCopy.getPixelsRef());
ofLog() << "the number is " << outputImage.getHeight();
ofLog() << "the number is " << diffCopy.getHeight();
ofSetColor(255);
img.update();
// cam.draw(0, 0, 800, 600);
outputImage.draw(0, 0, 800, 600);
// colorQuantizer.draw(ofPoint(0, cam.getHeight()-20));
colorQuantizer.draw(ofPoint(0, 600-20));
// use the [] operator to get elements from a Scalar
float diffRed = diffMean[0];
float diffGreen = diffMean[1];
float diffBlue = diffMean[2];
ofSetColor(255, 0, 0);
ofRect(0, 0, diffRed, 10);
ofSetColor(0, 255, 0);
ofRect(0, 15, diffGreen, 10);
ofSetColor(0, 0, 255);
ofRect(0, 30, diffBlue, 10);
}
//--------------------------------------------------------------
void ofApp::kMeansTest(){
cv::Mat samples = (cv::Mat_<float>(8, 1) << 31 , 2 , 10 , 11 , 25 , 27, 2, 1);
cv::Mat labels;
// double kmeans(const Mat& samples, int clusterCount, Mat& labels,
cv::TermCriteria termcrit;
int attempts, flags;
cv::Mat centers;
double compactness = cv::kmeans(samples, 3, labels, cv::TermCriteria(), 2, cv::KMEANS_PP_CENTERS, centers);
cout<<"labels:"<<endl;
for(int i = 0; i < labels.rows; ++i)
{
cout<<labels.at<int>(0, i)<<endl;
}
cout<<"\ncenters:"<<endl;
for(int i = 0; i < centers.rows; ++i)
{
cout<<centers.at<float>(0, i)<<endl;
}
cout<<"\ncompactness: "<<compactness<<endl;
}
Apologies in advance for the state of my code — it's getting late and I'm trying to get this done.
My question is what is the image format openFrameworks is using for grabbing the webcam image, what is the image format that openCV expects and what should I use to switch back from a mat image to an ofImage and is there a way to getPixelsRef from a mat image?
The area of code that I think I have something wrong is the following logic.
I have this line of code which gets the video frame from the webcam matA = ofxCv::toCv(cam.getPixelsRef());
Than do a couple ofxCv procedures on the frame such as ofxCv::pyrDown(matA, matB); which I think changes the image format or pixel format of the frame
Than I convert the frame back to OF with ofxCv::toOf(matA, outputImage);,
Next I get the difference in the pixels between the current frame and the last frame, create a copy of the difference between the two frames. Potentially the issue lies here with the diff output image format
Which I pass the diff copy to colorQuantizer.quantize(diffCopy.getPixelsRef()); to try and generate the color palette in for the change in pixels.
It is the colorQuantizer class and function call that is giving me an error which reads thread error [ error ] ofTexture: allocate(): ofTextureData has 0 width and/or height: 0x0
with an EXC_BAD_ACCESS
And lastly, could there be an alternative cause for the exc_bad_access thread error rather than image formatting? Being new to c++ I'm just guessing and going off instinct of what I think the rood cause of my problem is.
Many thanks.