I am currently doing real-time face evaluation and is trying to set the FPS of the Camera of my computer to 1 frame per second, followed by calling the cascade functions only once per second. (Currently using a While(true) loop) This is due to the limitation of my GPU.
I have tried to set the FPS of the camera by using
VideoCapture cap(0);
cap.set(CV_CAP_PROP_FPS, 1);
namedWindow("webcam",CV_WINDOW_AUTOSIZE);
but it is not working. The camera still process at a relative high FPS.
For the cascade function calling, I am doing it as below:
while ( true ){
cap >> frame;
vector<Rect> faces;
face_cascade.detectMultiScale( frame, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(30, 30) );
// Draw circles on the detected faces
for( int i = 0; i < faces.size(); i++ )
{
Point center( faces[i].x + faces[i].width*0.5, faces[i].y + faces[i].height*0.5 );
cout<<"Face location: "<<faces[i].x<<","<<faces[i].x + faces[i].width<<","<<faces[i].y<<","<<faces[i].y + faces[i].height;
ellipse( frame, center, Size( faces[i].width*0.5, faces[i].height*0.5), 0, 0, 360, Scalar( 255, 0, 255 ), 4, 8, 0 );
}
waitKey(30);
if ( !frame.data ){
cerr << "Cannot acquire frame from the webcam " << endl;
break;
}
imshow("webcam", frame);
}
I need the camera to go for only 1 frame per second, followed by calling the cascade functions once per second.
Edit: I have tried to display the FPS of the camera by using
int FPS = cap.get(CV_CAP_PROP_FPS);
It did show that FPS is currently at 1, but it seems that the camera is still moving at a relative high frame rate.
Setting the frame rate does not always work. Sometimes the camera simply does not respond to this change. However, you can do something to solve your problem in a tricky way. Measure the time that it takes to processing a frame then subtract it from 1000 mSec (1000 - Elapsce_Time) and make it wait for this time cv::waitKey(1000-Elapsce_Time). Finally, this is not a very way to do it. You should search for the actual problem with the camera and try to solve it.
Related
I am using haar cascading to detect frontal faces. I have below code:
int main()
{
Mat image;
cv::VideoCapture cap;
cap.open(1);
int frame_idx = 0;
time_t fpsStartTime, fpsEndTime;
time(&fpsStartTime);
for (;;)
{
frame_idx = frame_idx + 1;
cap.read(image);
CascadeClassifier face_cascade;
face_cascade.load("<PATH");
std::vector<Rect> faces;
face_cascade.detectMultiScale(image, faces, 1.1, 2, 0 | cv::CASCADE_SCALE_IMAGE, Size(30, 30));
// Draw circles on the detected faces
for (int i = 0; i < faces.size(); i++)
{
Point center(faces[i].x + faces[i].width*0.5, faces[i].y + faces[i].height*0.5);
ellipse(image, center, Size(faces[i].width*0.5, faces[i].height*0.5), 0, 0, 360, Scalar(255, 0, 255), 4, 8, 0);
}
cv::imshow("Detected Face", image);
char k = cv::waitKey(1);
if (k == 27)
break;
time(&fpsEndTime);
double seconds = difftime(fpsEndTime, fpsStartTime);
double fps = frame_idx / seconds;
std::string fps_txt = "FPS: " + std::to_string(fps); // fps_str.str();
cout << "FPS : " << fps_txt << endl;
}
return 0;
}
This code is working fine but giving very low FPS. FPS is ~1fps which is very slow. I am running this on Windows 10 laptop with intel i5 CPU. I believe this should not be this much slow.
In debug mode, it gives ~1fps but in release mode it is 4-5fps which again is very slow. I have run some openvino demo's like pedestrian detection which uses 2 openvino model on same hardware and it gives ~17-20fps which is very good.
I am using USB 3.0 logitech brio 4k camera so this cannot be a reason of low fps. My question is why haar cascading is performing very slow. Is there anyway we can enhance its speed and make it more usable. Please help. Thanks
You should not (re)load the classifier on every frame. It should load once before processing frames.
Move the following statements out of the for loop.
CascadeClassifier face_cascade;
face_cascade.load("<PATH");
See a demo on OpenCV Docs.
Can you confirm if you are using right .lib and .dll file?
I have checked and seen that the opencv_world440.lib & opencv_world440.dll provide great speed compared to opencv_world440d.lib & opencv_world440d.dll files.
My guess is that opencv_world440d.lib & opencv_world440d.dll are for debugging so slow speed.
Note::Your lib name may vary ie.., opencv_world<"SomeNumber">d.lib & opencv_world<"SomeNumber">.lib
I want to show up a cross in the middle of my screen and change the size of it pressing some keys on my keyboard.
For example ,
if I press b , the cross should become big.
if I press s, the cross should become small.
if I press m, the cross should become medium.
#include "opencv2/opencv.hpp"
#include <iostream>
using namespace std;
using namespace cv;
int main()
{
VideoCapture cap(0);
while(true)
{
Mat frame;
// Capture frame-by-frame
cap >> frame;
// If the frame is empty, break immediately
if (frame.empty())
break;
// for converting the frame to grayscale
Mat gray;
cvtColor( frame, gray , COLOR_BGR2GRAY);
line( frame, Point( 300, 240 ), Point( 340,240), Scalar( 0, 0, 255 ), 2, 4 );
line( frame, Point( 320, 260), Point( 320, 220), Scalar( 0, 0, 255 ), 2, 4);
imshow(" figure ",frame);
char c=(char)waitKey(25);
if(c==27)
break;
}
cap.release();
destroyAllWindows();
return 0;
}
please help me with this
My proposition is to introduce 'scale' variable that will be modified by key presses to calculate start and end points for both lines. Just assume those points are defined as [start point] = [middle point] - [scale] * [scale factor] and [end point] = [middle point] + [scale] * [scale factor]. So it would look like:
VideoCapture cap(0);
int size = 2;
bool drawCross = 1;
while(true)
{
Mat frame;
// Capture frame-by-frame
cap >> frame;
// If the frame is empty, break immediately
if (frame.empty())
break;
// for converting the frame to grayscale
Mat gray;
cvtColor( frame, gray , COLOR_BGR2GRAY);
if (drawCross) {
line( frame, Point( 320 - 10*size, 240 ), Point( 320 + 10*size,240), Scalar( 0, 0, 255 ), 2, 4 );
line( frame, Point( 320, 240 - 10*size), Point( 320, 240 + 10*size), Scalar( 0, 0, 255 ), 2, 4);
}
imshow(" figure ",frame);
char c=(char)waitKey(25);
if(c==27)
break;
else if (c==[whatever this "small" key is])
size = 1;
else if (c==[whatever this "medium" key is])
size = 2;
else if (c==[whatever this "large" key is])
size = 4;
else if (c==[whatever this "do not draw cross" key is])
drawCross = !drawCross;
}
==EDIT==
Solution is now fixed to be working with new 'requirements' given in comments below. This is my last input on this question, as SO is not 'Can you write that for me?' type of community. Your problems described in comments required me up to two minutes of googl-ing. And you need to read up on programming basics before you go on, like conditional branching and stuff. And I do not know if this code works 100% as I don't want to install C++ compiler at this moment.
I am receiving a videostream (640x480p) via UDP and use OpenCV's imdecode() to decode every frame in the same thread. If correctly decoded the frame is passed to a newly started thread for image processing (findChessboardCorners() and polylines()) and the thread is detached.
The receiving and decoding part works perfectly but I logged the execution time for polylines() and it starts at about 5ms and gets worse the longer the program runs (up to 4000ms and more). Visual Studio's performance profiler reported, that polylines() uses ~98% of the CPU. the vector with points to draw using polylines() consists of 40 points.
Even though i am detaching each thread, what could cause this performance loss? (even tested it with an Intel Xeon)
void decode(Mat videoFrame) {
Mat rotationMat;
Mat translationMat;
Mat chessboard;
resize(videoFrame, chessboard, Size(), resizeFactor, resizeFactor);
Size patternSize(chessboardSize.front(), chessboardSize.back());
vector<Point2f> corners;
vector<Point2f> imagePoints;
bool patternFound = findChessboardCorners(chessboard, patternSize, corners, CALIB_CB_ADAPTIVE_THRESH + CALIB_CB_FAST_CHECK);
if (patternFound) {
solvePnP(objectPoints, corners, cameraMatrix, distCoeffs, rotationMat, translationMat);
vector<Point3d> path_3d = fahrspur.computePath(steeringAngle);
vector<Point2d> path_2d;
projectPoints(path_3d, rotationMat, translationMat, cameraMatrix, distCoeffs, path_2d);
Mat curve(path_2d, true);
curve.convertTo(curve, CV_32S);
double t4 = getCurrentTime();
polylines(chessboard, curve, false, Scalar(0, 255, 0), 10, CV_AA);
double t5 = getCurrentTime();
cout << "time to execute polylines: " << t5-t4 << "ms" << endl;
assignFrameVideo(chessboard);
}
A new thread with this decode method is started from another thread, used for receiving the frames, in a while loop:
Mat frameVideo;
while(1) {
//code for receiving a single frame, decode it and store it in frameVideo.
thread decodeThread = thread(decode, frameVideo);
decodeThread.detach();
}
I also used the second option to use polylines() that way:
const Point *pts = (const Point*)Mat(path_2d).data;
int npts = Mat(path_2d).rows;
polylines(chessboard, &pts, &npts, 1, false, Scalar(0, 255, 0), 5);
But that does not work at all, the image is displayed without any lines.
i solved it by replacing CV_AA with LINE_4 as a parameter in polylines().
apparently the anti aliasing of the drawn line is the heavy part, now it runs within 0-1 ms.
I'm currently working on a project where I need to display a processed live video capture. Therefore, I'm using something similar to this:
cv::VideoCapture cap(0);
if (!cap.isOpened())
return -1;
cap.set(CV_CAP_PROP_FRAME_WIDTH, 1280);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 720);
cv::namedWindow("Current Capture");
for (;;)
{
cv::Mat frame;
cap >> frame;
cv::Mat mirrored;
cv::flip(frame, mirrored, 1);
cv::imshow("Current Capture", process_image(mirrored));
if (cv::waitKey(30) >= 0) break;
}
The problem I have is, that process_image, which perfomes a circle detection in the image, needs some time to finish and causes the displaying to be rather a slideshow then a video.
My Question is: How can I speed up the processing without manipulating the process_image function?
I thought about performing the image processing in another thread, but I'm not really sure how to start. Do you have any other idea than this?
PS.: I'm not expecting you to write code for me, I only need a point to start from ;)
EDIT:
Ok, if there is nothing i can do about the performance while capturing, I will need to change the process_image function.
cv::Mat process_image(cv::Mat img)
{
cv::Mat hsv;
cv::medianBlur(img, img, 7);
cv::cvtColor(img, hsv, cv::COLOR_BGR2HSV);
cv::Mat lower_hue_range; // lower and upper hue range in case of red color
cv::Mat upper_hue_range;
cv::inRange(hsv, cv::Scalar(LOWER_HUE1, 100, 100), cv::Scalar(UPPER_HUE1, 255, 255), lower_hue_range);
cv::inRange(hsv, cv::Scalar(LOWER_HUE2, 100, 100), cv::Scalar(UPPER_HUE1, 255, 255), upper_hue_range);
/// Combine the above two images
cv::Mat hue_image;
cv::addWeighted(lower_hue_range, 1.0, upper_hue_range, 1.0, 0.0, hue_image);
/// Reduce the noise so we avoid false circle detection
cv::GaussianBlur(hue_image, hue_image, cv::Size(13, 13), 2, 2);
/// store all found circles here
std::vector<cv::Vec3f> circles;
cv::HoughCircles(hue_image, circles, CV_HOUGH_GRADIENT, 1, hue_image.rows / 8, 100, 20, 0, 0);
for (size_t i = 0; i < circles.size(); i++)
{
/// circle center
cv::circle(hsv, cv::Point(circles[i][0], circles[i][1]), 3, cv::Scalar(0, 255, 0), -1, 8, 0);
/// circle outline
cv::circle(hsv, cv::Point(circles[i][0], circles[i][1]), circles[i][2], cv::Scalar(0, 0, 255), 3, 8, 0);
}
cv::Mat newI;
cv::cvtColor(hsv, newI, cv::COLOR_HSV2BGR);
return newI;
}
Is there a huge perfomance issue I can do anything about?
If you are sure that the process_image function is what is causing the bottle neck in your program, but you can't modify it, then there's not really a lot you can do. If that function takes longer to execute than the duration of a video frame then you will never get what you need.
How about reducing the quality of the video capture or reducing the size? At the moment I can see you have it set to 1280*720. If the process_image function has less data to work with it should execute faster.
I'm writing a program using C++ and OpenCV. It's actually my first so what I'm asking is probably something very basic I've overlooked. Much of it is copied - not copy+pasted mind you, but copied by hand, going line by line, understanding what each line was doing as I wrote it - from some of OpenCV's tutorials. I'll paste the code below.
The problem I'm encountering is that as soon as the webcam starts trying to implement facial recognition, everything just SLOWS. DOWN. As I understand it, its because the .exe is trying to read from two MASSIVE .xml files every frame update, but I don't have any idea how to fix it. It was worse before I constrained the height, width, and framerate of the video.
If anyone has any ideas at this point, I'd love to hear them. I'm very new to software programming - until now I've mostly done web development, so I'm not used to worrying about system memory and other factors.
Thanks in advance!
EDIT: Here are my system specs: Mac, OSX 10.9.4, 2.5 GHz Intel Core i5, 4 GB 1600 MHz DDR3 RAM.
#include "opencv2/objdetect.hpp"
#include "opencv2/videoio.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"
#include <iostream>
#include <stdio.h>
using namespace std;
using namespace cv;
/** Function Headers */
void detectAndDisplay( Mat frame );
/** Global variables */
String face_cascade_name = "haarcascade_frontalface_alt.xml";
String eyes_cascade_name = "haarcascade_eye_tree_eyeglasses.xml";
CascadeClassifier face_cascade;
CascadeClassifier eyes_cascade;
String window_name = "Capture - Face detection";
/** #function main */
int main( void )
{
cv::VideoCapture capture;
Mat frame;
//-- 1. Load the cascades
if( !face_cascade.load( face_cascade_name ) ){ printf("--(!)Error loading face cascade\n"); return -1; };
if( !eyes_cascade.load( eyes_cascade_name ) ){ printf("--(!)Error loading eyes cascade\n"); return -1; };
//-- 2. Read the video stream
capture.open( -1 );
if ( ! capture.isOpened() ) { printf("--(!)Error opening video capture\n"); return -1; }
capture.set(CV_CAP_PROP_FRAME_WIDTH,640);
capture.set(CV_CAP_PROP_FRAME_HEIGHT,480);
capture.set(CV_CAP_PROP_FPS, 15);
while ( capture.read(frame) )
{
if( frame.empty() )
{
printf(" --(!) No captured frame -- Break!");
break;
}
//-- 3. Apply the classifier to the frame
detectAndDisplay( frame );
int c = waitKey(10);
if( (char)c == 27 ) { break; } // escape
}
return 0;
}
/** #function detectAndDisplay */
void detectAndDisplay( Mat frame )
{
std::vector<Rect> faces;
Mat frame_gray;
cvtColor( frame, frame_gray, COLOR_BGR2GRAY );
equalizeHist( frame_gray, frame_gray );
//-- Detect faces
face_cascade.detectMultiScale( frame_gray, faces, 1.1, 2, 0|CASCADE_SCALE_IMAGE, Size(30, 30) );
for ( size_t i = 0; i < faces.size(); i++ )
{
Point center( faces[i].x + faces[i].width/2, faces[i].y + faces[i].height/2 );
ellipse( frame, center, Size( faces[i].width/2, faces[i].height/2 ), 0, 0, 360, Scalar( 255, 0, 255 ), 4, 8, 0 );
Mat faceROI = frame_gray( faces[i] );
std::vector<Rect> eyes;
//-- In each face, detect eyes
eyes_cascade.detectMultiScale( faceROI, eyes, 1.1, 2, 0 |CASCADE_SCALE_IMAGE, Size(30, 30) );
for ( size_t j = 0; j < eyes.size(); j++ )
{
Point eye_center( faces[i].x + eyes[j].x + eyes[j].width/2, faces[i].y + eyes[j].y + eyes[j].height/2 );
int radius = cvRound( (eyes[j].width + eyes[j].height)*0.25 );
circle( frame, eye_center, radius, Scalar( 255, 0, 0 ), 4, 8, 0 );
}
}
//-- Show what you got
imshow( window_name, frame );
}
A quick solution would be to replace:
eyes_cascade.detectMultiScale( faceROI, eyes, 1.1, 2, 0 |CASCADE_SCALE_IMAGE, Size(30, 30) );
by
eyes_cascade.detectMultiScale( faceROI, eyes, 1.3, 2, 0 |CASCADE_SCALE_IMAGE, Size(60, 60), Size(350, 350) );
1.3 is the scale factor, Size(60, 60) the min windows size and Size(350, 350) the max one. It means basically that it will start to search for 60*60 faces then increase size by oldWindowSize*1.3 until it reach 350*350. It is assumed there that your faces are min 60*60 and max 350 * 350.
You can tune it even more depending what you want. The minSize will have a the most impact on performance as well as scale (but 1.3 is already high). The maxSize will have less impact.
After this update, your prog should be twice faster or decrease CPU usage by half. However, I am still surprise that with your current tunings and you computer you have performances problems...
Give us a feedback if it works.