Object Counter in Opencv + BeagleBone Black Performance Issue - c++

I am facing performance issue in BeagleBone Black + Opencv Object Counter. I am using BackgroundSubtractorMOG2 for background subtraction and Contours Detection. Here is the code below:
cv::Mat frame;
cv::Mat resizedFrame;
cv::Mat back;
cv::Mat fore;
bool objStart = false;
bool objEnd = false;
bool start = true;
cv::Point startLine(0, 50); // this is the start of the line where I take decision
cv::Point endLine(1000, 50); // this is the end of the line
cv::VideoCapture cap("/home/moonzai/Videos/test.avi");
cv::BackgroundSubtractorMOG2 bg;
bg.set("nmixtures", 3);
vector<vector<cv::Point> > contours;
for(;;)
{
cap >> resizedFrame;
cv::resize(resizedFrame, frame, cv::Size(320, 240), 0, 0, cv::INTER_LINEAR); // I wrote this line when there were only 1 frame per second processing, I resized the frame to 320 X 240
if(start)
{
bg.operator ()(frame,fore);
bg.getBackgroundImage(back);
cv::erode(fore,fore,cv::Mat());
cv::dilate(fore,fore,cv::Mat());
cv::findContours(fore,contours,CV_RETR_EXTERNAL,CV_CHAIN_APPROX_NONE);
vector<cv::Rect> boundRect( contours.size() );
cv::Rect mainRect;
for( unsigned int i = 0; i < contours.size(); i++ )
{
boundRect[i] = boundingRect( cv::Mat(contours[i]) );
if(mainRect.area() < boundRect[i].area())
{
mainRect = boundRect[i];
}
}
if(LineIntersectsRect(startLine, endLine, mainRect)) // this function actually returns boolean, if rectangle is touching the line
{
objStart = true;
}
else
{
if(objStart)
{
objEnd = true;
}
}
if(objEnd && objStart)
{
counter ++;
cout << "Object: " << counter << endl;
objEnd = false;
objStart = false;
}
}
usleep(1000 * 33);
}
this code is working perfect on my desktop machine. but when I run this code on BeagleBone Black with Ubuntu 13.04 linux installed, this distribution has no GUI at all, I am working on terminal, it give me CPU usage of 80% with 2 frames per second of processing. Memory Usage is very low, about 8%, I am not getting my desired performance. so please guide me if I am doing something wrong.
The objective of my question is, is there any coding related issue or, BackgroundSubtractorMOG2 is resource hungry, so I have to use another way. If there is another way, then guide me what is that way?
thanks in advance...

I think that the best option is to use profiler(Very sleepy is quite easy to use, but still enough powerful for me, but i'm not sure whether there is linux version) and check in which part of your code there is a problem - take a look at this discussion How can I profile C++ code running in Linux? (accepted answer may not be good option in your situation, so look carefully at other answers too).
Also you may just try to decrease sleep time, it should increase fps and CPU usage.

C++ application performance is highly dependent on compiler options. Could you please provide gcc options that used to compile opencv library and your application?

Related

Why is haar cascade very slow opencv c++

I am using haar cascading to detect frontal faces. I have below code:
int main()
{
Mat image;
cv::VideoCapture cap;
cap.open(1);
int frame_idx = 0;
time_t fpsStartTime, fpsEndTime;
time(&fpsStartTime);
for (;;)
{
frame_idx = frame_idx + 1;
cap.read(image);
CascadeClassifier face_cascade;
face_cascade.load("<PATH");
std::vector<Rect> faces;
face_cascade.detectMultiScale(image, faces, 1.1, 2, 0 | cv::CASCADE_SCALE_IMAGE, Size(30, 30));
// Draw circles on the detected faces
for (int i = 0; i < faces.size(); i++)
{
Point center(faces[i].x + faces[i].width*0.5, faces[i].y + faces[i].height*0.5);
ellipse(image, center, Size(faces[i].width*0.5, faces[i].height*0.5), 0, 0, 360, Scalar(255, 0, 255), 4, 8, 0);
}
cv::imshow("Detected Face", image);
char k = cv::waitKey(1);
if (k == 27)
break;
time(&fpsEndTime);
double seconds = difftime(fpsEndTime, fpsStartTime);
double fps = frame_idx / seconds;
std::string fps_txt = "FPS: " + std::to_string(fps); // fps_str.str();
cout << "FPS : " << fps_txt << endl;
}
return 0;
}
This code is working fine but giving very low FPS. FPS is ~1fps which is very slow. I am running this on Windows 10 laptop with intel i5 CPU. I believe this should not be this much slow.
In debug mode, it gives ~1fps but in release mode it is 4-5fps which again is very slow. I have run some openvino demo's like pedestrian detection which uses 2 openvino model on same hardware and it gives ~17-20fps which is very good.
I am using USB 3.0 logitech brio 4k camera so this cannot be a reason of low fps. My question is why haar cascading is performing very slow. Is there anyway we can enhance its speed and make it more usable. Please help. Thanks
You should not (re)load the classifier on every frame. It should load once before processing frames.
Move the following statements out of the for loop.
CascadeClassifier face_cascade;
face_cascade.load("<PATH");
See a demo on OpenCV Docs.
Can you confirm if you are using right .lib and .dll file?
I have checked and seen that the opencv_world440.lib & opencv_world440.dll provide great speed compared to opencv_world440d.lib & opencv_world440d.dll files.
My guess is that opencv_world440d.lib & opencv_world440d.dll are for debugging so slow speed.
Note::Your lib name may vary ie.., opencv_world<"SomeNumber">d.lib & opencv_world<"SomeNumber">.lib

lag in opencv videocapture when i use rtsp camera stream

So i'm currently working on a project that needs to do a facial recognition on rtsp ip cam , i managed to get the rtsp feed with no problems, but when it comes to applying the face recognition the video feed gets too slow and shows a great delay, i even used multithreading to make it better but with no success,here is my code i'm still a beginner in multi threading matters so any help would be appreciated.
#include <iostream>
#include <thread>
#include "opencv2/opencv.hpp"
#include <vector>
using namespace std;
using namespace cv;
void detect(Mat img, String strCamera) {
string cascadeName1 = "C:\\ocv3.2\\Build\\install\\etc\\haarcascades\\haarcascade_frontalface_alt.xml";
CascadeClassifier facedetect;
bool loaded1 = facedetect.load(cascadeName1);
Mat original;
img.copyTo(original);
vector<Rect> human;
cvtColor(img, img, CV_BGR2GRAY);
equalizeHist(img, img);
facedetect.detectMultiScale(img, human, 1.1, 2, 0 | 1, Size(40, 80), Size(400, 480));
if (human.size() > 0)
{
for (int gg = 0; gg < human.size(); gg++)
{
rectangle(original, human[gg].tl(), human[gg].br(), Scalar(0, 0, 255), 2, 8, 0);
}
}
imshow("Detect " + strCamera, original);
int key6 = waitKey(40);
//End of the detect
}
void stream(String strCamera) {
VideoCapture cap(strCamera);
if (cap.isOpened()) {
while (true) {
Mat frame;
cap >> frame;
resize(frame, frame, Size(640, 480));
detect(frame, strCamera);
}
}
}
int main() {
thread cam1(stream, "rtsp://admin:password#ipaddress:554/live2.sdp?tcp");
thread cam2(stream, "rtsp://admin:password#ipaddress/live2.sdp?tcp");
cam1.join();
cam2.join();
return 0;
}
I had similar issues and was able to resolve them by completely isolating the frame capturing from processing of the images. I also updated OpenCV to the latest (3.2.0) available, but I think this will also resolve problems with earlier versions.
void StreamLoop(String strCamera, LFQueue1P1C<Mat> *imageQueue, bool *shutdown) {
VideoCapture cap(strCamera, CV_CAP_FFMPEG);
Mat image;
while(!(*shutdown) && cap.isOpened()){
*cap >> image;
imageQueue->Produce(image);
}
}
int main(){
Mat aImage1;
bool shutdown(false);
LFQueue1P1C<Mat> imageQueue;
string rstp("rtsp://admin:password#ipaddress:554/live2.sdp?tcp");
thread streamThread(StreamLoop, rtsp, &imageQueue, &shutdown);
...
while(!shutdownCondition){
if(imageQueue.Consume(aImage1)) {
// Process Image
resize(aImage1, aImage1, Size(640, 480));
detect(aImage1, rtsp);
}
}
shutdown = true;
if(streamThread.joinable()) streamThread.join();
...
return 0;
}
It seems that there is some issue with rtsp in OpenCV where it easily hangs up if there are even slight pauses while picking up the frames. As long as I pick up frames without much pause I have not seen a problem.
Also, I didn't have this issue when the video cameras where directly connected to my local network. It was not until we deployed them at a remote site that I started getting the hang ups. Separating frame retrieval and processing into separate threads resolved my issues, hopefully someone else might find this solution useful.
Note: The queue I used is a custom queue for passing items from one thread to another. The code I posted is modified from my original code to make it more readable and applicable to this problem.
i'm still a beginner in multi threading matters so any help would be appreciated
Having threads that have no way of exiting will cause you issues in the future. Even if it is test code, get in the habit of making sure the code has an exit path. As an example: You might copy and paste a section of code later on and forget there is an infinite loop in there and it will cause great grief later trying to track down why you have mysterious crashing or your resources are locked up.
I am not a C++ developer but I had the same problem in Java. I solved my issue by calling VideoCapture.grab() function before reading camera frame. According to OpenCV Doc, the use of the grab function is :
The primary use of the function is in multi-camera environments,
especially when the cameras do not have hardware synchronization.
Besides that, in java application, you should release your frame's Mat objects every time you read new frames.

OpenCV C++ Multi-threading to Improve Framerate

I have an opencv program where the image processing is sensitive to having a stable and relatively high framerate in the video capture. Unfortunately, all the image processing I do is slowing down the framerate significantly, resulting in erroneous behavior in my program. I believe that putting the camera on a separate thread and having the image processing happen on its own thread would improve framerate, but I am not sure how to do this. Can anyone guide me through the process?
UPDATE: After doing some research on threads, I managed to implement threading to where the final video feed post-processed is displayed. However, somehow my implementation of it is causing the image processing methods to fail (ex. before I could successfully track a moving object, now it is erroneous whether or not it is tracked). I suspect this has something to do with the image processing algorithms not being able to process each frame fast enough as new frames are read in. How can I improve this implementation so that my processing methods worked as they did without multithreading?
void CaptureFrames() {
VideoCapture capture(0);
if (!capture.isOpened()) {
cout << "cannot open camera" << endl;
}
while (true) {
//CAMERAFRAME is a global Mat defined at the top of my program
capture.read(CAMERAFRAME);
if (waitKey(30) >= 0) { break; }
}
}
void ProcessFrames() {
while (true) {
Mat hsvFrame;
Mat binMat;
Mat grayFrame;
Mat grayBinMat;
if (!CAMERAFRAME.empty()) {
//do image processing functions (these worked before implementing threads and are not causing errors)
imshow("gray binary", grayBinMat);
imshow("multithread test", CAMERAFRAME);
}
if (waitKey(30) >= 0) { break; }
}
}
int main(int argc, char** argv[]) {
thread t1(CaptureFrames);
thread t2(ProcessFrames);
while(true) {
if(waitKey(30) >= 0) {break;}
}
return 0;
}
Try the older version again but remove this last line from the ProcessFrames function.
if (waitKey(30) >= 0) { break; }
On showing images don't make it wait again for 30 m-seconds, the while loop will be enough

OpenCV Slow camera framerate

I just copied an program to detect faces through webcam but the video capture is really slow, and i dont know how to fix it!
Here is the code:
#include<stdio.h>
#include<math.h>
#include<opencv\cv.h>
#include<opencv\highgui.h>
#include<opencv2\objdetect\objdetect.hpp>
#include<opencv2\highgui\highgui.hpp>
#include<opencv2\imgproc\imgproc.hpp>
#include<vector>
using namespace cv;
using namespace std;
int main()
{
CascadeClassifier face_cascade;
if(!face_cascade.load("c:\\haar\\haarcascade_frontalface_alt2.xml")) {
printf("Error loading cascade file for the face");
return 1;
}
VideoCapture capture(0);
if(!capture.isOpened())
{
printf("Error trying to start the Camera");
return 1;
}
Mat cap_img,gray_img;
vector<Rect> faces;
while(1)
{
capture >> cap_img;
waitKey(10);
cvtColor(cap_img, gray_img, CV_BGR2GRAY);
cv::equalizeHist(gray_img,gray_img);
face_cascade.detectMultiScale(gray_img, faces, 1.1, 10, CV_HAAR_SCALE_IMAGE | CV_HAAR_DO_CANNY_PRUNING, cvSize(0,0), cvSize(300,300));
for(int i=0; i < faces.size();i++)
{
Point pt1(faces[i].x+faces[i].width, faces[i].y+faces[i].height);
Point pt2(faces[i].x,faces[i].y);
rectangle(cap_img, pt1, pt2, cvScalar(191,191,191), 2, 8, 0);
}
imshow("Result", cap_img);
waitKey(3);
char c = waitKey(3);
if(c == 27)
break;
}
return 0;
}
I am using Visual studio 2012 and this is the main.cpp file. Im using OpenCV 2.4.9!
OpenCV comes with prebuild libraries. When you use them in an application you actually want to deploy make sure you use the release libraries. In debug mode you have lots of additional checks in the form of asserts as well as debug symbols allowing you to step into the libraries with a debugger that are removed in release mode.
Another more specific advise regarding the code you posted: Avoid calls to cv::waitKey() as every single call, who would have guessed, makes your main thread wait the specified amount of time in milliseconds. Do not drop calls to it entirely as e.g. cv::imshow() will only work appropriately with these in place.
Edit:
Reduce your while loop to:
while ( true ) {
capture >> cap_img;
imshow("Result", cap_img);
if(waitKey(1) == 27)
break;
}
When you know how much time you need for capturing the image and just displaying it, then you can compare how much of an impact whatever algorithm you are running has on your performance.
Here is solution...everytime you read!!!..everytime you use vcap>>cap or cap.read(vcap)... you must set FPS ..25 is max..if i try to set 26 or higher it runs slow.
...this runs 30 fps for me
If you not set FPS...it runs slow too
if (cap.read(vcap)){
imshow("Cam Feed", vcap);
cap.set(CV_CAP_PROP_FPS, 25);
}
On top of the other answers, Multiprocessing is what fixed my issue. I split my program into 3 processes. One for grabbing the image, one for processing the image, and one for displaying the image. Referencing this page helped me a lot. I also followed the tips of the other answers here.

OpenCV Frame Rate Issue

I am working on a school project with OpenCV. A major part of the program will be a comparison of histograms. There will be a database of histograms and new histograms will be created from a live video feed then compared to the histograms in the database. Right now I am just trying to get the histograms created correctly from the video feed. My problem is that the program crashes or slows down dramatically at random intervals. So my question is how do I prevent the program from crashing or slowing down? OpenCV has always been kind of flaky for me, so I'm not sure if it is an issue with my code or if it is just the nature of OpenCV. If it is to do with my code I think the issue might have something to do with the frame rates (a guess/gut feeling). I am useing "cvWaitKey" to "pace" the loading of frames, but the "Learning OpenCV" book has this to say about "cvWaitKey"
c = cvWaitKey(33);
if( c == 27 ) break;
Once we have displayed the frame, we then wait for 33 ms. If the user hits a key, then c
will be set to the ASCII value of that key; if not, then it will be set to –1. If the user hits
the Esc key (ASCII 27), then we will exit the read loop. Otherwise, 33 ms will pass and
we will just execute the loop again.
It is worth noting that, in this simple example, we are not explicitly controlling
the speed of the video in any intelligent way. We are relying solely on the timer in
cvWaitKey() to pace the loading of frames. In a more sophisticated application it would
be wise to read the actual frame rate from the CvCapture structure (from the AVI) and
behave accordingly!
You will see in my code below (modified from here) that I my loop waits 10ms before starting the next execution. Often times the program will run with no issues at all, but sometimes it will crash less than a minute in, or five minutes in, there really is not pattern that I can detect. Any suggestions on how this crash( or slow down) can be prevented would be welcomed. Also I should add that I am using OpenCV 1.1 (can't ever get OpenCV 2.0 to work right), I am using Visual Studio 2008, and I create an .MSI installer package everytime I modify my code, that is, I do not debug in Visual Studio. Dependencies are cv110.dll, cxcore110.dll, and highgui110.dll. My code is below:
// SLC (Histogram).cpp : Defines the entry point for the console application.
#include "stdafx.h"
#include <cxcore.h>
#include <cv.h>
#include <cvaux.h>
#include <highgui.h>
#include <stdio.h>
#include <sstream>
#include <iostream>
using namespace std;
int main(){
CvCapture* capture = cvCaptureFromCAM(0);
if(!cvQueryFrame(capture)){
cout<<"Video capture failed, please check the camera."<<endl;
}
else{
cout<<"Video camera capture successful!"<<endl;
};
CvSize sz = cvGetSize(cvQueryFrame(capture));
IplImage* image = cvCreateImage(sz, 8, 3);
IplImage* imgHistogram = 0;
IplImage* gray = 0;
CvHistogram* hist;
cvNamedWindow("Image Source",1);
cvNamedWindow("Histogram",1);
for(;;){
image = cvQueryFrame(capture);
//Size of the histogram -1D histogram
int bins = 256;
int hsize[] = {bins};
//Max and min value of the histogram
float max_value = 0, min_value = 0;
//Value and normalized value
float value;
int normalized;
//Ranges - grayscale 0 to 256
float xranges[] = {0, 256};
float* ranges[] = {xranges};
//Create an 8 bit single channel image to hold a grayscale version of the original picture
gray = cvCreateImage(cvGetSize(image), 8, 1);
cvCvtColor(image, gray, CV_BGR2GRAY);
//Planes to obtain the histogram, in this case just one
IplImage* planes[] = {gray};
//Get the histogram and some info about it
hist = cvCreateHist(1, hsize, CV_HIST_ARRAY, ranges,1);
cvCalcHist(planes, hist, 0, NULL);
cvGetMinMaxHistValue(hist, &min_value, &max_value);
printf("Minimum Histogram Value: %f, Maximum Histogram Value: %f\n", min_value, max_value);
//Create an 8 bits single channel image to hold the histogram and paint it white
imgHistogram = cvCreateImage(cvSize(bins, 50),8,3);
cvRectangle(imgHistogram, cvPoint(0,0), cvPoint(256,50), CV_RGB(255,255,255),-1);
//Draw the histogram
for(int i=0; i < bins; i++){
value = cvQueryHistValue_1D(hist, i);
normalized = cvRound(value*50/max_value);
cvLine(imgHistogram,cvPoint(i,50), cvPoint(i,50-normalized), CV_RGB(0,0,0));
}
cvFlip(image, NULL, 1);
cvShowImage("Image Source", image);
cvShowImage("Histogram", imgHistogram);
//Page 19 paragraph 3 of "Learning OpenCV" tells us why we DO NOT use "cvReleaseImage(&image)" in this section
cvReleaseImage(&imgHistogram);
cvReleaseImage(&gray);
cvReleaseHist(&hist);
char c = cvWaitKey(10);
//if ASCII key 27 (esc) is pressed then loop breaks
if(c==27) break;
}
cvReleaseImage(&image);
cvReleaseCapture(&capture);
cvDestroyAllWindows();
}
Only a few things I can see or recommend:
Considering the build, make sure you're building in Release. Also, make sure the build of OpenCV you're using was built with OpenMP enabled, it makes an enormous difference.
Try moving your allocations outside the loop. Every loop you're re-creating gray and other images, when they should be re-used.
The other thing is your style, which makes it difficult to give good recommendations easily. It's poor style to pre-declare a bunch of variables, this is C-style. Declare your variables just prior to their use, and the code will be easier to read.
Update: I found the issue, it was actually my hardware (well the driver I think). I was using a PS3 Eye because of the amazing frame rates, but for some reason OpenCV does not like the PS3 Eye all the time. Sometimes it works great and other times not so great. I have verfied this on three computers, all of which run my code good with a standard web cam but randomly lock up when the PS3 Eye is used. Still, thank you for your suggestions GMan!