I am working on a school project with OpenCV. A major part of the program will be a comparison of histograms. There will be a database of histograms and new histograms will be created from a live video feed then compared to the histograms in the database. Right now I am just trying to get the histograms created correctly from the video feed. My problem is that the program crashes or slows down dramatically at random intervals. So my question is how do I prevent the program from crashing or slowing down? OpenCV has always been kind of flaky for me, so I'm not sure if it is an issue with my code or if it is just the nature of OpenCV. If it is to do with my code I think the issue might have something to do with the frame rates (a guess/gut feeling). I am useing "cvWaitKey" to "pace" the loading of frames, but the "Learning OpenCV" book has this to say about "cvWaitKey"
c = cvWaitKey(33);
if( c == 27 ) break;
Once we have displayed the frame, we then wait for 33 ms. If the user hits a key, then c
will be set to the ASCII value of that key; if not, then it will be set to –1. If the user hits
the Esc key (ASCII 27), then we will exit the read loop. Otherwise, 33 ms will pass and
we will just execute the loop again.
It is worth noting that, in this simple example, we are not explicitly controlling
the speed of the video in any intelligent way. We are relying solely on the timer in
cvWaitKey() to pace the loading of frames. In a more sophisticated application it would
be wise to read the actual frame rate from the CvCapture structure (from the AVI) and
behave accordingly!
You will see in my code below (modified from here) that I my loop waits 10ms before starting the next execution. Often times the program will run with no issues at all, but sometimes it will crash less than a minute in, or five minutes in, there really is not pattern that I can detect. Any suggestions on how this crash( or slow down) can be prevented would be welcomed. Also I should add that I am using OpenCV 1.1 (can't ever get OpenCV 2.0 to work right), I am using Visual Studio 2008, and I create an .MSI installer package everytime I modify my code, that is, I do not debug in Visual Studio. Dependencies are cv110.dll, cxcore110.dll, and highgui110.dll. My code is below:
// SLC (Histogram).cpp : Defines the entry point for the console application.
#include "stdafx.h"
#include <cxcore.h>
#include <cv.h>
#include <cvaux.h>
#include <highgui.h>
#include <stdio.h>
#include <sstream>
#include <iostream>
using namespace std;
int main(){
CvCapture* capture = cvCaptureFromCAM(0);
if(!cvQueryFrame(capture)){
cout<<"Video capture failed, please check the camera."<<endl;
}
else{
cout<<"Video camera capture successful!"<<endl;
};
CvSize sz = cvGetSize(cvQueryFrame(capture));
IplImage* image = cvCreateImage(sz, 8, 3);
IplImage* imgHistogram = 0;
IplImage* gray = 0;
CvHistogram* hist;
cvNamedWindow("Image Source",1);
cvNamedWindow("Histogram",1);
for(;;){
image = cvQueryFrame(capture);
//Size of the histogram -1D histogram
int bins = 256;
int hsize[] = {bins};
//Max and min value of the histogram
float max_value = 0, min_value = 0;
//Value and normalized value
float value;
int normalized;
//Ranges - grayscale 0 to 256
float xranges[] = {0, 256};
float* ranges[] = {xranges};
//Create an 8 bit single channel image to hold a grayscale version of the original picture
gray = cvCreateImage(cvGetSize(image), 8, 1);
cvCvtColor(image, gray, CV_BGR2GRAY);
//Planes to obtain the histogram, in this case just one
IplImage* planes[] = {gray};
//Get the histogram and some info about it
hist = cvCreateHist(1, hsize, CV_HIST_ARRAY, ranges,1);
cvCalcHist(planes, hist, 0, NULL);
cvGetMinMaxHistValue(hist, &min_value, &max_value);
printf("Minimum Histogram Value: %f, Maximum Histogram Value: %f\n", min_value, max_value);
//Create an 8 bits single channel image to hold the histogram and paint it white
imgHistogram = cvCreateImage(cvSize(bins, 50),8,3);
cvRectangle(imgHistogram, cvPoint(0,0), cvPoint(256,50), CV_RGB(255,255,255),-1);
//Draw the histogram
for(int i=0; i < bins; i++){
value = cvQueryHistValue_1D(hist, i);
normalized = cvRound(value*50/max_value);
cvLine(imgHistogram,cvPoint(i,50), cvPoint(i,50-normalized), CV_RGB(0,0,0));
}
cvFlip(image, NULL, 1);
cvShowImage("Image Source", image);
cvShowImage("Histogram", imgHistogram);
//Page 19 paragraph 3 of "Learning OpenCV" tells us why we DO NOT use "cvReleaseImage(&image)" in this section
cvReleaseImage(&imgHistogram);
cvReleaseImage(&gray);
cvReleaseHist(&hist);
char c = cvWaitKey(10);
//if ASCII key 27 (esc) is pressed then loop breaks
if(c==27) break;
}
cvReleaseImage(&image);
cvReleaseCapture(&capture);
cvDestroyAllWindows();
}
Only a few things I can see or recommend:
Considering the build, make sure you're building in Release. Also, make sure the build of OpenCV you're using was built with OpenMP enabled, it makes an enormous difference.
Try moving your allocations outside the loop. Every loop you're re-creating gray and other images, when they should be re-used.
The other thing is your style, which makes it difficult to give good recommendations easily. It's poor style to pre-declare a bunch of variables, this is C-style. Declare your variables just prior to their use, and the code will be easier to read.
Update: I found the issue, it was actually my hardware (well the driver I think). I was using a PS3 Eye because of the amazing frame rates, but for some reason OpenCV does not like the PS3 Eye all the time. Sometimes it works great and other times not so great. I have verfied this on three computers, all of which run my code good with a standard web cam but randomly lock up when the PS3 Eye is used. Still, thank you for your suggestions GMan!
Related
Thought I'd try my hand at a little (auto)correlation/convolution today in openCV and make my own 2D filter kernel.
Following openCV's 2D Filter Tutorial I discovered that making your own kernels for openCV's Filter2D might not be that hard. However I'm getting unhandled exceptions when I try to use one.
Code with comments relating to the issue here:
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <stdlib.h>
#include <stdio.h>
#include <iostream>
using namespace cv;
using namespace std;
int main(int argc, char** argv) {
//Loading the source image
Mat src;
src = imread( "1.png" );
//Output image of the same size and the same number of channels as src.
Mat dst;
//Mat dst = src.clone(); //didn't help...
//desired depth of the destination image
//negative so dst will be the same as src.depth()
int ddepth = -1;
//the convolution kernel, a single-channel floating point matrix:
Mat kernel = imread( "kernel.png" );
kernel.convertTo(kernel, CV_32F); //<<not working
//normalize(kernel, kernel, 1.0, 0.0, 4, -1, noArray()); //doesn't help
//cout << kernel.size() << endl; // ... gives 11, 11
//however, the example from tutorial that does work:
//kernel = Mat::ones( 11, 11, CV_32F )/ (float)(11*11);
//default value (-1,-1) here means that the anchor is at the kernel center.
Point anchor = Point(-1,-1);
//value added to the filtered pixels before storing them in dst.
double delta = 0;
//alright, let's do this...
filter2D(src, dst, ddepth , kernel, anchor, delta, BORDER_DEFAULT );
imshow("Source", src); //<< unhandled exception here
imshow("Kernel", kernel);
imshow("Destination", dst);
waitKey(1000000);
return 0;
}
As you can see, using the tutorials kernel works fine, but my image will crash the program, I've tried changing the bit-depth, normalizing, checking size and lots of commenting out blocks to see where it fails, but haven't cracked it yet.
The image is, '1.png':
And the kernel I want 'kernel.png':
I'm trying to see if I can get a hotspot in dst at the point where the eye catchlight is (the kernel I've chosen is the catchlight). I know there are other ways to do this, but I'm interested to see how effective convolving the catchlight over itself is. (autocorrelation I think that's called?)
Direct questions:
why the crash?
is the crash indicating a fundamental conceptual mistake?
or (hopefully) is it just some (silly) fault in the code?
Thanks in advance for any help :)
The assertion error should be posted which would help someone to answer you other than questioning why is the crash. Anyways, I have posted below the possible errors and solution for convolution filter2D.
Error 1:
OpenCV Error: Assertion failed (src.channels() == 1 && func != 0) in cv::countNo
nZero, file C:\builds\2_4_PackSlave-win32-vc12-shared\opencv\modules\core\src\st
at.cpp, line 549
Solution : Your input Image and the kernel should be grayscales. You can use the flag 0 in imread. (ex. cv::imread("kernel.png",0) to read the image as grayscale.) If you want to apply different kernels to different channels, split the image into separate color planes using split() and process them individually.
I don't see anything other than the obove error that may crash. Kernel size should in odd numbers and your kernel image is 11X11 which is fine. If it stills crashes kindly provide more information in order to help you out.
I'm new to OpenCV and am working on a video analysis project. Basically, I want to split my webcam into two sides (left and right), and have already figured out how to do this. However, I also want to analyze each side for red and green colors, and print out the amount of pixels that are red/green. I must have gone through every possible blog to figure this out, but alas it still doesn't work. The following code runs, however instead of detecting red as the code might suggest it seems to pick up white (all light sources and white walls). I have spent hours combing through the code but still cannot find the solution. Please help! Also note that this is being run on OSX 10.8, via Xcode. Thanks!
#include <iostream>
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/core/core.hpp"
using namespace cv;
using namespace std;
int main( int argc, char** argv )
{
VideoCapture cap(0); //capture the video from webcam
if ( !cap.isOpened() ) // if not success, exit program
{
cout << "Cannot open the web cam" << endl;
return -1;
}
namedWindow("HSVLeftRed", CV_WINDOW_AUTOSIZE);
namedWindow("HSVLeftGreen", CV_WINDOW_AUTOSIZE);
while (true) {
Mat image;
cap.read(image);
Mat HSV;
Mat threshold;
//Left Cropping
Mat leftimg = image(Rect(0, 0, 640, 720));
//Left Red Detection
cvtColor(leftimg,HSV,CV_BGR2HSV);
inRange(HSV,Scalar(0,0,150),Scalar(0,0,255),threshold);
imshow("HSVLeftRed",threshold);
//Left Green Detection
cvtColor(leftimg,HSV,CV_BGR2HSV);
inRange(HSV,Scalar(still need to find proper min values),Scalar(still need to find proper max values),threshold);
imshow("HSVLeftGreen",threshold);
}
return 0;
}
You're cropping a 640x720 area, which might not fit exactly your contents. Tip: Check your actual capture resolution with capture.get(CAP_PROP_FRAME_WIDTH) and capture.get(CAP_PROP_FRAME_HEIGHT). You might want to consider Mat threshold --> Mat thresholded. This is just some ranting :)
What I suspect is the actual issue is the threshold you use for HSV. According to cvtolor, section on RGB to HSV conversion,
On output 0 <= V <= 1.
so you should use a float representing your V threshold, i.e. 150 -> 150/255 ~= 0.58 etc.
I just copied an program to detect faces through webcam but the video capture is really slow, and i dont know how to fix it!
Here is the code:
#include<stdio.h>
#include<math.h>
#include<opencv\cv.h>
#include<opencv\highgui.h>
#include<opencv2\objdetect\objdetect.hpp>
#include<opencv2\highgui\highgui.hpp>
#include<opencv2\imgproc\imgproc.hpp>
#include<vector>
using namespace cv;
using namespace std;
int main()
{
CascadeClassifier face_cascade;
if(!face_cascade.load("c:\\haar\\haarcascade_frontalface_alt2.xml")) {
printf("Error loading cascade file for the face");
return 1;
}
VideoCapture capture(0);
if(!capture.isOpened())
{
printf("Error trying to start the Camera");
return 1;
}
Mat cap_img,gray_img;
vector<Rect> faces;
while(1)
{
capture >> cap_img;
waitKey(10);
cvtColor(cap_img, gray_img, CV_BGR2GRAY);
cv::equalizeHist(gray_img,gray_img);
face_cascade.detectMultiScale(gray_img, faces, 1.1, 10, CV_HAAR_SCALE_IMAGE | CV_HAAR_DO_CANNY_PRUNING, cvSize(0,0), cvSize(300,300));
for(int i=0; i < faces.size();i++)
{
Point pt1(faces[i].x+faces[i].width, faces[i].y+faces[i].height);
Point pt2(faces[i].x,faces[i].y);
rectangle(cap_img, pt1, pt2, cvScalar(191,191,191), 2, 8, 0);
}
imshow("Result", cap_img);
waitKey(3);
char c = waitKey(3);
if(c == 27)
break;
}
return 0;
}
I am using Visual studio 2012 and this is the main.cpp file. Im using OpenCV 2.4.9!
OpenCV comes with prebuild libraries. When you use them in an application you actually want to deploy make sure you use the release libraries. In debug mode you have lots of additional checks in the form of asserts as well as debug symbols allowing you to step into the libraries with a debugger that are removed in release mode.
Another more specific advise regarding the code you posted: Avoid calls to cv::waitKey() as every single call, who would have guessed, makes your main thread wait the specified amount of time in milliseconds. Do not drop calls to it entirely as e.g. cv::imshow() will only work appropriately with these in place.
Edit:
Reduce your while loop to:
while ( true ) {
capture >> cap_img;
imshow("Result", cap_img);
if(waitKey(1) == 27)
break;
}
When you know how much time you need for capturing the image and just displaying it, then you can compare how much of an impact whatever algorithm you are running has on your performance.
Here is solution...everytime you read!!!..everytime you use vcap>>cap or cap.read(vcap)... you must set FPS ..25 is max..if i try to set 26 or higher it runs slow.
...this runs 30 fps for me
If you not set FPS...it runs slow too
if (cap.read(vcap)){
imshow("Cam Feed", vcap);
cap.set(CV_CAP_PROP_FPS, 25);
}
On top of the other answers, Multiprocessing is what fixed my issue. I split my program into 3 processes. One for grabbing the image, one for processing the image, and one for displaying the image. Referencing this page helped me a lot. I also followed the tips of the other answers here.
I am doing a project on face detection from surveillance cameras.Now I am at the stage of face detection and I can detect faces from each frame.After detecting the face I need store that face to local folder.Now I can save each face in the specified folder.
Problem Now it is saving every faces,but I need to save identical faces only once.That means if saved one face as a jpeg image and when face detection progress again the same face is coming, so this time I don't want to save that particular face.
This is my code:
#include <cv.h>
#include <highgui.h>
#include <time.h>
#include <stdio.h>
using namespace std;
int ct=1;
int ct1=0;
IplImage *frame;
int frames;
void facedetect(IplImage* image);
void saveImage(IplImage *img,char *ex);
IplImage* resizeImage(const IplImage *origImg, int newWidth,int newHeight, bool keepAspectRatio);
const char* cascade_name="haarcascade_frontalface_default.xml";
int k=1;
int main(int argc, char** argv)
{
CvCapture *capture=cvCaptureFromFile("Arnab Goswami on Pepper spary rajagopal Complete NewsHour Debate (Mobile).3gp");
int count=1;
while(1)
{
frame = cvQueryFrame(capture);
if(count%30==0)
{
facedetect(frame);
}
count++;
}
cvReleaseCapture(&capture);
return 0;
}
void facedetect(IplImage* image)
{
ct1++;
cvNamedWindow("output");
int j=0,i,count=0,l=0,strsize;
char numstr[50];
int arr[100],arr1[100];
CvPoint ul,lr,w,h,ul1,lr1;
CvRect *r;
//int i=0;
IplImage* image1;IplImage* tmpsize;IplImage* reimg;
CvHaarClassifierCascade* cascade=(CvHaarClassifierCascade*) cvLoad(cascade_name);
CvMemStorage* storage=cvCreateMemStorage(0);
const char *extract;
if(!cascade)
{
cout<<"Coulid not load classifier cascade"<<endl;
}
if(cascade)
{
CvSeq*faces=cvHaarDetectObjects(image,cascade,storage,1.1,1,CV_HAAR_DO_CANNY_PRUNING,cvSize(10,10));
//function used for detecting faces.o/p is list of detected faces.
for(int i=0;i<(faces ? faces->total : 0);i++)
{
string s1="im",re,rename,ex=".jpeg";
sprintf(numstr, "%d", k);
re = s1 + numstr;
rename=re+ex;
char *extract1=new char[rename.size()+1];
extract1[rename.size()]=0;
memcpy(extract1,rename.c_str(),rename.size());
//Copies the values of rename.size from the location pointed by source //(rename.c_str)directly to the memory block pointed by destination(extract).
strsize=rename.size();
r=(CvRect*) cvGetSeqElem(faces,i);//draw rectangle outline around each image.
ul.x=r->x;
ul.y=r->y;
w.x=r->width;
h.y=r->height;
lr.x=(r->x + r->width);
lr.y=(r->y + r->height);
cvSetImageROI(image,cvRect(ul.x,ul.y,w.x,h.y));
image1=cvCreateImage(cvGetSize(image),image->depth,image->nChannels);
cvCopy(image, image1, NULL);
reimg=resizeImage(image1, 40, 40, true);
saveImage(reimg,extract1);
cvResetImageROI(image);
cvRectangle(image,ul,lr,CV_RGB(1,255,0),3,8,0);
j++,count++;
k++;
cout<<"frame"<<ct1<<" "<<"face"<<ct<<":"<<"x: "<<ul.x<<endl;
cout<<"frame"<<ct1<<" "<<"face"<<ct<<":"<<"y: "<<ul.y<<endl;
cout<<""<<endl;
ct++;
//cvShowImage("output",image);
}
//return image;
//cvNamedWindow("output");//creating a window.
cvShowImage("output",image);//showing resized image.
cvWaitKey(0);
}
}
void saveImage(IplImage *img,char *ex)
{
int i=0;
char path[255]="/home/athira/Image/OutputImage";
char *ext[200];
char buff[1000];
ext[i]=ex;
sprintf(buff,"%s/%s",path,ext[i]);//copy ext[i] to buff
strcat(path,buff);//concat path & buff
cvSaveImage(buff,img);
i++;
}
You are using the haar feature-based cascade classifier for object detection. As far as i know these xml files are only trained to detect the specific objects based on hundreds of evaluated pictures (see cascade classifier training).
So to compare saved images you will need another "detection" mode, because you have to compare if two faces are identical with respect to the view angle and so on (keyword: biometric data).
The keyword you're looking for is "face recognition" i think. Just build up a database based on your detected faces and use them for face recognition after that.
Edit:
Another maybe helpful link: www.shervinemami.info/faceRecognition.html
If I understood correctly, what you want is to detect faces in one frame, save a thumbnail of this face. Then, in the following frame, you want to detect faces again but only save the thumbnails for those that were not present in the first frame.
This problem is hard, because the faces captured in a video always change from one frame to the next. This is due to noise in the images, to the fact that the persons may be moving, etc. As a consequence, no two faces are ever identical in a surveillance video.
Hence, in order to achieve what you asked, you need to determine if the face you are considering has already been observed in previous frames. In its general form, this problem is not obvious one and is still the topic of a lot of research related to biometrics, pedestrian tracking and re-identification, etc. Therefore, you will have a hard time to achieve 100% effectiveness in detecting that a given face has already been observed.
However, if can accept a method that is not 100% effective, you could try the following approach:
Detect faces F0i in frame 0, with associated image position (x0i, y0i), and store the thumbnails
Compute sparse optical-flow (e.g. using KLT, see this link) on the positions (xn-1i, yn-1i) of the faces in previous frame n-1, in order to estimate their positions (xxni, yyni) in the current frame n.
Detect faces F0i in the current frame n, with associated image position (xni, yni), and save only the thumbnail of those which are not close to one of the predicted positions (xxni, yyni).
Increment n and repeat steps 2-3 using the next frame.
This is a simple algorithm using tracking to determine if a given face was already observed previously. It should be easier to implement than biometrics-based approaches, and also probably more appropriate in the context of video surveillance. However, it is not a 100% accurate, due to the limited effectivity of the optical-flow estimation and of the face detector.
I wanted to try my hand at text recognition, so i've used opencv to trace out the edges and c++ to find slopes, curves etc, the edge algorithm works well on big and uncluttered sets of characters but when it comes against small printed text or text with a lot of background noise like embedded in captcha it struggles and looks incomplete, my guess was i hadn't set the threshold values correctly and tried different values with no success.
Here is my code :
#include "cv.h"
#include "highgui.h"
using namespace cv;
const int low_threshold = 50;
const int high_threshold = 150;
int main()
{
IplImage* newImg;
IplImage* grayImg;
IplImage* cannyImg;
newImg = cvLoadImage("ocv.bmp",1);
grayImg = cvCreateImage( cvSize(newImg->width, newImg->height), IPL_DEPTH_8U, 1 );
cvCvtColor( newImg, grayImg, CV_BGR2GRAY );
cannyImg = cvCreateImage(cvGetSize(newImg), IPL_DEPTH_8U, 1);
cvCanny(grayImg, cannyImg, low_threshold, high_threshold, 3);
cvNamedWindow ("Source", 1);
cvNamedWindow ("Destination",1);
cvShowImage ("Source", newImg );
cvShowImage ("Destination", cannyImg );
cvWaitKey(0);
cvDestroyWindow ("Source" );
cvDestroyWindow ("Destination" );
cvReleaseImage (&newImg );
cvReleaseImage (&grayImg );
cvReleaseImage (&cannyImg );
return 0;
}
I've looked across the net and have seen some complicated thresholding conditions like in this code from this site :
% Set direction to either 0, 45, -45 or 90 depending on angle.
[x,y]=size(f1);
for i=1:x-1,
for j=1:y-1,
if ((gradAngle(i,j)>67.5 && gradAngle(i,j)<=90) || (gradAngle(i,j)>=-90 && gradAngle(i,j)<=-67.5))
gradDirection(i,j)=0;
elseif ((gradAngle(i,j)>22.5 && gradAngle(i,j)<=67.5))
gradDirection(i,j)=45;
elseif ((gradAngle(i,j)>-22.5 && gradAngle(i,j)<=22.5))
gradDirection(i,j)=90;
elseif ((gradAngle(i,j)>-67.5 && gradAngle(i,j)<=-22.5))
gradDirection(i,j)=-45;
end
end
end
If this is the solution can somebody provide me the c++ equivalent of this algorithm, if it's not what else can i do ?
Canny edge detector is a multi-step detector using hysteresis thresholding (it uses two threshold instead of one), and edge tracking (your last snippet is the part of this step). I suggest reading the wikipedia entry first. One possible solution could be to choose the high threshold, so e.g. 70% of the image pixels would be classified as edge (initially - you could do this quickly using histograms), than choose the low threshold as e.g. 40% of the high threshold. It might be a good idea to try to perform edge detection on image block rather than the whole image, so your algorithm could calculate different thresholds for different areas.
Note that CAPTCHA-s are designed to be hard to segment, and adding noise that broke edge detection is one technique to achive this (you might need to smooth the image first).