I'm working on a project where I will use homography as features in a classifier. My problem is in automatically calculating homographies, i'm using SIFT descriptors to find the points between the two images on which to calculate homography but SIFT are giving me very poor results, hence i can't use them in my work.
I'm using OpenCV 2.4.3.
At first I was using SURF, but I had similar results and I decided to use SIFT which are slower but more precise. My first guess was that the image resolution in my dataset was too low but i ran my algorithm on a state-of-the-art dataset (Pointing 04) and I obtained pretty much the same results, so the problem lies in what I do and not in my dataset.
The match between the SIFT keypoints found in each image is done with the FlannBased matcher, i tried the BruteForce one but the results were again pretty much the same.
This is an example of the match I found (image from Pointing 04 dataset)
The above image shows how poor is the match found with my program. Only 1 point is a correct match. I need (at least) 4 correct matches for what I have to do.
Here is the code i use:
This is the function that extracts SIFT descriptors from each image
void extract_sift(const Mat &img, vector<KeyPoint> &keypoints, Mat &descriptors, Rect* face_rec) {
// Create masks for ROI on the original image
Mat mask1 = Mat::zeros(img.size(), CV_8U); // type of mask is CV_8U
Mat roi1(mask1, *face_rec);
roi1 = Scalar(255, 255, 255);
// Extracts keypoints in ROIs only
Ptr<DescriptorExtractor> featExtractor;
Ptr<FeatureDetector> featDetector;
Ptr<DescriptorMatcher> featMatcher;
featExtractor = new SIFT();
featDetector = FeatureDetector::create("SIFT");
featDetector->detect(img,keypoints,mask1);
featExtractor->compute(img,keypoints,descriptors);
}
This is the function that matches two images' descriptors
void match_sift(const Mat &img1, const Mat &img2, const vector<KeyPoint> &kp1,
const vector<KeyPoint> &kp2, const Mat &descriptors1, const Mat &descriptors2,
vector<Point2f> &p_im1, vector<Point2f> &p_im2) {
// Matching descriptor vectors using FLANN matcher
Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("FlannBased");
std::vector< DMatch > matches;
matcher->match( descriptors1, descriptors2, matches );
double max_dist = 0; double min_dist = 100;
// Quick calculation of max and min distances between keypoints
for( int i = 0; i < descriptors1.rows; ++i ){
double dist = matches[i].distance;
if( dist < min_dist ) min_dist = dist;
if( dist > max_dist ) max_dist = dist;
}
// Draw only the 4 best matches
std::vector< DMatch > good_matches;
// XXX: DMatch has no sort method, maybe a more efficent min extraction algorithm can be used here?
double min=matches[0].distance;
int min_i = 0;
for( int i = 0; i < (matches.size()>4?4:matches.size()); ++i ) {
for(int j=0;j<matches.size();++j)
if(matches[j].distance < min) {
min = matches[j].distance;
min_i = j;
}
good_matches.push_back( matches[min_i]);
matches.erase(matches.begin() + min_i);
min=matches[0].distance;
min_i = 0;
}
Mat img_matches;
drawMatches( img1, kp1, img2, kp2,
good_matches, img_matches, Scalar::all(-1), Scalar::all(-1),
vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );
imwrite("imgMatch.jpeg",img_matches);
imshow("",img_matches);
waitKey();
for( int i = 0; i < good_matches.size(); i++ )
{
// Get the points from the best matches
p_im1.push_back( kp1[ good_matches[i].queryIdx ].pt );
p_im2.push_back( kp2[ good_matches[i].trainIdx ].pt );
}
}
And these functions are called here:
extract_sift(dataset[i].img,dataset[i].keypoints,dataset[i].descriptors,face_rec);
[...]
// Extract keypoints from i+1 image and calculate homography
extract_sift(dataset[i+1].img,dataset[i+1].keypoints,dataset[i+1].descriptors,face_rec);
dataset[front].points_r.clear(); // XXX: dunno if clearing the points every time is the best way to do it..
match_sift(dataset[front].img,dataset[i+1].img,dataset[front].keypoints,dataset[i+1].keypoints,
dataset[front].descriptors,dataset[i+1].descriptors,dataset[front].points_r,dataset[i+1].points_r);
dataset[i+1].H = findHomography(dataset[front].points_r,dataset[i+1].points_r, RANSAC);
Any help on how to improve the matching performance would be really appreciated, thanks.
You apparently use the "best four points" in your code w.r.t. the distance of the matches. In other words, you consider that a match is valid if both descriptors are really similar. I believe this is wrong. Did you try to draw all of the matches? Many of them should be wrong, but many should be good as well.
The distance of a match just tells how similar both points are. This doesn't tell if the match is coherent geometrically. Selecting the best matches should definitely consider the geometry.
Here is how I would do:
Detect the corners (you already do this)
Find the matches (you already do this)
Try to find a homography transform between both images by using the matches (don't filter them before!) using findHomography(...)
findHomography(...) will tell you which are the inliers. Those are your good_matches.
Related
I am trying to construct a panoromic view from different images.
Initially I tried to stitch two images as part of panoromic construction.
The two input images I am trying to stitch are:
I used ORB feature descriptor to find features in the image,then I found out Homography matrix between these two images.
My code is:
int main(int argc, char **argv){
Mat img1 = imread(argv[1],1);
Mat img2 = imread(argv[2],1);
//-- Step 1: Detect the keypoints using orb Detector
std::vector<KeyPoint> kp2,kp1;
// Default parameters of ORB
int nfeatures=500;
float scaleFactor=1.2f;
int nlevels=8;
int edgeThreshold=15; // Changed default (31);
int firstLevel=0;
int WTA_K=2;
int scoreType=ORB::HARRIS_SCORE;
int patchSize=31;
int fastThreshold=20;
Ptr<ORB> detector = ORB::create(
nfeatures,
scaleFactor,
nlevels,
edgeThreshold,
firstLevel,
WTA_K,
scoreType,
patchSize,
fastThreshold );
Mat descriptors_img1, descriptors_img2;
//-- Step 2: Calculate descriptors (feature vectors)
detector->detect(img1, kp1,descriptors_img1);
detector->detect(img2, kp2,descriptors_img2);
Ptr<DescriptorExtractor> extractor = ORB::create();
extractor->compute(img1, kp1, descriptors_img1 );
extractor->compute(img2, kp2, descriptors_img2 );
//-- Step 3: Matching descriptor vectors using FLANN matcher
if ( descriptors_img1.empty() )
cvError(0,"MatchFinder","1st descriptor empty",__FILE__,__LINE__);
if ( descriptors_img2.empty() )
cvError(0,"MatchFinder","2nd descriptor empty",__FILE__,__LINE__);
descriptors_img1.convertTo(descriptors_img1, CV_32F);
descriptors_img2.convertTo(descriptors_img2, CV_32F);
FlannBasedMatcher matcher;
std::vector<DMatch> matches;
matcher.match(descriptors_img1,descriptors_img2,matches);
double max_dist = 0; double min_dist = 100;
//-- Quick calculation of max and min distances between keypoints
for( int i = 0; i < descriptors_img1.rows; i++ )
{
double dist = matches[i].distance;
if( dist < min_dist )
min_dist = dist;
if( dist > max_dist )
max_dist = dist;
}
printf("-- Max dist : %f \n", max_dist );
printf("-- Min dist : %f \n", min_dist );
//-- Draw only "good" matches (i.e. whose distance is less than 3*min_dist )
std::vector< DMatch > good_matches;
for( int i = 0; i < descriptors_img1.rows; i++ )
{
if( matches[i].distance < 3*min_dist )
{
good_matches.push_back( matches[i]);
}
}
Mat img_matches;
drawMatches(img1,kp1,img2,kp2,good_matches,img_matches,Scalar::all(-1),
Scalar::all(-1),vector<char>(),DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );
std::vector<Point2f> obj;
std::vector<Point2f> scene;
for( int i = 0; i < good_matches.size(); i++ )
{
//-- Get the keypoints from the good matches
obj.push_back( kp1[ good_matches[i].queryIdx ].pt );
scene.push_back( kp2[ good_matches[i].trainIdx ].pt );
}
Mat H = findHomography( obj, scene, CV_RANSAC );
After wards some people told me to include the following code
cv::Mat result;
warpPerspective( img1, result, H, cv::Size( img1.cols+img2.cols, img1.rows) );
cv::Mat half(result, cv::Rect(0, 0, img2.cols, img2.rows) );
img2.copyTo(half);
imshow("result",result);
The result I got is
I also tried using inbuilt opencv stitch function. And I got the result
I am trying to implement stitch function so I dont want to use inbuilt opencv stitch function.
Can any one tell me where I went wrong and correct my code.Thanks in advance
Image stitching includes the following steps:
Feature finding
Find camera parameters
Warping
Exposure compensation
Seam Finding
Blending
You have to do all these steps in order to get the perfect result.
In your code you have only done the first part, that is feature finding.
You can find a detailed explanation on how image stitching works in Learn OpenCV
Also I have the code on Github
Hope this helps.
I'm trying to extract and match features with OpenCV using ORB for detecting and FLANN for matching, and i get a really weird result. After loading my 2 images and converting them to grayscale, here's my code:
// Initiate ORB detector
Ptr<FeatureDetector> detector = ORB::create();
// find the keypoints and descriptors with ORB
detector->detect(gray_image1, keypoints_object);
detector->detect(gray_image2, keypoints_scene);
Ptr<DescriptorExtractor> extractor = ORB::create();
extractor->compute(gray_image1, keypoints_object, descriptors_object );
extractor->compute(gray_image2, keypoints_scene, descriptors_scene );
// Flann needs the descriptors to be of type CV_32F
descriptors_scene.convertTo(descriptors_scene, CV_32F);
descriptors_object.convertTo(descriptors_object, CV_32F);
FlannBasedMatcher matcher;
vector<DMatch> matches;
matcher.match( descriptors_object, descriptors_scene, matches );
double max_dist = 0; double min_dist = 100;
//-- Quick calculation of max and min distances between keypoints
for( int i = 0; i < descriptors_object.rows; i++ )
{
double dist = matches[i].distance;
if( dist < min_dist ) min_dist = dist;
if( dist > max_dist ) max_dist = dist;
}
//-- Use only "good" matches (i.e. whose distance is less than 3*min_dist )
vector< DMatch > good_matches;
for( int i = 0; i < descriptors_object.rows; i++ )
{
if( matches[i].distance < 3*min_dist )
{
good_matches.push_back( matches[i]);
}
}
vector< Point2f > obj;
vector< Point2f > scene;
for( int i = 0; i < good_matches.size(); i++ )
{
//-- Get the keypoints from the good matches
obj.push_back( keypoints_object[ good_matches[i].queryIdx ].pt );
scene.push_back( keypoints_scene[ good_matches[i].trainIdx ].pt );
}
// Find the Homography Matrix
Mat H = findHomography( obj, scene, CV_RANSAC );
// Use the Homography Matrix to warp the images
cv::Mat result;
warpPerspective(image1,result,H,Size(image1.cols+image2.cols,image1.rows));
cv::Mat half(result,cv::Rect(0,0,image2.cols,image2.rows));
image2.copyTo(half);
imshow( "Result", result );
And this is a screen shot of the weird result i'm getting:
screen shot
What might be the problem?
Thanks!
You are experiencing the results of a bad matching: The homography which fits the data is not "realistic" and thus distorts the image.
You can debug your matching with imshow( "Good Matches", img_matches ); as done in the example.
There are multiple approaches to improve your matches:
Use the crossCheck option
Use the SIFT ratio test
Use the OutputArray mask in cv::findHompgraphy to identify totally wrong homography computations
... and so on...
ORB are binary feature vectors which don't work with Flann. Use Brute Force (BFMatcher) instead.
I am using FAST and FREAK to get the descriptors of a couple of images and then I apply knnMatch with a BruteForceMatcher matcher and next I am using a loop to separate the good matches:
float nndrRatio = 0.7f;
std::vector<KeyPoint> keypointsA, keypointsB;
Mat descriptorsA, descriptorsB;
std::vector< vector< DMatch > > matches;
int threshold=9;
// detect keypoints:
FAST(objectMat,keypointsA,threshold,true);
FAST(sceneMat,keypointsB,threshold,true);
FREAK extractor;
// extract descriptors:
extractor.compute( objectMat, keypointsA, descriptorsA );
extractor.compute( sceneMat, keypointsB, descriptorsB );
BruteForceMatcher<Hamming> matcher;
// match
matcher.knnMatch(descriptorsA, descriptorsB, matches, 2);
// good matches search:
vector< DMatch > good_matches;
for (size_t i = 0; i < matches.size(); ++i)
{
if (matches[i].size() < 2)
continue;
const DMatch &m1 = matches[i][0];
const DMatch &m2 = matches[i][1];
if(m1.distance <= nndrRatio * m2.distance)
good_matches.push_back(m1);
}
//If there are at least 7 good matches, then object has been found
if ( (good_matches.size() >=7))
{
cout << "OBJECT FOUND!" << endl;
}
I think the problem could be the good matches search method, because using it with the FlannBasedMatcher works fine but with the BruteForceMatcher very weirdly. I'm suspecting that I may be doing a nonsense with this method because the Hamming distance uses binary descriptors, but I can't think of a way to adapt it!
Any links, snippets, ideas,... please?
Your code is not bad, but I don't think it is what you want to do. Why did you choose this method?
If you want to detect an object in an image using OpenCV, you should maybe try the Cascade Classification. This link will explain how to train a classifier.
EDIT: If you think it is too complicated and if the object you want to detect is planar, you can try this tutorial (it basically computes the inliers by trying to find a homography transform between the object and the image). But the cascade classification is more general for object detection.
I am trying the quite new descriptor FREAK from the latest version of OpenCV following the freak_demo.cpp example. Instead of using SURF I use FAST. My basic code is something like this:
std::vector<KeyPoint> keypointsA, keypointsB;
Mat descriptorsA, descriptorsB;
std::vector<DMatch> matches;
FREAK extractor;
BruteForceMatcher<Hamming> matcher;
FAST(imgA,keypointsA,100);
FAST(imgB,keypointsB,20);
extractor.compute( imgA, keypointsA, descriptorsA );
extractor.compute( imgB, keypointsB, descriptorsB );
matcher.match(descriptorsA, descriptorsB, matches);
The algorithm finds a lot of matches, but there are a lot of outliers. Am I doing things right? Is there a way for tuning the algorithm?
When doing matching there are always some refinement steps for getting rid out of outliers.
What I usually do is discarding matches that have a distance over a threshold, for example:
for (int i = 0; i < matches.size(); i++ )
{
if(matches[i].distance > 200)
{
matches.erase(matches.begin()+i-1);
}
}
Then, I use RANSAC to see which matches fit the homography model. OpenCV has a function for this:
for( int i = 0; i < matches.size(); i++ )
{
trainMatches.push_back( cv::Point2f(keypointsB[ matches[i].trainIdx ].pt.x/500.0f, keypointsB[ matches[i].trainIdx ].pt.y/500.0f) );
queryMatches.push_back( cv::Point2f(keypointsA[ matches[i].queryIdx ].pt.x/500.0f, keypointsA[ matches[i].queryIdx ].pt.y/500.0f) );
}
Mat h = cv::findHomography(trainMatches,queryMatches,CV_RANSAC,0.005, status);
And I just draw the inliers:
for(size_t i = 0; i < queryMatches.size(); i++)
{
if(status.at<char>(i) != 0)
{
inliers.push_back(matches[i]);
}
}
Mat imgMatch;
drawMatches(imgA, keypointsA, imgB, keypointsB, inliers, imgMatch);
Just try different thresholds and distances until you get the desired resutls.
You can also train the descriptor by giving your own selected pairs. And tune the parameters in the constructor.
explicit FREAK( bool orientationNormalized = true
, bool scaleNormalized = true
, float patternScale = 22.0f
, int nbOctave = 4
, const vector<int>& selectedPairs = vector<int>()
);
BTW, a more efficient version of FREAK is on the way :-)
I am trying to use SURF but I am having trouble finding way to do so in C. The documentation only seems to have stuff for C++ in terms of.
I have been able to detect SURF feature:
IplImage *img = cvLoadImage("img5.jpg");
CvMat* image = cvCreateMat(img->height, img->width, CV_8UC1);
cvCvtColor(img, image, CV_BGR2GRAY);
// detecting keypoints
CvSeq *imageKeypoints = 0, *imageDescriptors = 0;
int i;
//Extract SURF points by initializing parameters
CvSURFParams params = cvSURFParams(1, 1);
cvExtractSURF( image, 0, &imageKeypoints, &imageDescriptors, storage, params );
printf("Image Descriptors: %d\n", imageDescriptors->total);
//draw the keypoints on the captured frame
for( i = 0; i < imageKeypoints->total; i++ )
{
CvSURFPoint* r = (CvSURFPoint*)cvGetSeqElem( imageKeypoints, i );
CvPoint center;
int radius;
center.x = cvRound(r->pt.x);
center.y = cvRound(r->pt.y);
radius = cvRound(r->size*1.2/9.*2);
cvCircle( image, center, radius, CV_RGB(0,255,0), 1, 8, 0 );
}
But I can't find the method that I need to compare the descriptors of 2 images. I found this code in C++ but I'm having trouble translating it:
// matching descriptors
BruteForceMatcher<L2<float> > matcher;
vector<DMatch> matches;
matcher.match(descriptors1, descriptors2, matches);
// drawing the results
namedWindow("matches", 1);
Mat img_matches;
drawMatches(img1, keypoints1, img2, keypoints2, matches, img_matches);
imshow("matches", img_matches);
waitKey(0);
I would appreciate if someone could lead me on to a descriptor matcher or even better, let me know where I can find the OpenCV documentation in C only.
This link might give you a hint. https://projects.developer.nokia.com/opencv/browser/opencv/opencv-2.3.1/samples/c/find_obj.cpp . Look in the function naiveNearestNeighbor
Check out the blog post from thioldhack. Contains a sample code. Its for QT, but you can easily do it for VC++ or any other. You will need to match the Key points using K-nearest neighbour algorithm. It has all.
The slightly longer but surest way is to compile OpenCV on your computer with debug information and just to step into the C++ implementation with a debugger. You can also copy it aside to your project and start peeling it like an onion till you get to pure C.