I want to check a scene image against two train images.
For that, I detect features and compute descriptors of both training images.
Before detecting, computing and matching the scene image, I will delete all matches of train1 and train2. Because these matches won't facilitate the matching of the scene image with train1 and train2.
So, I match train1 with train2 and get a vector of the matches with the trainIdx and queryIdx. But how can I delete these matches in the keypoints-vector and the descriptor matrix of train1 and train2?
Best regards,
dwi
I would have done like below:
std::vector<cv::KeyPoint> keypoints[2];
cv::Mat descriptor[2];
std::vector< cv::DMatch > matches;
/*
Write code to generate the keypoints, descriptors and matches here...
keypoint[0] -> Train Image 1 keypoints
keypoint[1] -> Train Image 2 keypoints
descriptor[0] -> Train Image 1 descriptors
descriptor[1] -> Train Image 2 descriptors
matches -> matched between train image 1 and 2
*/
// Logic to keep unmatched keypoints and corresponding descriptors
for (int idx = 0; idx < 2; idx++) {
std::vector<bool> isMatched(keypoints[idx].size(), false);
// Mark all matched keypoint as true
for (int i = 0; i < matches.size(); i++) {
if (idx == 0) {
isMatched[matches[i].queryIdx] = true;
}
else {
isMatched[matches[i].trainIdx] = true;
}
}
std::vector<cv::KeyPoint>::const_iterator itr = keypoints[idx].begin();
// New descriptor length will be old descriptor length minus matched keypoints size
int descriptor_length = keypoints[idx].size() - matches.size();
// Create temporary descriptor of new descriptor length
cv::Mat tempDescriptor(descriptor_length, descriptor[idx].cols, descriptor[idx].depth());
int count = 0;
for (int i = 0; i < isMatched.size(); i++) {
// Remove matched keypoints
if (isMatched[i] == true) {
itr = keypoints[idx].erase(itr);
}
else {
descriptor[idx].row(i).copyTo(tempDescriptor.row(count));
itr++;
count++;
}
}
descriptor[idx].release();
descriptor[idx] = tempDescriptor.clone();
}
I hope this will help.
OK, like Micka suggested, I iterate over all keypoints and descriptors and add all of them, except the matches, in a new vector/matrix.
There is no possibility to mark them unnecessary.
Related
I had a code which is getting ORB keypoints and descriptors of images and then it trains a bag of word using "BOWKMeansTrainer". the purpose of this code is to cluster photos using k-means. I don't have any problem using ORB.
Here is my problem:
I saw that openCV provides a VGG, You can find it HERE. I want to use VGG descriptor instead of ORB. Based on the documents, VGG doesn't have any function to make keypoints, so I used ORB keypoint and then used ORB keypoints to create VGG descriptors.
Ptr<ORB> detector = ORB::create();
Ptr<cv::xfeatures2d::VGG> detectorVGG = cv::xfeatures2d::VGG::create(VGG::VGG_120, 1.4f, true, true, 0.75f, true);
detector->detect(grayFrame, kp);
detectorVGG->compute(grayFrame, kp, descriptors);
This one also works and I can easily train bag of word and cluster them similar to the way I did using ORB. However when I want to test the bag of words, I'm getting this error:
error: (-215) _queryDescriptors.type() == trainDescType in function knnMatchImpl Aborttrap: 6
I tried to change the type of query descriptor Mat and descriptor Mat but I cannot solve it. In the best case , the error is changed to this:
(-215) (type == CV_8U && dtype == CV_32S) || dtype == CV_32F in function batchDistance
I am not sure if the problem is a small mistake in types or it is because of using ORB keypoints and VGG descriptors at the same time.
Part of my code is here:
vector<Mat> makeDescriptor(vector<Mat> frames)
{
int frameSize = frames.size();
vector<Mat> characteristics;
for (int i = 0 ; i < frameSize ; i++)
{
vector<KeyPoint> kp;
Mat descriptors;
Ptr<cv::xfeatures2d::VGG> detectorVGG = cv::xfeatures2d::VGG::create();
Ptr<ORB> detectorORB = ORB::create();
Mat frame = frames[i];
if (frame.empty()) {
cout << "Frame empty"<< "\n";
continue;
}
Mat grayFrame;
try
{
cvtColor(frame,grayFrame,CV_RGB2GRAY);
}
catch(Exception& e)
{
continue;
}
detectorORB->detect(grayFrame,kp);
detectorVGG->compute(grayFrame,kp,descriptors);
characteristics.push_back(descriptors);
}
return characteristics;
}
Mat makeDict(vector<Mat> characteristics, int _nCluster)
{
BOWKMeansTrainer bow(_nCluster);
for (int j = 0 ; j < characteristics.size() ; j++)
{
Mat descr = characteristics[j];
if (!descr.empty())
{
bow.add(descr);
}
}
Mat voc = bow.cluster();
return voc;
}
BOWImgDescriptorExtractor makeBow(Mat dict)
{
Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("BruteForce-Hamming");
Ptr<DescriptorExtractor> extractor = VGG::create();
BOWImgDescriptorExtractor bowDE(extractor,matcher);
Mat voc;
dict.convertTo(voc,CV_32F);
bowDE.setVocabulary(voc);
return bowDE;
}
void testBow(BOWImgDescriptorExtractor bowDE, vector<Mat> frames)
{
Mat features;
vector< vector<KeyPoint> > keypoints = makeORBKeyPoints(frames);
for (int i = 0 ; i < frames.size() ; i++)
{
Mat bowDesc;
if (!keypoints[i].empty())
{
cout << "inside if" << '\n';
Mat frame;
frames[i].convertTo(frame, CV_8U);
bowDE.compute(frame, keypoints[i], bowDesc); //This Line
bowDesc.convertTo(bowDesc,CV_32F);
}
features.push_back(bowDesc);
}
}
The runtime error is happening at this line:
bowDE.compute(frame, keypoints[i], bowDesc); //This Line
I appreciate if someone helps me out to find a solution for it.
Thanks,
Problem;
Since I didn't find an implementation of LBP in the OpenCV lib, I added the faces module to my OpenCV build. Which does have a LBP implementation.
I'm trying to do texture analysis using local binary patterns (lbp) to determine what is road and what are lines and what isn't road. So I divided my problem up in 3 'labels'. (Road / Lines / The rest )
After that I took 3 random sample images and made subimages with these labels. In other words I selected the road, lines, all the rest and used this as input for the training.
Training:
for (int i = 0; i < names.size(); i++) {
Mat trainingData = imread(names[i]), output;
if (!trainingData.empty()) {
cvtColor(trainingData, output, CV_RGB2GRAY);
int pos = names[i].find_last_of('_');
cv::String l = names[i].substr(pos + 1, 1);
const char *sub = l.c_str();
int label = atoi(sub);
grayscaleImages.push_back(output);
labels.push_back(label);
}
}
Ptr<LBPHFaceRecognizer> lbp = createLBPHFaceRecognizer(2, 4, 10, 10);
lbp->train(grayscaleImages, labels);
Using LBP to predict:
for (int i = 0; i < dataset.images.size(); i++) {
Mat image = (dataset.images[i]).clone();
Mat original = image.clone();
for (int i = 0 ; i < image.size().width; i += squareSize) {
for (int j = image.size().height / 2 ; j < image.size().height; j += squareSize) {
Mat subImage = image(Range(j, j + squareSize), Range(i, i + squareSize));
subImage = applySobel(subImage);
int predict = lbp->predict(subImage);
}
}
imshow("Image", original);
waitKey(0);
}
My question is why this isn't working? I'm kinda puzzeled about the prediction aswell. It doesn't seem to be anywhere near close to what I want it to do?
Do I have the implement LBP myself? Does the LBP implementation of the faces module do something special?
Do you have any suggestion on other methods for texture analysis specifically for the case of road / no road?
So here is a general overview of the files I'm using.
Input:
Selecting our training data:
Results of selecting: (every file has a _number to specify the label)
Final result after training on a random image of the same set. Purple represents road, green is line (or road marking).
I would like to know more on how to label pixels in images. I want to use the algorithms in Caffe in order to come up with an edge detection program. So far I have filtered the image to get the edges and successfully wrote code to detect whether a pixel is an edge or not(relying on its neighboring pixels). All I want to do now is label these as edges so I can feed them as training data in my algorithm(s). I am a beginner in C++ and Caffe. What I really need is someone to explain how labeling exactly works, would I say go to every pixel It considers to be an edge and give it the value 1 for edge and 0 for non-edge ? IF so how do I do this ? References and step by step guides are well appreciated since I have been struggling for some time on this task.
Ok for example I reached this code :
Mat pixelComparisonAlg(Mat img){
Mat img_gray;
img_gray = navigateImage(img.clone());
for(int x=0; x < img_gray.cols-1 ; x++){
for(int y=0; y < img_gray.rows-1; y++ ){
Vec3b value1 = img_gray.at<Vec3b>(Point(x+1,y+1));
Vec3b value2 = img_gray.at<Vec3b>(Point(x+1,y));
Vec3b value4 = img_gray.at<Vec3b>(Point(x,y+1));
Vec3b value = img_gray.at<Vec3b>(Point(x,y));
if( value1 != value || value != value2 || value!= value4){
//EDGE!
value.val[0] = 0;
value.val[1] = 0;
value.val[2] = 255;
value1.val[0] = 0;
value1.val[1] = 0;
value1.val[2] = 255;
img_gray.at<Vec3b>(Point(x,y)) = value;
}
else{
value;
}
}
}
return img_gray;
}
int main( int argc, char** argv )
{
Mat img = cv::imread("...", CV_LOAD_IMAGE_COLOR);
if(! img.data){
cout << "could not open or find the image" << endl;
return -1;
}
imshow("edges", pixelComparisonAlg(img.clone()));
cv::waitKey(0);
return 0;
}
is there a possibility to label these pixels as edges ? Lets say I take the code that changes the colors of the edges to red(after the if statement) modify it to label the pixel as edge or not. Thus I have a vector carrying information about the pixel : color (3 vector elements) and label ( 1 vector element)
Thank you
I'm trying to rewrite my very slow naive segmentation using floodFill to something faster. I ruled out meanShiftFiltering a year ago because of the difficulty in labelling the colours and then finding their contours.
The current version of opencv seems to have a fast new function that labels segments using mean shift: gpu::meanShiftSegmentation(). It produces images like the following:
(source: ekran.org)
So this looks to me pretty close to being able to generating contours. How can I run findContours to generate segments?
Seems to me, this would be done by extracting the labelled colours from the image, and then testing which pixel values in the image match each label colour to make a boolean image suitable for findContours. This is what I have done in the following (but its a bit slow and strikes me there should be a better way):
Mat image = imread("test.png");
...
// gpu operations on image resulting in gpuOpen
...
// Mean shift
TermCriteria iterations = TermCriteria(CV_TERMCRIT_ITER, 2, 0);
gpu::meanShiftSegmentation(gpuOpen, segments, 10, 20, 300, iterations);
// convert to greyscale (HSV image)
vector<Mat> channels;
split(segments, channels);
// get labels from histogram of image.
int size = 256;
labels = Mat(256, 1, CV_32SC1);
calcHist(&channels.at(2), 1, 0, Mat(), labels, 1, &size, 0);
// Loop through hist bins
for (int i=0; i<256; i++) {
float count = labels.at<float>(i);
// Does this bin represent a label in the image?
if (count > 0) {
// find areas of the image that match this label and findContours on the result.
Mat label = Mat(channels.at(2).rows, channels.at(2).cols, CV_8UC1, Scalar::all(i)); // image filled with label colour.
Mat boolImage = (channels.at(2) == label); // which pixels in labeled image are identical to this label?
vector<vector<Point>> labelContours;
findContours(boolImage, labelContours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
// Loop through contours.
for (int idx = 0; idx < labelContours.size(); idx++) {
// get bounds for this contour.
bounds = boundingRect(labelContours[idx]);
// create ROI for bounds to extract this region
Mat patchROI = image(bounds);
Mat maskROI = boolImage(bounds);
}
}
}
Is this the best approach or is there a better way to get the label colours? Seems it would be logical for meanShiftSegmentation to provide this information? (vector of colour values, or vector of masks for each label, etc.)
Thank you.
Following is another way of doing this without thowing away the colour information in the meanShiftSegmentation results. I did not compare the two for performance.
// Loop through whole image, pixel and pixel and then use the colour to index an array of bools indicating presence.
vector<Scalar> colours;
vector<Scalar>::iterator colourIter;
vector< vector< vector<bool> > > colourSpace;
vector< vector< vector<bool> > >::iterator colourSpaceBIter;
vector< vector<bool> >::iterator colourSpaceGIter;
vector<bool>::iterator colourSpaceRIter;
// Initialize 3D Vector
colourSpace.resize(256);
for (int i = 0; i < 256; i++) {
colourSpace[i].resize(256);
for (int j = 0; j < 256; j++) {
colourSpace[i][j].resize(256);
}
}
// Loop through pixels in the image (should be fastish, look into LUT for faster)
uchar r, g, b;
for (int i = 0; i < segments.rows; i++)
{
Vec3b* pixel = segments.ptr<Vec3b>(i); // point to first pixel in row
for (int j = 0; j < segments.cols; j++)
{
b = pixel[j][0];
g = pixel[j][1];
r = pixel[j][2];
colourSpace[b][g][r] = true; // this colour is in the image.
//cout << "BGR: " << int(b) << " " << int(g) << " " << int(r) << endl;
}
}
// Get all the unique colours from colourSpace
// loop through colourSpace
int bi=0;
for (colourSpaceBIter = colourSpace.begin(); colourSpaceBIter != colourSpace.end(); colourSpaceBIter++) {
int gi=0;
for (colourSpaceGIter = colourSpaceBIter->begin(); colourSpaceGIter != colourSpaceBIter->end(); colourSpaceGIter++) {
int ri=0;
for (colourSpaceRIter = colourSpaceGIter->begin(); colourSpaceRIter != colourSpaceGIter->end(); colourSpaceRIter++) {
if (*colourSpaceRIter)
colours.push_back( Scalar(bi,gi,ri) );
ri++;
}
gi++;
}
bi++;
}
// For each colour
int segmentCount = 0;
for (colourIter = colours.begin(); colourIter != colours.end(); colourIter++) {
Mat label = Mat(segments.rows, segments.cols, CV_8UC3, *colourIter); // image filled with label colour.
Mat boolImage = Mat(segments.rows, segments.cols, CV_8UC3);
inRange(segments, *colourIter, *colourIter, boolImage); // which pixels in labeled image are identical to this label?
vector<vector<Point> > labelContours;
findContours(boolImage, labelContours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
// Loop through contours.
for (int idx = 0; idx < labelContours.size(); idx++) {
// get bounds for this contour.
Rect bounds = boundingRect(labelContours[idx]);
float area = contourArea(labelContours[idx]);
// Draw this contour on a new blank image
Mat maskImage = Mat::zeros(boolImage.rows, boolImage.cols, boolImage.type());
drawContours(maskImage, labelContours, idx, Scalar(255,255,255), CV_FILLED);
Mat patchROI = frame(bounds);
Mat maskROI = maskImage(bounds);
}
segmentCount++;
}
I am working on a project using the Orb feature detector in OpenCV 2.3.1 . I am finding matches between 8 different images, 6 of which are very similar (20 cm difference in camera position, along a linear slider so there is no scale or rotational variance), and then 2 images taken from about a 45 degree angle from either side. My code is finding plenty of accurate matches between the very similar images, but few to none for the images taken from a more different perspective. I've included what I think are the pertinent parts of my code, please let me know if you need more information.
// set parameters
int numKeyPoints = 1500;
float distThreshold = 15.0;
//instantiate detector, extractor, matcher
detector = new cv::OrbFeatureDetector(numKeyPoints);
extractor = new cv::OrbDescriptorExtractor;
matcher = new cv::BruteForceMatcher<cv::HammingLUT>;
//Load input image detect keypoints
cv::Mat img1;
std::vector<cv::KeyPoint> img1_keypoints;
cv::Mat img1_descriptors;
cv::Mat img2;
std::vector<cv::KeyPoint> img2_keypoints
cv::Mat img2_descriptors;
img1 = cv::imread(fList[0].string(), CV_LOAD_IMAGE_GRAYSCALE);
img2 = cv::imread(fList[1].string(), CV_LOAD_IMAGE_GRAYSCALE);
detector->detect(img1, img1_keypoints);
detector->detect(img2, img2_keypoints);
extractor->compute(img1, img1_keypoints, img1_descriptors);
extractor->compute(img2, img2_keypoints, img2_descriptors);
//Match keypoints using knnMatch to find the single best match for each keypoint
//Then cull results that fall below given distance threshold
std::vector<std::vector<cv::DMatch> > matches;
matcher->knnMatch(img1_descriptors, img2_descriptors, matches, 1);
int matchCount=0;
for (int n=0; n<matches.size(); ++n) {
if (matches[n].size() > 0){
if (matches[n][0].distance > distThreshold){
matches[n].erase(matches[n].begin());
}else{
++matchCount;
}
}
}
I ended up getting enough useful matches by changing my process for filtering matches. My previous method was discarding a lot of good matches based solely on their distance value. This RobustMatcher class that I found in the OpenCV2 Computer Vision Application Programming Cookbook ended up working great. Now that all of my matches are accurate, I've been able to get good enough results by bumping up the number of keypoints that the ORB detector is looking. Using the RobustMatcher with SIFT or SURF still gives much better results, but I'm getting usable data with ORB now.
//RobustMatcher class taken from OpenCV2 Computer Vision Application Programming Cookbook Ch 9
class RobustMatcher {
private:
// pointer to the feature point detector object
cv::Ptr<cv::FeatureDetector> detector;
// pointer to the feature descriptor extractor object
cv::Ptr<cv::DescriptorExtractor> extractor;
// pointer to the matcher object
cv::Ptr<cv::DescriptorMatcher > matcher;
float ratio; // max ratio between 1st and 2nd NN
bool refineF; // if true will refine the F matrix
double distance; // min distance to epipolar
double confidence; // confidence level (probability)
public:
RobustMatcher() : ratio(0.65f), refineF(true),
confidence(0.99), distance(3.0) {
// ORB is the default feature
detector= new cv::OrbFeatureDetector();
extractor= new cv::OrbDescriptorExtractor();
matcher= new cv::BruteForceMatcher<cv::HammingLUT>;
}
// Set the feature detector
void setFeatureDetector(
cv::Ptr<cv::FeatureDetector>& detect) {
detector= detect;
}
// Set the descriptor extractor
void setDescriptorExtractor(
cv::Ptr<cv::DescriptorExtractor>& desc) {
extractor= desc;
}
// Set the matcher
void setDescriptorMatcher(
cv::Ptr<cv::DescriptorMatcher>& match) {
matcher= match;
}
// Set confidence level
void setConfidenceLevel(
double conf) {
confidence= conf;
}
//Set MinDistanceToEpipolar
void setMinDistanceToEpipolar(
double dist) {
distance= dist;
}
//Set ratio
void setRatio(
float rat) {
ratio= rat;
}
// Clear matches for which NN ratio is > than threshold
// return the number of removed points
// (corresponding entries being cleared,
// i.e. size will be 0)
int ratioTest(std::vector<std::vector<cv::DMatch> >
&matches) {
int removed=0;
// for all matches
for (std::vector<std::vector<cv::DMatch> >::iterator
matchIterator= matches.begin();
matchIterator!= matches.end(); ++matchIterator) {
// if 2 NN has been identified
if (matchIterator->size() > 1) {
// check distance ratio
if ((*matchIterator)[0].distance/
(*matchIterator)[1].distance > ratio) {
matchIterator->clear(); // remove match
removed++;
}
} else { // does not have 2 neighbours
matchIterator->clear(); // remove match
removed++;
}
}
return removed;
}
// Insert symmetrical matches in symMatches vector
void symmetryTest(
const std::vector<std::vector<cv::DMatch> >& matches1,
const std::vector<std::vector<cv::DMatch> >& matches2,
std::vector<cv::DMatch>& symMatches) {
// for all matches image 1 -> image 2
for (std::vector<std::vector<cv::DMatch> >::
const_iterator matchIterator1= matches1.begin();
matchIterator1!= matches1.end(); ++matchIterator1) {
// ignore deleted matches
if (matchIterator1->size() < 2)
continue;
// for all matches image 2 -> image 1
for (std::vector<std::vector<cv::DMatch> >::
const_iterator matchIterator2= matches2.begin();
matchIterator2!= matches2.end();
++matchIterator2) {
// ignore deleted matches
if (matchIterator2->size() < 2)
continue;
// Match symmetry test
if ((*matchIterator1)[0].queryIdx ==
(*matchIterator2)[0].trainIdx &&
(*matchIterator2)[0].queryIdx ==
(*matchIterator1)[0].trainIdx) {
// add symmetrical match
symMatches.push_back(
cv::DMatch((*matchIterator1)[0].queryIdx,
(*matchIterator1)[0].trainIdx,
(*matchIterator1)[0].distance));
break; // next match in image 1 -> image 2
}
}
}
}
// Identify good matches using RANSAC
// Return fundemental matrix
cv::Mat ransacTest(
const std::vector<cv::DMatch>& matches,
const std::vector<cv::KeyPoint>& keypoints1,
const std::vector<cv::KeyPoint>& keypoints2,
std::vector<cv::DMatch>& outMatches) {
// Convert keypoints into Point2f
std::vector<cv::Point2f> points1, points2;
cv::Mat fundemental;
for (std::vector<cv::DMatch>::
const_iterator it= matches.begin();
it!= matches.end(); ++it) {
// Get the position of left keypoints
float x= keypoints1[it->queryIdx].pt.x;
float y= keypoints1[it->queryIdx].pt.y;
points1.push_back(cv::Point2f(x,y));
// Get the position of right keypoints
x= keypoints2[it->trainIdx].pt.x;
y= keypoints2[it->trainIdx].pt.y;
points2.push_back(cv::Point2f(x,y));
}
// Compute F matrix using RANSAC
std::vector<uchar> inliers(points1.size(),0);
if (points1.size()>0&&points2.size()>0){
cv::Mat fundemental= cv::findFundamentalMat(
cv::Mat(points1),cv::Mat(points2), // matching points
inliers, // match status (inlier or outlier)
CV_FM_RANSAC, // RANSAC method
distance, // distance to epipolar line
confidence); // confidence probability
// extract the surviving (inliers) matches
std::vector<uchar>::const_iterator
itIn= inliers.begin();
std::vector<cv::DMatch>::const_iterator
itM= matches.begin();
// for all matches
for ( ;itIn!= inliers.end(); ++itIn, ++itM) {
if (*itIn) { // it is a valid match
outMatches.push_back(*itM);
}
}
if (refineF) {
// The F matrix will be recomputed with
// all accepted matches
// Convert keypoints into Point2f
// for final F computation
points1.clear();
points2.clear();
for (std::vector<cv::DMatch>::
const_iterator it= outMatches.begin();
it!= outMatches.end(); ++it) {
// Get the position of left keypoints
float x= keypoints1[it->queryIdx].pt.x;
float y= keypoints1[it->queryIdx].pt.y;
points1.push_back(cv::Point2f(x,y));
// Get the position of right keypoints
x= keypoints2[it->trainIdx].pt.x;
y= keypoints2[it->trainIdx].pt.y;
points2.push_back(cv::Point2f(x,y));
}
// Compute 8-point F from all accepted matches
if (points1.size()>0&&points2.size()>0){
fundemental= cv::findFundamentalMat(
cv::Mat(points1),cv::Mat(points2), // matches
CV_FM_8POINT); // 8-point method
}
}
}
return fundemental;
}
// Match feature points using symmetry test and RANSAC
// returns fundemental matrix
cv::Mat match(cv::Mat& image1,
cv::Mat& image2, // input images
// output matches and keypoints
std::vector<cv::DMatch>& matches,
std::vector<cv::KeyPoint>& keypoints1,
std::vector<cv::KeyPoint>& keypoints2) {
// 1a. Detection of the SURF features
detector->detect(image1,keypoints1);
detector->detect(image2,keypoints2);
// 1b. Extraction of the SURF descriptors
cv::Mat descriptors1, descriptors2;
extractor->compute(image1,keypoints1,descriptors1);
extractor->compute(image2,keypoints2,descriptors2);
// 2. Match the two image descriptors
// Construction of the matcher
//cv::BruteForceMatcher<cv::L2<float>> matcher;
// from image 1 to image 2
// based on k nearest neighbours (with k=2)
std::vector<std::vector<cv::DMatch> > matches1;
matcher->knnMatch(descriptors1,descriptors2,
matches1, // vector of matches (up to 2 per entry)
2); // return 2 nearest neighbours
// from image 2 to image 1
// based on k nearest neighbours (with k=2)
std::vector<std::vector<cv::DMatch> > matches2;
matcher->knnMatch(descriptors2,descriptors1,
matches2, // vector of matches (up to 2 per entry)
2); // return 2 nearest neighbours
// 3. Remove matches for which NN ratio is
// > than threshold
// clean image 1 -> image 2 matches
int removed= ratioTest(matches1);
// clean image 2 -> image 1 matches
removed= ratioTest(matches2);
// 4. Remove non-symmetrical matches
std::vector<cv::DMatch> symMatches;
symmetryTest(matches1,matches2,symMatches);
// 5. Validate matches using RANSAC
cv::Mat fundemental= ransacTest(symMatches,
keypoints1, keypoints2, matches);
// return the found fundemental matrix
return fundemental;
}
};
// set parameters
int numKeyPoints = 1500;
//Instantiate robust matcher
RobustMatcher rmatcher;
//instantiate detector, extractor, matcher
detector = new cv::OrbFeatureDetector(numKeyPoints);
extractor = new cv::OrbDescriptorExtractor;
matcher = new cv::BruteForceMatcher<cv::HammingLUT>;
rmatcher.setFeatureDetector(detector);
rmatcher.setDescriptorExtractor(extractor);
rmatcher.setDescriptorMatcher(matcher);
//Load input image detect keypoints
cv::Mat img1;
std::vector<cv::KeyPoint> img1_keypoints;
cv::Mat img1_descriptors;
cv::Mat img2;
std::vector<cv::KeyPoint> img2_keypoints
cv::Mat img2_descriptors;
std::vector<std::vector<cv::DMatch> > matches;
img1 = cv::imread(fList[0].string(), CV_LOAD_IMAGE_GRAYSCALE);
img2 = cv::imread(fList[1].string(), CV_LOAD_IMAGE_GRAYSCALE);
rmatcher.match(img1, img2, matches, img1_keypoints, img2_keypoints);
I had a similar problem with opencv python and came here via google.
To solve my problem I wrote python code for matching-filtering based on #KLowes solution. I will share it here in case someone else has the same problem:
""" Clear matches for which NN ratio is > than threshold """
def filter_distance(matches):
dist = [m.distance for m in matches]
thres_dist = (sum(dist) / len(dist)) * ratio
sel_matches = [m for m in matches if m.distance < thres_dist]
#print '#selected matches:%d (out of %d)' % (len(sel_matches), len(matches))
return sel_matches
""" keep only symmetric matches """
def filter_asymmetric(matches, matches2, k_scene, k_ftr):
sel_matches = []
for match1 in matches:
for match2 in matches2:
if match1.queryIdx < len(k_ftr) and match2.queryIdx < len(k_scene) and \
match2.trainIdx < len(k_ftr) and match1.trainIdx < len(k_scene) and \
k_ftr[match1.queryIdx] == k_ftr[match2.trainIdx] and \
k_scene[match1.trainIdx] == k_scene[match2.queryIdx]:
sel_matches.append(match1)
break
return sel_matches
def filter_ransac(matches, kp_scene, kp_ftr, countIterations=2):
if countIterations < 1 or len(kp_scene) < minimalCountForHomography:
return matches
p_scene = []
p_ftr = []
for m in matches:
p_scene.append(kp_scene[m.queryIdx].pt)
p_ftr.append(kp_ftr[m.trainIdx].pt)
if len(p_scene) < minimalCountForHomography:
return None
F, mask = cv2.findFundamentalMat(np.float32(p_ftr), np.float32(p_scene), cv2.FM_RANSAC)
sel_matches = []
for m, status in zip(matches, mask):
if status:
sel_matches.append(m)
#print '#ransac selected matches:%d (out of %d)' % (len(sel_matches), len(matches))
return filter_ransac(sel_matches, kp_scene, kp_ftr, countIterations-1)
def filter_matches(matches, matches2, k_scene, k_ftr):
matches = filter_distance(matches)
matches2 = filter_distance(matches2)
matchesSym = filter_asymmetric(matches, matches2, k_scene, k_ftr)
if len(k_scene) >= minimalCountForHomography:
return filter_ransac(matchesSym, k_scene, k_ftr)
To filter matches filter_matches(matches, matches2, k_scene, k_ftr) has to be called where matches, matches2 represent matches obtained by orb-matcher and k_scene, k_ftr are corresponding keypoints.
I don't think there is anything very wrong with your code. From my experience opencv's ORB is sensitive to scale variations.
You can probably confirm this with a small test, make some images with rotation only and some with scale variations only. The rotation ones will probably match fine but the scale ones won't (i think decreasing scale is the worst).
I also advise you to try the opencv version from the trunk (see opencv's site for compile instructions), ORB as been updated since 2.3.1 and performs a little better but still has those scale problems.