I am doing a project on face recognition from video images.I extracted the features,now I need to compare the feature.So I found FlannBasedMatcher is a good method, also it is very fast.FlannBasedMatcher is already in the opencv (I am using opencv),but like to implement it myself with out any opencv help.Please help me to find what is exactly happening inside FlannBasedMatcher.Any response will be greatly appreciated.
Features are typically compared using some distance metrics such as Euclidian distance between features that are considered to be points in some multi-demnsional space; one can use the angle between two vectors (that is feature vectors) that is independent of vector scaling; one can use a Humming distance for comparing binary strings, etc. The best way depends on the structure and the meaning of your feature vector. For faces it can be an angle between two vectors expressed through a dot product.
Now, flann is used for finding nearest neighbors and as such is not directly related to feature comparison though it can help to speed up finding similar features that are worth comparison (flann=fast library for nearest neighbors). Thus you won’t need to search through all your vectors trying to select the one that has highest dot product with the query vector, but instead directly compare a given face (vector) with just a few closest faces (vectors).
Finally, addressing a previous answer, in some cases one can use sparse arrays instead of KD trees. They are part of openCV too but can be implented through hash tables or trees. In sparse arrays you can check indices of neighboring elements which is analogous to flann nearest neighbors. Of course, sparse arrays are more limited than flann - for example, they require an exhaustive search in the neighborhood to get a nearest neighbors list but this is still faster than global search. Here is an example:
int dims = 3;
int sz[] = {1000, 1000, 1000}; // memory efficient
SparseMat M3d(dims, sz, CV_32F);
Point3i idx_sparse;
Vec3f p;
//set the element of a sparse 3D Mat
M3d.ref<Vec3f>(idx_sparse.x, idx_sparse.y, idx_sparse.z) = p;
// iterate
SparseMatIterator it = M3d.begin();
SparseMatIterator it_end = M3d.end();
for (; it != it_end; ++it) {
// access existing element through iterator
Vec3f vec = it.value<Vec3f>();
// check neighbors if they exist
int* idx = it.node()->index;
idx[0]++; idx[1]--; idx[2]+=2;
if (M3d.find(idx) != M3d.end()) {
Vec3f vec = M3d.ref<Vec3f>(idx);
}
}
It is not that easy. You have to implement kd-tree with aproximated nearest neighbor search. It is described in paper "An Optimal Algorithm for Approximate Nearest
Neighbor Searching in Fixed Dimensions" by Arya et al.
If you don`t want to do it from the scratch and just want to get rid of OpenCV, you can take original FLANN implementation.
Related
I'm currently trying to find the nearest neighbours (NNs) of a point in its feature space, using the FLANN module from PCL library.
The points that I'm trying to compare are RIFT descriptors, in the form of pcl::Histogram<32>, as I use 4 bins for the distance and 8 for the gradient (like in the original article).
I'm wondering what distance measure should I use, as the default one is the L2 norm, which seems kind of weak when matching points in a high dimensionnal feature space.
I use the KdTreeMultiIndexCreator index of FLANN to speed-up the search.
I'm bound to use the flann module as below:
// Useful types
typedef pcl::Histogram<32> FeatureT;
typedef flann::L2<float> DistanceT;
typedef pcl::search::FlannSearch<FeatureT, DistanceT> SearchT;
typedef typename SearchT::FlannIndexCreatorPtr CreatorPtrT;
typedef typename SearchT::KdTreeMultiIndexCreator IndexT;
typedef typename SearchT::PointRepresentationPtr RepresentationPtrT;
// Instantiate search object with 4 randomized trees and 128 checks
SearchT search (true, CreatorPtrT (new IndexT (4)));
search.setPointRepresentation (RepresentationPtrT (new DefaultFeatureRepresentation<FeatureT>));
search.setChecks (128); // The more checks the more precise the solution
// search_cloud is filled with the keypoints to match
search.setInputCloud (search_cloud);
search.nearestKSearch(point_to_match, 1, indices, distances);
So, what is the best distance measure available in FLANN to fit my problem ?
I am trying to find the exact number of neighbour nodes in a big 3D points dataset. The goal is for each point of the dataset to retrieve all the possible neighbours in a region with a given radius. FLANN ensures that for lower dimensional data can retrieve the exact neighbors while comparing with brute force search it seems to not be the case. The neighbors are essential for further calculations and therefore I need the exact number. I tested increasing the radius a little bit but doesn't seem to be this the problem. Is anyone aware how to calculate the exact neighbors with FLANN or other C++ library?
The code:
// All nodes to be tested for inclusion in support domain.
flann::Matrix<double> query_nodes = flann::Matrix<double>(&nodes_pos[0].x, nodes_pos.size(), 3);
// Set default search parameters
flann::SearchParams search_parameters = flann::SearchParams();
search_parameters.checks = -1;
search_parameters.sorted = false;
search_parameters.use_heap = flann::FLANN_True;
flann::KDTreeSingleIndexParams index_parameters = flann::KDTreeSingleIndexParams();
flann::KDTreeSingleIndex<flann::L2_3D<double> > index(query_nodes, index_parameters);
index.buildIndex();
//FLANN uses L2 for radius search.
double l2_radius = (this->support_layer_*grid.spacing)*(this->support_layer_*grid.spacing);
double extension = l2_radius/10.;
l2_radius+= extension;
index.radiusSearch(query_nodes, indices, dists, l2_radius, search_parameters);
Try nanoflann. It is designed for low dimensional spaces and gives exact nearest neighbors. Furthermore, it is just one header file that you can either "install" or just copy to your project.
You should check page 6+ from the flann-manual, to fine-tune your search parameters, such as target_precision, which should be set to 1, for "maximum" accuracy.
That parameter is often found as epsilon (ε) in Approximate Nearest Neighbor Search (ANNS), which is used in high dimensional spaces, in order to (try) to beat the curse of dimensionality. FLANN is usually used in 128 dimensions, not 3, as far as I can tell, which may explain the bad performance you are experiencing.
A c++ library that works well in 3 dimensions is CGAL. However, it's much larger than FLANN, because it is a library for computational geometry, thus it provides functionality for many problems, not just NNS.
Is there an OpenCV function that can give me a list of all the local maxima for a histogram? Maybe there is a function that lets me specify a minimum peak/threshold and will tell me the bins of all those local maxima above that threshold.
If not, is there a function that can sort the bins from highest(most frequent) to lowest (least frequent). I can then grab all the first 20 or so bins and I have my 20 biggest local maxima.
Opencv minMaxLoc can be used in this context with a sliding window. If the location of the maxima is on an edge then ignore the maxima, otherwise record as maxima. You can use something like the function below (Note: this code is more like psuedocode it has not been tested)
/**
* Assumes a 1 channel histogram
*/
vector<int> findMaxima(Mat histogram, int windowsize, int histbins){
vector<int> maximas;
int lastmaxima;
for(int i = 0; i < histbins - windowsize; i++){
//Just some Local variables, only maxloc and maxval are used.
int maxval,minval;
Point* maxloc, maxloc;
//Crop the windows
Rect window(i,0,windowsize,1);
//Get the maxima
minMaxLoc(histogram(window), minval,maxval,maxloc,minloc);
//Check if its not on the side
if(maxloc.x != 0&&maxloc.x != windowsize-1){
//Translate from cropped window into real position
int originalposition = maxloc.x+i;
//Check that this is a new maxima and not already recorded
if(lastmaxima != originalposition){
maximas.push(originalposition);
lastmaxima = originalposition;
}
}
}
return maximas;
}
Of course this is a very simplistic system. You might want to use a multiscale approach with different sliding window sizes. You may also need to apply gaussian smoothing depending on your data. Another approach could be to run this for a small window size like 3 or 4 (you need a mimimum of 3). Then you could use something else for non maxima-suppression.
For your approach in which you suggested
Maybe there is a function that lets me specify a minimum peak/threshold and will tell me the bins of all those local maxima above that threshold.
You could simply perform a threshold before finding the maxima with the above function.
threshold(hist,res ...aditional parameters...);
vector<int> maximas = findMaximas(hist, ...other parameters...);
AFAIK OpenCV doesn't have such functionality, but it is possible do implement something similar yourself.
In order to sort histogram bins you can possibly use sortIdx, but as a result you will obtain list of largest bins, which is different than local maxima (those should be "surrounded" by smaller values).
To obtain local maxima you can compare each bin with its neighbors (2 in 1D case). A bin should be larger than neighbors with some margin to be considered a local maximum.
Depending on the size of the bins, you may want to filter the histogram before this step (for example convolve it with Gaussian kernel), since otherwise you'd obtain too much of these maxima, especially for small bin sizes. If you've used Gaussian kernel - it's sigma would be related to the size of the neighborhood in which detected local maxima are "global".
Once you detect those points - you may want to perform non-maximal suppression, to replace groups of points that lie very close together with a single point. A simple strategy for that would be to sort those maxima according to some criteria (for example difference with neighbors), then take one maximum and remove all the points in its neighborhood (its size can be related the the Gaussian kernel sigma), take next remaining maximum and again remove points in its neighborhood and so on until you run out of points or go below some meaningful difference values.
Finally, you may want to sort remaining candidate points by their absolute values (to get "largest" local maxima), or by their differences with neighbors (to get "sharpest" ones).
You may try another approach. We can use this definition of local maximum to implement a simpler algorithm: just move a sliding window of size S along the histogram and pick maximum in each position. This will have some problems:
in locations with prominent maximum multiple window positions will generate points that correspond to the same maximum (can be fixed with non maximum suppression),
in locations with no or small variation it will return
semi-random maxima (can be fixed with threshold on variance in
window or difference between maximum and neighborhood),
in regions with monotonic histogram it will return a largest value (which is not necessarily a maximum).
Once you perform all the "special case" handling - those 2 approaches would be quite similar I believe.
Another thing to implement may be "multi scale" approach, which can be considered as an extension if those 2. Basically it boils down to detecting local maxima for different neighborhood sizes, and then storing them all along with corresponding neighborhood size, which can be helpful for some purposes.
As you can see, this is a quite vague guide, and there's a reason for that: the type and amount of local maximas you want to get will most likely depend on the problem you have in mind. There's no hard and easy rule to decide if the point should be considered a local maxima, so you should probably start with some simple approach and then refine it for your specific case.
I want to match features in two images to detect copy-move forgery. I used the PCA-SIFT code to detect image features. But, I am having trouble in matching the PCA-SIFT features. According to several papers, similar matching process is used for PCA-SIFT as is used in SIFT. I have used the following code snippet to match features.
%des1 and des2 are the PCA-SIFT descriptors obtained from two images
% Precompute matrix transpose
des2t = des2';
matchTable = zeros(1,size(des1,1));
cnt=0; %no. of matches
%ration of ditances
distRatio = 0.5;
%normalising features
m1=max(max(des1));
m2=max(max(des2));
m=max(m1,m2);
des1=des1./m;
des2=des2./m;
for i = 1 : size(des1,1)
%finding eucledian distance of a vector in one image to all features in second image
A=des1(i,:);
D = des2-repmat(A,size(des2,1),1);
[vals,indx] = sort((sum(D.^2,2)).^(1/2)); %sort distances
% Check if nearest neighbor has angle less than distRatio times 2nd.
if (vals(1) < distRatio * vals(2))
matchTable(i) = indx(1);
cnt=cnt+1;
else
matchTable(i) = 0;
end
end
cnt
The above code works fine for SIFT features. But I am not able to get correct results for PCA-SIFT features even after trying several values of distRatio(0-1). I'm also not sure if the matlab central code for PCA-SIFT(mentioned above) does the exact process as mentioned in this paper
If somebody has any idea about the above problem then please comment.
The problem is, PCA does not preserve euclidean distance between 2 vectors. Take a simple example where your data is along the line y = x. The distance between 2 points along the line will depend on both co-ordinates, even if all your data is 1 dimensional, i.e. lying along the line. When you apply PCA, the new euclidean distance will only take the principle component into account, which would be the line y=x, so distance between (1,1), (2,2) would just be 1 instead of sqrt(2).
However, if you normalize the features by their euclidean norm, nearest neighbor using euclidean distance is equivalent to computing cosine-similarity (dot-product) between features.
https://en.wikipedia.org/wiki/Cosine_similarity
Therefore I would first recommend you to test if matching for sift features works if you normalize them by their L2 norm. If yes, you could apply PCA on those features, again normalize the PCA features by their L2 norm and then compute euclidean distance. As far as I remember, L2 norm of a sift vector is 1. So, you only need normalize your PCA-SIFT features by their L2 norm and compute euclidean distance.
I have been developing an application to count circular objects such as bacterial colonies from pictures.
What make it easy is the fact that the objects are generally well distinct from the background.
However, few difficulties make the analysis tricky:
The background will present gradual as well as rapid intensity change.
In the edges of the container, the object will be elliptic rather than circular.
The edges of the objects are sometimes rather fuzzy.
The objects will cluster.
The object can be very small (6px of diameter)
Ultimately, the algorithms will be used (via GUI) by people that do not have deep understanding of image analysis, so the parameters must be intuitive and very few.
The problem has been address many times in the scientific literature and "solved", for instance, using circular Hough transform or watershed approaches, but I have never been satisfied by the results.
One simple approach that was described is to get the foreground by adaptive thresholding and split (as I described in this post) the clustered objects using distance transform.
I have successfully implemented this method, but it could not always deal with sudden change in intensity. Also, I have been asked by peers to come out with a more "novel" approach.
I therefore was looking for a new method to extract foreground.
I therefore investigated other thresholding/blob detection methods.
I tried MSERs but found out that they were not very robust and quite slow in my case.
I eventually came out with an algorithm that, so far, gives me excellent results:
I split the three channels of my image and reduce their noise (blur/median blur). For each channel:
I apply a manual implementation of the first step of adaptive thresholding by calculating the absolute difference between the original channel and a convolved (by a large kernel blur) one. Then, for all the relevant values of threshold:
I apply a threshold on the result of 2)
find contours
validate or invalidate contours on the grant of their shape (size, area, convexity...)
only the valid continuous regions (i.e. delimited by contours) are then redrawn in an accumulator (1 accumulator per channel).
After accumulating continuous regions over values of threshold, I end-up with a map of "scores of regions". The regions with the highest intensity being those that fulfilled the the morphology filter criteria the most often.
The three maps (one per channel) are then converted to grey-scale and thresholded (the threshold is controlled by the user)
Just to show you the kind of image I have to work with:
This picture represents part of 3 sample images in the top and the result of my algorithm (blue = foreground) of the respective parts in the bottom.
Here is my C++ implementation of : 3-7
/*
* cv::Mat dst[3] is the result of the absolute difference between original and convolved channel.
* MCF(std::vector<cv::Point>, int, int) is a filter function that returns an positive int only if the input contour is valid.
*/
/* Allocate 3 matrices (1 per channel)*/
cv::Mat accu[3];
/* We define the maximal threshold to be tried as half of the absolute maximal value in each channel*/
int maxBGR[3];
for(unsigned int i=0; i<3;i++){
double min, max;
cv::minMaxLoc(dst[i],&min,&max);
maxBGR[i] = max/2;
/* In addition, we fill accumulators by zeros*/
accu[i]=cv::Mat(compos[0].rows,compos[0].cols,CV_8U,cv::Scalar(0));
}
/* This loops are intended to be multithreaded using
#pragma omp parallel for collapse(2) schedule(dynamic)
For each channel */
for(unsigned int i=0; i<3;i++){
/* For each value of threshold (m_step can be > 1 in order to save time)*/
for(int j=0;j<maxBGR[i] ;j += m_step ){
/* Temporary matrix*/
cv::Mat tmp;
std::vector<std::vector<cv::Point> > contours;
/* Thresholds dst by j*/
cv::threshold(dst[i],tmp, j, 255, cv::THRESH_BINARY);
/* Finds continous regions*/
cv::findContours(tmp, contours, CV_RETR_LIST, CV_CHAIN_APPROX_TC89_L1);
if(contours.size() > 0){
/* Tests each contours*/
for(unsigned int k=0;k<contours.size();k++){
int valid = MCF(contours[k],m_minRad,m_maxRad);
if(valid>0){
/* I found that redrawing was very much faster if the given contour was copied in a smaller container.
* I do not really understand why though. For instance,
cv::drawContours(miniTmp,contours,k,cv::Scalar(1),-1,8,cv::noArray(), INT_MAX, cv::Point(-rect.x,-rect.y));
is slower especially if contours is very long.
*/
std::vector<std::vector<cv::Point> > tpv(1);
std::copy(contours.begin()+k, contours.begin()+k+1, tpv.begin());
/* We make a Roi here*/
cv::Rect rect = cv::boundingRect(tpv[0]);
cv::Mat miniTmp(rect.height,rect.width,CV_8U,cv::Scalar(0));
cv::drawContours(miniTmp,tpv,0,cv::Scalar(1),-1,8,cv::noArray(), INT_MAX, cv::Point(-rect.x,-rect.y));
accu[i](rect) = miniTmp + accu[i](rect);
}
}
}
}
}
/* Make the global scoreMap*/
cv::merge(accu,3,scoreMap);
/* Conditional noise removal*/
if(m_minRad>2)
cv::medianBlur(scoreMap,scoreMap,3);
cvtColor(scoreMap,scoreMap,CV_BGR2GRAY);
I have two questions:
What is the name of such foreground extraction approach and do you see any reason for which it could be improper to use it in this case ?
Since recursively finding and drawing contours is quite intensive, I would like to make my algorithm faster. Can you indicate me any way to achieve this goal ?
Thank you very much for you help,
Several years ago I wrote an aplication that detects cells in a microscope image. The code is written in Matlab, and I think now that is more complicated than it should be (it was my first CV project), so I will only outline tricks that will actually be helpful for you. Btw, it was deadly slow, but it was really good at separating large groups of twin cells.
I defined a metric by which to evaluate the chance that a given point is the center of a cell:
- Luminosity decreases in a circular pattern around it
- The variance of the texture luminosity follows a given pattern
- a cell will not cover more than % of a neighboring cell
With it, I started to iteratively find the best cell, mark it as found, then look for the next one. Because such a search is expensive, I employed genetic algorithms to search faster in my feature space.
Some results are given below: