I'm trying to use BRISK implementation of OpenCV (for C++) in order to check in a photo if an image (or a part of an image) is included in. For example, I take a photo, and I try to match it with a set of images in database, and I would like to select the best corresponding image (or an error message if none of all the images is good enough).
So, I'm just testing OpenCV for the moment. I've simply taken the sample included in the framework (matching_to_many_images), and change the detector and descriptor from SURF to BRISK.
However, I have weird results. These are the results of matching (BruteForce Hamming):
In the first one, the scenes are entirely different, but there are a lot of matches!
In the second one, the scenes are pretty similar, but some matches are wrong.
I think this is a parameters issue- because on demo videos of BRISK, the results are significant.
Have you seen the OpenCV documentation for BRISK? I'm not sure what parameters you're using now, but you can specify the threshold and octaves, as well as the pattern. Documentation at
http://docs.opencv.org/modules/features2d/doc/feature_detection_and_description.html#brisk
Also you could try a different feature matching algorithm, although it appears that in the BRISK paper they also used hamming distance
Lastly, it's not too unexpected to have erroneous feature matches; try out different scenes as well as different feature parameters and see how your results are
There are commonly many incorrect initial matches when doing feature-feature matching using SIFT, SURF, BRISK, or any other local descriptor.
Many of these initial matches will be incorrect due to ambiguous features or features that arise from background clutter. [From Distinctive Image Features from Scale Invariant Keypoints]
The next step is to select only a subset of those matches that all agree on a common transformation between the two images. This is explained in sections 7.3 and 7.4 of Distinctive Image Features from Scale Invariant Keypoints.
The OpenCV Tutorial gives an excellent example of how to extract features and calculate a homography (a transformation that tells you how to to transform each point from one image to the other one).
You can replace the feature-detector/descriptor with any other one, which will result in different robustness to certain transformations like rotation, scaling or errors like blur or lumination change. The basic implementation of BRISK already has meaningful parameters defined.
Last but not least, if you try to match two completely different images, what would you expect as a result? The algorithm will try to find similarities, and therefore always calculate a result, even if it is non-sense and the scores are very low. Just keep in mind: Garbage in -> Garbage out.
Related
I have 4 types of symbols of musical notes of the same color: Whole note, half note, Crotchet and quaver. I need to classify an image and tell if it has one of this symbols (just one for now) and which one. for example, if i have an image with just the musical staff (but nothing else in it) it should tell me that the image is empty, but if i have an image with a Half note symbol in it, it should tell me something like "it is a half note".
Suppose i have 20 sample images for each possible symbol and 20 with the base case (nothing in it), i want to train a SVM to classify any input image. I've read about how i could do it, but i still have certain doubts. i think the process is something like this (and please correct me if i'm wrong):
extract the descriptors of all the sample images.
put those descriptors inside different Mat Objects (one for each symbol).
feed those Mats to the SVM to train it.
Use the SVM to classify the images.
i have specific doubts about what i think is the process:
is what i described the correct process for what i need to do?
should i pre-process the sample images (say extract the background and apply canny edges) before i feed them to the descriptor extractor? o can i leave them as they are?
i have read about three methods of extracting the descriptors: HOG, BOW (Bag of Words) and SIFT. i think they all do what i need but i don't know which one to use. i see that HOG is mostly (if not all times) for face and pedestrians detection and i don't know if it could be used for my case. Any advice of which one should use?
how many sample images should i have for every case?
i dont need specific details of the implementation, but i do need answers to these questions, thank you in advance
I'm not an expert of SIFT and BOW but I know something about HOG and SVM.
1 Is what i described the correct process for what i need to do?
If you are using OpenCV and HOG no that is not correct. Have a look to the sample code for HOG in OpenCV samples and you will find that, once extracted, the descriptors directly feed the SVM without filling a MAT element.
2 should i pre-process the sample images (say extract the background and apply canny edges) before i feed them to the descriptor extractor? o can i leave them as they are?
This is not mandatory. Preprocessing has been proved to be very useful but for your simple case you wont need it. On the other hand, if your wall presents draws, stickers or something that can confuse the detector then yes. It can be a good solution to decrease the number of false positives.
3 i have read about three methods of extracting the descriptors: HOG, BOW (Bag of Words) and SIFT. i think they all do what i need but i don't know which one to use. i see that HOG is mostly (if not all times) for face and pedestrians detection and i don't know if it could be used for my case. Any advice of which one should use?
I have direct knowledge only of HOG. You can easily implement your own detector with HOG without any problem, I'm currently using it for traffic signs. Pay attention to the detection window that you want to use. You can leave all the other parameters as they are, it will work for simple cases.
4 how many sample images should i have for every case?
Once again it depends on the situation. I would say that 200 images (try also with less) for class will do the trick but you can always increase the number by applying some transformation on the positives. Try to flip, saturate or blur the images.
Some more considerations. I think that you can work with grey scale images due to the fact that color is not important to distinguish the notes (all the same color right?). If you have problem with false positives you can try to use the HSV color space to filter out patches that you will then use to detect the notes (it really works well with red!!). The easiest way to train your SVM is using a linear kernel and then train a model for each class.
Working on a small side project related to Computer Vision, mostly to try playing around with OpenCV. It lead me to an interesting question:
Using feature detection to find known objects in an image isn't always easy- objects are hard to find, especially if the features of the target object aren't great.
But if I could choose ahead of time what it is I'm looking for, then in theory I could generate for myself an optimal image for detection. Any quality that makes feature detection hard would be absent, and all the qualities that make it easy would exist.
I suspect this sort of thought went into things like QR codes, but with the limitations that they wanted QR codes to be simple, and small.
So my question for you: How would you generate an optimal image for later recognition by a camera? What if you already know that certain problems like skew, or partial obscuring would occur?
Thanks very much
I think you need something like AR markers.
Take a look at ArToolkit, ArToolkitPlus or Aruco libraries, they have marker generators and detectors.
And papeer about marker generation: http://www.uco.es/investiga/grupos/ava/sites/default/files/GarridoJurado2014.pdf
If you plan to use feature detection, than marker should be specific to used feature detector. Common practice for detector design is good response to "corners" or regions with high x,y gradients. Also you should note the scaling of target.
The simplest detection can be performed with BLOBS. It can be faster and more robust than feature points. For example you can detect circular blobs or rectangular.
Depending on the distance you want to see your markers from and viewing conditions/backgrounds you typically use and camera resolution/noise you should choose different images/targets. Under moderate perspective from a longer distance a color target is pretty unique, see this:
https://surf-it.soe.ucsc.edu/sites/default/files/velado_report.pdf
at close distances various bar/QR codes may be a good choice. Other than that any flat textured object will be easy to track using homography as opposed to 3D objects.
http://docs.opencv.org/trunk/doc/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html
Even different views of 3d objects can be quickly learned and tracked by such systems as Predator:
https://www.youtube.com/watch?v=1GhNXHCQGsM
then comes the whole field of hardware, structured light, synchronized markers, etc, etc. Kinect, for example, uses a predefined pattern projected on the surface to do stereo. This means it recognizes and matches million of micro patterns per second creating a depth map from the matched correspondences. Note that one camera sees the pattern and while another device - a projector generates it working as a virtual camera, see
http://article.wn.com/view/2013/11/17/Apple_to_buy_PrimeSense_technology_from_the_360s_Kinect/
The quickest way to demonstrate good tracking of a standard checkerboard pattern is to use pNp function of open cv:
http://www.juergenwiki.de/work/wiki/lib/exe/fetch.php?media=public:cameracalibration_detecting_fieldcorners_of_a_chessboard.gif
this literally can be done by calling just two functions
found = findChessboardCorners(src, chessboardSize, corners, camFlags);
drawChessCornersDots(dst, chessboardSize, corners, found);
To sum up, your question is very broad and there are multiple answers and solutions. Formulate your viewing condition, camera specs, backgrounds, distances, amount of motion and perspective you expect to have indoors vs outdoors, etc. There is no such a thing as a general average case in computer vision!
I need to know what's the best way to match certain shape (template) in the image.
I know there is several ways, but some of them did not lead to a very good results and the another need a lot of process time, so anyone tried a good and fast way to do the matching with short process time.
For example this is the template...
And I have a sample and I want to compare the sample with the template and return true if the sample is similar to the template else return false.
Note: I tried contour matching, Cascade Classification, and SURF, but all of them is not very good or the process time is not so good.
Matching things with eachother can be a rather difficult task, mainly due to the fact that different techniques have very different characteristics and can yield almost perfect results on some categories and very bad results on others.
This said, I don't think you'll ever get an answer to your question, at least not one that says "Use xyx method from [cited paper], that will solve all your problems". I'll try to point out some examples for you hoping that it'll help.
Template matching operator: compare a template with a sliding window on your image, can achieve very good results if your template is very similar to the object you are looking for in the image, no matter how complex it is. Can be very fast, it's not invariant to basically anything, so if you plan to have rotations, significant changes in lighting or something else, this is probably not going to work for you. here you can find out some code. Watch out which color space are you using, different color spaces can achieve very different results if used right (e.g. for face analysis HSV can be better that RGB in some cases)
Keypoint matching like SIFT or SURF: I used this a lot with very good results. You'll need to decide what descriptor to use and what matcher. OpenCV has some nice examples,here you can find one. Not going to be the fastest way to match your object since these descriptors can take some time to be extracted, it's good if you don't know much about the conditions you'll be working in though: it's usually robust to scale, rotation and lightning changes as long that keypoints can be correctly found on both the template and the image.
Shape matching: I was rather surprised when, in an image classification competition i participated in, I had been able to use a simple HOG descriptor to obtain very discriminating information about my images. Histograms of Oriented Gradients are a rather powerful tool for describing the shape of an object, it uses edge orientation and magnitude to describe your image. They can be fast to compute (OpenCV has a a GPU implementation I think), configurable (you can decide how thick your grid can be and how many cells, resulting in very different information). HOGs are not invariant to rotation, seen the object from a different angle will likely produce a different histogram, but they are very robust to lighting changes due to the fact that doesn't use color.
HOGs are just an example, there are a lot of shape and contour descriptors but basically they offer pretty much the same I think.
Histogram matching: not my first choice, it can be useful if you know something about the object and the rest of image. For example, if you know you are looking for your pink flower in a jungle image where it's the only pink thing there, a simple color histogram matching will do just fine. Pick up a sliding window, run it over your image, compare your histograms and you'll be done. Very fast, very simple, it doesn't use the shape at all so no matter how complex your object is, you'll find it. Not using shape makes it robust to rotations, watch out for lighting changes though. A very big limitations of this method is that if there are other pink things in your jungle you won't be able to distinguish.
Hybrid approaches: here is where you can get the best out of the techniques cited above. As you have seen, most of them work well in a certain environment and quite bad in others. You can use a combination of the techniques you know and obtain something much better than the sum of the parts. I worked a lot with HOGs and head pose estimation and a real breakthrough came when we started extracting HOGs not in a dense way but around certain keypoints. You'll need to know your problem, find out what do you need and adapt a bunch of methods to it. In general, hybrid methods can work a lot better and a lot slower.
Hope this helps you a bit, I don't think that, given the information you gave us, I could give you a much better answer..(probably someone else can, that's why I'm still a student :) )
I have a simple template grayscale image, with white background and black shape over it, and I have several similar test images, I want to compare these two images and see if template matches any of the test images. Can you please suggest a simple(easy to use) pattern recognition library for C++ which takes two images and compares them and shows the result?
Just do image1-image2 for all pixels. Then sum up all the differences. The lower the results, the closer the images.
If your pattern could be of several sizes, then you have to resize it and check it for each positions.
Implement a Neural Network on the image. Inputs should be the greyscales of your image. you should train your network to a train set, chose proper regularization parameters using a cross validation set, and finally test your network on a test set.
http://www.codeproject.com/Articles/13582/Back-propagation-Neural-Net
(I have done this myself to train a network to recognise hand written digits - it works very well.)
How simple the library you need is depends on the specific parameters of your problem. OpenCV is a great image processing library that should be able to do what you need it to. Here is a tutorial on template matching in OpenCV. It makes it very easy to switch between matching metrics and choose the best one for your problem.
I have 2 images with the same content but might have different scale or rotation. The problem is, I have to find the regions of these images and match them with one another. For example, if I have a circle on image1, i have to find the corresponding circle in image2.
I just like to ask what the proper way of solving this is. I am looking at the matchShapes of opencv. I believe this problem is image correspondence but I really have no idea how to solve it!
Thanks in advance!
I have the following images:
Template Image => https://lh6.googleusercontent.com/-q5qeExXUlpc/T7SbL9yWmCI/AAAAAAAAByg/gV_vM1kyLnU/w348-h260-n-k/1.labeled.jpg
Sample Image => https://lh4.googleusercontent.com/-x0IWxV7JdbI/T7SbNjG5czI/AAAAAAAAByw/WSu-y5O7ee4/w348-h260-n-k/2.labeled.jpg
Note that the numbers on the images correspond to the proper matching of regions. These are not present when comaparing the images.
As usually with computer vision problems, you can never provide too much information and make too many assumptions about the data you intend to analyze. Solving the general problem is close to impossible as we can't do human level pattern recognition with computers. How does your problem set look like? A few examples would be very helpful in trying to provide good answers.
You mention that the images have the same content, but with different colors. If that means it's the same scene photographed under different lighting conditions and from possibly different angles, you might need to do a rigid image registration first, so the feature points in the two images should overlap. If the shapes on your images might have multiple distortions compared to each other, you might be interested in non-rigid image registration.
If you already know the objects you are looking for, you can simply do a search for these objects in both images, for example with chamfer matching or any other matching algorithm.
Use ORB feature detector from OpenCV. Once you have the descriptors use BFMatcher with norm type NORM_HAMMING.