OpenCV C++ - Find an image contained within an image? - c++

I have code for searching one small image in bigger another one image:
int* MyLib::MatchingMethod(int, void*)
{
/// Source image to display
img.copyTo(img_display);
/// Create the result matrix
int result_cols = img.cols - templ.cols + 1;
int result_rows = img.rows - templ.rows + 1;
result.create(result_rows, result_cols, CV_32FC1);
match_method = 0;
/// Do the Matching and Normalize
matchTemplate(img, templ, result, match_method);
normalize(result, result, 0, 1, cv::NORM_MINMAX, -1, cv::Mat());
/// Localizing the best match with minMaxLoc
double minVal;
double maxVal;
cv::Point minLoc;
cv::Point maxLoc;
cv::Point matchLoc;
minMaxLoc(result, &minVal, &maxVal, &minLoc, &maxLoc, cv::Mat());
/// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better
if (match_method == CV_TM_SQDIFF || match_method == CV_TM_SQDIFF_NORMED)
{
matchLoc = minLoc;
}
else
{
matchLoc = maxLoc;
}
if (showOpenCVWindow) {
/// Show me what you got
rectangle(img_display, matchLoc, cv::Point(matchLoc.x + templ.cols, matchLoc.y + templ.rows), cv::Scalar(255, 0, 0, 255), 2, 8, 0);
rectangle(result, matchLoc, cv::Point(matchLoc.x + templ.cols, matchLoc.y + templ.rows), cv::Scalar(255, 0, 0, 255), 2, 8, 0);
imshow(image_window, img_display);
imshow(result_window, result);
}
double myX = (matchLoc.x + (templ.cols) / 2);
double myY = (matchLoc.y + (templ.rows) / 2);
static int o[2];
o[0] = myX;
o[1] = myY;
return o;
}
But this code could mistakenly "found" any area, even if bigger image doesn't contains small image.
How to change this code, to force it to "exactly" searching of the small image. For example, if smaller image is not on the bigger image, this code must show any info message "Image not found".
Update 1. It looks, like matchTemplate doesn't work good. For example, I have 3 images - one template ( http://s6.postimg.org/nj2ts3lf5/image.png ) , one image, that contains image from template ( http://s6.postimg.org/fp6tkg301/image.png ), and one image, that doesn't contains template ( http://s6.postimg.org/9x23zk3sh/image.png ).
For first image, that contains template, maxVal=0.99999994039535522 and it correctly selected area: http://s6.postimg.org/65x4qzfht/image.png
But for image, that doesn't contains template, maxVal=1.0000000000000000 and it incorrectly selected area, that doesn't contains template image: http://s6.postimg.org/5132llt0x/screenshot_544.png
Thank you!

You are visualizing the result regardless of the certainty with which the algorithm performed matching. Template matching will always give you an output - what you want to do is to try to figure out if it's valid or not.
Try outputing minVal or maxVal depending on the match_method. You should compare the value in the cases when the correct match was found and in the cases when it gave you a false positive. Those experiments should allow you to establish a threshold, that distinguishes between true hits and false positives. Thus, you will be able to say how big - for example - the maxVal has to be to be sure that it was a match. Pseudo code would go something like this:
if maxVal > threshold:
match_found = true
match_position = maxLoc
Now that's a theoretical approach. Since you didn't provide any images, it might or might not be the solution for your problem.
EDIT:
If you cannot find a definite threshold value (which in my opinion should be possible in most cases, if you maintain quality, size, etc), try doing one of two things:
Try looking at all obtained results, before minMaxLoc, calculate the mean value and see if the maxVal found is much bigger than the mean value in the true positive cases. Maybe you can define the threshold as the % of the mean value, thus saying: if maxVal > meanVal + meanVal * n%: match_found = true
It is a common situation, that template matching works better with edges than with the real image. Again, you haven't provided samples, so it's hard to say how reliable will that approach be here. But if you have enough high frequencies, to light up an image with Canny Edges, that might give you a much clearer threshold for discriminating between true and false positives.
EDIT2:
Since you're using match_method = 0, that means CV_TM_SQDIFF. For more control over the process, use the name explicitly. Find information on the methods here.
Also, put the cout inside the if statement, so that you print the correct value, that actually idicates the match (in your case, it's minVal).
if (match_method == CV_TM_SQDIFF || match_method == CV_TM_SQDIFF_NORMED)
{
matchLoc = minLoc;
std::cout << minVal << std::endl;
}
else
{
matchLoc = maxLoc;
std::cout << maxVal << std::endl;
}
And again: fairly tuned contours detection should almost certainly help if this doesn't give you the expected results.

Related

OpenCV Template Matching for N best matches not working

I have been struggling with this for quite a long while and can't seem to find the problem here. Let me try to walk you through the process.
I am trying to obtain the 10 best bounding boxes for my template matching.
Not going to show the entire image here, but here's my cv::mat when I loop through the cv::mat to search for the lowest values (results)
int a,b;
for ( a = 0; a < final_image_height; a++){
for ( b = 0; b < final_image_width; b++){
if (result_scores_mat.at<float>(a, b) < 70 ){
printf("%.2f ", result_scores_mat.at<float>(a, b));
}
}
}
This gives me some values present in the cv::mat with "lower values"
68.50 68.93 54.50 68.92 64.62 57.12 62.69 65.86 63.52 68.35 68.65 61.93 69.18 67.69
I then move into a loop whereby I called minMaxLoc() to find the minVal (using TM_SQDIFF, so minVal)
for ( i = 0; i < 10; i++){
minMaxLoc(result_scores_mat, &minVal, &maxVal, &minLoc, &maxLoc, cv::Mat());
cout<<minVal<<endl;
// for match_method TM_SQDIFF we take lowest values
matchLoc = minLoc;
drawRadius = result_radius_mat.at<float>(matchLoc.x, matchLoc.y);
// display source image and result matrix , draw rectangle around highest possible matching area
cv::rectangle( img_display_mat, matchLoc, cv::Point( matchLoc.x + 2*drawRadius, matchLoc.y + 2*drawRadius), cv::Scalar::all(255), 2, 8, 0);
result_scores_mat.at<float>(minLoc.x, minLoc.y)= 255;
}
What I am doing in the last line is replacing the value of the float at the minLoc, so that the next loop I run through minMaxLoc doesn't locate that point and gives me the next minimum value in the cv::mat.
However, I keep receiving such an output with cout<<minVal<<endl;
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
It detects rightly the 54.50 that is present in the cv::mat, but I want all the minimum values in the loop.
Is there something wrong I'm doing, or any way around this? Thank you so much!
It seems that you are replacing minimum value with 255 in wrong way, it means that you detect minimum value correctly, but you replace 255 in wrong position, you should replace below line
result_scores_mat.at<float>(minLoc.x, minLoc.y)= 255;
by this line:
result_scores_mat.at<float>(minLoc.y, minLoc.x)= 255;
For knowing that why we replace this lines by each other, you can take a look at the source code of .at method.It is like below
template<typename _Tp> _Tp& at(int row, int col);
/** #overload
#param row Index along the dimension 0
#param col Index along the dimension 1
*/

How to detect a multiple-bar pattern?

I'd like to detect a custom "multiple-bar pattern" in an image.
The pattern looks like this, kind of a group of parallel black bars with the same width but different height, see this image:
This pattern could be on the image or even not but if it is - I'd like to get it's position.
Note: The color of the pattern is black in every case.
Note: The size of the pattern is unknown, so it could be big or could be super small.
Note: The pattern bar count is a fixed number. It will be the same ( in this case 7) for every occurrence.
An image could look like this:
And after performing the code search algorithm this should happen:
Any help would be very appreciated. Thanks a million in advance, Tempi.
Note: The code I got so far (not working)
Mat myImage; // this is the mat of the photo you can see above
Mat algorithmImage;
myImage.coptyTo(algorithmImage);
cvtColor(algorithmImage, algorithmImage, CV_RGB2HSV);
double imgThreshold = 20;
cv::inRange(algorithmImage, cv::Scalar(0, 0, 0, 0), cv::Scalar(180, 255, 30, 0), 20);
Mat canny;
vector<vector<Point>> contours;
vector<Vec4i> hierarchy;
Canny( algorithmImage, canny, 3, 6, 3 );
findContours( canny, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, Point(0, 0) );
for( int i = 0; i<contours.size(); i++ ) {
// ??
}
bool isLineAlreadyFound(const Vec4i& _l1, const Vec4i& _l2) {
Vec4i l1(_l1), l2(_l2);
float length1 = sqrtf((l1[2] - l1[0])*(l1[2] - l1[0]) + (l1[3] - l1[1])*(l1[3] - l1[1]));
float length2 = sqrtf((l2[2] - l2[0])*(l2[2] - l2[0]) + (l2[3] - l2[1])*(l2[3] - l2[1]));
float product = (l1[2] - l1[0])*(l2[2] - l2[0]) + (l1[3] - l1[1])*(l2[3] - l2[1]);
if (fabs(product / (length1 * length2)) < cos(CV_PI / 30))
return false;
float mx1 = (l1[0] + l1[2]) * 0.5f;
float mx2 = (l2[0] + l2[2]) * 0.5f;
float my1 = (l1[1] + l1[3]) * 0.5f;
float my2 = (l2[1] + l2[3]) * 0.5f;
float dist = sqrtf((mx1 - mx2)*(mx1 - mx2) + (my1 - my2)*(my1 - my2));
if (dist > std::max(length1, length2) * 0.5f)
return false;
return true;
}
Here's how I would approach your problem.
Since your pattern is totally black, may be you can benefit from that
and just do a threshold. Try a very low threshold (if it's a perfect
black, use 1 as a threshold).
After above step, using findContours should already do a good job in
detecting shapes. But what you'll need is a way to recognize your
pattern. Since you your patterns may show up in different scale and/or
orientation, you will need a scale/orientation invariant descriptor. I
mean here, a shape descriptor. You can look at
https://en.wikipedia.org/wiki/Image_moment for Hu moments.
That will be for only one bar in each pattern. You should complete
your search by designing a specific feature for your pattern with the
help of the features extracted from each bar.
Th trick here is that you have 14 parallel edges. Those are impossible to miss. As noted in other answer, threshold to isolate them. Directly after that, run the edge detector. This turns black&white edges into lines, while the rest of the image might produce a few scattered pixels. Visualize this to see for yourself.
Next, run the Hough Transform from OpenCV. This gives you lines in the format {offset, direction}. Even if a few scattered background pixels would coincidentally line up, they wouldn't form 14 lines with the same direction.
You can calculate the maximum offset difference to find the scale of the bar pattern, and double-check the relative offsets to check individual bar widths and spacings. Remember, the 14 lines are edges, so you need to pair them.
At this point you've identified the direction and scale of the bar pattern. it might be a bit surprising to realize you haven't actually identified the actual bar positions. The reason for this procedure is that we tackle the hard problem first. We now go back to the output of the edge detection, and divide all the edges there into three categories: the parallel edges, the rounded end caps between two pairs of parallel edges (follow the contours), and random background pixels. You might need a MORPH_CLOSE operation to close gaps in the contour.

face and eye detection not working properly

using this code below I want to detect face and eyes from a video,
the code runs without error but the video and detection result is not displayed when I run it , what is the problem ?
I tried it using images it works fine on some images and other images just detect faces .
#include <opencv2\opencv.hpp>
using namespace cv;
using namespace std;
int main()
{
float EYE_SX = 0.16f;
float EYE_SY = 0.26f;
float EYE_SW = 0.30f;
float EYE_SH = 0.28f;
Mat dest, gray,frame;
VideoCapture capture("m.mp4");
CascadeClassifier detector, eyes_detector;
if (!capture.isOpened()) // check if we succeeded
return -1;
if(!detector.load("haarcascade_frontalface_alt2.xml"))
cout << "No se puede abrir clasificador." << endl;
if(!eyes_detector.load("haarcascade_eye_tree_eyeglasses.xml"))
cout << "No se puede abrir clasificador para los ojos." << endl;
for (;;)
{
capture >> frame;
cvtColor(frame, gray, CV_BGR2GRAY);
equalizeHist(gray, dest);
vector<Rect> rect;
detector.detectMultiScale(dest, rect);
for (Rect rc : rect)
{
rectangle(frame,
Point(rc.x, rc.y),
Point(rc.x + rc.width, rc.y + rc.height),
CV_RGB(0, 255, 0), 2);
}
if (rect.size() > 0)
{
Mat face = dest(rect[0]).clone();
vector<Rect> leftEye, rightEye;
int leftX = cvRound(face.cols * EYE_SX);
int topY = cvRound(face.rows * EYE_SY);
int widthX = cvRound(face.cols * EYE_SW);
int heightY = cvRound(face.rows * EYE_SH);
int rightX = cvRound(face.cols * (1.0 - EYE_SX - EYE_SW));
Mat topLeftOfFace = face(Rect(leftX, topY, widthX, heightY));
Mat topRightOfFace = face(Rect(rightX, topY, widthX, heightY));
eyes_detector.detectMultiScale(topLeftOfFace, leftEye);
eyes_detector.detectMultiScale(topRightOfFace, rightEye);
if ((int)leftEye.size() > 0)
{
rectangle(frame,
Point(leftEye[0].x + leftX + rect[0].x, leftEye[0].y + topY + rect[0].y),
Point(leftEye[0].width + widthX + rect[0].x - 5, leftEye[0].height + heightY + rect[0].y),
CV_RGB(0, 255, 255), 2);
}
if ((int)rightEye.size() > 0)
{
rectangle(frame,
Point(rightEye[0].x + rightX + leftX + rect[0].x, rightEye[0].y + topY + rect[0].y),
Point(rightEye[0].width + widthX + rect[0].x + 5, rightEye[0].height + heightY + rect[0].y),
CV_RGB(0, 255, 255), 2);
}
}
}
imshow("Ojos", frame);
waitKey(0);
return 1;
}
So, right now, the imshow("Ojos", frame); and the waitKey(0); are only getting called right before the program ends. That's fine for images, but not for video, since you want it to happen once per frame.
If you move it up a few lines, inside that for loop (basically, just put the bracket that's one line up from it one line below it), it should start working better for videos.
However, there are a couple of other things you might want to tweak in the code - it's only going to show one right eye and one left eye. This is normally what you want to happen, but if you have false positives you might end up with somebody's hair or skin being labeled an eye, and you're none-the-wiser as to how it happens. I'd reccommend displaying all of the items in the lefteye and righteye vectors. This can be done simply by replacing those if statements (if (int)rightEye.size() > 0, etc) with
for (int i = 0; i < rightEye.size(); i++) {
rectangle(frame,
Point(rightEye[i].x + rightX + leftX + rect[i].x,
rightEye[i].y + topY + rect[i].y),
Point(rightEye[i].width + widthX + rect[i].x + 5,
rightEye[i].height + heightY + rect[i].y),
CV_RGB(0, 255, 255), 2);
}
If you're having problems with false positives or negatives, you might want to look into tweaking the parameters on detectMultiscale - right now, you're leaving everything to the defaults. Multiscale has a number of parameters that can be put in. Image and objects you already have, but there are other, such as:
scaleFactor – Parameter specifying how much the image size is
reduced at each image scale. The default is 1.1. The bigger that is, the larger a scale will happen, which will take the cascade less time but there will be more false positives.
minNeighbors – Parameter specifying how
many neighbors each candidate rectangle should have to retain it. Default is 3. The bigger that is, the more neighbors it'll search for, resulting in less false positives, but it'll take longer. Tweak it too high, and it'll start giving false negatives.
flags – Parameter with the same meaning for an old cascade as in the
function cvHaarDetectObjects. It is not used for a new cascade. Default's zero, just leave it at zero for the most part.
minSize – Minimum possible object size. Objects smaller than that are
ignored. Default is Size(0,0). I tend to bump it up just a little. Again, bigger means it's faster and less false positives, but too big will skip over whatever you're looking for.
maxSize – Maximum possible object size. Objects larger than
that are ignored. Default is I believe the max size of the image passed in. I tend to limit it to smaller than that. Smaller on this means it's faster and less false positives, but too small will skip over whatever you're looking for.
cascade_name.detectMultiScale( frame_gray, frame_rectangle, 1.1, 2, 0, Size(30, 30) ); , as an example.

Template Matching for Coins with OpenCV

I am undertaking a project that will automatically count values of coins from an input image. So far I have segmented the coins using some pre-processing with edge detection and using the Hough-Transform.
My question is how do I proceed from here? I need to do some template matching on the segmented images based on some previously stored features. How can I go about doing this.
I have also read about something called K-Nearest Neighbours and I feel it is something I should be using. But I am not too sure how to go about using it.
Research articles I have followed:
Coin
Detector
Coin
Recognition
One way of doing pattern matching is using cv::matchTemplate.
This takes an input image and a smaller image which acts as template. It compares the template against overlapped image regions computing the similarity of the template with the overlapped region. Several methods for computing the comparision are available.
This methods does not directly support scale or orientation invariance. But it is possible to overcome that by scaling candidates to a reference size and by testing against several rotated templates.
A detailed example of this technique is shown to detect pressence and location of 50c coins. The same procedure can be applied to the other coins.
Two programs will be built. One to create templates from the big image template for the 50c coin. And another one which will take as input those templates as well as the image with coins and will output an image where the 50c coin(s) are labelled.
Template Maker
#define TEMPLATE_IMG "50c.jpg"
#define ANGLE_STEP 30
int main()
{
cv::Mat image = loadImage(TEMPLATE_IMG);
cv::Mat mask = createMask( image );
cv::Mat loc = locate( mask );
cv::Mat imageCS;
cv::Mat maskCS;
centerAndScale( image, mask, loc, imageCS, maskCS);
saveRotatedTemplates( imageCS, maskCS, ANGLE_STEP );
return 0;
}
Here we load the image which will be used to construct our templates.
Segment it to create a mask.
Locate the center of masses of said mask.
And we rescale and copy that mask and the coin so that they ocupy a square of fixed size where the edges of the square are touching the circunference of mask and coin. That is, the side of the square has the same lenght in pixels as the diameter of the scaled mask or coin image.
Finally we save that scaled and centered image of the coin. And we save further copies of it rotated in fixed angle increments.
cv::Mat loadImage(const char* name)
{
cv::Mat image;
image = cv::imread(name);
if ( image.data==NULL || image.channels()!=3 )
{
std::cout << name << " could not be read or is not correct." << std::endl;
exit(1);
}
return image;
}
loadImage uses cv::imread to read the image. Verifies that data has been read and the image has three channels and returns the read image.
#define THRESHOLD_BLUE 130
#define THRESHOLD_TYPE_BLUE cv::THRESH_BINARY_INV
#define THRESHOLD_GREEN 230
#define THRESHOLD_TYPE_GREEN cv::THRESH_BINARY_INV
#define THRESHOLD_RED 140
#define THRESHOLD_TYPE_RED cv::THRESH_BINARY
#define CLOSE_ITERATIONS 5
cv::Mat createMask(const cv::Mat& image)
{
cv::Mat channels[3];
cv::split( image, channels);
cv::Mat mask[3];
cv::threshold( channels[0], mask[0], THRESHOLD_BLUE , 255, THRESHOLD_TYPE_BLUE );
cv::threshold( channels[1], mask[1], THRESHOLD_GREEN, 255, THRESHOLD_TYPE_GREEN );
cv::threshold( channels[2], mask[2], THRESHOLD_RED , 255, THRESHOLD_TYPE_RED );
cv::Mat compositeMask;
cv::bitwise_and( mask[0], mask[1], compositeMask);
cv::bitwise_and( compositeMask, mask[2], compositeMask);
cv::morphologyEx(compositeMask, compositeMask, cv::MORPH_CLOSE,
cv::Mat(), cv::Point(-1, -1), CLOSE_ITERATIONS );
/// Next three lines only for debugging, may be removed
cv::Mat filtered;
image.copyTo( filtered, compositeMask );
cv::imwrite( "filtered.jpg", filtered);
return compositeMask;
}
createMask does the segmentation of the template. It binarizes each of the BGR channels, does the AND of those three binarized images and performs a CLOSE morphologic operation to produce the mask.
The three debug lines copy the original image into a black one using the computed mask as a mask for the copy operation. This helped in chosing the proper values for the threshold.
Here we can see the 50c image filtered by the mask created in createMask
cv::Mat locate( const cv::Mat& mask )
{
// Compute center and radius.
cv::Moments moments = cv::moments( mask, true);
float area = moments.m00;
float radius = sqrt( area/M_PI );
float xCentroid = moments.m10/moments.m00;
float yCentroid = moments.m01/moments.m00;
float m[1][3] = {{ xCentroid, yCentroid, radius}};
return cv::Mat(1, 3, CV_32F, m);
}
locate computes the center of mass of the mask and its radius. Returning those 3 values in a single row mat in the form { x, y, radius }.
It uses cv::moments which calculates all of the moments up to the third order of a polygon or rasterized shape. A rasterized shape in our case. We are not interested in all of those moments. But three of them are useful here. M00 is the area of the mask. And the centroid can be calculated from m00, m10 and m01.
void centerAndScale(const cv::Mat& image, const cv::Mat& mask,
const cv::Mat& characteristics,
cv::Mat& imageCS, cv::Mat& maskCS)
{
float radius = characteristics.at<float>(0,2);
float xCenter = characteristics.at<float>(0,0);
float yCenter = characteristics.at<float>(0,1);
int diameter = round(radius*2);
int xOrg = round(xCenter-radius);
int yOrg = round(yCenter-radius);
cv::Rect roiOrg = cv::Rect( xOrg, yOrg, diameter, diameter );
cv::Mat roiImg = image(roiOrg);
cv::Mat roiMask = mask(roiOrg);
cv::Mat centered = cv::Mat::zeros( diameter, diameter, CV_8UC3);
roiImg.copyTo( centered, roiMask);
cv::imwrite( "centered.bmp", centered); // debug
imageCS.create( TEMPLATE_SIZE, TEMPLATE_SIZE, CV_8UC3);
cv::resize( centered, imageCS, cv::Size(TEMPLATE_SIZE,TEMPLATE_SIZE), 0, 0 );
cv::imwrite( "scaled.bmp", imageCS); // debug
roiMask.copyTo(centered);
cv::resize( centered, maskCS, cv::Size(TEMPLATE_SIZE,TEMPLATE_SIZE), 0, 0 );
}
centerAndScale uses the centroid and radius computed by locate to get a region of interest of the input image and a region of interest of the mask such that the center of the such regions is also the center of the coin and mask and the side length of the regions are equal to the diameter of the coin/mask.
These regions are later scaled to a fixed TEMPLATE_SIZE. This scaled region will be our reference template. When later on in the matching program we want to check if a detected candidate coin is this coin we will also take a region of the candidate coin, center and scale that candidate coin in the same way before performing template matching. This way we achieve scale invariance.
void saveRotatedTemplates( const cv::Mat& image, const cv::Mat& mask, int stepAngle )
{
char name[1000];
cv::Mat rotated( TEMPLATE_SIZE, TEMPLATE_SIZE, CV_8UC3 );
for ( int angle=0; angle<360; angle+=stepAngle )
{
cv::Point2f center( TEMPLATE_SIZE/2, TEMPLATE_SIZE/2);
cv::Mat r = cv::getRotationMatrix2D(center, angle, 1.0);
cv::warpAffine(image, rotated, r, cv::Size(TEMPLATE_SIZE, TEMPLATE_SIZE));
sprintf( name, "template-%03d.bmp", angle);
cv::imwrite( name, rotated );
cv::warpAffine(mask, rotated, r, cv::Size(TEMPLATE_SIZE, TEMPLATE_SIZE));
sprintf( name, "templateMask-%03d.bmp", angle);
cv::imwrite( name, rotated );
}
}
saveRotatedTemplates saves the previous computed template.
But it saves several copies of it, each one rotated by an angle, defined in ANGLE_STEP. The goal of this is to provide orientation invariance. The lower that we define stepAngle the better orientation invariance we get but it also implies a higher computational cost.
You may download the whole template maker program here.
When run with ANGLE_STEP as 30 I get the following 12 templates :
Template Matching.
#define INPUT_IMAGE "coins.jpg"
#define LABELED_IMAGE "coins_with50cLabeled.bmp"
#define LABEL "50c"
#define MATCH_THRESHOLD 0.065
#define ANGLE_STEP 30
int main()
{
vector<cv::Mat> templates;
loadTemplates( templates, ANGLE_STEP );
cv::Mat image = loadImage( INPUT_IMAGE );
cv::Mat mask = createMask( image );
vector<Candidate> candidates;
getCandidates( image, mask, candidates );
saveCandidates( candidates ); // debug
matchCandidates( templates, candidates );
for (int n = 0; n < candidates.size( ); ++n)
std::cout << candidates[n].score << std::endl;
cv::Mat labeledImg = labelCoins( image, candidates, MATCH_THRESHOLD, false, LABEL );
cv::imwrite( LABELED_IMAGE, labeledImg );
return 0;
}
The goal here is to read the templates and the image to be examined and determine the location of coins which match our template.
First we read into a vector of images all the template images we produced in the previous program.
Then we read the image to be examined.
Then we binarize the image to be examined using exactly the same function as in the template maker.
getCandidates locates the groups of points which are toghether forming a polygon. Each of these polygons is a candidate for coin. And all of them are rescaled and centered in a square of size equal to that of our templates so that we can perform matching in a way invariant to scale.
We save the candidate images obtained for debugging and tuning purposes.
matchCandidates matches each candidate with all the templates storing for each the result of the best match. Since we have templates for several orientations this provides invariance to orientation.
Scores of each candidate are printed so we can decide on a threshold to separate 50c coins from non 50c coins.
labelCoins copies the original image and draws a label over the ones which have a score greater than (or lesser than for some methods) the threshold defined in MATCH_THRESHOLD.
And finally we save the result in a .BMP
void loadTemplates(vector<cv::Mat>& templates, int angleStep)
{
templates.clear( );
for (int angle = 0; angle < 360; angle += angleStep)
{
char name[1000];
sprintf( name, "template-%03d.bmp", angle );
cv::Mat templateImg = cv::imread( name );
if (templateImg.data == NULL)
{
std::cout << "Could not read " << name << std::endl;
exit( 1 );
}
templates.push_back( templateImg );
}
}
loadTemplates is similar to loadImage. But it loads several images instead of just one and stores them in a std::vector.
loadImage is exactly the same as in the template maker.
createMask is also exactly the same as in the tempate maker. This time we apply it to the image with several coins. It should be noted that binarization thresholds were chosen to binarize the 50c and those will not work properly to binarize all the coins in the image. But that is of no consequence since the program objective is only to identify 50c coins. As long as those are properly segmented we are fine. It actually works in our favour if some coins are lost in this segmentation since we will save time evaluating them (as long as we only lose coins which are not 50c).
typedef struct Candidate
{
cv::Mat image;
float x;
float y;
float radius;
float score;
} Candidate;
void getCandidates(const cv::Mat& image, const cv::Mat& mask,
vector<Candidate>& candidates)
{
vector<vector<cv::Point> > contours;
vector<cv::Vec4i> hierarchy;
/// Find contours
cv::Mat maskCopy;
mask.copyTo( maskCopy );
cv::findContours( maskCopy, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, cv::Point( 0, 0 ) );
cv::Mat maskCS;
cv::Mat imageCS;
cv::Scalar white = cv::Scalar( 255 );
for (int nContour = 0; nContour < contours.size( ); ++nContour)
{
/// Draw contour
cv::Mat drawing = cv::Mat::zeros( mask.size( ), CV_8UC1 );
cv::drawContours( drawing, contours, nContour, white, -1, 8, hierarchy, 0, cv::Point( ) );
// Compute center and radius and area.
// Discard small areas.
cv::Moments moments = cv::moments( drawing, true );
float area = moments.m00;
if (area < CANDIDATES_MIN_AREA)
continue;
Candidate candidate;
candidate.radius = sqrt( area / M_PI );
candidate.x = moments.m10 / moments.m00;
candidate.y = moments.m01 / moments.m00;
float m[1][3] = {
{ candidate.x, candidate.y, candidate.radius}
};
cv::Mat characteristics( 1, 3, CV_32F, m );
centerAndScale( image, drawing, characteristics, imageCS, maskCS );
imageCS.copyTo( candidate.image );
candidates.push_back( candidate );
}
}
The heart of getCandidates is cv::findContours which finds the contours of areas present in its input image. Which here is the mask previously computed.
findContours returns a vector of contours. Each contour itself being a vector of points which form the outer line of the detected polygon.
Each polygon delimites the region of each candidate coin.
For each contour we use cv::drawContours to draw the filled polygon over a black image.
With this drawn image we use the same procedure earlier explained to compute centroid and radius of the polygon.
And we use centerAndScale, the same function used in the template maker, to center and scale the image contained in that poligon in an image which will have the same size as our templates. This way we will later on be able to perform a proper matching even for coins from photos of different scales.
Each of these candidate coins is copied in a Candidate structure which contains :
Candidate image
x and y for centroid
radius
score
getCandidates computes all these values except for score.
After composing the candidate it is put in a vector of candidates which is the result we get from getCandidates.
These are the 4 candidates obtained :
void saveCandidates(const vector<Candidate>& candidates)
{
for (int n = 0; n < candidates.size( ); ++n)
{
char name[1000];
sprintf( name, "Candidate-%03d.bmp", n );
cv::imwrite( name, candidates[n].image );
}
}
saveCandidates saves the computed candidates for debugging purpouses. And also so that I may post those images here.
void matchCandidates(const vector<cv::Mat>& templates,
vector<Candidate>& candidates)
{
for (auto it = candidates.begin( ); it != candidates.end( ); ++it)
matchCandidate( templates, *it );
}
matchCandidates just calls matchCandidate for each candidate. After completion we will have the score for all candidates computed.
void matchCandidate(const vector<cv::Mat>& templates, Candidate& candidate)
{
/// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better
candidate.score;
if (MATCH_METHOD == CV_TM_SQDIFF || MATCH_METHOD == CV_TM_SQDIFF_NORMED)
candidate.score = FLT_MAX;
else
candidate.score = 0;
for (auto it = templates.begin( ); it != templates.end( ); ++it)
{
float score = singleTemplateMatch( *it, candidate.image );
if (MATCH_METHOD == CV_TM_SQDIFF || MATCH_METHOD == CV_TM_SQDIFF_NORMED)
{
if (score < candidate.score)
candidate.score = score;
}
else
{
if (score > candidate.score)
candidate.score = score;
}
}
}
matchCandidate has as input a single candidate and all the templates. It's goal is to match each template against the candidate. That work is delegated to singleTemplateMatch.
We store the best score obtained, which for CV_TM_SQDIFF and CV_TM_SQDIFF_NORMED is the smallest one and for the other matching methods is the biggest one.
float singleTemplateMatch(const cv::Mat& templateImg, const cv::Mat& candidateImg)
{
cv::Mat result( 1, 1, CV_8UC1 );
cv::matchTemplate( candidateImg, templateImg, result, MATCH_METHOD );
return result.at<float>( 0, 0 );
}
singleTemplateMatch peforms the matching.
cv::matchTemplate uses two imput images, the second smaller or equal in size to the first one.
The common use case is for a small template (2nd parameter) to be matched against a larger image (1st parameter) and the result is a bidimensional Mat of floats with the matching of the template along the image. Locating the maximun (or minimun depending on the method) of this Mat of floats we get the best candidate position for our template in the image of the 1st parameter.
But we are not interested in locating our template in the image, we already have the coordinates of our candidates.
What we want is to get a measure of similitude between our candidate and template. Which is why we use cv::matchTemplate in a way which is less usual; we do so with a 1st parameter image of size equal to the 2nd parameter template. In this situation the result is a Mat of size 1x1. And the single value in that Mat is our score of similitude (or dissimilitude).
for (int n = 0; n < candidates.size( ); ++n)
std::cout << candidates[n].score << std::endl;
We print the scores obtained for each of our candidates.
In this table we can see the scores for each of the methods available for cv::matchTemplate. The best score is in green.
CCORR and CCOEFF give a wrong result, so those two are discarded. Of the remaining 4 methods the two SQDIFF methods are the ones with higher relative difference between the best match (which is a 50c) and the 2nd best (which is not a 50c). Which is why I have choosen them.
I have chosen SQDIFF_NORMED but there is no strong reason for that. In order to really chose a method we should test with a higher ammount of samples, not just one.
For this method a working threshold could be 0.065. Selection of a proper threshold also requires many samples.
bool selected(const Candidate& candidate, float threshold)
{
/// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better
if (MATCH_METHOD == CV_TM_SQDIFF || MATCH_METHOD == CV_TM_SQDIFF_NORMED)
return candidate.score <= threshold;
else
return candidate.score>threshold;
}
void drawLabel(const Candidate& candidate, const char* label, cv::Mat image)
{
int x = candidate.x - candidate.radius;
int y = candidate.y;
cv::Point point( x, y );
cv::Scalar blue( 255, 128, 128 );
cv::putText( image, label, point, CV_FONT_HERSHEY_SIMPLEX, 1.5f, blue, 2 );
}
cv::Mat labelCoins(const cv::Mat& image, const vector<Candidate>& candidates,
float threshold, bool inverseThreshold, const char* label)
{
cv::Mat imageLabeled;
image.copyTo( imageLabeled );
for (auto it = candidates.begin( ); it != candidates.end( ); ++it)
{
if (selected( *it, threshold ))
drawLabel( *it, label, imageLabeled );
}
return imageLabeled;
}
labelCoins draws a label string at the location of candidates with a score bigger than ( or lesser than depending on the method) the threshold.
And finally the result of labelCoins is saved with
cv::imwrite( LABELED_IMAGE, labeledImg );
The result being :
The whole code for the coin matcher can be downloaded here.
Is this a good method?
That is hard to tell.
The method is consistent. It correctly detects the 50c coin for the sample and input image provided.
But we have no idea if the method is robust because it has not been tested with a proper sample size. And even more important is to test it against samples which were not available when the program was being coded, that is the true measure of robustness when done with a large enough sample size.
I am rather confident in the method not having false positives from silver coins. But I am not so sure about other copper coins like the 20c. As we can see from the scores obtained the 20c coin gets a score very similar to the 50c.
It is also quite possible that false negatives will happen under varying lighting conditions. Which is something that can and should be avoided if we have control over lighting conditions such as when we are designing a machine to take photos of coins and count them.
If the method works the same method can be repeated for each type of coin leading to full detection of all coins.
Code in this answer is also available under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
If you detect all coins correctly Its better to use size(radial) and RGB features to recognize its value. Its not a good idea that concatenate these features because their number are not equal ( size is one number and number of RGB features are much larger than one). I recommend you to use two classifier for this purpose. One for size and another for RGB features.
You have to classify all coins into for example 3 (It depends on type
of your coins) size class. You can do this with a simple 1NN
classifier (just calculate the radial of test coin and classify it to
nearest predefined radial)
Then you should have some templates in each size and use template matching to recognize its value.(all templates and detected coins should be resize to a particular size. e.g. (100,100) ) For template
matching you can use matchtemplate function. I thing that the CV_TM_CCOEFF method may be the best one, but you can test all methods
to get a good result. (Note you don't need to search on image for coin because you detect the coin previously as you mentioned in your
question. You just need to use this function to get one number as a similarity/difference between two image and classify the test coin to a class which the similarity is maximized or difference is minimized)
EDIT1: You should have all rotations in your templates in each class to compensate the rotation of test coin.
EDIT2: If all coins are in different sizes the first step is enough. Otherwise you should patch the similar sizes to one class and classify the test coin using the second step (RGB features).
(1) Find the coins edge, using Hough Transform Algorithm.
(2) Determine the origin dot of the coins. I don't know how you'll do this.
(3) You can use k from KNN Algorithm for comparing the diameter or of the coins. Don't forget to set the bias value.
You could try and set up a training set of coin images and generate SIFT/SURF etc. descriptors of it. (EDIT: OpenCV feature detectors
Using these data you could set up a kNN classifier, using the coins values as training labels.
Once you perform kNN classification on you segmented coin images, your classification result would yield the coins value.

How to detect the Sun from the space sky in OpenCv?

I need to detect the Sun from the space sky.
These are examples of the input images:
I've got such results after Morphologic filtering ( open operation for twice )
Here's the algorithm code of this processing:
// Color to Gray
cvCvtColor(image, gray, CV_RGB2GRAY);
// color threshold
cvThreshold(gray,gray,150,255,CV_THRESH_BINARY);
// Morphologic open for 2 times
cvMorphologyEx( gray, dst, NULL, CV_SHAPE_RECT, CV_MOP_OPEN, 2);
Isn't it too heavy processing for such a simple task? And how to find the center of the Sun? If I find white points, than I'll find white points of big Earth ( left top corner on first example image )
Please advise me please my further action to detect the Sun.
UPDATE 1:
Trying algorithm of getting centroid by formula : {x,y} = {M10/M00, M01/M00}
CvMoments moments;
cvMoments(dst, &moments, 1);
double m00, m10, m01;
m00 = cvGetSpatialMoment(&moments, 0,0);
m10 = cvGetSpatialMoment(&moments, 1,0);
m01 = cvGetSpatialMoment(&moments, 0,1);
// calculating centroid
float centroid_x = m10/m00;
float centroid_y = m01/m00;
cvCircle( image,
cvPoint(cvRound(centroid_x), cvRound(centroid_y)),
50, CV_RGB(125,125,0), 4, 8,0);
And where Earth is in the photo, I got such a result:
So, centroid is on the Earth. :(
UPDATE 2:
Trying cvHoughCircles:
CvMemStorage* storage = cvCreateMemStorage(0);
CvSeq* circles = cvHoughCircles(dst, storage, CV_HOUGH_GRADIENT, 12,
dst->width/2, 255, 100, 0, 35);
if ( circles->total > 0 ) {
// getting first found circle
float* circle = (float*)cvGetSeqElem( circles, 0 );
// Drawing:
// green center dot
cvCircle( image, cvPoint(cvRound(circle[0]),cvRound(circle[1])),
3, CV_RGB(0,255,0), -1, 8, 0 );
// wrapping red circle
cvCircle( image, cvPoint(cvRound(circle[0]),cvRound(circle[1])),
cvRound(circle[2]), CV_RGB(255,0,0), 3, 8, 0 );
}
First example: bingo, but the second - no ;(
I've tried different configuration of cvHoughCircles() - couldn't find configuration to fit every my example photo.
UPDATE3:
matchTemplate approach worked for me ( response of mevatron ). It worked with big number of tests.
How about trying a simple matchTemplate approach. I used this template image:
And, it detected the 3 out of 3 of the sun images I tried:
This should work due to the fact that circles (in your case the sun) are rotationally invariant, and since you are so far away from the sun it should be roughly scale invariant as well. So, template matching will work quite nicely here.
Finally, here is the code that I used to do this:
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
using namespace cv;
using namespace std;
int main(int argc, char* argv[])
{
/// Load image and template
string inputName = "sun2.png";
string outputName = "sun2_detect.png";
Mat img = imread( inputName, 1 );
Mat templ = imread( "sun_templ.png", 1 );
/// Create the result matrix
int result_cols = img.cols - templ.cols + 1;
int result_rows = img.rows - templ.rows + 1;
Mat result( result_cols, result_rows, CV_32FC1 );
/// Do the Matching and Normalize
matchTemplate(img, templ, result, CV_TM_CCOEFF);
normalize(result, result, 0, 1, NORM_MINMAX, -1, Mat());
Point maxLoc;
minMaxLoc(result, NULL, NULL, NULL, &maxLoc);
rectangle(img, maxLoc, Point( maxLoc.x + templ.cols , maxLoc.y + templ.rows ), Scalar(0, 255, 0), 2);
rectangle(result, maxLoc, Point( maxLoc.x + templ.cols , maxLoc.y + templ.rows ), Scalar(0, 255, 0), 2);
imshow("img", img);
imshow("result", result);
imwrite(outputName, img);
waitKey(0);
return 0;
}
Hope you find that helpful!
Color Segmentation Approach
Do a color segmentation on the images to identify objects on the black background. You may identify the sun according to its area (given this uniquely identifies it, resp. don't varies largely accross images).
A more sophisticated approach could compute image moments, e.g. hu moments of the objects. See this page for these features.
Use a classification algorithm of your choice to do the actual classification of the objects found. The most simple approach is to manually specify thresholds, resp. value ranges that turn out to work for all(most) of your object/image combinations.
You may compute the actual position from the raw moments, as for the circular sun the position is equal to the center of mass
Centroid: {x, y } = { M10/M00, M01/M00 }
Edge Map Approach
Another option would be a circle hough transformation of the edge map, this will hopefully return some candidate circles (by position and radius). You may select the sun-circle according to the radius you expect (if you are lucky there is at most one).
A simple addition to your code is to filter out objects based on their size. If you always expect the earth to be much bigger than the sun, or the sun to have almost the same area in each picture, you can filter it by area.
Try Blob detector for this task.
And note that it may be good to apply a morphological opening/closing instead of simple erode or dilate, so your sun will have almost the same area before and after processing.