OpenCV Template Matching for N best matches not working - c++

I have been struggling with this for quite a long while and can't seem to find the problem here. Let me try to walk you through the process.
I am trying to obtain the 10 best bounding boxes for my template matching.
Not going to show the entire image here, but here's my cv::mat when I loop through the cv::mat to search for the lowest values (results)
int a,b;
for ( a = 0; a < final_image_height; a++){
for ( b = 0; b < final_image_width; b++){
if (result_scores_mat.at<float>(a, b) < 70 ){
printf("%.2f ", result_scores_mat.at<float>(a, b));
}
}
}
This gives me some values present in the cv::mat with "lower values"
68.50 68.93 54.50 68.92 64.62 57.12 62.69 65.86 63.52 68.35 68.65 61.93 69.18 67.69
I then move into a loop whereby I called minMaxLoc() to find the minVal (using TM_SQDIFF, so minVal)
for ( i = 0; i < 10; i++){
minMaxLoc(result_scores_mat, &minVal, &maxVal, &minLoc, &maxLoc, cv::Mat());
cout<<minVal<<endl;
// for match_method TM_SQDIFF we take lowest values
matchLoc = minLoc;
drawRadius = result_radius_mat.at<float>(matchLoc.x, matchLoc.y);
// display source image and result matrix , draw rectangle around highest possible matching area
cv::rectangle( img_display_mat, matchLoc, cv::Point( matchLoc.x + 2*drawRadius, matchLoc.y + 2*drawRadius), cv::Scalar::all(255), 2, 8, 0);
result_scores_mat.at<float>(minLoc.x, minLoc.y)= 255;
}
What I am doing in the last line is replacing the value of the float at the minLoc, so that the next loop I run through minMaxLoc doesn't locate that point and gives me the next minimum value in the cv::mat.
However, I keep receiving such an output with cout<<minVal<<endl;
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
54.5003
It detects rightly the 54.50 that is present in the cv::mat, but I want all the minimum values in the loop.
Is there something wrong I'm doing, or any way around this? Thank you so much!

It seems that you are replacing minimum value with 255 in wrong way, it means that you detect minimum value correctly, but you replace 255 in wrong position, you should replace below line
result_scores_mat.at<float>(minLoc.x, minLoc.y)= 255;
by this line:
result_scores_mat.at<float>(minLoc.y, minLoc.x)= 255;
For knowing that why we replace this lines by each other, you can take a look at the source code of .at method.It is like below
template<typename _Tp> _Tp& at(int row, int col);
/** #overload
#param row Index along the dimension 0
#param col Index along the dimension 1
*/

Related

OpenCV C++ - Find an image contained within an image?

I have code for searching one small image in bigger another one image:
int* MyLib::MatchingMethod(int, void*)
{
/// Source image to display
img.copyTo(img_display);
/// Create the result matrix
int result_cols = img.cols - templ.cols + 1;
int result_rows = img.rows - templ.rows + 1;
result.create(result_rows, result_cols, CV_32FC1);
match_method = 0;
/// Do the Matching and Normalize
matchTemplate(img, templ, result, match_method);
normalize(result, result, 0, 1, cv::NORM_MINMAX, -1, cv::Mat());
/// Localizing the best match with minMaxLoc
double minVal;
double maxVal;
cv::Point minLoc;
cv::Point maxLoc;
cv::Point matchLoc;
minMaxLoc(result, &minVal, &maxVal, &minLoc, &maxLoc, cv::Mat());
/// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better
if (match_method == CV_TM_SQDIFF || match_method == CV_TM_SQDIFF_NORMED)
{
matchLoc = minLoc;
}
else
{
matchLoc = maxLoc;
}
if (showOpenCVWindow) {
/// Show me what you got
rectangle(img_display, matchLoc, cv::Point(matchLoc.x + templ.cols, matchLoc.y + templ.rows), cv::Scalar(255, 0, 0, 255), 2, 8, 0);
rectangle(result, matchLoc, cv::Point(matchLoc.x + templ.cols, matchLoc.y + templ.rows), cv::Scalar(255, 0, 0, 255), 2, 8, 0);
imshow(image_window, img_display);
imshow(result_window, result);
}
double myX = (matchLoc.x + (templ.cols) / 2);
double myY = (matchLoc.y + (templ.rows) / 2);
static int o[2];
o[0] = myX;
o[1] = myY;
return o;
}
But this code could mistakenly "found" any area, even if bigger image doesn't contains small image.
How to change this code, to force it to "exactly" searching of the small image. For example, if smaller image is not on the bigger image, this code must show any info message "Image not found".
Update 1. It looks, like matchTemplate doesn't work good. For example, I have 3 images - one template ( http://s6.postimg.org/nj2ts3lf5/image.png ) , one image, that contains image from template ( http://s6.postimg.org/fp6tkg301/image.png ), and one image, that doesn't contains template ( http://s6.postimg.org/9x23zk3sh/image.png ).
For first image, that contains template, maxVal=0.99999994039535522 and it correctly selected area: http://s6.postimg.org/65x4qzfht/image.png
But for image, that doesn't contains template, maxVal=1.0000000000000000 and it incorrectly selected area, that doesn't contains template image: http://s6.postimg.org/5132llt0x/screenshot_544.png
Thank you!
You are visualizing the result regardless of the certainty with which the algorithm performed matching. Template matching will always give you an output - what you want to do is to try to figure out if it's valid or not.
Try outputing minVal or maxVal depending on the match_method. You should compare the value in the cases when the correct match was found and in the cases when it gave you a false positive. Those experiments should allow you to establish a threshold, that distinguishes between true hits and false positives. Thus, you will be able to say how big - for example - the maxVal has to be to be sure that it was a match. Pseudo code would go something like this:
if maxVal > threshold:
match_found = true
match_position = maxLoc
Now that's a theoretical approach. Since you didn't provide any images, it might or might not be the solution for your problem.
EDIT:
If you cannot find a definite threshold value (which in my opinion should be possible in most cases, if you maintain quality, size, etc), try doing one of two things:
Try looking at all obtained results, before minMaxLoc, calculate the mean value and see if the maxVal found is much bigger than the mean value in the true positive cases. Maybe you can define the threshold as the % of the mean value, thus saying: if maxVal > meanVal + meanVal * n%: match_found = true
It is a common situation, that template matching works better with edges than with the real image. Again, you haven't provided samples, so it's hard to say how reliable will that approach be here. But if you have enough high frequencies, to light up an image with Canny Edges, that might give you a much clearer threshold for discriminating between true and false positives.
EDIT2:
Since you're using match_method = 0, that means CV_TM_SQDIFF. For more control over the process, use the name explicitly. Find information on the methods here.
Also, put the cout inside the if statement, so that you print the correct value, that actually idicates the match (in your case, it's minVal).
if (match_method == CV_TM_SQDIFF || match_method == CV_TM_SQDIFF_NORMED)
{
matchLoc = minLoc;
std::cout << minVal << std::endl;
}
else
{
matchLoc = maxLoc;
std::cout << maxVal << std::endl;
}
And again: fairly tuned contours detection should almost certainly help if this doesn't give you the expected results.

Why does StereoSGBM give negative and above numberOfDisparities numbers

I'm writing a function in OpenCV to compute v and u-disparities, so I need first the disparity image. I set sgbm.minDisparity = 0 and numberOfDisparities = 160.
The disparity image is CV_16SC1, and I need Unsigned values to go on programming my function.
I printed the whole Mat and there are negative values and values above 160. If I understood well the documentation, the disparity image represents the disparity values*16. Does that mean that the maximum value is 16*160 in my case?. If not, what could be wrong?. And anyway, why there are values less than zero when minDisparity is set to 0? Here's the code:
void Stereo_SGBM(){
int numberOfDisparities;
StereoSGBM sgbm;
Mat img1, img2;
img1=left_frame; //left and right frames are global variables
img2=right_frame;
Size img_size = img1.size();
//I make sure the number of disparities is divisible by 16
numberOfDisparities = 160;
int cn=1; //Grayscale
sgbm.preFilterCap = 63;
sgbm.SADWindowSize = 3;
sgbm.P1 = 8*cn*sgbm.SADWindowSize*sgbm.SADWindowSize;
sgbm.P2 = 32*cn*sgbm.SADWindowSize*sgbm.SADWindowSize;
sgbm.minDisparity = 0;
sgbm.numberOfDisparities = numberOfDisparities;
sgbm.uniquenessRatio = 10;
sgbm.speckleWindowSize = 100;
sgbm.speckleRange = 2;
sgbm.disp12MaxDiff = 1;
sgbm.fullDP = false;
Mat disp; // CV_16SC1
Mat disp8; //CV_8UC1 (used later in the code
sgbm(img1, img2, disp);
//disp contains negative values and larger than 160!!!
//img1 and img2 are left and right channels of size 1242x375 grayscale
}
The way I see it, the disparity is meant to be a float, and it reflects on the parameters. If you convert the result to float, and divide by 16, things makes a little more sense:
The algorithm apparently reports -1 (actually minDisparity - 1) where it could not match. And numberOfDisparities is more "max disparity - min disparity", rather than an actual number of values.
For example, if you give minDisparity=2 and numberOfDisparities=144, you will get results in the range: 1.0 - 145.0. The number of different values will actually be 144*16 because it goes in 1/16 increments.
So yes, in your case, using integers, this means you will get 16*160 max value.

OpenCV Sobel Filters resulting in almost completely black images

I am having some issues with my sobel_y (and sobel_x, but I figure they are having the same issue) filter in that it keeps giving me an image that it basically only black and white. I am having to rewrite this function for a class, so no I cannot use the built-in one, and had it working, minus some minor tweaks because the output image looked a little strange with still being black and white even though it was supposed to be converted back. I figured out how to fix that, and in the process I messed with something and broke it and cannot seem to get it back to working even with the black and white image output only. I keep getting a black image, with some white lines here and there near the top. I have tried changing the Mat grayscale type (third parameter) to all different values, as my professor mentioned in the class that we are using 32 bit floating point images, but that did not help either.
Even though the issue occurs after running the Studentfilter2D, I think it is a problem with the grayscaling of the image, although whenever I debug, it seems to work just fine. This is also because I have 2 other filtering functions I had to write that use Studentfilter2D, and they both give me the expected results. My sobel_y function is shown below:
// Convert the image in bgr to grayscale OK to use the OpenCV function.
// Find the coefficients used by the OpenCV function, and give a link where you found it.
// Note: This student function expects the matrix gray to be preallocated with the same width and
// height, but with 1 channel.
void BGR2Gray(Mat& bgr, Mat& gray)
{
// Y = .299 * R + .587 * G + .114 * B, from http://docs.opencv.org/modules/imgproc/doc/miscellaneous_transformations.html#cvtcolor
// Some extra assistance, for the third parameter for the InputArray, from http://docs.opencv.org/trunk/modules/core/doc/basic_structures.html#inputarray
// Not sure about the fourth parameter, but was just trying it to see if that may be the issue as well
cvtColor(bgr, gray, CV_BGR2GRAY, 1);
return;
}
// Convolve image with kernel - this routine will be called from the other
// subroutines! (gaussian, sobel_x and sobel_y)
// image is single channel. Do not use the OpenCV filter2D!!
// Implementation can be with the .at or similar to the
// basic method found in the Chapter 2 of the OpenCV tutorial in CANVAS,
// or online at the OpenCV documentation here:
// http://docs.opencv.org/doc/tutorials/core/mat-mask-operations/mat-mask operations.html
// In our code the image and the kernel are both floats (so the sample code will need to change)
void Studentfilter2D (Mat& image, Mat& kernel)
{
int kCenterX = kernel.cols / 2;
int kCenterY = kernel.rows / 2;
// Algorithm help from http://www.songho.ca/dsp/convolution/convolution.html
for (int iRows = 0; iRows < image.rows; iRows++)
{
for (int iCols = 0; iCols < image.cols; iCols++)
{
float result = 0.0;
for (int kRows = 0; kRows < kernel.rows; kRows++)
{
// Flip the rows for the convolution
int kRowsFlipped = kernel.rows - 1 - kRows;
for (int kCols = 0; kCols < kernel.cols; kCols++)
{
// Flip the columns for the convolution
int kColsFlipped = kernel.cols - 1 - kCols;
// Indices of shifting around the convolution
int iRowsIndex = iRows + kRows - kCenterY;
int iColsIndex = iCols + kCols - kCenterX;
// Check bounds using the indices
if (iRowsIndex >= 0 && iRowsIndex < image.rows && iColsIndex >= 0 && iColsIndex < image.cols)
{
result += image.at<float>(iRowsIndex, iColsIndex) * kernel.at<float>(kRowsFlipped, kColsFlipped);
}
}
}
image.at<float>(iRows, iCols) = result;
}
}
return;
}
void sobel_y (Mat& image, int)
{
// Note, the filter parameter int is unused.
Mat mask = (Mat_<float>(3, 3) << 1, 2, 1,
0, 0, 0,
-1, -2, -1) / 3;
//Mat grayscale(image.rows, image.cols, CV_32FC1);
BGR2Gray(image, image);
Studentfilter2D(image, mask);
// Here is the documentation on normalize http://docs.opencv.org/modules/core/doc/operations_on_arrays.html#normalize
normalize(image, image, 0, 1, NORM_MINMAX);
cvtColor(image, image, CV_GRAY2BGR);
return;
}
Like I said, I had this working before, just looking for some fresh eyes to look at it and see what I may be missing. I have been looking at this same code so much for the past 4 days that I think I am just missing things. In case anyone is wondering, I have also tried changing the mask values of the filter, but to no avail.
There are two things that are worth mentioning.
The first is that you are not taking proper care of the type of your matrices/images.
The input to Studentfilter2D in sobel_y is an 8-bit grayscale image of type CV_8UC1 meaning that the data is an array of unsigned char.
Your Studentfilter2D function, however, is indexing this input image as though it was of type float. This means it is picking the wrong pixels to work with.
If the above does not immediately solve your problem, you should consider the range of your final derivative image. Since it is a derivative it will no longer be in the range [0, 255]. Instead, it might even contain negative numbers. When you try to visualize this, you will run into problems unless you first normalize your image.
There are built in functions to do this in OpenCV if you look around in the documentation.

Thresholding a range of colors from an image

The plan
My project is able to capture the bitmap of a target window and convert it into an IplImage, and then display that image in a cvNamedWindow, where further processing can take place.
For the sake of testing, I've loaded an image into MSPaint like so:
The user is then allowed to click and drag the mouse over any number of pixels within the image to create a vector<cv::Scalar_<BYTE>> containing these RGB color values.
Then, with the help of ColorRGBToHLS(), this array is then sorted from left to right by hue, like so:
// PixelColor is just a cv::Scalar_<BYTE>
bool comparePixelColors( PixelColor& pc1, PixelColor& pc2 ) {
WORD h1 = 0, h2 = 0;
WORD s1 = 0, s2 = 0;
WORD l1 = 0, l2 = 0;
ColorRGBToHLS(RGB(pc1.val[2], pc1.val[1], pc1.val[0]), &h1, &l1, &s1);
ColorRGBToHLS(RGB(pc2.val[2], pc2.val[1], pc2.val[0]), &h2, &l2, &s2);
return ( h1 < h2 );
}
//..(elsewhere in code)
std::sort(m_colorRange.begin(), m_colorRange.end(), comparePixelColors);
...and then shown in a new cvNamedWindow, which looks something like:
The problem
Now, the idea here is to create a binary threshold image (or "mask") where this selected range of colors become white, and the rest of the source image becomes black... similar to the way the "Select By Color" tool operates in GIMP, or the "magic wand" tool works in Photoshop... except instead of limiting ourselves to a specific contoured selection, we are literally operating on the image as a whole.
I've read into cvInRangeS, and it sounds like it's precisely what I need.
However, and for whatever reason, the thresholded image always ends up being totally black...
VOID ShowThreshedImage(const IplImage* src, const PixelColor& min, const PixelColor& max)
{
IplImage* imgHSV = cvCreateImage(cvGetSize(src), IPL_DEPTH_8U, 3);
cvCvtColor(src, imgHSV, CV_RGB2HLS);
cvNamedWindow("T1");
cvShowImage("T1", imgHSV); // <-- Shows up like the image below
IplImage* imgThreshed = cvCreateImage(cvGetSize(src), IPL_DEPTH_8U, 1);
cvInRangeS(imgHSV, min, max, imgThreshed);
cvNamedWindow("T2");
cvShowImage("T2", imgThreshed); // <-- SHOWS UP PITCH BLACK!
}
This is what the "T1" window ends up looking like (which I suppose is correct?):
Bearing in mind that because the color range vector is stored as RGB (and that OpenCV internally reverses this order into BGR), I have converted the min/max values into HLS before passing them into ShowThreshedImage() like so:
CvScalar rgbPixelToHSV(const PixelColor& pixelColor)
{
WORD h = 0, s = 0, l = 0;
ColorRGBToHLS(RGB(pixelColor.val[2], pixelColor.val[1], pixelColor.val[0]), &h, &l, &s);
return PixelColor(h, s, l);
}
//...(elsewhere in code)
if(m_colorRange.size() > 0)
m_minHSV = rgbPixelToHSV(m_colorRange[0]);
if(m_colorRange.size() > 1)
m_maxHSV = rgbPixelToHSV(m_colorRange[m_colorRange.size() - 1]);
ShowThreshedImage(m_imgSrc, m_minHSV, m_maxHSV);
...But even without this conversion and simply passing RGB values instead, the result is still an entirely black image. I've even tried manually plugging in certain min/max values, and the best result I got was a few lit pixels (albeit, the incorrect ones).
The question:
What am I doing wrong here?
Is there something that I don't understand about the cvInRangeS method?
Do I need to step through each and every single color in order to properly threshold the selected range out of the source image?
Are there any other ways of accomplishing this?
Thank you for your time.
Update:
I have discovered that cvInRangeS expects all values for min to be lower than that of max. But when a range of colors are selected, there doesn't appear to be any guarantee that this will be the case, often resulting in a black thresholded image.
And swapping values to enforce this rule may result in unwanted colors within the new range (in some cases, this could include all colors instead of just the desired ones).
So I suppose the real question here would be:
"How do you segment an array of RGB colors, and use them to threshold an image?"
Your problem might be caused by the simple fact that OpenCV maintains a different range for values than for instanc MSpaint. For instance the HSV color space in paint is 360,100,100 while in OpenCV it is 180,255,255. Check your input values in openCV bu outputting the pixel value when clicking on a certain pixel. inRangeS should be the correct tool for the job. That said, in RGB it should work just as well because the range is the same as in paint.
cvSetMouseCallback("MyWindow", mouseEvent, (void*) &myImage);
void mouseEvent(int evt, int x, int y, int flags, void *param) {
if (evt == CV_EVENT_LBUTTONDOWN) {
printf("%d %d\n", x, y);
IplImage* imageSource = (IplImage*) param;
Mat image(imageSource);
cout << "Image cols " << image.cols << " rows " << image.rows << endl;
Mat imageHSV;
cvtColor(image, imageHSV, CV_BGR2HSV);
Vec3b p = imageHSV.at<Vec3b > (y, x);
char text[20];
sprintf(text, "H=%d, S=%d, V=%d", p[0], p[1], p[2]);
cout << text << endl;
}
}
When you have an idea about the HSV values by using this values, use these as lower and upper bounds for the in range method after converting the image to HSV by using cvtColor(image, imageHSV, CV_BGR2HSV). That should make you able to get the desired result.
It is not going to be too inefficient to iterate through every pixel. That is exactly what cvInRangeS would do - see this: http://docs.opencv.org/doc/tutorials/core/how_to_scan_images/how_to_scan_images.html#the-efficient-way (I do this all the time and it is instantaneous for reasonable size images).
I would treat the color in the array as points in 3D RGB space. Find two color points that specify a prism that includes all other color points. That is just finding the min and max of all r,g, and b values. If this idea is not ok then you might have to check every image pixel against every pixel in the vector.
Then for each pixel in the image: result is black if (pixel.r < min.r) || (pixel.r > max.r) || (pixel.g < min.g) || (pixel.g > max.g) || (pixel.b < min.b) || (pixel.b > max.b), result is the pixel value otherwise.
This all should be very easy, so long as it is actually what you want.

Video Processing OpenCV?

I am trying to pass in a HSV frame from a video to the function, but the function does not seem to do anything to it. What am I doing wrong? The function is supposed to go through each pixel, and depending on its hue range supposed to make it black or white, leaving me with a binary image. Instead it doesn't seem to affect the HSV image at all....
Thanks
PS sorry for the bad code formatting, StackOverflow isn't allowing me to post the original format.
void sort (IplImage *skinmask)
{
for (int row=0; row<=skinmask->height;row++)
{
uchar* pixelrow=(uchar*)(skinmask->imageData+(row*(skinmask->widthStep)));
for (int column=0; column<=skinmask->width; column++)
{
if (6<pixelrow[3*column]<36)
{
pixelrow[3*column]=256;
pixelrow[(3*column)+1]=256;
pixelrow[(3*column)+2]=256;
}
else
{
pixelrow[3*column]=0;
pixelrow[(3*column)+1]=0;
pixelrow[(3*column)+2]=0;
}
column++;
}
row++;
}
cvMorphologyEx(skinmask,skinmask,NULL,NULL,CV_MOP_CLOSE,1);
}
Doing an operation like thresholding pixel-by-pixel is usually the wrong way to go about achieving this in OpenCV - there are functions that work on whole image arrays that are simpler and are already optimized for speed.
In this case try first splitting the image to separate out the H/S/V channels, then threshold on the Hue channel to get a mask (you may have to use the intersection of two masks, which you can do using a multiply or "bitwise and") - the resulting mask is your black and white image.
(I realise I've linked to the C++ documentation, but I'm sure you can find the equivalent functions in the old-style OpenCV docs)
Update
Ok, I'll try to write some code to show what I mean. I also found the function I was looking for, which is better than Threshold, it is InRangeS. This lets you put upper and lower bounds on all the channels at once, and it applies them all into your mask for you.
void HSVImageToMask(IplImage * image, cvMat * mask)
/* mask should be the same size as image, and of type CV_8UC1 */
/* e.g. cvMat * mask = cvCreateMat(image->width, image->height, CV8UC1); */
{
double hMin = 6;
double hMax = 36;
double sMin = 10; /* not sure what value you need */
double sMax = 245; /* not sure what value you need */
double vMin = 0;
double vMax = 255;
CvScalar hsvMin = cvScalar(hMin, sMin, vMin);
CvScalar hsvMax = cvScalar(hMax, sMax, vMax);
cvInRangeS(image, hsvMin, hsvMax, mask);
}
PS. I figured out the problem with your original code - you should be using 255 instead of 256 as your "white" value. This method is still better though :)
PPS. We didn't need them after all but for future reference:
"bitwise and":
cvAnd(const CvArr* src1, const CvArr* src2, CvArr* dst)
If you have two black and white masks, this will give you the intersection. Use cvOr to get the Union.