I'm currently making a program to track 4 paddles, with 3 different colors. I'm having trouble understanding how best to proceed, with the knowledge I have now, and how to reduce the computational costs of running the project. There are code examples of the steps listed at the end of this post.
The program contains a class file called Controllers, that has simple get and set functions for things such as X and Y position, and which HSV values are used for thresholding.
The program in its unoptimized state now does the following:
Reads image from webcam
Converts image to HSV colorspace
Uses the inRange function of OpenCV, together with some previously defined max/min values for HSV, to threshold the HSV image 3 times, one for each colored paddle. This saves to seperate Mat arrays.
(This step is problematic for me) - Performs erosion and dilation of EACH of the three thresholded images.
Passes the image into a function, that uses Moments to create a vector of point describing the contours, and then uses moments to calculate the X and Y location, which is saved as an object and pushed back into a vector of these paddle objects.
Everything technically works at this point, but the resources required to perform the morphological operations three times each loop through the while loop that reads images from the webcam is slowing the program immensely.(Applying 2 iterations of erosion, and 3 of dilation on 3 640*480 images at an acceptable frame rate.)
Threshold Images for different paddles
inRange(HSV, playerOne.getHSVmin(), playerOne.getHSVmax(), threshold1);
inRange(HSV, playerTwo.getHSVmin(), playerTwo.getHSVmax(), threshold2);
inRange(HSV, powerController.getHSVmin(), powerController.getHSVmax(), threshold3);
Perform morphological operations
morphOps(threshold1);
void morphOps(Mat &thresh)
{
//Create a structuring element to be used for morph operations.
Mat structuringElement = getStructuringElement(MORPH_RECT, Size(3,3));
Mat dilateElement = getStructuringElement(MORPH_RECT, Size(6, 6));
//Perform the morphological operations, using two/three iterations because the noise is horrible.
erode(thresh, thresh, structuringElement, Point(-1, -1), 3);
dilate(thresh, thresh, dilateElement, Point(-1, -1), 2);
}
Track the image
trackFilteredObject(playerOne, threshold1, cameraFeed);
trackFilteredObject(playerTwo, threshold2, cameraFeed);
trackFilteredObject(powerController, threshold3, cameraFeed);
void trackFilteredObject(Controllers theControllers, Mat threshold, Mat HSV, Mat &cameraFeed)
{
vector <Controllers> players;
Mat temp;
threshold.copyTo(temp);
//these vectors are needed to save the output of findCountours
vector< vector<Point> > contours;
vector<Vec4i> hierarchy;
//Find the contours of the image
findContours(temp, contours, hierarchy, CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE);
//Moments are used to find the filtered objects.
double refArea = 0;
bool objectFound = false;
if (hierarchy.size() > 0)
{
int numObjects = hierarchy.size();
//If there are more objects than the maximum number of objects we want to track, the filter may be noisy.
if (numObjects < MAX_NUM_OBJECTS)
{
for (int i = 0; i >= 0; i = hierarchy[i][0])
{
Moments moment = moments((Mat)contours[i]);
double area = moment.m00;
//If the area is less than min area, then it is probably noise
if (area > MIN_AREA)
{
Controllers player;
player.setXPos(moment.m10 / area);
player.setYPos(moment.m01 / area);
player.setType(theControllers.getType());
player.setColor(theControllers.getColor());
players.push_back(player);
objectFound = true;
}
else objectFound = false;
}
//Draw the object location on screen if an object is found
if (objectFound)
{
drawObject(players, cameraFeed);
}
}
}
}
The idea is that I want to be able to isolate each object, and use the X and Y positions as points of a triangle, and use the information to calculate angle and power of an arrow shot. So I want to know if there is a better way to isolate the colored paddles and remove the noise, that doesn't require me to perform these morphological operations for each color.
Related
I want to detect the bounding rectangle of an German ID card within an image by using OpenCV.
This is what my code looks like:
capture >> frame;
cv::resize(frame, frame, cv::Size(512,256));
cv::Mat grayScaledFrame, blurredFrame, cannyFrame;
cv::cvtColor(frame, grayScaledFrame, cv::COLOR_BGR2GRAY);
cv::GaussianBlur(grayScaledFrame, blurredFrame, cv::Size(9,9), 1);
cv::Canny(blurredFrame, cannyFrame, 40, 70);
// CONTOURS
std::vector<std::vector<cv::Point>> contours;
cv::findContours(cannyFrame, contours, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_SIMPLE);
// SORT
int maxArea = 0;
std::vector<cv::Point> contour;
for(int i = 0; i < contours.size(); i++) {
int thisArea = cv::contourArea(contours.at(i));
if(thisArea > maxArea) {
maxArea = thisArea;
contour = contours.at(i);
}
}
cv::Rect borderBox = cv::boundingRect(contour);
cv::rectangle(cannyFrame, borderBox, cv::Scalar{255, 32, 32}, 8);
cv::imshow("Webcam", cannyFrame);
The result looks like this:
RESULT
There are some rectangles detected but not the big one I'm interested in.
I've already tried different thresholds for Canny and also different kernel sizes for Gaussian Blur.
Best regards
First of all, as the environmental conditions change, the parameters of the code change, so it is necessary to standardize the environment (light, distance to the object, etc.).
To get this detection right, put the card at a fixed distance from the camera and calculate the area of the rectangles.
When the card is at a certain distance from the camera, you get approximate reference values of the card's area. Then, when drawing a rectangle, you use values within a specified tolerance range.
How to use the Hungarian algorithm to correlate the corresponding targets between two consecutive frames, and finally realize that it can judge whether the moving target in the next frame is already existing or new? I do not how to start in the actual implementation of the program using C++ and Opencv3 library?
Now I can get the centroids of the moving target detected by kalman filter in each frame.
//This is part of the main function.
kalmanv.clear();
initKalman(0, 0);
Point s, p;//s:kalmanCorrect,p:kalmanPredict
//variable definition
vector<vector<Point>> contours;//rectangular frame position around the contour
vector<Vec4i> hierarchy;
int count = 0;
Mat frame, gray, mogMask;
char numText[8];
while (capture.read(frame)) {
...
findContours(mogMask, contours, hierarchy, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE, Point(0, 0));
//definition of moment and center moment
vector<Moments> mu(contours.size());
vector<Point2f> mc(contours.size());
...
mu[t] = moments(contours[t], false);//calculate moment
mc[t] = Point2f(mu[t].m10 / mu[t].m00, mu[t].m01 / mu[t].m00);//calculate center moment
measurement(0) = mc[t].x;
measurement(1) = mc[t].y;
p = kalmanPredict();
Point center = Point(selection.x + (selection.width / 2), selection.y + (selection.height / 2));//calculate centroid
s = kalmanCorrect(center.x, center.y);
//I don't know what to do next.
}
I have got the set of centroids points for the moving target, and then how to use the data in conjunction with the Hungarian algorithm?
I'm using opencv 2.4.13
I'm trying to find the perimeter of a connected component, I was thinking of using ConnectedComponentWithStats but it doesn't return the perimeter, only the area, width, etc...
There is a method to find the area with the contour but not the opposite (with one component i mean, not the entire image).
The method arcLength doesn't work as well beause i have all the points of the component, not only the contour.
I know there is a BF way to find it by iterating through each pixel of the component and see if he has neighbors which aren't in the same component. But I'd like a function which costs less.
Otherwise, if you know a way to link a component with the contours found by the method findContours, it suits me as well.
Thanks
Adding to #Miki's answer, This is a faster way to find the perimeter of the connected component
//getting the connected components with statistics
cv::Mat1i labels, stats;
cv::Mat centroids;
int lab = connectedComponentsWithStats(img, labels, stats, centroids);
for (int i = 1; i < lab; ++i)
{
//Rectangle around the connected component
cv::Rect rect(stats(i, 0), stats(i, 1), stats(i, 2), stats(i, 3));
// Get the mask for the i-th contour
Mat1b mask_i = labels(rect) == i;
// Compute the contour
vector<vector<Point>> contours;
findContours(mask_i, contours, RETR_EXTERNAL, CHAIN_APPROX_NONE);
if(contours.size() <= 0)
continue;
//Finding the perimeter
double perimeter = contours[0].size();
//you can use this as well for measuring perimeter
//double perimeter = arcLength(contours[0], true);
}
The easiest thing is probably to use findContours.
You can compute the contour on the i-th component computed by connectedComponents(WithStats) , so they are aligned with your labels. Using CHAIN_APPROX_NONE you'll get all the points in the contour, so the size() of the vector is already a measure of the perimeter. You can eventually use arcLength(...) to get a more accurate result:
Mat1i labels;
int n_labels = connectedComponents(img, labels);
for (int i = 1; i < n_labels; ++i)
{
// Get the mask for the i-th contour
Mat1b mask_i = labels == i;
// Compute the contour
vector<vector<Point>> contours;
findContours(mask_i.clone(), contours, RETR_EXTERNAL, CHAIN_APPROX_NONE);
if (!contours.empty())
{
// The first contour (and probably the only one)
// is the one you're looking for
// Compute the perimeter
double perimeter_i = contours[0].size();
}
}
What I'm trying to do is measure the thickness of the eyeglasses frames. I had the idea to measure the thickness of the frame's contours (may be a better way?). I have so far outlined the frame of the glasses, but there are gaps where the lines don't meet. I thought about using HoughLinesP, but I'm not sure if this is what I need.
So far I have conducted the following steps:
Convert image to grayscale
Create ROI around the eye/glasses area
Blur the image
Dilate the image (have done this to remove any thin framed glasses)
Conduct Canny edge detection
Found contours
These are the results:
This is my code so far:
//convert to grayscale
cv::Mat grayscaleImg;
cv::cvtColor( img, grayscaleImg, CV_BGR2GRAY );
//create ROI
cv::Mat eyeAreaROI(grayscaleImg, centreEyesRect);
cv::imshow("roi", eyeAreaROI);
//blur
cv::Mat blurredROI;
cv::blur(eyeAreaROI, blurredROI, Size(3,3));
cv::imshow("blurred", blurredROI);
//dilate thin lines
cv::Mat dilated_dst;
int dilate_elem = 0;
int dilate_size = 1;
int dilate_type = MORPH_RECT;
cv::Mat element = getStructuringElement(dilate_type,
cv::Size(2*dilate_size + 1, 2*dilate_size+1),
cv::Point(dilate_size, dilate_size));
cv::dilate(blurredROI, dilated_dst, element);
cv::imshow("dilate", dilated_dst);
//edge detection
int lowThreshold = 100;
int ratio = 3;
int kernel_size = 3;
cv::Canny(dilated_dst, dilated_dst, lowThreshold, lowThreshold*ratio, kernel_size);
//create matrix of the same type and size as ROI
Mat dst;
dst.create(eyeAreaROI.size(), dilated_dst.type());
dst = Scalar::all(0);
dilated_dst.copyTo(dst, dilated_dst);
cv::imshow("edges", dst);
//join the lines and fill in
vector<Vec4i> hierarchy;
vector<vector<Point>> contours;
cv::findContours(dilated_dst, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE);
cv::imshow("contours", dilated_dst);
I'm not entirely sure what the next steps would be, or as I said above, if I should use HoughLinesP and how to implement it. Any help is very much appreciated!
I think there are 2 main problems.
segment the glasses frame
find the thickness of the segmented frame
I'll now post a way to segment the glasses of your sample image. Maybe this method will work for different images too, but you'll probably have to adjust parameters, or you might be able to use the main ideas.
Main idea is:
First, find the biggest contour in the image, which should be the glasses. Second, find the two biggest contours within the previous found biggest contour, which should be the glasses within the frame!
I use this image as input (which should be your blurred but not dilated image):
// this functions finds the biggest X contours. Probably there are faster ways, but it should work...
std::vector<std::vector<cv::Point>> findBiggestContours(std::vector<std::vector<cv::Point>> contours, int amount)
{
std::vector<std::vector<cv::Point>> sortedContours;
if(amount <= 0) amount = contours.size();
if(amount > contours.size()) amount = contours.size();
for(int chosen = 0; chosen < amount; )
{
double biggestContourArea = 0;
int biggestContourID = -1;
for(unsigned int i=0; i<contours.size() && contours.size(); ++i)
{
double tmpArea = cv::contourArea(contours[i]);
if(tmpArea > biggestContourArea)
{
biggestContourArea = tmpArea;
biggestContourID = i;
}
}
if(biggestContourID >= 0)
{
//std::cout << "found area: " << biggestContourArea << std::endl;
// found biggest contour
// add contour to sorted contours vector:
sortedContours.push_back(contours[biggestContourID]);
chosen++;
// remove biggest contour from original vector:
contours[biggestContourID] = contours.back();
contours.pop_back();
}
else
{
// should never happen except for broken contours with size 0?!?
return sortedContours;
}
}
return sortedContours;
}
int main()
{
cv::Mat input = cv::imread("../Data/glass2.png", CV_LOAD_IMAGE_GRAYSCALE);
cv::Mat inputColors = cv::imread("../Data/glass2.png"); // used for displaying later
cv::imshow("input", input);
//edge detection
int lowThreshold = 100;
int ratio = 3;
int kernel_size = 3;
cv::Mat canny;
cv::Canny(input, canny, lowThreshold, lowThreshold*ratio, kernel_size);
cv::imshow("canny", canny);
// close gaps with "close operator"
cv::Mat mask = canny.clone();
cv::dilate(mask,mask,cv::Mat());
cv::dilate(mask,mask,cv::Mat());
cv::dilate(mask,mask,cv::Mat());
cv::erode(mask,mask,cv::Mat());
cv::erode(mask,mask,cv::Mat());
cv::erode(mask,mask,cv::Mat());
cv::imshow("closed mask",mask);
// extract outermost contour
std::vector<cv::Vec4i> hierarchy;
std::vector<std::vector<cv::Point>> contours;
//cv::findContours(mask, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE);
cv::findContours(mask, contours, hierarchy, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
// find biggest contour which should be the outer contour of the frame
std::vector<std::vector<cv::Point>> biggestContour;
biggestContour = findBiggestContours(contours,1); // find the one biggest contour
if(biggestContour.size() < 1)
{
std::cout << "Error: no outer frame of glasses found" << std::endl;
return 1;
}
// draw contour on an empty image
cv::Mat outerFrame = cv::Mat::zeros(mask.rows, mask.cols, CV_8UC1);
cv::drawContours(outerFrame,biggestContour,0,cv::Scalar(255),-1);
cv::imshow("outer frame border", outerFrame);
// now find the glasses which should be the outer contours within the frame. therefore erode the outer border ;)
cv::Mat glassesMask = outerFrame.clone();
cv::erode(glassesMask,glassesMask, cv::Mat());
cv::imshow("eroded outer",glassesMask);
// after erosion if we dilate, it's an Open-Operator which can be used to clean the image.
cv::Mat cleanedOuter;
cv::dilate(glassesMask,cleanedOuter, cv::Mat());
cv::imshow("cleaned outer",cleanedOuter);
// use the outer frame mask as a mask for copying canny edges. The result should be the inner edges inside the frame only
cv::Mat glassesInner;
canny.copyTo(glassesInner, glassesMask);
// there is small gap in the contour which unfortunately cant be closed with a closing operator...
cv::dilate(glassesInner, glassesInner, cv::Mat());
//cv::erode(glassesInner, glassesInner, cv::Mat());
// this part was cheated... in fact we would like to erode directly after dilation to not modify the thickness but just close small gaps.
cv::imshow("innerCanny", glassesInner);
// extract contours from within the frame
std::vector<cv::Vec4i> hierarchyInner;
std::vector<std::vector<cv::Point>> contoursInner;
//cv::findContours(glassesInner, contoursInner, hierarchyInner, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE);
cv::findContours(glassesInner, contoursInner, hierarchyInner, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
// find the two biggest contours which should be the glasses within the frame
std::vector<std::vector<cv::Point>> biggestInnerContours;
biggestInnerContours = findBiggestContours(contoursInner,2); // find the one biggest contour
if(biggestInnerContours.size() < 1)
{
std::cout << "Error: no inner frames of glasses found" << std::endl;
return 1;
}
// draw the 2 biggest contours which should be the inner glasses
cv::Mat innerGlasses = cv::Mat::zeros(mask.rows, mask.cols, CV_8UC1);
for(unsigned int i=0; i<biggestInnerContours.size(); ++i)
cv::drawContours(innerGlasses,biggestInnerContours,i,cv::Scalar(255),-1);
cv::imshow("inner frame border", innerGlasses);
// since we dilated earlier and didnt erode quite afterwards, we have to erode here... this is a bit of cheating :-(
cv::erode(innerGlasses,innerGlasses,cv::Mat() );
// remove the inner glasses from the frame mask
cv::Mat fullGlassesMask = cleanedOuter - innerGlasses;
cv::imshow("complete glasses mask", fullGlassesMask);
// color code the result to get an impression of segmentation quality
cv::Mat outputColors1 = inputColors.clone();
cv::Mat outputColors2 = inputColors.clone();
for(int y=0; y<fullGlassesMask.rows; ++y)
for(int x=0; x<fullGlassesMask.cols; ++x)
{
if(!fullGlassesMask.at<unsigned char>(y,x))
outputColors1.at<cv::Vec3b>(y,x)[1] = 255;
else
outputColors2.at<cv::Vec3b>(y,x)[1] = 255;
}
cv::imshow("output", outputColors1);
/*
cv::imwrite("../Data/Output/face_colored.png", outputColors1);
cv::imwrite("../Data/Output/glasses_colored.png", outputColors2);
cv::imwrite("../Data/Output/glasses_fullMask.png", fullGlassesMask);
*/
cv::waitKey(-1);
return 0;
}
I get this result for segmentation:
the overlay in original image will give you an impression of quality:
and inverse:
There are some tricky parts in the code and it's not tidied up yet. I hope it's understandable.
The next step would be to compute the thickness of the the segmented frame. My suggestion is to compute the distance transform of the inversed mask. From this you will want to compute a ridge detection or skeletonize the mask to find the ridge. After that use the median value of ridge distances.
Anyways I hope this posting can help you a little, although it's not a solution yet.
Depending on lighting, frame color etc this may or may not work but how about simple color detection to separate the frame ? Frame color will usually be a lot darker than human skin. You'll end up with a binary image (just black and white) and by calculating the number (area) of black pixels you get the area of the frame.
Another possible way is to get better edge detection, by adjusting/dilating/eroding/both until you get better contours. You will also need to differentiate the contour from the lenses and then apply cvContourArea.
My work is based on images with an array of dots (Fig. 1), and the final result is shown in Fig. 4. I will explain my work step by step.
Fig. 1 Original image
Step 1: Detect the edge of every object, including the dots and a "ring" that I want to delete for better performance. And the result of edge detection is shown in Fig.2. I used Canny edge detector but it didn't work well with some light-gray dots. My first question is how to close the contours of dots and reduce other noise as much as possible?
Fig. 2 Edge detection
Step 2: Dilate every object. I didn't find a good way to fill holes, so I dilate them directly. As shown in Fig.3, holes seem to be enlarged too much and so does other noise. My second question is how to fill or dilate the holes in order to make them be filled circles in the same/similar size?
Fig. 3 Dilation
Step 3: Find and draw the mass center of every dot. As shown in Fig. 4, due to the coarse image processing, there exist mark of the "ring" and some of dots are shown in two white pixels. The result wanted should only show the dots and one white pixel for one dot.
Fig. 4: Mass centers
Here is my code for these 3 steps. Can anyone help to make my work better?
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <stdlib.h>
#include <stdio.h>
#include <cv.h>
#include <highgui.h>
using namespace std;
using namespace cv;
// Global variables
Mat src, edge, dilation;
int dilation_size = 2;
// Function header
void thresh_callback(int, void*);
int main(int argc, char* argv)
{
IplImage* img = cvLoadImage("c:\\dot1.bmp", 0); // dot1.bmp = Fig. 1
// Perform canny edge detection
cvCanny(img, img, 33, 100, 3);
// IplImage to Mat
Mat imgMat(img);
src = img;
namedWindow("Step 1: Edge", CV_WINDOW_AUTOSIZE);
imshow("Step 1: Edge", src);
// Apply the dilation operation
Mat element = getStructuringElement(2, Size(2 * dilation_size + 1, 2 * dilation_size + 1),
Point(dilation_size, dilation_size)); // dilation_type = MORPH_ELLIPSE
dilate(src, dilation, element);
// imwrite("c:\\dot1_dilate.bmp", dilation);
namedWindow("Step 2: Dilation", CV_WINDOW_AUTOSIZE);
imshow("Step 2: Dilation", dilation);
thresh_callback( 0, 0 );
waitKey(0);
return 0;
}
/* function thresh_callback */
void thresh_callback(int, void*)
{
vector<vector<Point>> contours;
vector<Vec4i> hierarchy;
// Find contours
findContours(dilation, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, Point(0, 0));
// Get the moments
vector<Moments> mu(contours.size());
for(int i = 0; i < contours.size(); i++) {
mu[i] = moments(contours[i], false);
}
// Get the mass centers
vector<Point2f> mc(contours.size());
for(int i = 0; i < contours.size(); i++) {
mc[i] = Point2f(mu[i].m10/mu[i].m00 , mu[i].m01/mu[i].m00);
}
// Draw mass centers
Mat drawing = Mat::zeros(dilation.size(), CV_8UC1);
for( int i = 0; i< contours.size(); i++ ) {
Scalar color = Scalar(255, 255, 255);
line(drawing, mc[i], mc[i], color, 1, 8, 0);
}
namedWindow("Step 3: Mass Centers", CV_WINDOW_AUTOSIZE);
imshow("Step 3: Mass Centers", drawing);
}
There are a few things you can do to improve your results. To reduce noise in the image, you can apply a median blur before applying the Canny operator. This is a common de-noising technique. Also, try to avoid using the C API and IplImage.
cv::Mat img = cv::imread("c:\\dot1.bmp", 0); // dot1.bmp = Fig. 1
cv::medianBlur(img, img, 7);
// Perform canny edge detection
cv::Canny(img, img, 33, 100);
This significantly reduces the amount of noise in your edge image:
To better retain the original sizes of your dots, you can perform a few iterations of morphological closing with a smaller kernel rather than dilation. This will also reduce joining of the dots with the circle:
// This replaces the call to dilate()
cv::morphologyEx(src, dilation, MORPH_CLOSE, cv::noArray(),cv::Point(-1,-1),2);
This will perform two iterations with a 3x3 kernel, indicated by using cv::noArray().
The result is cleaner, and the dots are completely filled:
Leaving the rest of your pipeline unmodified gives the final result. There are still a few spurious mass centers from the circle, but considerably fewer than the original method:
If you wanted to attempt removing the circle from the results entirely, you could try using cv::HoughCircles() and adjusting the parameters until you get a good result. This might have some difficulties because the entire circle is not visible in the image, only segments, but I recommend you experiment with it. If you did detect the innermost circle, you could use it as a mask to filter out external mass centers.
how to close contours of dots? use drawContours method with filled drawing option (CV_FILLED or thickness = -1)
reduce noise? use one of the blurring (low pass filtering) methods.
similar size? use erosion after dilation = morphological closing.
one dot for one circle, output without outer ring? find average of all contour areas. erase contours having big difference to this value. output the remaining centers.
Aurelius already mentioned most of these, but since this problem is quiet interesting, I will probably try and post a complete solution when I have enough time. Good luck.