How to extend a drawn line in OpenCV? - c++

I have a project to create a face detection program. One of the requirements is for me to draw a line across the center of the face.
Now the way I have done this is by first drawing a line across the two eyes (by getting each eye's coordinates and using them as line points) and then drawing a perpendicular line to that. This way I can account for face rotation aswell.
However this line is the same length as the line across the eyes. I want this perpendicular line to cut across the entire face if possible. Attached is an example of my output for one the rotated face images. I need the green line to be longer.

unfortunately your image is a jpeg so it has compression artifacts.
I first tried to extract the line, but that didnt give good angle information, so I use cv::minAreaRect after green area segmentation.
cv::Mat green;
cv::inRange(input, cv::Scalar(0,100,0), cv::Scalar(100,255,100),green);
// instead of this, maybe try to find contours and choose the right one, if there are more green "objects" in the image
std::vector<cv::Point2i> locations;
cv::findNonZero(green, locations);
// find the used color to draw in same color later
cv::Vec3d averageColorTmp(0,0,0);
for(unsigned int i=0; i<locations.size(); ++i)
{
averageColorTmp += input.at<cv::Vec3b>(locations[i]);
}
averageColorTmp *= 1.0/locations.size();
// compute direction of the segmented region
cv::RotatedRect line = cv::minAreaRect(locations);
// extract directions:
cv::Point2f start = line.center;
cv::Point2f rect_points[4];
line.points(rect_points);
cv::Point2f direction1 = rect_points[0] - rect_points[1];
cv::Point2f direction2 = rect_points[0] - rect_points[3];
// use dominant direction
cv::Point2f lineWidthDirection;
lineWidthDirection = (cv::norm(direction1) < cv::norm(direction2)) ? direction1 : direction2;
double lineWidth = cv::norm(lineWidthDirection);
cv::Point2f lineLengthDirection;
lineLengthDirection = (cv::norm(direction1) > cv::norm(direction2)) ? direction1 : direction2;
lineLengthDirection = 0.5*lineLengthDirection; // because we operate from center;
// input parameter:
// how much do you like to increase the line size?
// value of 1.0 is original size
// value of > 1 is increase of line
double scaleFactor = 3.0;
// draw the line
cv::line(input, start- scaleFactor*lineLengthDirection, start+ scaleFactor*lineLengthDirection, cv::Scalar(averageColorTmp[0],averageColorTmp[1],averageColorTmp[2]), lineWidth);
giving this result:
This method should be quite robust, IF there is no other green in the image but only the green line. If there are other green parts in the image you should extract contours first and choose the right contour to assign the variable locations

Ill write it out in pseudocode
Take the furthest right hand edge coordinate (row1,col1) in your image
Take the right hand point of your line (row2,col2)
Draw a line from that your line (row2,col2) to (row2,col1)
Repeat for the left hand side
This will result in drawing two additional lines from either side of your line to the edge of your image.

Related

Hausdorff Distance Object Detection

I have been struggling trying to implement the outlining algorithm described here and here.
The general idea of the paper is determining the Hausdorff distance of binary images and using it to find the template image from a test image.
For template matching, it is recommended to construct image pyramids along with sliding windows which you'll use to slide over your test image for detection. I was able to do both of these as well.
I am stuck on how to move forward from here on. Do I slide my template over the test image from different pyramid layers? Or is it the test image over the template? And with regards to the sliding window, is/are they meant to be a ROI of the test or template image?
In a nutshell, I have pieces to the puzzle but no idea of which direction to take to solve the puzzle
int distance(vector<Point>const& image, vector<Point>const& tempImage)
{
int maxDistance = 0;
for(Point imagePoint: image)
{
int minDistance = numeric_limits<int>::max();
for(Point tempPoint: tempImage)
{
Point diff = imagePoint - tempPoint;
int length = (diff.x * diff.x) + (diff.y * diff.y);
if(length < minDistance) minDistance = length;
if(length == 0) break;
}
maxDistance += minDistance;
}
return maxDistance;
}
double hausdorffDistance(vector<Point>const& image, vector<Point>const& tempImage)
{
double maxDistImage = distance(image, tempImage);
double maxDistTemp = distance(tempImage, image);
return sqrt(max(maxDistImage, maxDistTemp));
}
vector<Mat> buildPyramids(Mat& frame)
{
vector<Mat> pyramids;
int count = 6;
Mat prevFrame = frame, nextFrame;
while(count > 0)
{
resize(prevFrame, nextFrame, Size(), .85, .85);
prevFrame = nextFrame;
pyramids.push_back(nextFrame);
--count;
}
return pyramids;
}
vector<Rect> slidingWindows(Mat& image, int stepSize, int width, int height)
{
vector<Rect> windows;
for(size_t row = 0; row < image.rows; row += stepSize)
{
if((row + height) > image.rows) break;
for(size_t col = 0; col < image.cols; col += stepSize)
{
if((col + width) > image.cols) break;
windows.push_back(Rect(col, row, width, height));
}
}
return windows;
}
Edit I: More analysis on my solution can be found here
This is a bi-directional task.
Forward Direction
1. Translation
For each contour, calculate its moment. Then for each point in that contour, translate it about the moment i.e. contour.point[i] = contour.point[i] - contour.moment[i]. This moves all of the contour points to the origin.
PS: You need to keep track of each contour's produced moment because it will be used in the next section
2. Rotation
With the newly translated points, calculate their rotated rect. This will give you the angle of rotation. Depending on this angle, you would want to calculate the new angle which you want to rotate this contour by; this answer would be helpful.
After attaining the new angle, calculate the rotation matrix. Remember that your center here will be the origin i.e. (0, 0). I did not take scaling into account (that's where the pyramids come into play) when calculating the rotation matrix hence I passed 1.
PS: You need to keep track of each contour's produced matrix because it will be used in the next section
Using this matrix, you can go ahead and rotate each point in the contour by it as shown here*.
Once all of this is done, you can go ahead and calculate the Hausdorff distance and find contours which pass your set threshold.
Back Direction
Everything done in the first section, has to be undone in order for us to draw the valid contours onto our camera feed.
1. Rotation
Recall that each detected contour produced a rotation matrix. You want to undo the rotation of the valid contours. Just perform the same rotation but using the inverse matrix.
For each valid contour and corresponding matrix
inverse_matrix = matrix[i].inv(cv2.DECOMP_SVD)
Use * to rotate the points but with inverse_matrix as parameter
PS: When calculating the inverse, if the produced matrix was not a square one, it would fail. cv2.DECOMP_SVD will produce an inverse matrix even if the original matrix was a non-square.
2. Translation
With the valid contours' points rotated back, you just have to undo the previously performed translation. Instead of subtracting, just add the moment to each point.
You can now go ahead and draw these contours to your camera feed.
Scaling
This is were image pyramids come into play.
All you have to do is resize your template image by a fixed size/ratio upto your desired number of times (called layers). The tutorial found here does a good job of explaining how to do this in OpenCV.
It goes without saying that the values you choose to resize your image by and number of layers will and do play a huge role in how robust your program will be.
Put it all together
Template Image Operations
Create a pyramid consisting of n layers
For each layer in n
Find contours
Translate the contour points
Rotate the contour points
This operation should only be performed once and only store the results of the rotated points.
Camera Feed Operations
Assumptions
Let the rotated contours of the template image at each level be stored in templ_contours. So if I say templ_contours[0], this is going to give me the rotated contours at pyramid level 0.
Let the image's translated, rotated contours and moments be stored in transCont, rotCont and moment respectively.
image_contours = Find Contours
for each contour detected in image
moment = calculate moment
for each point in image_contours
transCont.thisPoint = forward_translate(image_contours.thisPoint)
rotCont.thisPoint = forward_rotate(transCont.thisPoint)
for each contour_layer in templ_contours
for each contour in rotCont
calculate Hausdorff Distance
valid_contours = contours_passing_distance_threshold
for each point in valid_contours
valid_point = backward_rotate(valid_point)
for each point in valid_contours
valid_point = backward_translate(valid_point)
drawContours(valid_contours, image)

How do I find if the object ball I track crosses a line that I have drawn?

I am using c++ with OpenCV 3.0 to create a basic form of SimulCam.
I am currently stuck on finding a way to check when the object ball has crossed/intersected with a line that I have drawn on the output window.
The ball is being tracked using contours, and ultimately I would like to work out the exact frame number this intersect happens at.
But first, I would like to understand how to perform the check to see when the Object ball has crossed/intersected with the drawn line.
Scene with ball moving towards line
I have the contours for the object, I would like to understand how to perform the check of an intersection.
Code for finding contours and Object Tracking:
findContours(resizedThresh, contourVector, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));
contourVector.resize(contourVector.size());
line(resizedF_Fast, Point(300, 0), Point(300, 360), Scalar(255), 2, 8);
for (size_t i = 0; i < contourVector.size(); i++) {
approxPolyDP(Mat(contourVector[i]), contourVector[i], 0.01*arcLength(contourVector[i], true), true);
double area = contourArea(contourVector[i]);
if (contourVector[i].size() > 5 && (area > 200)) {
++circlesC;
drawContours(resizedF_Fast, contourVector, i, Scalar(255, 255, 255), 2, CV_AA, hierarchy, abs(1));
searchForMovement(resizedThresh, resizedF_Fast);
}
}
I have done some other research, and I have been looking into using lineIterator, but i'm not entirely sure..
Apologies for the potential crude code, novice here. Any help would be greatly appreciated.
My first approach would be to fit a circle into your contour points and then compute the distance between the line and your circle center with the dot product. Maybe like this (didnt tried it out):
Point Pc; // circle center
Point L0(300,0);
Point L1(300,360);
double v[] = {L1.x-L0.x,L1.y-L0.y};
double w[] = {Pc.x-L0.x,Pc.y-L0.y};
Mat v(1,2,CV_32F,v);
Mat w(1,2,CV_32F,w);
double c1 = w.dot(v);
double c2 = v.dot(v);
double b = c1 / c2;
Mat Pb = L0 + b * v;
double distance = norm(Pc,Pb);
Then you check if your distance minus your circle radius is less equal zero.
But due to perspective transformation of your camera the ball becomes an ellipse and my assumption becomes less accurate.
If you need a more accurate solution you need to check every contour point and take the minimum distance.
This link shows some code and further explanations.
I finally worked through this, i'll post the general idea here.
For each frame, calculate the object contours.
each contour will have an x and y coordinate stored
Used LineIterator (e.g. lineIt) to cycle through all values of a line.
if (xpos_contour < lineIt.pos().x) {
// Object is on the left of the line
}
else if (xpos_contour > lineIt.pos().x) {
// Object is to the right of the line
}
Bear in mind the input video im using filmed top down, so only the x coordinate mattered.

OpenCV: how to detect lines of a specific colour?

I am working on a small OpenCV project to detect lines of a certain colour from a mobile phone camera.
In short would like to:
Transform the input image into an image of a certain colour (e.g. Red from a specific upper and lower range)
Apply Hough line transformation to the resulting image so that it detects only lines of that specific colour
Superimpose on the original image the lines detected
Those are the functions that I'd like to use but not quiet sure how to fill the missing bits.
This is the processImage function called from an smartphone app when processing images from an instance of CvVideoCamera
- (void)processImage:(Mat&)image;
{
cv::Mat orig_image = image.clone();
cv::Mat red_image = ??
// Apply houghes transformation to detect lines between a minimum length and a maximum length (I was thinking of using the CV_HOUGH_PROBABILISTIC method..)
// Comment.. see below..
I am unable to understand the documentation here as the C++
method signature does not have a method field
vector<Vec2f> lines;
From the official documentation:
C++: void HoughLines(InputArray image, OutputArray lines, double rho, double theta, int threshold, double srn=0, double stn=0 )
HoughLines(dst, lines, 1, CV_PI/180, 100, 0, 0 );
Taken from sample code, haven't understood properly how it works..
(e.g. what's the usage of theta? How does giving a different angle
reflect into line detection?)
for( size_t i = 0; i < lines.size(); i++ )
{
Here I should only consider lines above a certain size.. (no idea how)
}
Here I should then add the resulting lines to original image (no idea how) so that they can be shown on the screen..
Any help would be greatly appreciated.
You can use HSV color space to extract color tone information.
Here's some code with comments, if there are any questions feel free to ask:
int main(int argc, char* argv[])
{
cv::Mat input = cv::imread("C:/StackOverflow/Input/coloredLines.png");
// convert to HSV color space
cv::Mat hsvImage;
cv::cvtColor(input, hsvImage, CV_BGR2HSV);
// split the channels
std::vector<cv::Mat> hsvChannels;
cv::split(hsvImage, hsvChannels);
// hue channels tells you the color tone, if saturation and value aren't too low.
// red color is a special case, because the hue space is circular and red is exactly at the beginning/end of the circle.
// in literature, hue space goes from 0 to 360 degrees, but OpenCV rescales the range to 0 up to 180, because 360 does not fit in a single byte. Alternatively there is another mode where 0..360 is rescaled to 0..255 but this isn't as common.
int hueValue = 0; // red color
int hueRange = 15; // how much difference from the desired color we want to include to the result If you increase this value, for example a red color would detect some orange values, too.
int minSaturation = 50; // I'm not sure which value is good here...
int minValue = 50; // not sure whether 50 is a good min value here...
cv::Mat hueImage = hsvChannels[0]; // [hue, saturation, value]
// is the color within the lower hue range?
cv::Mat hueMask;
cv::inRange(hueImage, hueValue - hueRange, hueValue + hueRange, hueMask);
// if the desired color is near the border of the hue space, check the other side too:
// TODO: this won't work if "hueValue + hueRange > 180" - maybe use two different if-cases instead... with int lowerHueValue = hueValue - 180
if (hueValue - hueRange < 0 || hueValue + hueRange > 180)
{
cv::Mat hueMaskUpper;
int upperHueValue = hueValue + 180; // in reality this would be + 360 instead
cv::inRange(hueImage, upperHueValue - hueRange, upperHueValue + hueRange, hueMaskUpper);
// add this mask to the other one
hueMask = hueMask | hueMaskUpper;
}
// now we have to filter out all the pixels where saturation and value do not fit the limits:
cv::Mat saturationMask = hsvChannels[1] > minSaturation;
cv::Mat valueMask = hsvChannels[2] > minValue;
hueMask = (hueMask & saturationMask) & valueMask;
cv::imshow("desired color", hueMask);
// now perform the line detection
std::vector<cv::Vec4i> lines;
cv::HoughLinesP(hueMask, lines, 1, CV_PI / 360, 50, 50, 10);
// draw the result as big green lines:
for (unsigned int i = 0; i < lines.size(); ++i)
{
cv::line(input, cv::Point(lines[i][0], lines[i][1]), cv::Point(lines[i][2], lines[i][3]), cv::Scalar(0, 255, 0), 5);
}
cv::imwrite("C:/StackOverflow/Output/coloredLines_mask.png", hueMask);
cv::imwrite("C:/StackOverflow/Output/coloredLines_detection.png", input);
cv::imshow("input", input);
cv::waitKey(0);
return 0;
}
using this input image:
Will extract this "red" color (adjust hueValue and hueRange to detect different colors):
and HoughLinesP detects those lines from the mask (should work with HoughLines similarly):
Here's another set of images with non-lines too...
About your different questions:
There are two functions HoughLines and HoughLinesP. HoughLines does not extract a line length, but you can compute it in a post-processing by checking again, which pixels of the edge-mask (HoughLines input) correspond to the extracted line.
parameters:
image - edge image (should be clear?)
lines - lines given by angle and position, no length or sth. they are interpreted infinitely long
rho - the accumulator resolution. The bigger, the more robust in case of slightly distorted lines it should be, but the less accurate in the extracted lines' position/angle
threshold - the bigger the less false positives, but you might miss some lines
theta - angle resolution: the smaller, the more different lines (depending on the orientation) can be detected. If your line's orientation does not fit in the angle steps, the line might not be detected. For example if you CV_PI/180 will detect in 1° resolution, if your line has a 0.5° (e.g. 33.5°) orientation, it might be missed.
I'm not so extremely sure about all the parameters, maybe you'll have to look at the literature about hough line detection, or someone else can add some hints here.
If you instead use cv::HoughLinesP, line segments with start- and end-point will be detected, which is easier to interpret and you can compute the line length from cv::norm(cv::Point(lines[i][0], lines[i][1]) - cv::Point(lines[i][2], lines[i][3]))
I will not show the code but the steps with some tricks.
Assume that you want to detect road lanes (which are lines with white or light yellow color and have some specific properties).
Original Image( I add some extra lines to make noise )
Step 1: Remove parts of image that not need to be used, save the CPU usage (simple but useful)
Step 2: Convert to gray image
Step 3:threshold
Using threshold according to the color of your line, the color will become white and others will become black
Step 4: Using Contours to find the bound of objects
Step 5: Using Fitline with Contours in previous step are input to return equations of lines
Fitline returns (x0,y0) and vector v = (a,b)
Step 6: With the equations of lines you can draw in any line you want

Matching small grayscale images

I want to test whether two images match. Partial matches also interest me.
The problem is that the images suffer from strong noise. Another problem is that the images might be rotated with an unknown angle. The objects shown in the images will roughly always have the same scale!
The images show area scans from a top-shot perspective. "Lines" are mostly walls and other objects are mostly trees and different kinds of plants.
Another problem was, that the left image was very blurry and the right one's lines were very thin.
To compensate for this difference I used dilation. The resulting images are the ones I uploaded.
Although It can easily be seen that these images match almost perfectly I cannot convince my algorithm of this fact.
My first idea was a feature based matching, but the matches are horrible. It only worked for a rotation angle of -90°, 0° and 90°. Although most descriptors are rotation invariant (in past projects they really were), the rotation invariance seems to fail for this example.
My second idea was to split the images into several smaller segments and to use template matching. So I segmented the images and, again, for the human eye they are pretty easy to match. The goal of this step was to segment the different walls and trees/plants.
The upper row are parts of the left, and the lower are parts of the right image. After the segmentation the segments were dilated again.
As already mentioned: Template matching failed, as did contour based template matching and contour matching.
I think the dilation of the images was very important, because it was nearly impossible for the human eye to match the segments without dilation before the segmentation. Another dilation after the segmentation made this even less difficult.
Your first job should be to fix the orientation. I am not sure what is the best algorithm to do that but here is an approach I would use: fix one of the images and start rotating the other. For each rotation compute a histogram for the color intense on each of the rows/columns. Compute some distance between the resulting vectors(e.g. use cross product). Choose the rotation that results in smallest cross product. It may be good idea to combine this approach with hill climbing.
Once you have the images aligned in approximately the same direction, I believe matching should be easier. As the two images are supposed to be at the same scale, compute something analogous to the geometrical center for both images: compute weighted sum of all pixels - a completely white pixel would have a weight of 1, and a completely black - weight 0, the sum should be a vector of size 2(x and y coordinate). After that divide those values by the dimensions of the image and call this "geometrical center of the image". Overlay the two images in a way that the two centers coincide and then once more compute cross product for the difference between the images. I would say this should be their difference.
You can also try following methods to find rotation and similarity.
Use image moments to get the rotation as shown here.
Once you rotate the image, use cross-correlation to evaluate the similarity.
EDIT
I tried this with OpenCV and C++ for the two sample images. I'm posting the code and results below as it seems to work well at least for the given samples.
Here's the function to calculate the orientation vector using image moments:
Mat orientVec(Mat& im)
{
Moments m = moments(im);
double cov[4] = {m.mu20/m.m00, m.mu11/m.m00, m.mu11/m.m00, m.mu02/m.m00};
Mat covMat(2, 2, CV_64F, cov);
Mat evals, evecs;
eigen(covMat, evals, evecs);
return evecs.row(0);
}
Rotate and match sample images:
Mat im1 = imread(INPUT_FOLDER_PATH + string("WojUi.png"), 0);
Mat im2 = imread(INPUT_FOLDER_PATH + string("XbrsV.png"), 0);
// get the orientation vector
Mat v1 = orientVec(im1);
Mat v2 = orientVec(im2);
double angle = acos(v1.dot(v2))*180/CV_PI;
// rotate im2. try rotating with -angle and +angle. here using -angle
Mat rot = getRotationMatrix2D(Point(im2.cols/2, im2.rows/2), -angle, 1.0);
Mat im2Rot;
warpAffine(im2, im2Rot, rot, Size(im2.rows, im2.cols));
// add a border to rotated image
int borderSize = im1.rows > im2.cols ? im1.rows/2 + 1 : im1.cols/2 + 1;
Mat im2RotBorder;
copyMakeBorder(im2Rot, im2RotBorder, borderSize, borderSize, borderSize, borderSize,
BORDER_CONSTANT, Scalar(0, 0, 0));
// normalized cross-correlation
Mat& image = im2RotBorder;
Mat& templ = im1;
Mat nxcor;
matchTemplate(image, templ, nxcor, CV_TM_CCOEFF_NORMED);
// take the max
double max;
Point maxPt;
minMaxLoc(nxcor, NULL, &max, NULL, &maxPt);
// draw the match
Mat rgb;
cvtColor(image, rgb, CV_GRAY2BGR);
rectangle(rgb, maxPt, Point(maxPt.x+templ.cols-1, maxPt.y+templ.rows-1), Scalar(0, 255, 255), 2);
cout << "max: " << max << endl;
With -angle rotation in code, I get max = 0.758. Below is the rotated image in this case with the matching region.
Otherwise max = 0.293

OpenCV Image to Black and White Shape

i want the hand image to be a black and white shape of the hand. here's a sample of the input and the desired output:
using a threshold doesn't give the desired output because some of the colors inside the hand are the same with the background color. how can i get the desired output?
Adaptive threshold, find contours, floodfill?
Basically, adaptive threshold turns your image into black and white, but takes the threshold level based on local conditions around each pixel - that way, you should avoid the problem you're experiencing with an ordinary threshold. In fact, I'm not sure why anyone would ever want to use a normal threshold.
If that doesn't work, an alternative approach is to find the largest contour in the image, draw it onto a separate matrix and then floodfill everything inside it with black. (Floodfill is like the bucket tool in MSPaint - it starts at a particular pixel, and fills in everything connected to that pixel which is the same colour with another colour of your choice.)
Possibly the most robust approach against various lighting conditions is to do them all in the sequence at the top. But you may be able to get away with only the threshold or the countours/floodfill.
By the way, perhaps the trickiest part is actually finding the contours, because findContours returns an arraylist/vector/whatever (depends on the platform I think) of MatOfPoints. MatOfPoint is a subclass of Mat but you can't draw it directly - you need to use drawContours. Here's some code for OpenCV4Android that I know works:
private Mat drawLargestContour(Mat input) {
/** Allocates and returns a black matrix with the
* largest contour of the input matrix drawn in white. */
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
Imgproc.findContours(input, contours, new Mat() /* hierarchy */,
Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
double maxArea = 0;
int index = -1;
for (MatOfPoint contour : contours) { // iterate over every contour in the list
double area = Imgproc.contourArea(contour);
if (area > maxArea) {
maxArea = area;
index = contours.indexOf(contour);
}
}
if (index == -1) {
Log.e(TAG, "Fatal error: no contours in the image!");
}
Mat border = new Mat(input.rows(), input.cols(), CvType.CV_8UC1); // initialized to 0 (black) by default because it's Java :)
Imgproc.drawContours(border, contours, index, new Scalar(255)); // 255 = draw contours in white
return border;
}
Two quick things you can try:
After thresholding you can:
Do a morphological closing,
or, the most straightforward: cv::findContours, keep the largest if it's more than one, then draw it using cv::fillConvexPoly and you will get this mask. (fillConvexPoly will fill the holes for you)