I need to be able to detect the edges of a card, currently it works when the background is non-disruptive and best when it is contrasting but still works pretty well on a non-contrasting background.
The problem occurs when the card is on a disruptive background the bilateral filter lets in too much noise and causes inaccurate edge detection.
Here is the code I am using:
bilateralFilter(imgGray, detectedEdges, 0, 175, 3, 0);
Canny( detectedEdges, detectedEdges, 20, 65, 3 );
dilate(detectedEdges, detectedEdges, Mat::ones(3,3,CV_8UC1));
The imgGray being the grayscale version of the original image.
Here are some tests on a disruptive background and the results (contact info distorted in all images):
Colored card:
Result:
And here is a white card:
Results:
Can anyone tell me how I can preserve the edges of the card no matter the background, color while removing the noise?
Find the edges using canny which you are already doing, then find the contour in image and find the suitable rectangle using bounding box and apply some threshold on the occupancy and the dimensions of rectangle. This should zero down to your rectangle i.e. your card edges and take it as ROI on which you can further work.
Related
What I need
I'm currently working on an augmented reality kinda game. The controller that the game uses (I'm talking about the physical input device here) is a mono colored, rectangluar pice of paper. I have to detect the position, rotation and size of that rectangle in the capture stream of the camera. The detection should be invariant on scale and invariant on rotation along the X and Y axes.
The scale invariance is needed in case that the user moves the paper away or towards the camera. I don't need to know the distance of the rectangle so scale invariance translates to size invariance.
The rotation invariance is needed in case the user tilts the rectangle along its local X and / or Y axis. Such a rotation changes the shape of the paper from rectangle to trapezoid. In this case, the object oriented bounding box can be used to measure the size of the paper.
What I've done
At the beginning there is a calibration step. A window shows the camera feed and the user has to click on the rectangle. On click, the color of the pixel the mouse is pointing at is taken as reference color. The frames are converted into HSV color space to improve color distinguishing. I have 6 sliders that adjust the upper and lower thresholds for each channel. These thresholds are used to binarize the image (using opencv's inRange function).
After that I'm eroding and dilating the binary image to remove noise and unite nerby chunks (using opencv's erode and dilate functions).
The next step is finding contours (using opencv's findContours function) in the binary image. These contours are used to detect the smallest oriented rectangles (using opencv's minAreaRect function). As final result I'm using the rectangle with the largest area.
A short conclusion of the procedure:
Grab a frame
Convert that frame to HSV
Binarize it (using the color that the user selected and the thresholds from the sliders)
Apply morph ops (erode and dilate)
Find contours
Get the smallest oriented bouding box of each contour
Take the largest of those bounding boxes as result
As you may noticed, I don't make an advantage of the knowledge about the actual shape of the paper, simply because I don't know how to use this information properly.
I've also thought about using the tracking algorithms of opencv. But there were three reasons that prevented me from using them:
Scale invariance: as far as I read about some of the algorithms, some don't support different scales of the object.
Movement prediction: some algorithms use movement prediction for better performance, but the object I'm tracking moves completely random and therefore unpredictable.
Simplicity: I'm just looking for a mono colored rectangle in an image, nothing fancy like car or person tracking.
Here is a - relatively - good catch (binary image after erode and dilate)
and here is a bad one
The Question
How can I improve the detection in general and especially to be more resistant against lighting changes?
Update
Here are some raw images for testing.
Can't you just use thicker material?
Yes I can and I already do (unfortunately I can't access these pieces at the moment). However, the problem still remains. Even if I use material like cartboard. It isn't bent as easy as paper, but one can still bend it.
How do you get the size, rotation and position of the rectangle?
The minAreaRect function of opencv returns a RotatedRect object. This object contains all the data I need.
Note
Because the rectangle is mono colored, there is no possibility to distinguish between top and bottom or left and right. This means that the rotation is always in range [0, 180] which is perfectly fine for my purposes. The ratio of the two sides of the rect is always w:h > 2:1. If the rectangle would be a square, the range of roation would change to [0, 90], but this can be considered irrelevant here.
As suggested in the comments I will try histogram equalization to reduce brightness issues and take a look at ORB, SURF and SIFT.
I will update on progress.
The H channel in the HSV space is the Hue, and it is not sensitive to the light changing. Red range in about [150,180].
Based on the mentioned information, I do the following works.
Change into the HSV space, split the H channel, threshold and normalize it.
Apply morph ops (open)
Find contours, filter by some properties( width, height, area, ratio and so on).
PS. I cannot fetch the image you upload on the dropbox because of the NETWORK. So, I just use crop the right side of your second image as the input.
imgname = "src.png"
img = cv2.imread(imgname)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
## Split the H channel in HSV, and get the red range
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
h,s,v = cv2.split(hsv)
h[h<150]=0
h[h>180]=0
## normalize, do the open-morp-op
normed = cv2.normalize(h, None, 0, 255, cv2.NORM_MINMAX, cv2.CV_8UC1)
kernel = cv2.getStructuringElement(shape=cv2.MORPH_ELLIPSE, ksize=(3,3))
opened = cv2.morphologyEx(normed, cv2.MORPH_OPEN, kernel)
res = np.hstack((h, normed, opened))
cv2.imwrite("tmp1.png", res)
Now, we get the result as this (h, normed, opened):
Then find contours and filter them.
contours = cv2.findContours(opened, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
print(len(contours))[-2]
bboxes = []
rboxes = []
cnts = []
dst = img.copy()
for cnt in contours:
## Get the stright bounding rect
bbox = cv2.boundingRect(cnt)
x,y,w,h = bbox
if w<30 or h < 30 or w*h < 2000 or w > 500:
continue
## Draw rect
cv2.rectangle(dst, (x,y), (x+w,y+h), (255,0,0), 1, 16)
## Get the rotated rect
rbox = cv2.minAreaRect(cnt)
(cx,cy), (w,h), rot_angle = rbox
print("rot_angle:", rot_angle)
## backup
bboxes.append(bbox)
rboxes.append(rbox)
cnts.append(cnt)
The result is like this:
rot_angle: -2.4540319442749023
rot_angle: -1.8476102352142334
Because the blue rectangle tag in the source image, the card is splited into two sides. But a clean image will have no problem.
I know it's been a while since I asked the question. I recently continued on the topic and solved my problem (although not through rectangle detection).
Changes
Using wood to strengthen my controllers (the "rectangles") like below.
Placed 2 ArUco markers on each controller.
How it works
Convert the frame to grayscale,
downsample it (to increase performance during detection),
equalize the histogram using cv::equalizeHist,
find markers using cv::aruco::detectMarkers,
correlate markers (if multiple controllers),
analyze markers (position and rotation),
compute result and apply some error correction.
It turned out that the marker detection is very robust to lighting changes and different viewing angles which allows me to skip any calibration steps.
I placed 2 markers on each controller to increase the detection robustness even more. Both markers has to be detected only one time (to measure how they correlate). After that, it's sufficient to find only one marker per controller as the other can be extrapolated from the previously computed correlation.
Here is a detection result in a bright environment:
in a darker environment:
and when hiding one of the markers (the blue point indicates the extrapolated marker postition):
Failures
The initial shape detection that I implemented didn't perform well. It was very fragile to lighting changes. Furthermore, it required an initial calibration step.
After the shape detection approach I tried SIFT and ORB in combination with brute force and knn matcher to extract and locate features in the frames. It turned out that mono colored objects don't provide much keypoints (what a surprise). The performance of SIFT was terrible anyway (ca. 10 fps # 540p).
I drew some lines and other shapes on the controller which resulted in more keypoints beeing available. However, this didn't yield in huge improvements.
As the graph shows, I implemented 3 kinds of blurring to my source image.
Gaussian & Median blurs look OK, but the Meanshift result confused me.
There are discontinuous areas on the right side.
My implemented code is
pyrMeanShiftFiltering(src2, dst2, 5, 25, 3);, and it's still the same after being modified to pyrMeanShiftFiltering(src2, dst2, 2, 25, 3);.
So I think it's got nothing to do with Spatial Window Radius.
Can anyone explain why this happened, and how to avoid it for me?
Thanks a lot!
========================================================
8/13
added code:
int border = 1;
copyMakeBorder(src2, src2, border, border, border, border, BORDER_CONSTANT, 0);
Unfortunately, it's still the same..
Here I can see the same effect, so looks like a bug in implementation (looks like convolution edge effect). You can try to add border to image, before applying the filter, and remove border after processing finished (see copyMakeBorder opencv's method).
update - in the process of troubleshooting, trial and error I got my mats mixed up. Its working now
I was wondering if anyone can help troubleshoot my program. Here's some background. I'm in the early stages of writing a program that will detect parts along a conveyor belt for an automated pick and place robot. Right now I'm trying differentiate an upside down part to a right side up part. I figured I can do this by querying pixels at a certain radius and angle from the part center point. (ie, if pixels 1,2, and 3 are black, part must be upside down)
Here is a screenshot of what I have so far:
Here is a screenshot of it sometimes working:
The goal here is if the pixel is black, draw a black circle, if white, draw a green circle. As you can see, all the circles are black. I can occasionally get a green circle when the pixel im querying lays on the part edge, but not in the middle. To the right, you can see the b/w image that I am reading, named "filter".
Here is the code for one of the circles. Radian is the orientation of the part, and radius is the distance away from the part centerpoint for my pixel query.
// check1 is the coordinate for 1st pixel query
Point check1 = Point(pos.x + radius * cos(radian), pos.y + radius* sin (radian)); Scalar intensity1 = filter.at<uchar>(Point(check1)); // pixel should be 0-255
if (intensity1.val[0]>50)
{
circle(screenshot, check1, 8, CV_RGB(0, 255, 0), 2); //green circle
}
else
{
circle(screenshot, check1, 8, CV_RGB(0, 0, 0), 2); //black circle
}
Here are the steps of how I created filter mat
Webcam stream -
cvtcolor (bgr2hsv) -
GaussianBlur -->
inRange (becomes 8u I think) -
erode -
dilate =
Filter mat (this is the mat that I use findContours on and also the pixel querying.)
I've also tried making a copy of filter, and using findContours and pixel querying on separate mats with no luck. I'm not sure what is wrong. Any suggestions is appreciated. I have a feeling that I might be using the incorrect mat format.
For my college project I need to identify a species of a plant from plant leaf shape by detecting edges of a leaf. (I use OpenCV 2.4.9 and C++), but the source image has taken in the real environment of the plant and has more than one leaf. See the below example image. So here I need to extract the edge pattern of just one leaf to process further.
Using Canny Edge Detector I can identify edges of the whole image.
But I don't know how to proceed from here to extract edge pattern of just one leaf, may be more clear and complete leaf. I don't know even if this is possible also. Can anyone please tell me if this is possible how to extract edges of one leaf I just want to know the image peocessing steps that I need to apply to the image. I don't want any code samples. I'm new to image processing and OpenCV and learning by doing experiments.
Thanks in advance.
Edit
As Luis said said I have done Morphological close to the image after doing edge detection using Canny edge detection, but it seems still it is difficult me to find the largest contour from the image.
Here are the steps I have taken to process the image
Apply Bilateral Filter to reduce noise
bilateralFilter(img_src, img_blur, 31, 31 * 2, 31 / 2);
Adjust contrast by histogram equaliztion
cvtColor(img_blur,img_equalized,CV_BGR2GRAY);
Apply Canny edge detector
Canny(img_equalized, img_edge_detected, 20, 60, 3);
Threshold binary image to remove some background data
threshold(img_edge_detected, img_threshold, 1, 255,THRESH_BINARY_INV);
Morphological close of the image
morphologyEx(img_threshold, img_closed, MORPH_CLOSE, getStructuringElement(MORPH_ELLIPSE, Size(2, 2)));
Following are the resulting images I'm getting.
This result I'm getting for the above original image
Source image and result for second image
Source :
Result :
Is there any way to detect the largest contour and extract it from the image ?
Note that my final target is to create a plant identification system using real environmental image, but here I cannot use template matching or masking kind of things because the user has to take an image and upload it so the system doesn't have any prior idea about the leaf.
Here is the full code
#include <opencv\cv.h>
#include <opencv\highgui.h>
using namespace cv;
int main()
{
Mat img_src, img_blur,img_gray,img_equalized,img_edge_detected,img_threshold,img_closed;
//Load original image
img_src = imread("E:\\IMAG0196.jpg");
//Apply Bilateral Filter to reduce noise
bilateralFilter(img_src, img_blur, 31, 31 * 2, 31 / 2);
//Adjust contrast by histogram equaliztion
cvtColor(img_blur,img_equalized,CV_BGR2GRAY);
//Apply Canny edge detector
Canny(img_equalized, img_edge_detected, 20, 60, 3);
//Threshold binary image to remove some background data
threshold(img_edge_detected, img_threshold, 15, 255,THRESH_BINARY_INV);
//Morphological close of the image
morphologyEx(img_threshold, img_closed, MORPH_CLOSE, getStructuringElement(MORPH_ELLIPSE, Size(2, 2)));
imshow("Result", img_closed);
waitKey(0);
return 0;
}
Thank you.
Well there is a similar question that was asked here:
opencv matching edge images
It seems that edge information is not a good descriptor for the image, still if you want to try it I'll do the following steps:
Load image and convert it to grayscale
Detect edges - Canny, Sobel try them and find what it suits you best
Set threshold to a given value that eliminates most background - Binarize image
Close the image - Morphological close dont close the window!
Count and identify objects in the image (Blobs, Watershed)
Check each object for a shape (assuming you have described shapes of the leaf you could find before or a standard shape like an ellipse) features like:
http://docs.opencv.org/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html
http://www.math.uci.edu/icamp/summer/research_11/park/shape_descriptors_survey.pdf
If a given object has a given shape that you described as a leaf then you detected the leaf!.
I believe that given images are taken in the real world these algorithm will perform poorly but it's a start. Well hope it helps :).
-- POST EDIT 06/07
Well since you have no prior information about the leaf, I think the best we could do is the following:
Load image
Bilateral filter
Canny
Extract contours
Assume: that the contour with the largest perimeter is the leaf
Convex hull the 3 or 2 largest contours (the blue line is the convex hull done)
Use this convex hull to do a graph cut on the image and segmentate it
If you do those steps, you'll end up with images like these:
I won't post the code here, but you can check it out in my messy github. I hope you don't mind it was made in python.
Leaf - Github
Still, I have a couple of things to finish that could improve the result.. Roadmap would be:
Define the mask in the graphcut (like its described in the doc)
Apply region grow may give a better convex hull
Remove all edges that touch the border of the image can help to identify larger edges
Well, again, I hope it helps
I have been trying to detect the iris region of an eye and thereafter draw a circle around the detected area. I have managed to obtain a clear black and white eye image containing just the pupil, upper eyelid line and eyebrow using threshold function.
Once this is achieved HoughCircles is applied to detect if there are circles appearing in the image. However, it never detects any circular regions. After reading up on HoughCircles, it states that
the Hough gradient method works as follows:
First the image needs to be passed through an edge detection phase (in this case, cvCanny()).
I then added a canny detector after the threshold function. This still produced zero circles detected. If I remove the threshold function, the eye image becomes busy with unnecessary lines; hence I included it in.
cv::equalizeHist(gray, img);
medianBlur(img, img, 1);
IplImage img1 = img;
cvAddS(&img1, cvScalar(70,70,70), &img1);
//converting IplImage to cv::Mat
Mat imgg = cvarrToMat(&img1);
medianBlur(imgg, imgg, 1);
cv::threshold(imgg, imgg, 120, 255, CV_THRESH_BINARY);
cv::Canny(img, img, 0, 20);
medianBlur(imgg, imgg, 1);
vector<Vec3f> circles;
/// Apply the Hough Transform to find the circles
HoughCircles(imgg, circles, CV_HOUGH_GRADIENT, 1, imgg.rows/8, 100, 30, 1, 5);
How can I overcome this problem?
Would hough circle method work?
Is there a better solution to detecting the iris region?
Are the parameters chosen correct?
Also note that the image is directly obtained from the webcam.
Try using Daugman's Integro differential operator. It calculates the centre of the iris and pupil and draws an accurate circle on the iris and pupil boundaries. The MATLAB code is available here iris boundary detection using Daugman's method. Since I'm not familiar with OpenCV you could convert it.
The binary eye image contained three different aspects eyelashes ,the eye and the eyebrow.The main aim is to get to the region of interest which is the eye/iris, excluding eyebrows and eyelashes.I followed these steps:-
Step 1: Discard the upper half of the eye image ,therefore we are left with eyelashes,eye region and small shadow regions .
Step 2:Find the contours
Step 3:Find largest contour so that we have just the eye region
Step 4:Use bounding box to create a rectangle around the eye area
http://docs.opencv.org/doc/tutorials/imgproc/shapedescriptors/bounding_rects_circles/bounding_rects_circles.html
Now we have the region of interest.From this point I am now exacting these images and using neural network to train the system to emulate properties of a mouse. Im currently learning about the neural network link1 and how to use it in opencv.
Using the previous methods which included detecting the iris point,creating an eye vector,tracking it and calculating the graze on the screen is time consuming .Also there is light reflected on the iris making it difficult to detect.