I have images of a smelting cube, forming into a droplet over time. So far, i extracted the contour of it, but next i'd need to to distinguish between object and surface. My Idea is to detect the corners where object touches surface, but i am struggling to find a reasonable approach how to do so (preferably using the c++ interface of opencv). I'd appreciate any suggestions.
Here are some examples of the extracted contour:
edit:
#Haris:
i have tried a variant of your suggestion and it is doing the job for me:
In the approximated contour i approach from the left, looking for the first angle with a value in a specified range, then the same from the right. As the approximated contour points are a subset of the original contour points, I then identify the 2 corner points in the original sequence, and cut it at both corners. The middle part i take as the droplet, and the left and right part, i reassamble to be my surface line. There might be better, more stable approaches, but this works for me. Thanks!
You can try this approach,
Find contour and approxPolyDP.
Suppose you got approxPolyDP point like P1,P2,P3 etc...
Now calculate angle between consecutive line, that is angle between line(P1,P2), line(P2,P3) etc.. and check the difference in angle for each adjustment line, if the difference comes close to 90 degree you can say there is a corner.
For Angle you can use the equation
double Angle = atan2(y2 - y1, x2 - x1) * 180.0 / CV_PI;
Related
I am trying to understand how Centroid to Contour (CtC) detector works. Here is a sample code that I have found on Github, and I am trying to understand what is the idea behind this. In this sample, the author trying to detect speed sign using CtC. There are just two important functions:
pre_process()
CtC_features
I have understood a part of the code and how it works but I have some problems in understanding how CtC_features function works.
If you can help me I would like to understand the following parts (just 3 points):
Why if centroid.x > curr.x we need to add PI value to the angle result ( if (centroid.x > curr.x) ang += 3.14159; //PI )
When we start selecting the features on line 97 we set the start angle ( double ang = - 1.57079; ). Why is this half the pi value and negative? How was this value chosen?
And a more general question, how can you know that what feature you select are related to speed limit sign? You find the centroid of the image and adjust the angle in the first step, but in the second step how can you know if ( while (feature_v[i].first > ang) ) the current angle is bigger than your hardcode angle ( in first case ang = - 1.57079) then we add that distance as a feature.
I would like to understand the idea behind this code and if someone with more experience and with some knowledge about trigonometry would help me it will be just great.
Thank you.
The code you provided is not the best, but let's see what happens.
I took this starting image of a sign:
Then, pre_process is called, which basically runs a Canny edge detector, along with some tricks which should lead to a better edge detection. I won't look into them, but this is what it returns:
Not the greatest. Maybe some parameter tuning would help.
But now, CtC_features is called, which is the scope of the question. The role of CtC_features is to obtain some features for a machine learning algorithms. This amounts to finding a numerical description of the image which would help the ML algorithm detect the sign. Such a description can be anything. Think about how someone who never saw a STOP sign and does not know how to read would describe it. They would say something like "A red, flat plate, with 8 sides and some white stuff in the middle". Based on this description, someone might be able to tell it's a STOP sign. We want to do the same, but since computers are computers, we look for numerical features. And, with them, some algorithm could be trained to "learn" what features each sign has.
So, let's see what features does CtC_features obtains from the contours.
The first thing it does is to call findContours. This function takes a binary image and returns arrays of points representing the contours of the image. Basically, it takes the edges and puts them into arrays of points. With connectivity, so we basically know which points are connected. If we use the code from here for visualization, we can see what happens:
So, the array contours is a std::vector<std::vector<cv::Point>> and you have in each sub-array a continuous contour, here drawn with a different color.
Next, we compute the number of points (edge pixels), and do an average over their coordinates to find the centroid of the edge image. The centroid is the filled circle:
Then, we iterate over all points, and create a vector of std::pair<double, double>, recording for each point the distance from the centroid and the angle. The angle function is defined at the bottom of the file as
double angle(Point2f a, Point2f b) {
return atan((a.y - b.y) / (a.x - b.x));
}
It basically computes the angle of the line from a to b with respect to the x axis, using the arctangent function. I'll let you watch a video on arctangent, but tl;dr is that it gives you the angle from a ratio. In radians (a circle is 2 PI radians, half a circle is PI radians). The problem is that the function is periodic, with a period of PI. This means that there are 2 angles on the circle (the circle of all points at the same distance around the centroid) which give you the same value. So, we compute the ratio (the ratio is btw known as the tangent of the angle), apply the inverse function (arctangent) and we get an angle (corresponding to a point). But what if it's the other point? Well, we know that the other point is exactly with PI degrees offset (it is diametrically opposite), so we add PI if we detect that it's the other point.
The picture below also helps understand why there are 2 points:
The tangent of the angle is highlighted vertical distance,. But the angle on the other side of the diagonal line, which intersects the circle in the bottom left, also has the same tangent. The atan function gives the tangents only for angles on the left side of the center. Note that there are no 2 directions with the same tangent.
What the check does is to ask whether the point is on the right of the centroid. This is done in order to be able to add a half a circle (PI radians or 180 degrees) to correct for the result of atan.
Now, we know the distance (a simple formula) and we have found (and corrected) for the angle. We insert this pair into the vector feature_v, and we sort it. The sort function, called like that, sorts after the first element of the pair, so we sort after the angle, then after distance.
The interval variable:
int degree = 10;
double interval = double((double(degree) / double(360)) * 2 * 3.14159); //5 degrees interval
simply is value of degree, converted from degrees into radians. We need radians since the angles have been computed so far in radians, and degrees are more user friendly. Yep, the comment is wrong, the interval is 10 degrees, not 5.
The ang variable defined below it is -PI / 2 (a quarter of a circle):
double ang = - 1.57079;
Now, what it does is to divide the points around the centroid into bins, based on the angle. Each bin is 10 degrees wide. This is done by iterating over the points sorted after angle, all are accumulated until we get to the next bin. We are only interested in the largest distance of a point in each bin. The starting point should be small enough that all the direction (points) are captured.
In order to understand why it starts from -PI/2, we have to get back at the trigonometric function diagram above. What happens if the angle goes like this:
See how the highlighted vertical segment goes "downwards" on the y axis. This means that its length (and implicitly the tangent) is negative here. Also, the angle is considered to be negative (otherwise there would be 2 angles on the same side of the center with the same tangent). Now, we are interested in the range of angles we have. It's all the angles on the right side of the centroid, starting from the bottom at -PI/2 to the top at PI/2. A range of PI radians, or 180 degrees. This is also written in the documentation of atan:
If no errors occur, the arc tangent of arg (arctan(arg)) in the range [-PI/2, +PI/2] radians, is returned.
So, we simply split all the possible directions (360 degrees) into buckets of 10 degrees, and take the distance of the farthest point in each bin. Since the circle has 360 degrees, we'll get 360 / 10 = 36 bins. Then, these are normalized such that the greatest value is 1. This helps a bit with the machine learning algorithm.
How can we know if the point we selected belongs to the sign? We don't. Most computer vision make some assumptions regarding the image in order to simplify the problem. The idea of the algorithm is to determine the shape of the sign by recording the distance from the center to the edges. This makes the assumption that the centroid is roughly in the middle of the sign. Depending on the ML algorithm used, and on the training data, different levels of robustness can be obtained.
Also, it assumes that (some of) the edges can be reliably identified. See how in my image, the algorithm was not able to detect the upper left edge?
The good news is that this doesn't have to be perfect. ML algorithms know how to handle this variation (up to some extent) provided that they are appropriately trained. It doesn't have to be perfect, but it has to be good enough. In order to answer what good enough means, what are the actual limitations of the algorithm, some more testing needs to be done, as well as some understanding of the ML algorithm used. But this is also why ML is so popular in vision: it can handle a lot of variation quite well.
At the end, we basically get an array of 36 numbers, one for each of the 36 bins of 10 degrees, representing the maximum distance of a point in the bin. I assume this is because the developer of the algorithm wanted a way to capture the shape of the sign, by looking at distances from center in various directions. This assumes that no edges are detected in the background, and the sign looks something like:
The max distance is used to pick the border, and not the or other symbols on the sign.
It is not directly used here, but a possibly related reading is the Hough transform, which uses a similar particularization to detect straight lines in an image.
I am working on a program which draws shapes based on a cgm file input. I am trying to draw elliptical arc and it gives the opening portion in terms of a start and end vector from the center of the arc. I need help calculating the angle to the vector so I can draw.
I have been trying to use the standard atan2(y/x) but then I found it is valid for circles and not ellipses.
This image gives an example of what I'm trying to do. I am looking for angles A and B.
edit: This is related to my other question here. (Also note, this question is based on the math behind my problem while the other question was for programming help with qt.)
The wiki page on ellipses kind of shows why the math isn't working but I'm not sure how to implement it.
The angles A and B you were drawing in your picture in fact have nothing to do with the ellipse.
Just calculate once the angle between the x-axis and the line from origin to point (75,50). This is given by arctan(50/75) = 33.69°. And by symmetry, it is the same as the angle to point (75, -50).
Then, by simple trigonometry, for angle A you get A = 360° - 33.69°, whereas for B you get B= 180° + 33.69°.
Considering A, this is the same information that is obtained by atan2(-50, 75). However, the result of atan2 is in (i) in radians and (ii) in the range [-pi, pi]. You could add 2*pi and express it in angles and you get the same result as above.
I want to follow the rightmost edge in the following picture with a line following robot.
I tried simple "thresholding", but unfortunately, it includes the blurry white halo:
The reason I threshold is to obtain a clean line from the Sobel edge detector:
Is there a good algorithm which I can use to isolate this edge/move along this edge? The one I am using currently seems error prone, but it's the best one I've been able to figure out so far.
Note: The edge may curve or be aligned in any direction, but a point on the edge will always lie very close to the center of the image. Here's a video of what I'm trying to do. It doesn't follow the edge after (1:35) properly due to the halo screwing up the thresholding.
Here's another sample:
Here, I floodfill the centermost edge to separate it from the little bump in the bottom right corner:
Simplest method (vertical line)
If you know that your image will have black on the right side of the line, here's a simple method:
1) apply the Sobel operator to find the first derivative in the x direction. The result will be an image that is most negative where your gradient is strongest. (Use a large kernel size to average out the halo effect. You can even apply a Gaussian blur to the image first, to get even more averaging if the 7x7 kernel isn't enough.)
2) For each row of the image, find the index of the minimum (i.e. most negative) value. That's your estimate of line position in that row.
3) Do whatever you want with that. (Maybe take the median of those line positions, on the top half and the bottom half of the image, to get an estimate of 2 points that describe the line.)
Slightly more advanced (arbitrary line)
Use this if you don't know the direction of the line, but you do know that it's straight enough that you can approximate it with a straight line.
1)
dx = cv2.Sobel(grayscaleImg,cv2.cv.CV_32F,1,0,ksize=7)
dy = cv2.Sobel(grayscaleImg,cv2.cv.CV_32F,0,1,ksize=7)
angle = np.atan2(dy,dx)
magnitudeSquared = np.square(dx)+np.square(dy)
You now have the angle (in radians) and magnitude of the gradient at each point in your image.
2) From here you can use basic numpy operations to find the line: Filter the points to only keep points where magnitudeSquared > some threshold. Then grab the most common angle (np.bincount() is useful for that). Now you know your line's angle.
3) Further filter the points to only keep points that are close to that angle. You now have all the points on your line. Fit a line through the coordinates of those points.
Most advanced and brittle (arbitrary curve)
If you really need to handle a curve, here's one way:
1) Use your method above to threshold the image. Manually tune the threshold until the white/black division happens roughly where you want it. (Probably 127 is not the right threshold. But if your lighting conditions are consistent, you might be able to find a threshold that works. Confirm it works across multiple images.)
2) Use OpenCV's findcontours() to fit a curve to the white/black boundary. If it's too choppy, use approxPolyDP() to simplify it.
I have a set of points on the unit sphere and a corresponding set of values being equal, for simplicity, to 0 and 1. Thus I'm constructing the characteristic function of a set on the sphere. Typically, I have several such sets, which form a partition of the sphere. An example is given in the figure.
I was wondering if paraview can find boundaries between the cells and compute the length and the curvature of the boundaries.
I read in a paper that using gradient reconstruction the guys managed to find the curvature of such contours. I imagine that if the curvature can be found, the length should be somewhat simpler. If the answer to the above question is yes, where should I look for the corresponding documentation?
For points on the sphere if they are build based on great-circle distance principle, it means all lines connecting points are of a shortest distance and plane goes through sphere center. In such case angle could be computed as arccos of scalar product.
R = 1;
angle = arccos(x1*x2 + y1*y2 + z1*z2);
length = R*angle;
And parametric line from p1 to p2 could be build using slerp interpolation.
slerp(t) = sin((1.0-t)*angle)/sin(angle)*p1 + sin(t*angle)/sin(angle)*p2;
where t is in [0...1] range
In such case curvature is 1/R for all great circle lines. That would be first thing I would try - try to match actual boundaries with those made from great-circle approach. If they match, that's the answer
Links
https://en.wikipedia.org/wiki/Great_circle
https://en.wikipedia.org/wiki/Great-circle_distance
https://en.wikipedia.org/wiki/Slerp
UPDATE
In case of non-great arcs I would propose following modification. Build great arc plane which goes through sphere center and on intersection with surface makes great arc between the points. Fix axis as a line going through those two points. Start rotating great arc plane along above mentioned axis till you get the exactly your arc of circle connecting two points. At this moment you could get rotation angle, compute your circle plane position and radius r, curvature as 1/r etc
I can successfully threshold images and find edges in an image. What I am struggling with is trying to extract the angle of the black edges accurately.
I am currently taking the extreme points of the black edge and calculating the angle with the atan2 function, but because of aliasing, depending on the point you choose the angle can come out with some degree of variation. Is there a reliable programmable way of choosing the points to calculate the angle from?
Example image:
For example, the Gimp Measure tool angle at 3.12°,
If you're writing your own library, then creating a robust solution for this problem will allow you to develop several independent chunks of code that you can string together to solve other problems, too. I'll assume that you want to find the corners of the checkerboard under arbitrary rotation, under varying lighting conditions, in the presence of image noise, with a little nonlinear pincushion/barrel distortion, and so on.
Although there are simple kernel-based techniques to find whole pixels as edge pixels, when working with filled polygons you'll want to favor algorithms that can find edges with sub-pixel accuracy so that you can perform accurate line fits. Even though the gradient from dark square to white square crosses several pixels, the "true" edge will be found at some sub-pixel point, and very likely not the point you'd guess by manually clicking.
I tried to provide a simple summary of edge finding in this older SO post:
what is the relationship between image edges and gradient?
For problems like yours, a robust solution is to find edge points along the dark-to-light transitions with sub-pixel accuracy, then fit lines to the edge points, and use the line angles. If you are processing a true camera image, and if there is an uncorrected radial distortion in the image, then there are some potential problems with measurement accuracy, but we'll ignore those.
If you want to find an accurate fit for an edge, then it'd be great to scan for sub-pixel edges in a direction perpendicular to that edge. That presupposes that we have some reasonable estimate of the edge direction to begin with. We can first find a rough estimate of the edge orientation, then perform an accurate line fit.
The algorithm below may appear to have too many steps, but my purpose is point out how to provide a robust solution.
Perform a few iterations of erosion on black pixels to separate the black boxes from one another.
Run a connected components algorithm (blob-finding algorithm) to find the eroded black squares.
Identify the center (x,y) point of each eroded square as well as the (x,y) end points defining the major and minor axes.
Maintain the data for each square in a structure that has the total area in pixels, the center (x,y) point, the (x,y) points of the major and minor axes, etc.
As needed, eliminate all components (blobs) that are too small. For example, you would want to exclude all "salt and pepper" noise blobs. You might also temporarily ignore checkboard squares that are cut off by the image edges--we can return to those later.
Then you'll loop through your list of blobs and do the following for each blob:
Determine the direction roughly perpendicular to the edges of the checkerboard square. How you accomplish this depends in part on what data you calculate when you run your connected components algorithm. In a general-purpose image processing library, a standard connected components algorithm will determine dozens of properties and measurements for each individual blob: area, roundness, major axis direction, minor axis direction, end points of the major and minor axis, etc. For rectangular figures, it can be sufficient to calculate the topmost, leftmost, rightmost, and bottommost points, as these will define the four corners.
Generate edge scans in the direction roughly perpendicular to the edges. These must be performed on the original, unmodified image. This generally assumes you have bilinear interpolation implemented to find the grayscale values of sub-pixel (x,y) points such as (100.35, 25.72) since your scan lines won't fall exactly on whole pixels.
Use a sub-pixel edge point finding technique. In general, you'll perform a curve fit to the edge points in the direction of the scan, then find the real-valued (x,y) point at maximum gradient. That's the edge point.
Store all sub-pixel edge points in a list/array/collection.
Generate line fits for the edge points. These can use Hough, RANSAC, least squares, or other techniques.
From the line equations for each of your four line fits, calculate the line angle.
That algorithm finds the angles independently for each black checkerboard square. It may be overkill for this one application, but if you're developing a library maybe it'll give you some ideas about what sub-algorithms to implement and how to structure them. For example, the algorithm would rely on implementations of these techniques:
Image morphology (e.g. erode, dilate, close, open, ...)
Kernel operations to implement morphology
Thresholding to binarize an image -- the Otsu method is worth checking out
Connected components algorithm (a.k.a blob finding, or the OpenCV contours function)
Data structure for blob
Moment calculations for blob data
Bilinear interpolation to find sub-pixel (x,y) values
A linear ray-scanning technique to find (x,y) gray values along a specific direction (which will also rely on bilinear interpolation)
A curve fitting technique and means to determine steepest tangent to find edge points
Robust line fit technique: Hough, RANSAC, and/or least squares
Data structure for line equation, related functions
All that said, if you're willing to settle for a slight loss of accuracy, and if you know that the image does not suffer from radial distortion, etc., and if you just need to find the angle of the parallel lines defined by all checkboard edges, then you might try..
Simple kernel-based edge point finding technique (Laplacian on Gaussian-smoothed image)
Hough line fit to edge points
Choose the two line fits with the greatest number of votes, which should be one set of horizontal-ish lines and the other set of vertical-ish lines
There are also other techniques that are less accurate but easier to implement:
Use a kernel-based corner-finding operator
Find the angles between corner points.
And so on and so on. As you're developing your library and creating robust implementations of standalone functions that you can string together to create application-specific solutions, you're likely to find that robust solutions rely on more steps than you would have guessed, but it'll also be more clear what the failure mode will be at each incremental step, and how to address that failure mode.
Can I ask, what C++ library are you using to code this?
Jerry is right, if you actually apply a threshold to the image it would be in 2bit, black OR white. What you may have applied is a kind of limiter instead.
You can make a threshold function (if you're coding the image processing yourself) by applying the limiter you may have been using and then turning all non-white pixels black. If you have the right settings, the squares should be isolated and you will be able to calculate the angle.
Once this is done you can use a path finding algorithm to find some edge, any edge will do. If you find a more or less straight path, you can use the extreme points as you are doing now to determine the angle. Since the checker-board rotation is only relevant within 90 degrees, your angle should be modulo 90 degrees or pi over 2 radians.
I'm not sure it's (anywhere close to) the right answer, but my immediate reaction would be to threshold twice: once where anything but black is treated as white, and once where anything but white is treated as black.
Find the angle for each, then interpolate between the two angles.
Your problem have few solutions but all have one very important issue which you seem to neglect. Note: When you are trying to make geometrical calculation in the image, the points you use must be as far as possible one from the other. You are taking 2 points inside a single square. Those points are very close one to another, so a slight error in pixel location of of the points leads to a large error in the angle. Why do you use only a single square, when you have many squares in the image?
Here are few solutions:
Find the line angle of every square. You have at least 9 squares in the image, 4 lines in each square which give you total of 36 angles (18 will be roughly at 3[deg] and 18 will be ~93[deg]). Remove the 90[degrees] and you get 36 different measurements of the angle. Sort them and take the average of the middle 30 (disregarding the lower 3 and higher 3 measurements). This will give you an accurate result
Second solution, find the left extreme point of the leftmost square and the right extreme point of the rightmost square. Now calculate the angle between them. The result will be much more accurate because the points are far away.
A third algorithm which will give you accurate results because it doesn't involve finding any points and no need for thresholding. Just smooth the image, calculate gradients in X and Y directions (gx,gy), calculate the angle of the gradient in each pixel atan(gy,gx) and make histogram of the angles. You will have 2 significant peaks near the 3[deg] and 93[deg]. Just find the peaks by searching the maximum in the histogram. This will work even if you have a lot of noise in the image, even with antialising and jpg artifacts, and even if you have other drawings on the image. But remember, you must smooth the image a lot before calculating the derivatives.