I have points in 3D which make 2 or 3 side of rectangle. How I can calculate the coordinates of the cube's corners? Is it possible?
Updated: https://github.com/CPIGroup/3d-Camera-scanDimensions
This is just an idea, not a proven method.
First, find the planes. Randomly select 3 points, find a plane that passes through them, normalize the 4 parameters. Repeat 1000 or so times. You will end up with 1000 4-tuples of numbers. Use one of the clustering analysis methods to find 2 or 3 groups of 4-tuples that are very close together. Average each of the groups. These will be, approximately, planes of your box's sides.
Now make them more precise. For each plane, find all points that are close to it but not close to other planes (for some value of "close", perhaps to be found using a clustering method too). For each such group of points, find a best fit plane using least squares.
If you have three planes, great; intersect them and you have a vertex and three edges. For two planes, you only have one edge. Either way, you can now try to find other edges. For simplicity, consider your plane to be an XY plane and your known edge an X axis. You now need to find the leftmost (rightmost) vertical line such that most of the points are to the left (resp. right) of it. Project all the points to the X axis. You now have a 1-dimensional case of your original problem: there is a lot of random points on some interval, find the interval. Use a clustering method again.
I'm not super experienced with this, but possibly you could use RANSAC ?
There seem to be many papers on the plane detection from pointclouds using RANSAC
Also you might want to have a look at the Point Clouds Library(PCL).It's a pretty impressive project with many useful features including also planar segmentation
As soon as the planes are detected, it should be a matter of finding the edges/corners which should be a lot simpler.
Related
I have a set of point cloud, and I would like to test if there is a corner in a 3D room. So I would like to discuss my approach and if there is a better approach or not in terms of speed, because I want to test it on mobile phones.
I will try to use hough tranform to detect lines, then I will try to see if there are three lines that are intersecting and they make a two plane that are intersecting too.
If the point cloud data comes from a depth sensor, then you have a relatively dense sampling of your walls. One thing I found that works well with depth sensors (e.g. Kinect or DepthSense) is a robust version of the RANSAC procedure that #MartinBeckett suggested.
Instead of picking 3 points at random, pick one point at random, and get the neighboring points in the cloud. There are two ways to do that:
The proper way: use a 3D nearest neighbor query data structure, like a KD-tree, to get all the points within some small distance from your query point.
The sloppy but faster way: use the pixel grid neighborhood of your randomly selected pixel. This may include points that are far from it in 3D, because they are on a different plane/object, but that's OK, since this pixel will not get much support from the data.
The next step is to generate a plane equation from that group of 3D points. You can use PCA on their 3D coordinates to get the two most significant eigenvectors, which define the plane surface (the last eigenvector should be the normal).
From there, the RANSAC algorithm proceeds as usual: check how many other points in the data are close to that plane, and find the plane(s) with maximal support. I found it better to find the largest support plane, remove the supporting 3D points, and run the algorithm again to find other 'smaller' planes. This way you may be able to get all the walls in your room.
EDIT:
To clarify the above: the support of a hypothesized plane is the set of all 3D points whose distance from that plane is at most some threshold (e.g. 10 cm, should depend on the depth sensor's measurement error model).
After each run of the RANSAC algorithm, the plane that had the largest support is chosen. All the points supporting that plane may be used to refine the plane equation (this is more robust than just using the neighboring points) by performing PCA/linear regression on the support set.
In order to proceed and find other planes, the support of the previous iteration should be removed from the 3D point set, so that remaining points lie on other planes. This may be repeated as long as there are enough points and best plane fit error is not too large.
In your case (looking for a corner), you need at least 3 perpendicular planes. If you find two planes with large support which are roughly parallel, then they may be the floor and some counter, or two parallel walls. Either the room has no visible corner, or you need to keep looking for a perpendicular plane with smaller support.
Normal approach would be ransac
Pick 3 points at random.
Make a plane.
Check if each other point lies on the plane.
If enough are on the plane - recalculate a best plane from all these points and remove them from the set
If not try another 3 points
Stop when you have enough planes, or too few points left.
Another approach if you know that the planes are near vertical or near horizontal.
pick a small vertical range
Get all the points in this range
Try and fit 2d lines
Repeat for other Z ranges
If you get a parallel set of lines in each Z slice then they are probably have a plane - recalculate the best fit plane for the points.
I would first like to point out
Even though this is an old post, I would like to present a complementary approach, similar to Hough Voting, to find all corner locations, composed of plane intersections, jointly:
Uniformly sample the space. Ensure that there is at least a distance $d$ between the points (e.g. you can even do this is CloudCompare with a 'space' subsampling)
Compute the point cloud normals at these points.
Randomly pick 3 points from this downsampled cloud.
Each oriented point (point+plane) defines a hypothetical plane. Therefore, each 3 point picked define 3 planes. Those planes, if not parallel and not intersecting at a line, always intersect at a single point.
Create a voting space to describe the corner: The intersection of the 3 planes (the point) might a valid parameterization. So our parameter space has 3 free parameters.
For each 3 points cast a vote in the accumulator space to the corner point.
Go to (2) and repeat until all sampled points are exhausted, or enough iterations are done. This way we'll be casting votes for all possible corner locations.
Take the local maxima of the accumulator space. Depending on the votes, we'll be selecting the corners from intersection of the largest planes (as they'll receive more votes) to the intersection of small planes. The largest 4 are probably the corners of the room. If not, one could also consider the other local maxima.
Note that the voting space is a quantized 3D space and the corner location will be a rough estimate of the actual one. If desired, one could store the planes intersection at that very location and refine them (with iterative optimization similar to ICP or etc) to get a very fine corner location.
This approach will be quite fast and probably very accurate, given that you could refine the location. I believe it's the best algorithm presented so far. Of course this assumes that we could compute the normals of the point clouds (we can always do that at sample locations with the help of the eigenvectors of the covariance matrix).
Please also look here, where I have put out a list of plane-fitting related questions at stackoverflow:
3D Plane fitting algorithms
I can successfully threshold images and find edges in an image. What I am struggling with is trying to extract the angle of the black edges accurately.
I am currently taking the extreme points of the black edge and calculating the angle with the atan2 function, but because of aliasing, depending on the point you choose the angle can come out with some degree of variation. Is there a reliable programmable way of choosing the points to calculate the angle from?
Example image:
For example, the Gimp Measure tool angle at 3.12°,
If you're writing your own library, then creating a robust solution for this problem will allow you to develop several independent chunks of code that you can string together to solve other problems, too. I'll assume that you want to find the corners of the checkerboard under arbitrary rotation, under varying lighting conditions, in the presence of image noise, with a little nonlinear pincushion/barrel distortion, and so on.
Although there are simple kernel-based techniques to find whole pixels as edge pixels, when working with filled polygons you'll want to favor algorithms that can find edges with sub-pixel accuracy so that you can perform accurate line fits. Even though the gradient from dark square to white square crosses several pixels, the "true" edge will be found at some sub-pixel point, and very likely not the point you'd guess by manually clicking.
I tried to provide a simple summary of edge finding in this older SO post:
what is the relationship between image edges and gradient?
For problems like yours, a robust solution is to find edge points along the dark-to-light transitions with sub-pixel accuracy, then fit lines to the edge points, and use the line angles. If you are processing a true camera image, and if there is an uncorrected radial distortion in the image, then there are some potential problems with measurement accuracy, but we'll ignore those.
If you want to find an accurate fit for an edge, then it'd be great to scan for sub-pixel edges in a direction perpendicular to that edge. That presupposes that we have some reasonable estimate of the edge direction to begin with. We can first find a rough estimate of the edge orientation, then perform an accurate line fit.
The algorithm below may appear to have too many steps, but my purpose is point out how to provide a robust solution.
Perform a few iterations of erosion on black pixels to separate the black boxes from one another.
Run a connected components algorithm (blob-finding algorithm) to find the eroded black squares.
Identify the center (x,y) point of each eroded square as well as the (x,y) end points defining the major and minor axes.
Maintain the data for each square in a structure that has the total area in pixels, the center (x,y) point, the (x,y) points of the major and minor axes, etc.
As needed, eliminate all components (blobs) that are too small. For example, you would want to exclude all "salt and pepper" noise blobs. You might also temporarily ignore checkboard squares that are cut off by the image edges--we can return to those later.
Then you'll loop through your list of blobs and do the following for each blob:
Determine the direction roughly perpendicular to the edges of the checkerboard square. How you accomplish this depends in part on what data you calculate when you run your connected components algorithm. In a general-purpose image processing library, a standard connected components algorithm will determine dozens of properties and measurements for each individual blob: area, roundness, major axis direction, minor axis direction, end points of the major and minor axis, etc. For rectangular figures, it can be sufficient to calculate the topmost, leftmost, rightmost, and bottommost points, as these will define the four corners.
Generate edge scans in the direction roughly perpendicular to the edges. These must be performed on the original, unmodified image. This generally assumes you have bilinear interpolation implemented to find the grayscale values of sub-pixel (x,y) points such as (100.35, 25.72) since your scan lines won't fall exactly on whole pixels.
Use a sub-pixel edge point finding technique. In general, you'll perform a curve fit to the edge points in the direction of the scan, then find the real-valued (x,y) point at maximum gradient. That's the edge point.
Store all sub-pixel edge points in a list/array/collection.
Generate line fits for the edge points. These can use Hough, RANSAC, least squares, or other techniques.
From the line equations for each of your four line fits, calculate the line angle.
That algorithm finds the angles independently for each black checkerboard square. It may be overkill for this one application, but if you're developing a library maybe it'll give you some ideas about what sub-algorithms to implement and how to structure them. For example, the algorithm would rely on implementations of these techniques:
Image morphology (e.g. erode, dilate, close, open, ...)
Kernel operations to implement morphology
Thresholding to binarize an image -- the Otsu method is worth checking out
Connected components algorithm (a.k.a blob finding, or the OpenCV contours function)
Data structure for blob
Moment calculations for blob data
Bilinear interpolation to find sub-pixel (x,y) values
A linear ray-scanning technique to find (x,y) gray values along a specific direction (which will also rely on bilinear interpolation)
A curve fitting technique and means to determine steepest tangent to find edge points
Robust line fit technique: Hough, RANSAC, and/or least squares
Data structure for line equation, related functions
All that said, if you're willing to settle for a slight loss of accuracy, and if you know that the image does not suffer from radial distortion, etc., and if you just need to find the angle of the parallel lines defined by all checkboard edges, then you might try..
Simple kernel-based edge point finding technique (Laplacian on Gaussian-smoothed image)
Hough line fit to edge points
Choose the two line fits with the greatest number of votes, which should be one set of horizontal-ish lines and the other set of vertical-ish lines
There are also other techniques that are less accurate but easier to implement:
Use a kernel-based corner-finding operator
Find the angles between corner points.
And so on and so on. As you're developing your library and creating robust implementations of standalone functions that you can string together to create application-specific solutions, you're likely to find that robust solutions rely on more steps than you would have guessed, but it'll also be more clear what the failure mode will be at each incremental step, and how to address that failure mode.
Can I ask, what C++ library are you using to code this?
Jerry is right, if you actually apply a threshold to the image it would be in 2bit, black OR white. What you may have applied is a kind of limiter instead.
You can make a threshold function (if you're coding the image processing yourself) by applying the limiter you may have been using and then turning all non-white pixels black. If you have the right settings, the squares should be isolated and you will be able to calculate the angle.
Once this is done you can use a path finding algorithm to find some edge, any edge will do. If you find a more or less straight path, you can use the extreme points as you are doing now to determine the angle. Since the checker-board rotation is only relevant within 90 degrees, your angle should be modulo 90 degrees or pi over 2 radians.
I'm not sure it's (anywhere close to) the right answer, but my immediate reaction would be to threshold twice: once where anything but black is treated as white, and once where anything but white is treated as black.
Find the angle for each, then interpolate between the two angles.
Your problem have few solutions but all have one very important issue which you seem to neglect. Note: When you are trying to make geometrical calculation in the image, the points you use must be as far as possible one from the other. You are taking 2 points inside a single square. Those points are very close one to another, so a slight error in pixel location of of the points leads to a large error in the angle. Why do you use only a single square, when you have many squares in the image?
Here are few solutions:
Find the line angle of every square. You have at least 9 squares in the image, 4 lines in each square which give you total of 36 angles (18 will be roughly at 3[deg] and 18 will be ~93[deg]). Remove the 90[degrees] and you get 36 different measurements of the angle. Sort them and take the average of the middle 30 (disregarding the lower 3 and higher 3 measurements). This will give you an accurate result
Second solution, find the left extreme point of the leftmost square and the right extreme point of the rightmost square. Now calculate the angle between them. The result will be much more accurate because the points are far away.
A third algorithm which will give you accurate results because it doesn't involve finding any points and no need for thresholding. Just smooth the image, calculate gradients in X and Y directions (gx,gy), calculate the angle of the gradient in each pixel atan(gy,gx) and make histogram of the angles. You will have 2 significant peaks near the 3[deg] and 93[deg]. Just find the peaks by searching the maximum in the histogram. This will work even if you have a lot of noise in the image, even with antialising and jpg artifacts, and even if you have other drawings on the image. But remember, you must smooth the image a lot before calculating the derivatives.
I'm trying to figure out how to decide how many vertices I need to have to make my circle look as smooth as possible.
Here is an example of two circles, both having 24 vertices:
As you see, the bigger the circle becomes, the more vertices I need to hide the straight lines.
At first I thought that the minimum length of one line on the edge should be 6px, but that approach failed when I increased the circle size: I got way too many vertices. I also thought about calculating the angles, but I quickly realised that angles doesn't differ on different sized circles. I also checked this answer, but I don't have a clue how to convert it into code (and some weird stuff there: th uses itself for calculating itself), and I think it doesn't even work, since the author is using the angle from one slice to the middle of circle, which doesn't change if the circle gets larger.
Then I realised that maybe the solution is to check the angle between two vertices at the edges, in this way:
As you see, the fewer vertices, the bigger the lengths are for those triangles. So this has to be the answer, I just don't know how to calculate the number of vertices by using this information.
The answer you link to actually implements exactly the idea you propose at the end of your question.
The decisive formula that you need from that answer is this one:
th = arccos(2 * (1 - e / r)^2 - 1)
This tells you the angle between two vertices, where r is the radius of the circle and e is the maximum error you're willing to tolerate, i.e. the maximum deviation of your polygon from the circle -- this is the error marked in your diagram. For example, you might choose to set e to 0.5 of a pixel.
Because th is measured in radians, and 360 degrees (a full circle) is equal to 2*pi in radians, the number of vertices you need is
num_vertices = ceil(2*pi/th)
In case you want to draw the polygon triangles from the center of the circle, the formula for the required number of sides is:
sides = PI / arccos(1 - error / radius)
where error is the maximum deviation of polygon from the circle, in pixels and radius is also expressed in pixels.
Error 0.33 seems to produce results indistinguishable from an ideal circle. Circles drawn at error 0.5, on close inspection still show some subtly visible angles between sides, especially visible in small circles.
This function obviously breaks down when radius is much smaller than error, and may produce NaN values. You may want to use a special case (for example draw 3 sides) in this situation.
The graph below shows number of sides obtained from the function, with error set to 0.33:
First of all, if you are using OpenGL or DirectX you can significantly decrease the number of vertices by using a triangle fan structure.
As for the problem of the amount of vertices, I would imagine the number of vertices required for a smooth circle to scale with the circumference. This scales with r, so I would advice to find a good factor A such that:
#vertices = A * r
The angles are the same in the two cases of 24 vertices.
But with larger circle, the human eye is able to better see the individual straight lines.
So you need some heuristic that takes into account
the angle between two consecutive line segments in the curve, and
the size, and possibly
the scaling for display.
The third point is difficult since one does not in general know the size at which some graphic will be displayed. E.g., an SVG format picture can be displayed at any size. The most general solution, I think, is to have direct support for various figures (Bezier lines, circles, etc.) in the renderer, and then define the figure with a few parameters instead of as a sequence of points. Or, define it in terms of some figure that the renderer supports, e.g. as a sequence of connected Bezier curves. That way, the renderer can add the necessary number of points to make it look smooth and nice.
However, I guess that you're not creating a renderer, so then perhaps only the first two points above are relevant.
I know the distance to various points on a plane, as it is being viewed from an angle. I want to find the equation for this plane from just that information (5 to 15 different points, as many as necessary).
I will later use the equation for the plane to estimate what the distance to the plane should be at different points; in order to prove that it is roughly flat.
Unfortunately, a google search doesn't bring much up. :(
If you, indeed, know distances and not coordinates, then it is ill-posed problem - there is infinite number of planes that will have points with any number of given distances from origin.
This is easy to verify. Let's take shortest distance D0, from set of given distances {D0..DN-1} , and construct a plane with normal vector {D0,0,0} (vector of length D0 along x-axis). For each of remaining lengths we now have infinite number of points that will lie in this plane (forming circles in-plane around (D0,0,0) point). Moreover, we can rotate all vectors by an arbitrary angle and get a new plane.
Here is simple picture in 2D (distances to a line; it's simpler to draw ;) ).
As we can see, there are TWO points on the line for each distance D1..DN-1 > D0 - one is shown for D1 and D2, and the two other for these distances would be placed in 4th quadrant (+x, -y). Moreover, we can rotate our line around origin by an arbitrary angle and still satisfy given distances.
I'm going to skip over the process of finding the best fit plane, it's been handled in some other answers, and talk about something else.
"Prove" takes us into statistical inference. The way this is done is you make a formal hypothesis "the surface is flat" and then see if the data supports rejecting this hypothesis at some confidence level.
So you can wind up saying "I'm not even 1% sure that the surface isn't flat" -- but you can't ever prove that it's flat.
Geometry? Sounds like a job for math.SE! What form will the equation take? Will it be a plane?
I will assume you want an accurate solution.
Find the absolute positions with geometry
Make a best fit regression line in C++ in 2 of the 3 dimensions.
I have 2 six faced solids. The only guarantee is that they each have 8 vertex3f's (verticies with x,y and z components). Given this, how can I find out if these are colliding?
I'm hesitant to answer after you deleted your last question while I was trying to answer it and made me lose my post. Please don't do that again. Anyway:
Not necessarily optimal, but obviously correct, based on constructive solid geometry:
Represent the two solids each as an intersection of 6 half-spaces. Note that this depends on convexity but nothing else, and extends to solids with more sides. My preferred representation for halfspaces is to choose a point on each surface (for example, a vertex) and the outward-pointing unit normal vector to that surface.
Intersect the two solids by treating all 12 half-spaces as the defining half-spaces for a new solid. (This step is purely conceptual and might not involve any actual code.)
Compute the surface/edge representation of the new solid and check that it's non-empty. One approach to doing this is to initially populate your surface/edge representation with one surface for each of the 12 half-spaces with edges outside the bounds of the 2 solids, then intersect its edges with each of the remaining 11 half-spaces.
It sounds like a bit of work, but there's nothing complicated. Only dot products, cross products (to get the initial representation), and projections.
It seems I'm too dumb to quit.
Consider this. If any edge of solid 1 intersects any face of solid 2, you have a collision. That's not quite comprehensive because there are case when one is is fully contained in the other, which you can test by determining if the center of either is contained in the other.
Checking edge face intersection works like this.
Define the edge as vector starting from one vertex running to the other. Take note of the length, L, of the edge.
Define the plane segments by a vertex, a normal, an in-plane basis, and the positions of the remaining vertices in that basis.
Find the intersection of the line and the plane. In the usual formulation you will be able to get both the length along the line, and the in-plane coordinates of the intersection in the basis that you have chosen.
The intersection must line as length [0,L], and must lie inside the figure in the plane. That last part is a little harder, but has a well known general solution.
This will work. For eloquence, I rather prefer R..'s solution. If you need speed...well, you'll just have to try them and see.
Suppose one of your hexahedrons H1 has vertices (x_1, y_1, z_1), (x_2, y_2, z_2), .... Find the maximum and minimum in each coordinate: x_min = min(x_1, x_2, ...), x_max = max(x_1, x_2,...), and so on. Do the same for the other hexahedron H2.
If the interval [x_min(H1), x_max(H1)] and the interval [x_min(H2), x_max(H2)] do not intersect (that is, either x_max(H1) < x_min(H2) or x_max(H2) < x_min(H1)), then the hexahedrons cannot possibly collide. Repeat this for the y and z coordinates. Qualitatively, this is like looking at the shadow of each hexahedron on the x-axis. If they don't overlap, the polyhedrons can't collide.
If any of the intervals do overlap, you'll have to move on to more precise collision detection. This will be a lot trickier. The obvious brute force method is to check if any of the edges of one intersects any of the faces of the other, but I imagine you can do a lot better than that.
The brute force way to check if an edge intersects a face... First you'd find the intersection of the line defined by the edge with the plane defined by the face (see wikipedia, for example). Then you have to check if that point is actually on the edge and the face. The edge is easy - just see if the coordinates are between the coordinates of the two vertices defining the edge. The face is trickier, especially with no guarantees about it being convex. In the general case, you'll have to just see which side of the half-planes defined by each edge it's on. If it's on the inside half-plane for all of them, it's inside the face. I unfortunately don't have time to type that all up now, but I bet googling could aid you some there. But of course, this is all brute force, and there may be a better way. (And dmckee points out a special case that this doesn't handle)