I am looking for a way of approximating a surface based on a set of 3D data points. For this purpose I would like to use a method based on radial basis functions but I cannot find a free implementation in C++.
I looked in ITK, VTK and open CV but I did not find anything...
Does anyone knows a free implementation of such an algorithm ?
Any suggestion about the reconstruction of a surface based on a set of 3D data points is also more than welcome ! :)
3D surface reconstruction can be challenging. I would first recommend taking a look at PCL. The Point Cloud Library has grown into a nice set of tools for 3D point management and interpretation, and its license and API sound compatible with your needs. The surface reconstruction features of the library appear to be most applicable. In fact, RBF reconstruction is supported.
If PCL doesn't work, there are other options:
MeshLab,
This SO post provides a nice summary, and
of course, Wikipedia provides some links
Finally, you might search CiteSeerX, Google Scholar, etc. for papers like this one. As an example, a search for "3D Surface Reconstruction" at CiteSeerX yields many hits. RBF-based reconstruction is just one of many methods: is your application truly limited to radial basis functions? If not, there are many choices, (i.e. Ball Pivoting Algorithm). See this survey paper for some comparisons.
Related
I'm a dev with about a decade of enterprise software engineering under his belt, and my hobbyist interests have steered me into the vast and scary realm of computer vision (CV).
One thing that is not immediately clear to me is the division of labor between Structure from Motion (SFM) tools and Multi View Stereo (MVS) tools.
Specifically, CMVS appears to be the best-in-show MVS tool, and Bundler seems to be one of the better open source SFM tools out there.
Taken from CMVS's own homepage:
You should ALWAYS use CMVS after Bundler and before PMVS2
I'm wondering: why?!? My understanding of SFM tools is that they perform the 3D reconstruction for you, so why do we need MVS tools in the first place? What value/processing/features do they add that SFM tools like Bundler can't address? Why the proposed pipeline of:
Bundler -> CMVS -> PMVS2
?
Quickly put, Structure from Motion (SfM) and MultiView Stereo (MVS) techniques are complementary, as they do not deal with the same assumptions. They also differ slightly in their inputs, MVS requiring camera parameters to run, which is estimated (output) by SfM. SfM only gives a coarse 3D output, whereas PMVS2 gives a more dense output, and finally CMVS is there to circumvent some limitations of PMVS2.
The rest of the answer provides an high-level overview of how each method works, explaining why it is this way.
Structure from Motion
The first step of the 3D reconstruction pipeline you highlighted is a SfM algorithm that could be done using Bundler, VisualSFM, OpenMVG or the like. This algorithm takes in input some images and outputs the camera parameters of each image (more on this later) as well as a coarse 3D shape of the scene, often called the sparse reconstruction.
Why does SfM outputs only a coarse 3D shape? Basically, SfM techniques begins by detecting 2D features in every input image and matching those features between pairs of images. The goal is, for example, to tell "this table corner is located at those pixels locations in those images." Those features are described by what we call descriptors (like SIFT or ORB). Those descriptors are built to represent a small region (ie. a bunch of neighboring pixels) in images. They can represent reliably highly textured or rough geometries (e.g., edges), but these scene features need to be unique (in the sense distinguishing) throughout the scene to be useful. For example (maybe oversimplified), a wall with repetitive patterns would not be very useful for the reconstruction, because even though it is highly textured, every region of the wall could potentially match pretty much everywhere else on the wall. Since SfM is performing a 3D reconstruction using those features, the vertices of the 3D scene reconstruction will be located on those unique textures or edges, giving a coarse mesh as output. SfM won't typically produce a vertex in the middle of surface without precise and distinguishing texture. But, when many matches are found between the images, one can compute a 3D transformation matrix between the images, effectively giving the relative 3D position between the two camera poses.
MultiView Stereo
Afterwards, the MVS algorithm is used to refine the mesh obtained by the SfM technique, resulting in what is called a dense reconstruction. This algorithm requires the camera parameters of each image to work, which is output by the SfM algorithm. As it works on a more constrained problem (since they already have the camera parameters of every image like position, rotation, focal, etc.), MVS will compute 3D vertices on regions which were not (or could not be) correctly detected by descriptors or matched. This is what PMVS2 does.
How can PMVS work on regions where 2D feature descriptor would difficultly match? Since you know the camera parameters, you know a given pixel in an image is the projection of a line in another image. This approach is called epipolar geometry. Whereas SfM had to seek through the entire 2D image for every descriptor to find a potential match, MVS will work on a single 1D line to find matches, simplifying the problem quite a deal. As such, MVS usually takes into account illumination and object materials into its optimization, which SfM does not.
There is one issue, though: PMVS2 performs a quite complex optimization that can be dreadfully slow or take an astronomic amount of memory on large image sequences. This is where CMVS comes into play, clustering the coarse 3D SfM output into regions. PMVS2 will then be called (potentially in parallel) on each cluster, simplifying its execution. CMVS will then merge each PMVS2 output in an unified detailed model.
Conclusion
Most of the information provided in this answer and many more can be found in this tutorial from Yasutaka Furukawa, author of CMVS and PMVS2:
http://www.cse.wustl.edu/~furukawa/papers/fnt_mvs.pdf
In essence, both techniques emerge from two different approaches: SfM aims to perform a 3D reconstruction using a structured (but theunknown) sequence of images while MVS is a generalization of the two-view stereo vision, based on human stereopsis.
I want to create a random fractal terrain on the gpu (with a compute shader). I've started with implementing marching cubes: Generating Complex Procedural Terrains Using the GPU, and it works really good: marching cubes on the gpu. However, marching cubes can't extract sharp features or use an adaptive resolution.
So I looked for an advanced isosurface extraction algorithm and found Dual Contouring of Hermite Data. So I implemented dc in Java to test this algorithm and it looks great: dc on the cpu. (There are some holes in the mesh and no sharp features, but I was too lazy to implement/fix this because it's only a prototype.)
But I've noticed some negative aspects:
Intercell-dependent. (I have no idea how to port this to the gpu: the only resource I've found is Dual Contouring with OpenCL.)
I don't know how to create a chunk system, because there are no "clear" borders https://i.stack.imgur.com/62dy6.png.
So I continued my search for a better algorithm and found Cubical Marching Squares: Adaptive Feature Preserving Surface Extraction from Volume Data. This seems to be the perfect algorithm for me: intercell-independent, adaptive, sharp features, primal structure and even manifold. Unfortunately, there are no resources on how to implement this algorithm except for this Cubical Marching Squares Implementation. I think I understand the two parts of the algorithm: create a grid, for each cell:
Subdivide until the maximum depth is reached or there is no need to do so.
Split each cell into 6 faces, extract their surface and stitch them together.
But I don't know how to connect those two parts (especially the part with transitional faces, page 38).
So does anybody know how to implement dc as a shader, how to implement cms or a better algorithm (maybe dual marching cubes, I think it has the same problem as dc, but I haven't tested it yet)?
As you have already mentioned for the GPU you would need no intercell-dependencies... I'm sure people have come up with workarounds for that on DC but CMS should be one of the best for all those things that you need, namely intercell-independent (by definition), preserves sharp features, creates manifold geometry, and supports adaptive resolution.
In terms of resources on CMS I agree they are quite limited.
The original paper: http://graphics.csie.ntu.edu.tw/CMS/
This project by Matt Keeter has a 'C' CMS implementation in it (https://www.mattkeeter.com/projects/kokopelli/)
https://github.com/mkeeter/kokopelli <-- code
I used Matt Keeter's implementation as a reference for my partial 'C++' implementation (the thesis that you linked), here is the code for it in case you didn't find it:
https://bitbucket.org/GRassovsky/cubical-marching-squares
However keep in mind that it is indeed partial, that is to say it has the main algorithm working (adaptive and manifold), but I haven't got around to implement the sharp-feature preservation, 2D and 3D disambiguation, etc. It is also currently a basic CPU implementation... I had the good intentions to implement all those things and make a GPU implementation but for now have had no time.
"But I don't know how to connect those two parts" - that only goes to say how badly I have written my thesis, because I try to explain how I do this :D (Mind you, I am not sure exactly how this was done in the original paper... should we say that some things are not very clear there and you would have to use your imagination :))
Working on a small side project related to Computer Vision, mostly to try playing around with OpenCV. It lead me to an interesting question:
Using feature detection to find known objects in an image isn't always easy- objects are hard to find, especially if the features of the target object aren't great.
But if I could choose ahead of time what it is I'm looking for, then in theory I could generate for myself an optimal image for detection. Any quality that makes feature detection hard would be absent, and all the qualities that make it easy would exist.
I suspect this sort of thought went into things like QR codes, but with the limitations that they wanted QR codes to be simple, and small.
So my question for you: How would you generate an optimal image for later recognition by a camera? What if you already know that certain problems like skew, or partial obscuring would occur?
Thanks very much
I think you need something like AR markers.
Take a look at ArToolkit, ArToolkitPlus or Aruco libraries, they have marker generators and detectors.
And papeer about marker generation: http://www.uco.es/investiga/grupos/ava/sites/default/files/GarridoJurado2014.pdf
If you plan to use feature detection, than marker should be specific to used feature detector. Common practice for detector design is good response to "corners" or regions with high x,y gradients. Also you should note the scaling of target.
The simplest detection can be performed with BLOBS. It can be faster and more robust than feature points. For example you can detect circular blobs or rectangular.
Depending on the distance you want to see your markers from and viewing conditions/backgrounds you typically use and camera resolution/noise you should choose different images/targets. Under moderate perspective from a longer distance a color target is pretty unique, see this:
https://surf-it.soe.ucsc.edu/sites/default/files/velado_report.pdf
at close distances various bar/QR codes may be a good choice. Other than that any flat textured object will be easy to track using homography as opposed to 3D objects.
http://docs.opencv.org/trunk/doc/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.html
Even different views of 3d objects can be quickly learned and tracked by such systems as Predator:
https://www.youtube.com/watch?v=1GhNXHCQGsM
then comes the whole field of hardware, structured light, synchronized markers, etc, etc. Kinect, for example, uses a predefined pattern projected on the surface to do stereo. This means it recognizes and matches million of micro patterns per second creating a depth map from the matched correspondences. Note that one camera sees the pattern and while another device - a projector generates it working as a virtual camera, see
http://article.wn.com/view/2013/11/17/Apple_to_buy_PrimeSense_technology_from_the_360s_Kinect/
The quickest way to demonstrate good tracking of a standard checkerboard pattern is to use pNp function of open cv:
http://www.juergenwiki.de/work/wiki/lib/exe/fetch.php?media=public:cameracalibration_detecting_fieldcorners_of_a_chessboard.gif
this literally can be done by calling just two functions
found = findChessboardCorners(src, chessboardSize, corners, camFlags);
drawChessCornersDots(dst, chessboardSize, corners, found);
To sum up, your question is very broad and there are multiple answers and solutions. Formulate your viewing condition, camera specs, backgrounds, distances, amount of motion and perspective you expect to have indoors vs outdoors, etc. There is no such a thing as a general average case in computer vision!
I am wondering whether there is any existing library or software which is on tetrahedral mesh simplification and written in C/C++. You may know that there is a famous algorithm on triangular mesh based on Quadric Error Metrics, called Surface Simpliļ¬cation Using Quadric Error Metrics. I reckon there should be something similar but applied to tetrahedral mesh. Thanks!
You should probably search in CGAL library CGAL. Its the best for computational geometry algorithms. I am sure that someone should have already implemented mesh simplification in CGAL.
The closest I can find to what you are looking for exists here it allows tetrahedral volumetric rendering.
Not sure if this is what you are looking for, as the original question and the comments seem to differ slightly in the end aim.
Let me know if you need any further info:)
I am writing a very simple 3d particle software rendering system, but I am only really interested in coding the particle system and the rasterizer, so what I am looking for, is the easiest way to go from 3d particle coordinates, through camera, to screen coordinates, but with features like variable FOV, and targeted (look at) camera.
Any additional features such as distance from point to point, bounding volumes etc. would be a bonus, but ease of use is more important to me than features.
The only license requirement is that it's free (as in beer).
You probably want a scenegraph library. Many C++ scene graph libraries exist. OpenScenegraph is popular, Coin3D (free for non-commercial use) is an implementation of the OpenInventor spec, any of them would probably fit your need as it doesn't sound like you need any cutting-edge support. There is also Panda3D, which I've heard is good if you're into Python.
You could do all of this in a straight low-level toolkit like OpenGL but without prior experience it takes a lot longer to get a handle on using OpenGL than any of the scenegraph libraries.
When choosing a scenegraph library it will probably just come down to personal preference as to which API you prefer.
Viewing is done with elementary transformations, the same way that model transformations are done. If you want some convenience functions like gluLookAt() in GLU, then I don't know, but it would be really easy to make your own.
If you do want to make your own Look At function etc. then I can recommend eigen which is a really easy to use linear algebra library for C++.
If you're trying to just focus on the rendering portion of a particle system, I'd go with an established 3D rendering library.
Given your description, you could consider trying to just add your particle rasterization to one or both of the software renderers in Irrlicht. One other advantage of this would be that you could compare your results to the DX/OGL particle effect renderers that already exist. All of the plumbing/camera management/etc would be done for you.
You may also want to have a look at the Armadillo C++ library