I'm a dev with about a decade of enterprise software engineering under his belt, and my hobbyist interests have steered me into the vast and scary realm of computer vision (CV).
One thing that is not immediately clear to me is the division of labor between Structure from Motion (SFM) tools and Multi View Stereo (MVS) tools.
Specifically, CMVS appears to be the best-in-show MVS tool, and Bundler seems to be one of the better open source SFM tools out there.
Taken from CMVS's own homepage:
You should ALWAYS use CMVS after Bundler and before PMVS2
I'm wondering: why?!? My understanding of SFM tools is that they perform the 3D reconstruction for you, so why do we need MVS tools in the first place? What value/processing/features do they add that SFM tools like Bundler can't address? Why the proposed pipeline of:
Bundler -> CMVS -> PMVS2
?
Quickly put, Structure from Motion (SfM) and MultiView Stereo (MVS) techniques are complementary, as they do not deal with the same assumptions. They also differ slightly in their inputs, MVS requiring camera parameters to run, which is estimated (output) by SfM. SfM only gives a coarse 3D output, whereas PMVS2 gives a more dense output, and finally CMVS is there to circumvent some limitations of PMVS2.
The rest of the answer provides an high-level overview of how each method works, explaining why it is this way.
Structure from Motion
The first step of the 3D reconstruction pipeline you highlighted is a SfM algorithm that could be done using Bundler, VisualSFM, OpenMVG or the like. This algorithm takes in input some images and outputs the camera parameters of each image (more on this later) as well as a coarse 3D shape of the scene, often called the sparse reconstruction.
Why does SfM outputs only a coarse 3D shape? Basically, SfM techniques begins by detecting 2D features in every input image and matching those features between pairs of images. The goal is, for example, to tell "this table corner is located at those pixels locations in those images." Those features are described by what we call descriptors (like SIFT or ORB). Those descriptors are built to represent a small region (ie. a bunch of neighboring pixels) in images. They can represent reliably highly textured or rough geometries (e.g., edges), but these scene features need to be unique (in the sense distinguishing) throughout the scene to be useful. For example (maybe oversimplified), a wall with repetitive patterns would not be very useful for the reconstruction, because even though it is highly textured, every region of the wall could potentially match pretty much everywhere else on the wall. Since SfM is performing a 3D reconstruction using those features, the vertices of the 3D scene reconstruction will be located on those unique textures or edges, giving a coarse mesh as output. SfM won't typically produce a vertex in the middle of surface without precise and distinguishing texture. But, when many matches are found between the images, one can compute a 3D transformation matrix between the images, effectively giving the relative 3D position between the two camera poses.
MultiView Stereo
Afterwards, the MVS algorithm is used to refine the mesh obtained by the SfM technique, resulting in what is called a dense reconstruction. This algorithm requires the camera parameters of each image to work, which is output by the SfM algorithm. As it works on a more constrained problem (since they already have the camera parameters of every image like position, rotation, focal, etc.), MVS will compute 3D vertices on regions which were not (or could not be) correctly detected by descriptors or matched. This is what PMVS2 does.
How can PMVS work on regions where 2D feature descriptor would difficultly match? Since you know the camera parameters, you know a given pixel in an image is the projection of a line in another image. This approach is called epipolar geometry. Whereas SfM had to seek through the entire 2D image for every descriptor to find a potential match, MVS will work on a single 1D line to find matches, simplifying the problem quite a deal. As such, MVS usually takes into account illumination and object materials into its optimization, which SfM does not.
There is one issue, though: PMVS2 performs a quite complex optimization that can be dreadfully slow or take an astronomic amount of memory on large image sequences. This is where CMVS comes into play, clustering the coarse 3D SfM output into regions. PMVS2 will then be called (potentially in parallel) on each cluster, simplifying its execution. CMVS will then merge each PMVS2 output in an unified detailed model.
Conclusion
Most of the information provided in this answer and many more can be found in this tutorial from Yasutaka Furukawa, author of CMVS and PMVS2:
http://www.cse.wustl.edu/~furukawa/papers/fnt_mvs.pdf
In essence, both techniques emerge from two different approaches: SfM aims to perform a 3D reconstruction using a structured (but theunknown) sequence of images while MVS is a generalization of the two-view stereo vision, based on human stereopsis.
I'm trying to use g2o for refine the 3D points obtained through a multi view system.
I have obtained the 3D points through triangulation, and for each acquisition I have the relative poses and the 2D features.
I would refine the 3D points using g2o for graph optimization, in order to reduce the reprojection error.
Anyone have experience in it? I saw that there are a great number of types of vertices and edges but I can't understand what are the one that can be the more suitable for my application.
I tried to follow the 'sba_demo.cpp' in the examples folder of g2o but it seems to not use the 2D corresponces.
I am looking for a way of approximating a surface based on a set of 3D data points. For this purpose I would like to use a method based on radial basis functions but I cannot find a free implementation in C++.
I looked in ITK, VTK and open CV but I did not find anything...
Does anyone knows a free implementation of such an algorithm ?
Any suggestion about the reconstruction of a surface based on a set of 3D data points is also more than welcome ! :)
3D surface reconstruction can be challenging. I would first recommend taking a look at PCL. The Point Cloud Library has grown into a nice set of tools for 3D point management and interpretation, and its license and API sound compatible with your needs. The surface reconstruction features of the library appear to be most applicable. In fact, RBF reconstruction is supported.
If PCL doesn't work, there are other options:
MeshLab,
This SO post provides a nice summary, and
of course, Wikipedia provides some links
Finally, you might search CiteSeerX, Google Scholar, etc. for papers like this one. As an example, a search for "3D Surface Reconstruction" at CiteSeerX yields many hits. RBF-based reconstruction is just one of many methods: is your application truly limited to radial basis functions? If not, there are many choices, (i.e. Ball Pivoting Algorithm). See this survey paper for some comparisons.
I am wondering whether there is any existing library or software which is on tetrahedral mesh simplification and written in C/C++. You may know that there is a famous algorithm on triangular mesh based on Quadric Error Metrics, called Surface Simpliļ¬cation Using Quadric Error Metrics. I reckon there should be something similar but applied to tetrahedral mesh. Thanks!
You should probably search in CGAL library CGAL. Its the best for computational geometry algorithms. I am sure that someone should have already implemented mesh simplification in CGAL.
The closest I can find to what you are looking for exists here it allows tetrahedral volumetric rendering.
Not sure if this is what you are looking for, as the original question and the comments seem to differ slightly in the end aim.
Let me know if you need any further info:)
I'm making a small demo to suggest Qt and OSG at my company to make some things easier to everyone working with me. But I need to do it rather quickly. I need a library for 2D and 3D binary operations on 2D and 3D geometry that will give a result mesh that can draw in OSG at the output. I also need a simple solver and mesher with which I could just define temperature on a tetrahedras within a mesh or nodes within a geometry and view a solution with OSG. I don't make solvers, I work with UI and some inner routines thus I need these libraries or something I can make a demo of a simple CAE software based on Qt and OSG. Can you suggest something?