Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
The findHomography() function in OpenCV finds a perspective transformation between two planes.The computed homography matrix is refined further (using inliers only in case of a robust method) with the Levenberg-Marquardt method to reduce the re-projection error even more. Can anyone provide any links to C/C++ code to the Levenberg Marquardt algorithm which is required to minimize the error function, as it will help me to understand the mathematics behind the algorithm. (Everywhere in the net ,only libraries or specific codes have been uploaded but not the detailed code to the algorithm).
I know it's not C/C++, but the file on the Matlab file exchange might have source code that could help you:
http://www.mathworks.com/matlabcentral/fileexchange/16063
If you understand C/C++ you will probably have no problem understanding Matlab, especially if you're looking at source code for the purposes of furthering your understanding.
The openCV code is open so check modules. The cheatsheet for openCV has Levenberg Marquardt formulation, see column 3 page 1. You can use grep to search for that cheatsheet formulation or just for the name of the method:
grep -rnw . -e "Levenberg"
For example, you can discover a lot of levmar*.c files in the modules/legacy folder. The reason for using LMA after solving a system of linear equations with RANSAC is because the former linear equations optimize a wrong metrics - an algebraic error in parameters of Homography. A better metrics is the sum of squared residual in terms of point coordinates. In this metrics the solution is non-linear and has to be found iteratively. The linear solution is used to initialize a non-linear algorithm and increase its chances to converge to a global minima.
Seeing the code, however, is not always the easiest way to understand a complex concept. Here is an 'evolutionary' line up of optimization algorithms that hopefully may facilitate your understanding:
Optimization can always be cast as minimization of cost or residuals
If the function convex (a bowl shape) it has only one global minima and it can be found by sliding down. Since gradient points up we take it with a minus sign:
param[t+1]=param[t]-k*cost', where t-iteration, k-step and ' stands for a gradient or a derivative w.r.t. parameters
So far so good but in some cases (imagine a cost function that looks like a snowboarding tube) going along gradient will make you overshoot and zig-zag from one side of the valley to the other while only slowly going down (to global min). Solution - use a better function approximation (Taylor expansion up to a second derivative) to arrive at this
param[t+1]=param[t]-(k/cost'') *cost', where '' is second derivative
Intuitively fast function means large second derivative thus we slow down convergence by reducing our step (dividing it by a large number) and vise versa.
4. For quadratic error which often represents a sum of squared residuals or some distances we can come up with an approximation of the second derivative as a product of first derivatives (called Jacobians shown as J which is generally a matrix); second derivative or Hessian = JTJ, so optimization looks like
param[t+1]=param[t]-(JTJ)-1 *cost' - called Gauss-Newton
Here we dropped k as a step for simplicity; Compare this and previous formulas, we are almost there! Also note, in wikipedia and other sites you may see that instead of k they use a residual yi-f(x, param) but you can ignore it for now as well. Go further only if you are comfortable with the last formulation.
5. Here comes Levenberg and says - it is great but may be unstable. We assumed too much with those second derivatives and it would be nice to dump them and make things look like first derivative as before. To do that we add a dumping factor that can make optimization look like
param[t+1]=param[t]-(JTJ + lambda*I)-1 *cost', where I is identity matrix and lambda is a scalar
Note that when lambda is large you can discard JTJ and you are left with standard gradient descent. They do it when convergence slows down, otherwice they use small lambda which makes the algorithm look more like Gauss-Newton. Note that J = d_z/d_param, cost’=d_f/d_param and f=zT*z
Here comes Marquardt and says, great but I like our gradient descent go faster - for small diagonal(JTJ) I'd like larger step, hence a final formula where we devide by small diagonal elements resulting in faster convergence when gradient is slow:
param[t+1]=param[t]-(JTJ + lambda*diagonal(JTJ))-1 *cost'
Here is a quick summary using downhill skiing analogy where we show the expression for parameter change param[t+1]-param[t]:
1. Gradient J points up but you want to ski down so go against gradient -kJ;
2. If zig-zagging in a 'snow tube' move in steps inverse to a second derivative (H); you will hopefully go faster straight down: -H-1J
3. Approximate a second derivative with the product of first derivatives -(JTJ)-1J ;
4. Dump it if it is too much and go back to a first derivative -(JTJ + lambdaI)-1 J;
5. don't want to get stuck on a flat slope? scale your step inversely to diagonal of approximated Hessian: -(JTJ + lambdadiag(JTJ))-1 J
It is a big mystery why all these work since my intuition gives up after a while. May be it is a good time to look at this philosophically and apply a principle of Occam's Razor - the less we assume the better we off but if assume too little we may not get too far. All we tried to do is to predict (assume) the global behaviour of the function looking at it locally (imaging getting married after being together for 1 week). Using Hessian is like having more assumptions in terms of its numerous parameters as opposed to more conservative Jacobian that has a few parameters (assumptions). Finally we may want to be flexible in being either conservative or risky and that's what Levenberg-Marquardt methods tries to accomplish.
Related
I need to solve some large (N~1e6) Laplacian matrices that arise in the study of resistor networks. The rest of the network analysis is being handled with boost graph and I would like to stay in C++ if possible. I know there are lots and lots of C++ matrix libraries but no one seems to be a clear leader in speed or usability. Also, the many questions on the subject, here and elsewhere seem to rapidly devolve into laundry lists which are of limited utility. In an attempt to help myself and others, I will try to keep the question concise and answerable:
What is the best library that can effectively handle the following requirements?
Matrix type: Symmetric Diagonal Dominant/Laplacian
Size: Very large (N~1e6), no dynamic resizing needed
Sparsity: Extreme (maximum 5 nonzero terms per row/column)
Operations needed: Solve for x in A*x=b and mat/vec multiply
Language: C++ (C ok)
Priority: Speed and simplicity to code. I would really rather avoid having to learn a whole new framework for this one problem or have to manually write too much helper code.
Extra love to answers with a minimal working example...
If you want to write your own solver, in terms of simplicity, it's hard to beat Gauss-Seidel iteration. The update step is one line, and it can be parallelized easily. Successive over-relaxation (SOR) is only slightly more complicated and converges much faster.
Conjugate gradient is also straightforward to code, and should converge much faster than the other iterative methods. The important thing to note is that you don't need to form the full matrix A, just compute matrix-vector products A*b. Once that's working, you can improve the convergance rate again by adding a preconditioner like SSOR (Symmetric SOR).
Probably the fastest solution method that's reasonable to write yourself is a Fourier-based solver. It essentially involves taking an FFT of the right-hand side, multiplying each value by a function of its coordinate, and taking the inverse FFT. You can use an FFT library like FFTW, or roll your own.
A good reference for all of these is A First Course in the Numerical Analysis of Differential Equations by Arieh Iserles.
Eigen is quite nice to use and one of the fastest libraries I know:
http://eigen.tuxfamily.org/dox/group__TutorialSparse.html
There is a lot of related post, you could have look.
I would recommend C++ and Boost::ublas as used in UMFPACK and BOOST's uBLAS Sparse Matrix
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I am trying to fit some points to an inverse parabola, in the form of F(x)=1/(ax^2+bx+c).
My objective is to program a function in c++ that would take a set of 10-30 points and fit them to the inverse parabola.
I started trying to get an analytical expression using Least squares, but I can't reach to get a result. I tried by hand (little crazy) and then I tried to solve analytically the expressions for a,b and c, but mupad doesn't give me a result (I am pretty new to Matlab's mupad so maybe i am not doing it correctly).
i don't know anymore how to approach the problem.
Can I get a analytical expression for this specific problem? I have also seen algorithms for general least squares fitting but I don't need a so complicated algorithm, I just need it for this equation.
If not, how would StackOverflow people approach the problem?
If needed I can post the equations, and the small Mupad code I've tried, but I think is unnecessary.
EDIT: some example
Im sorry the image is a little bit messy but it is the thing i need.
The data is in blue (this data is particularly noisy). I need to use only the data that is between the vertical lines (a bunch of data in the left and another one in the right).
The result of the fit is in de red line.
all this has been made with matlab, but I need to make it in c++.
I'll try to post some data...
Edit 2: I actually did the fitting in Matlab as follows (not the actual code):
create linear system Ax = b, with
A = [x² x 1]
x = [a; b; c]
b = 1/y;
It should work, shouldn't it? I can solve then using Moore-Penrose pseudoinv calculated with SVD. Isn't it?
There is no analytic solution of least squares; it is an minimisation problem, and requires clever iterative methods to solve. (nonlinear LS - thanks #insilico)
You could try a Newton style iterative method (by rearranging your equation) but I don't think it will converge easily for your function -- which is going to be highly nonlinear at two points!
I'd recommend using a library for this. such as nelder-mead search
http://www.codecogs.com/code/maths/optimization/nelder.php
you simply provide your error function - which is
sum( pow(F(x) - dataY(x), 2) )
and provide a set of initial values (a stab in the dark at the solution);
I've had good success with nelder-mead.
I don't think you will find a good plain-coded solution.
If I understand you correctly, you just need to know the formula for a fit for a particular data set, right?
If so, then you just need to get a curve fitting program and fit the curve using your desired method. Then, implement the formula shown by the curve fit.
There are a few curve fit programs out there:
Curve Expert
http://www.curveexpert.net/
* Eurequa *
http://creativemachines.cornell.edu/eureqa
Additionally, some spreadsheet packages may have the curve fitting facilities you need.
I would be happy to try to do a fit for you if you provide the data. No guarantees on getting the fit you want.
So I have an iterative closest point (ICP) algorithm that has been written and will fit a model to a point cloud. As a quick tutorial for those not in the know ICP is a simple algorithm that fits points to a model ultimately providing a homogeneous transform matrix between the model and points.
Here is a quick picture tutorial.
Step 1. Find the closest point in the model set to your data set:
Step 2: Using a bunch of fun maths (sometimes based on gradiant descent or SVD) pull the clouds closer together and repeat untill a pose is formed:
![Figure 2][2]
Now that bit is simple and working, what i would like help with is:
How do I tell if the pose that I have is a good one?
So currently I have two ideas, but they are kind of hacky:
How many points are in the ICP Algorithm. Ie, if I am fitting to almost no points, I assume that the pose will be bad:
But what if the pose is actually good? It could be, even with few points. I dont want to reject good poses:
So what we see here is that low points can actually make a very good position if they are in the right place.
So the other metric investigated was the ratio of the supplied points to the used points. Here's an example
Now we exlude points that are too far away because they will be outliers, now this means we need a good starting position for the ICP to work, but i am ok with that. Now in the above example the assurance will say NO, this is a bad pose, and it would be right because the ratio of points vs points included is:
2/11 < SOME_THRESHOLD
So thats good, but it will fail in the case shown above where the triangle is upside down. It will say that the upside down triangle is good because all of the points are used by ICP.
You don't need to be an expert on ICP to answer this question, i am looking for good ideas. Using knowledge of the points how can we classify whether it is a good pose solution or not?
Using both of these solutions together in tandem is a good suggestion but its a pretty lame solution if you ask me, very dumb to just threshold it.
What are some good ideas for how to do this?
PS. If you want to add some code, please go for it. I am working in c++.
PPS. Someone help me with tagging this question I am not sure where it should fall.
One possible approach might be comparing poses by their shapes and their orientation.
Shapes comparison can be done with Hausdorff distance up to isometry, that is poses are of the same shape if
d(I(actual_pose), calculated_pose) < d_threshold
where d_threshold should be found from experiments. As isometric modifications of X I would consider rotations by different angles - seems to be sufficient in this case.
Is poses have the same shape, we should compare their orientation. To compare orientation we could use somewhat simplified Freksa model. For each pose we should calculate values
{x_y min, x_y max, x_z min, x_z max, y_z min, y_z max}
and then make sure that each difference between corresponding values for poses does not break another_threshold, derived from experiments as well.
Hopefully this makes some sense, or at least you can draw something useful for your purpose from this.
ICP attempts to minimize the distance between your point-cloud and a model, yes? Wouldn't it make the most sense to evaluate it based on what that distance actually is after execution?
I'm assuming it tries to minimize the sum of squared distances between each point you try to fit and the closest model point. So if you want a metric for quality, why not just normalize that sum, dividing by the number of points it's fitting. Yes, outliers will disrupt it somewhat but they're also going to disrupt your fit somewhat.
It seems like any calculation you can come up with that provides more insight than whatever ICP is minimizing would be more useful incorporated into the algorithm itself, so it can minimize that too. =)
Update
I think I didn't quite understand the algorithm. It seems that it iteratively selects a subset of points, transforms them to minimize error, and then repeats those two steps? In that case your ideal solution selects as many points as possible while keeping error as small as possible.
You said combining the two terms seemed like a weak solution, but it sounds to me like an exact description of what you want, and it captures the two major features of the algorithm (yes?). Evaluating using something like error + B * (selected / total) seems spiritually similar to how regularization is used to address the overfitting problem with gradient descent (and similar) ML algorithms. Selecting a good value for B would take some experimentation.
Looking at your examples, it seems that one of the things that determines whether the match is good or not, is the quality of the points. Could you use/calculate a weighting factor in calculating your metric?
For example, you could weight down points which are co-linear / co-planar, or spatially close, as they probably define the same feature. That would perhaps allow your upside-down triangle to be rejected (as the points are in a line, and that not a great indicator of the overall pose) but the corner-case would be ok, as they roughly define the hull.
Alternatively, maybe the weighting should be on how distributed the points are around the pose, again trying to ensure you have good coverage, rather than matching small indistinct features.
I have nonlinear equations such as:
Y = f1(X)
Y = f2(X)
...
Y = fn(X)
In general, they don't have exact solution, therefore I use Newton's method to solve them. Method is iteration based and I'm looking for way to optimize calculations.
What are the ways to minimize calculation time? Avoid calculation of square roots or other math functions?
Maybe I should use assembly inside C++ code (solution is written in C++)?
A popular approach for nonlinear least squares problems is the Levenberg-Marquardt algorithm. It's kind of a blend between Gauss-Newton and a Gradient-Descent method. It combines the best of both worlds (navigates well the search space for for ill-posed problems and converges quickly). But there's lots of wiggle room in terms of the implementation. For example, if the square matrix J^T J (where J is the Jacobian matrix containing all derivatives for all equations) is sparse you could use the iterative CG algorithm to solve the equation systems quickly instead of a direct method like a Cholesky factorization of J^T J or a QR decomposition of J.
But don't just assume that some part is slow and needs to be written in assembler. Assembler is the last thing to consider. Before you go that route you should always use a profiler to check where the bottlenecks are.
Are you talking about a number of single parameter functions to solve one at a time or a system of multi-parameter equations to solve together?
If the former, then I've often found that a finding a better initial approximation (from where the Newton-Raphson loop starts) can save more execution time than polishing the loop itself, because convergence in the loop can be slow initially but is fast later. If you know nothing about the functions then finding a decent initial approximation is hard, but it might be worth trying a few secant iterations first. You might also want to look at Brent's method
Consider using Rational Root Test in parallel. If impossible to use values of absolute precision then use closest to zero results as the best fit to continue by Newton method.
Once single root found, you may decrease the equation grade by dividing it with monom (x-root).
Dividing and rational root test are implemented here https://github.com/ohhmm/openmind/blob/sh/omnn/math/test/Sum_test.cpp#L260
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I have a "continuous" linear programming problem that involves maximizing a linear function over a curved convex space. In typical LP problems, the convex space is a polytope, but in this case the convex space is piecewise curved -- that is, it has faces, edges, and vertices, but the edges aren't straight and the faces aren't flat. Instead of being specified by a finite number of linear inequalities, I have a continuously infinite number. I'm currently dealing with this by approximating the surface by a polytope, which means discretizing the continuously infinite constraints into a very large finite number of constraints.
I'm also in the situation where I'd like to know how the answer changes under small perturbations to the underlying problem. Thus, I'd like to be able to supply an initial condition to the solver based on a nearby solution. I believe this capability is called a "warm start."
Can someone help me distinguish between the various LP packages out there? I'm not so concerned with user-friendliness as speed (for large numbers of constraints), high-precision arithmetic, and warm starts.
Thanks!
EDIT: Judging from the conversation with question answerers so far, I should be clearer about the problem I'm trying to solve. A simplified version is the following:
I have N fixed functions f_i(y) of a single real variable y. I want to find x_i (i=1,...,N) that minimize \sum_{i=1}^N x_i f_i(0), subject to the constraints:
\sum_{i=1}^N x_i f_i(1) = 1, and
\sum_{i=1}^N x_i f_i(y) >= 0 for all y>2
More succinctly, if we define the function F(y)=\sum_{i=1}^N x_i f_i(y), then I want to minimize F(0) subject to the condition that F(1)=1, and F(y) is positive on the entire interval [2,infinity). Note that this latter positivity condition is really an infinite number of linear constraints on the x_i's, one for each y. You can think of y as a label -- it is not an optimization variable. A specific y_0 restricts me to the half-space F(y_0) >= 0 in the space of x_i's. As I vary y_0 between 2 and infinity, these half-spaces change continuously, carving out a curved convex shape. The geometry of this shape depends implicitly (and in a complicated way) on the functions f_i.
As for LP solver recommendations, two of the best are Gurobi and CPLEX (google them). They are free for academic users, and are capable of solving large-scale problems. I believe they have all the capabilities that you need. You can get sensitivity information (to a perturbation) from the shadow prices (i.e. the Lagrange multipliers).
But I'm more interested in your original problem. As I understand it, it looks like this:
Let S = {1,2,...,N} where N is the total number of functions. y is a scalar. f_{i}:R^{1} -> R^{1}.
minimize sum{i in S} (x_{i} * f_{i}(0))
x_{i}
s.t.
(1) sum {i in S} x_{i} * f_{i}(1) = 1
(2) sum {i in S} x_{i} * f_{i}(y) >= 0 for all y in (2,inf]
It just seems to me that you might want to try solve this problem as an convex NLP rather than an LP. Large-scale interior point NLP solvers like IPOPT should be able to handle these problems easily. I strongly recommended trying IPOPT http://www.coin-or.org/Ipopt
From a numerical point of view: for convex problems, warm-starting is not necessary with interior point solvers; and you don't have to worry about the combinatorial cycling of active sets. What you've described as "warm-starting" is actually perturbing the solution -- that's more akin to sensitivity analysis. In optimization parlance, warm-starting usually means supplying a solver with an initial guess -- the solver will take that guess and end up at the same solution, which isn't really what you want. The only exception is if the active set changes with a different initial guess -- but for a convex problem with a unique optimum, this cannot happen.
If you need any more information, I'd be pleased to supply it.
EDIT:
Sorry about the non-standard notation -- I wish I could type in LaTeX like on MathOverflow.net. (Incidentally, you might try posting this there -- I think the mathematicians there would be interested in this problem)
Ah now I see about the "y > 2". It isn't really an optimization constraint so much as an interval defining a space (I've edited my description above). My mistake. I'm wondering if you could somehow transform/project the problem from an infinite to a finite one? I can't think of anything right now, but I'm just wondering if that's possible.
So your approach is to discretize the problem for y in (2,inf]. I'm guessing you're choosing a very big number to represent inf and a fine discretization grid. Oooo tricky. I suppose discretization is probably your best bet. Maybe guys who do real analysis have ideas.
I've seen something similar being done for problems involving Lyapunov functions where it was necessary to enforce a property in every point within a convex hull. But that space was finite.
I encountered a similar problem. I searched the web and found just now that this problem may be classified as "semi-infinite" problem. MATLAB has tools to solve this kind of problems (function "fseminf"). But I haven't checked this in detail. Sure people have encountered this kind of questions.
You shouldn't be using an LP solver and doing the discretization yourself. You can do much better by using a decent general convex solver. Check out, for example, cvxopt. This can handle a wide variety of different functions in your constraints, or allow you to write your own. This will be far better than attempting to do the linearization yourself.
As to warm start, it makes more sense for an LP than a general convex program. While warm start could potentially be useful if you hand code the entire algorithm yourself, you typically still need several Newton steps anyway, so the gains aren't that significant. Most of the benefit of warm start comes in things like active set methods, which are mostly only used for LP.