Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I am trying to fit some points to an inverse parabola, in the form of F(x)=1/(ax^2+bx+c).
My objective is to program a function in c++ that would take a set of 10-30 points and fit them to the inverse parabola.
I started trying to get an analytical expression using Least squares, but I can't reach to get a result. I tried by hand (little crazy) and then I tried to solve analytically the expressions for a,b and c, but mupad doesn't give me a result (I am pretty new to Matlab's mupad so maybe i am not doing it correctly).
i don't know anymore how to approach the problem.
Can I get a analytical expression for this specific problem? I have also seen algorithms for general least squares fitting but I don't need a so complicated algorithm, I just need it for this equation.
If not, how would StackOverflow people approach the problem?
If needed I can post the equations, and the small Mupad code I've tried, but I think is unnecessary.
EDIT: some example
Im sorry the image is a little bit messy but it is the thing i need.
The data is in blue (this data is particularly noisy). I need to use only the data that is between the vertical lines (a bunch of data in the left and another one in the right).
The result of the fit is in de red line.
all this has been made with matlab, but I need to make it in c++.
I'll try to post some data...
Edit 2: I actually did the fitting in Matlab as follows (not the actual code):
create linear system Ax = b, with
A = [x² x 1]
x = [a; b; c]
b = 1/y;
It should work, shouldn't it? I can solve then using Moore-Penrose pseudoinv calculated with SVD. Isn't it?
There is no analytic solution of least squares; it is an minimisation problem, and requires clever iterative methods to solve. (nonlinear LS - thanks #insilico)
You could try a Newton style iterative method (by rearranging your equation) but I don't think it will converge easily for your function -- which is going to be highly nonlinear at two points!
I'd recommend using a library for this. such as nelder-mead search
http://www.codecogs.com/code/maths/optimization/nelder.php
you simply provide your error function - which is
sum( pow(F(x) - dataY(x), 2) )
and provide a set of initial values (a stab in the dark at the solution);
I've had good success with nelder-mead.
I don't think you will find a good plain-coded solution.
If I understand you correctly, you just need to know the formula for a fit for a particular data set, right?
If so, then you just need to get a curve fitting program and fit the curve using your desired method. Then, implement the formula shown by the curve fit.
There are a few curve fit programs out there:
Curve Expert
http://www.curveexpert.net/
* Eurequa *
http://creativemachines.cornell.edu/eureqa
Additionally, some spreadsheet packages may have the curve fitting facilities you need.
I would be happy to try to do a fit for you if you provide the data. No guarantees on getting the fit you want.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I have to apply a heuristic algorithm for finding a minimum or maximum from a function.
I understood what heuristic means, but where can I find an algorithm for applying it on Rosenbrock function, for example.
(C++,JAVA,C# or even pseudocode could be very helpful).
The simplest, and most obvious, solution would be to use the Random walk algorithm. It works by starting with a random point within the search space and then visiting it's random neighbor.
A similar, but more reasonable, algorithm is the Hill climbing. Again, you start at a random point but this time you chose the best neighbor.
Another, technically, heuristic algorithm is Random sampling, which just means picking any point from the search space and remembering the best one you found.
An improvement over Random sampling is the Simulated annealing algorithm. It's a kind of a fusion of Random sampling and Hill climbing: you start with picking random points in the search space but as the time goes on you tend to stick with the higher quality ones.
You can find more information and pseudo code samples of all of the above on Wikipedia.
A whole different class of solutions are the Genetic algorithms. You can start learning about them by reading http://www.obitko.com/tutorials/genetic-algorithms/index.php. Unfortunately it does not seem to have any code samples.
The Wikipedia article that you reference mentions adaptive coordinate descent, which is a state-of-the-art evolutionary algorithm, as a technique for minimizing the Rosenbrock function. Googling that finds several papers with pseudocode and algorithms including this one. The paper even includes a reference to actual code in Matlab.
You could also use Expectation–maximization with random restart although that's probably significantly less efficient than adaptive coordinate descent.
You need to use partial derivatives of the function to find its extrema. You start from a random point on the plane and use these derivatives (calculated numerically) to find a direction to move to a next point. Repeat this process until you find a minimum (justified by second derivatives).
By the way, the function you are talking about is a known difficult case for such iterative search, so you'll need to experiment with step size and other parameters of your algorithm.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I was curious if it would be possible to count the number of things in a picture, let's say the number of ducks, by first taking a sample picture and then seeing where it matched in a separate picture. So to clarify, we would have 2 pictures (one picture with a single duck, and one picture with 4 ducks for the sake of the argument) and the program would see how many matches it could make in the 4 duck picture by overlaying the one duck picture--thereby counting how many ducks there are in the picture. I've been reading up on computer vision a little bit, and I know that opencv's site talked about using a Fourier transform to break an image into its magnitude and phase. I was hoping to possibly take the magnitude of the one duck picture into a matrix and then compare it to a series of matrices from the four duck picture.
I imagine this would be quite difficult, seeing as how I would have to somehow tell the program the shape of the initial duck and then store that duck's broken down image information into a matrix and then compare that to matrices broken down from the other picture. Any ideas/suggestions? I thought this would be a good learning experience, since I'm an electrical engineering student and I learned Fourier Transforms, DFTs, etc. last semester--it'd just be cool to actually apply them to something.
You are talking about object recognition - one of fundamental problems in computer vision. Your main idea - take a picture of the object, get some features from it and then find same set of features on other image - is correct. However, pixel by pixel comparison (no matter in time or frequency domain) is very error-prone and normally gives poor results. In most cases more high-level features give much better results.
To get started, take a look at Cascade Classifier in OpenCV which uses Haar-like features (small rectangles with particular gray level). It is most well-known for face detection and recognition, but can also be trained for other objects.
You may also be interested in SURF method, which searches for points with similar characteristics, or even AAMs, which try to model shape and appearance of an object.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
The findHomography() function in OpenCV finds a perspective transformation between two planes.The computed homography matrix is refined further (using inliers only in case of a robust method) with the Levenberg-Marquardt method to reduce the re-projection error even more. Can anyone provide any links to C/C++ code to the Levenberg Marquardt algorithm which is required to minimize the error function, as it will help me to understand the mathematics behind the algorithm. (Everywhere in the net ,only libraries or specific codes have been uploaded but not the detailed code to the algorithm).
I know it's not C/C++, but the file on the Matlab file exchange might have source code that could help you:
http://www.mathworks.com/matlabcentral/fileexchange/16063
If you understand C/C++ you will probably have no problem understanding Matlab, especially if you're looking at source code for the purposes of furthering your understanding.
The openCV code is open so check modules. The cheatsheet for openCV has Levenberg Marquardt formulation, see column 3 page 1. You can use grep to search for that cheatsheet formulation or just for the name of the method:
grep -rnw . -e "Levenberg"
For example, you can discover a lot of levmar*.c files in the modules/legacy folder. The reason for using LMA after solving a system of linear equations with RANSAC is because the former linear equations optimize a wrong metrics - an algebraic error in parameters of Homography. A better metrics is the sum of squared residual in terms of point coordinates. In this metrics the solution is non-linear and has to be found iteratively. The linear solution is used to initialize a non-linear algorithm and increase its chances to converge to a global minima.
Seeing the code, however, is not always the easiest way to understand a complex concept. Here is an 'evolutionary' line up of optimization algorithms that hopefully may facilitate your understanding:
Optimization can always be cast as minimization of cost or residuals
If the function convex (a bowl shape) it has only one global minima and it can be found by sliding down. Since gradient points up we take it with a minus sign:
param[t+1]=param[t]-k*cost', where t-iteration, k-step and ' stands for a gradient or a derivative w.r.t. parameters
So far so good but in some cases (imagine a cost function that looks like a snowboarding tube) going along gradient will make you overshoot and zig-zag from one side of the valley to the other while only slowly going down (to global min). Solution - use a better function approximation (Taylor expansion up to a second derivative) to arrive at this
param[t+1]=param[t]-(k/cost'') *cost', where '' is second derivative
Intuitively fast function means large second derivative thus we slow down convergence by reducing our step (dividing it by a large number) and vise versa.
4. For quadratic error which often represents a sum of squared residuals or some distances we can come up with an approximation of the second derivative as a product of first derivatives (called Jacobians shown as J which is generally a matrix); second derivative or Hessian = JTJ, so optimization looks like
param[t+1]=param[t]-(JTJ)-1 *cost' - called Gauss-Newton
Here we dropped k as a step for simplicity; Compare this and previous formulas, we are almost there! Also note, in wikipedia and other sites you may see that instead of k they use a residual yi-f(x, param) but you can ignore it for now as well. Go further only if you are comfortable with the last formulation.
5. Here comes Levenberg and says - it is great but may be unstable. We assumed too much with those second derivatives and it would be nice to dump them and make things look like first derivative as before. To do that we add a dumping factor that can make optimization look like
param[t+1]=param[t]-(JTJ + lambda*I)-1 *cost', where I is identity matrix and lambda is a scalar
Note that when lambda is large you can discard JTJ and you are left with standard gradient descent. They do it when convergence slows down, otherwice they use small lambda which makes the algorithm look more like Gauss-Newton. Note that J = d_z/d_param, cost’=d_f/d_param and f=zT*z
Here comes Marquardt and says, great but I like our gradient descent go faster - for small diagonal(JTJ) I'd like larger step, hence a final formula where we devide by small diagonal elements resulting in faster convergence when gradient is slow:
param[t+1]=param[t]-(JTJ + lambda*diagonal(JTJ))-1 *cost'
Here is a quick summary using downhill skiing analogy where we show the expression for parameter change param[t+1]-param[t]:
1. Gradient J points up but you want to ski down so go against gradient -kJ;
2. If zig-zagging in a 'snow tube' move in steps inverse to a second derivative (H); you will hopefully go faster straight down: -H-1J
3. Approximate a second derivative with the product of first derivatives -(JTJ)-1J ;
4. Dump it if it is too much and go back to a first derivative -(JTJ + lambdaI)-1 J;
5. don't want to get stuck on a flat slope? scale your step inversely to diagonal of approximated Hessian: -(JTJ + lambdadiag(JTJ))-1 J
It is a big mystery why all these work since my intuition gives up after a while. May be it is a good time to look at this philosophically and apply a principle of Occam's Razor - the less we assume the better we off but if assume too little we may not get too far. All we tried to do is to predict (assume) the global behaviour of the function looking at it locally (imaging getting married after being together for 1 week). Using Hessian is like having more assumptions in terms of its numerous parameters as opposed to more conservative Jacobian that has a few parameters (assumptions). Finally we may want to be flexible in being either conservative or risky and that's what Levenberg-Marquardt methods tries to accomplish.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I am looking for C++ open-source library (or just open-source Unix tool) to do: Equality test on Equations .
Equations can be build during runtime as AST Trees, string or other format.
Equations will mostly be simple algebra ones, with some assumptions about unknown functions. Domain, will be integer arithmetic (no floating point issues, as related issues are well known - Thanks #hardmath for stressing it out, I've assumed it's known).
Example: Input might contain function phi, with assumptions about it (most cases) phi(x,y)=phi(y,x) and try to solve :
equality_test( phi( (a+1)*(a+1) , a+b ) = phi( b+a, a*a + 2a + 1 )
It can be fuzzy or any equality test - what I mean is that, it does not have to always succeed (It may return "false" even if equations are equal).
If there would be problem with supporting assumptions like above about phi function, I can handle this, so just simple linear algebra equations equality testers are welcome as well.
Could you recommend some C/C++ programming libraries or Unix tools ? (open-source)
If possible, could you attach some example how such equality test, might look like in given library/tool ?
P.S. If such equality_test could (in case of success) return isomorphism - (what I mean, a kind of "mapping") - between two given equations, would be highly welcome. But tools without such capabilities are highly welcome as well.
P.S. By "fuzzy tester" I mean that in internals equation solver will be "fuzzy" in terms of looking for "isomorphism" of two functions, not in terms of testing against random inputs - I could implement this, for sure, but I try to find something with better precision.
P.P.S. There is another issue, why I need better performance solution, than brute-force "all inputs testing". Above equation is simplyfied form of my internal problem, where I do not have mapping between variables in equations. That is, I have eq1=phi( (a+1)*(a+1) , a+b ) and eq2=phi( l+k, k*k + 2k + 1 ) , and I have to find out that a==k and b==l. But this sub-problem I can handle with "brute-force" approach (even asymptotic complexity of this approach), case there is just a few variables, let it be 8. So I would need to do this equation_test for each possible mapping. If there is a tool that that whole job, I would be highly thankful, and could contribute to such project. But I don't require such functionality, simply equation_test() will be enough, I can handle rest easily.
To sum it up:
equality_test() is only one of many subproblems I have to solve, so computational complexity matters.
it does not have to be 100% reliable, but higher likelihood, than just testing equations with a few random inputs and variable mapping is highly welcome :).
output of "yes" or "no" (all additional information might be useful but in future, at this stage I need "Yes"/"No")
Your topic is one of automated theorem proving, for which a number of free/open source software packages have been developed. Many of these are meant for proof verification, but what you ask for is proof searching.
Dealing with the abstract topic of equations would be the theories mathematicians call varieties. These theories have nice properties with respect to the existence and regularity of their models.
It is possible you have in mind equations that deal specifically with real numbers or other system, which would add some axioms to the theory in which a proof is sought.
If in principle an algorithm exists to determine whether or not a logical statement can be proven in a theory, that theory is called decidable. For example, the theory of real closed fields is decidable, as Tarski showed in 1951. However a practical implementation of such an algorithm is lacking and perhaps impossible.
Here are a few open source packages that might be worth learning something about to guide your design and development:
Tac: A generic and adaptable interactive theorem prover
Prover9: An automated theorem prover for first-order and equational logic
E(quational) Theorem Prover
I am not sure for any library but how about you do it yourself by generating a random set of inputs for your equation and substituting it in both equations which have to be compared. This would give you a almost correct result given you generate considerable amount of random data.
Edit:
Also you can try http://www.wolframalpha.com/
with
(x+1)*(y+1) equals x+y+xy+2
and
(x+1)*(y+1) equals x+y+xy+1
I think you can get pretty far with using Reverse Polish Notation.
Write out your equation using RPN
Apply transformations to bring all expressions to the same form, e.g. *A+BC --> +*AB*AC (which is the RPN equivalent of A*(B+C) --> A*B+A*C), ^*BCA --> *^BA^CA (i.e. (B*C)^A --> B^A * C^A)
"Sort" the arguments of symmetric binary operator so that "lighter" operations appear on one side (e.g. A*B + C --> C + A*B)
You will have problem with dummy variables, for example sum indices. There is no other way, I think, but to try every combination of matching them on both sides of the equation.
In general, the problem is very complicated.
You can try a hack, though: use an optimizing compiler (C,Fortran) and compile both sides of the equation to optimized machine code and compare the outputs. It may work, or may not.
Opensource (GPL) project Maxima has tool simmilar to Wolfram Alpha's equals tool :
(a+b+c)+(x+y)**2 equals (x**2+b+c+a+2*x*y+y**2)
Which is is(equal()), that solves formulas :
(%i1) is(equal( (a+b+c)+(x+y)**2 , (x**2+b+c+a+2*x*y+y**2) ));
(%o1) true
For this purpose, it uses rational simplifier - ratsimp, in order to simplify the difference of two equations. When difference of two equations is simplified to zero, we know they are equal for all possible values:
(%i2) ratsimp( ((a+b+c)+(x+y)**2) - ((x**2+b+c+a+2*x*y+y**2)) );
(%o2) 0
This answer, just shows direction (like other answers). If you know about something similar, that can be used as a part of C++ Unix program - programming library ? Good C/C++ binding similar tool to this. Please, post new answer.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I have a "continuous" linear programming problem that involves maximizing a linear function over a curved convex space. In typical LP problems, the convex space is a polytope, but in this case the convex space is piecewise curved -- that is, it has faces, edges, and vertices, but the edges aren't straight and the faces aren't flat. Instead of being specified by a finite number of linear inequalities, I have a continuously infinite number. I'm currently dealing with this by approximating the surface by a polytope, which means discretizing the continuously infinite constraints into a very large finite number of constraints.
I'm also in the situation where I'd like to know how the answer changes under small perturbations to the underlying problem. Thus, I'd like to be able to supply an initial condition to the solver based on a nearby solution. I believe this capability is called a "warm start."
Can someone help me distinguish between the various LP packages out there? I'm not so concerned with user-friendliness as speed (for large numbers of constraints), high-precision arithmetic, and warm starts.
Thanks!
EDIT: Judging from the conversation with question answerers so far, I should be clearer about the problem I'm trying to solve. A simplified version is the following:
I have N fixed functions f_i(y) of a single real variable y. I want to find x_i (i=1,...,N) that minimize \sum_{i=1}^N x_i f_i(0), subject to the constraints:
\sum_{i=1}^N x_i f_i(1) = 1, and
\sum_{i=1}^N x_i f_i(y) >= 0 for all y>2
More succinctly, if we define the function F(y)=\sum_{i=1}^N x_i f_i(y), then I want to minimize F(0) subject to the condition that F(1)=1, and F(y) is positive on the entire interval [2,infinity). Note that this latter positivity condition is really an infinite number of linear constraints on the x_i's, one for each y. You can think of y as a label -- it is not an optimization variable. A specific y_0 restricts me to the half-space F(y_0) >= 0 in the space of x_i's. As I vary y_0 between 2 and infinity, these half-spaces change continuously, carving out a curved convex shape. The geometry of this shape depends implicitly (and in a complicated way) on the functions f_i.
As for LP solver recommendations, two of the best are Gurobi and CPLEX (google them). They are free for academic users, and are capable of solving large-scale problems. I believe they have all the capabilities that you need. You can get sensitivity information (to a perturbation) from the shadow prices (i.e. the Lagrange multipliers).
But I'm more interested in your original problem. As I understand it, it looks like this:
Let S = {1,2,...,N} where N is the total number of functions. y is a scalar. f_{i}:R^{1} -> R^{1}.
minimize sum{i in S} (x_{i} * f_{i}(0))
x_{i}
s.t.
(1) sum {i in S} x_{i} * f_{i}(1) = 1
(2) sum {i in S} x_{i} * f_{i}(y) >= 0 for all y in (2,inf]
It just seems to me that you might want to try solve this problem as an convex NLP rather than an LP. Large-scale interior point NLP solvers like IPOPT should be able to handle these problems easily. I strongly recommended trying IPOPT http://www.coin-or.org/Ipopt
From a numerical point of view: for convex problems, warm-starting is not necessary with interior point solvers; and you don't have to worry about the combinatorial cycling of active sets. What you've described as "warm-starting" is actually perturbing the solution -- that's more akin to sensitivity analysis. In optimization parlance, warm-starting usually means supplying a solver with an initial guess -- the solver will take that guess and end up at the same solution, which isn't really what you want. The only exception is if the active set changes with a different initial guess -- but for a convex problem with a unique optimum, this cannot happen.
If you need any more information, I'd be pleased to supply it.
EDIT:
Sorry about the non-standard notation -- I wish I could type in LaTeX like on MathOverflow.net. (Incidentally, you might try posting this there -- I think the mathematicians there would be interested in this problem)
Ah now I see about the "y > 2". It isn't really an optimization constraint so much as an interval defining a space (I've edited my description above). My mistake. I'm wondering if you could somehow transform/project the problem from an infinite to a finite one? I can't think of anything right now, but I'm just wondering if that's possible.
So your approach is to discretize the problem for y in (2,inf]. I'm guessing you're choosing a very big number to represent inf and a fine discretization grid. Oooo tricky. I suppose discretization is probably your best bet. Maybe guys who do real analysis have ideas.
I've seen something similar being done for problems involving Lyapunov functions where it was necessary to enforce a property in every point within a convex hull. But that space was finite.
I encountered a similar problem. I searched the web and found just now that this problem may be classified as "semi-infinite" problem. MATLAB has tools to solve this kind of problems (function "fseminf"). But I haven't checked this in detail. Sure people have encountered this kind of questions.
You shouldn't be using an LP solver and doing the discretization yourself. You can do much better by using a decent general convex solver. Check out, for example, cvxopt. This can handle a wide variety of different functions in your constraints, or allow you to write your own. This will be far better than attempting to do the linearization yourself.
As to warm start, it makes more sense for an LP than a general convex program. While warm start could potentially be useful if you hand code the entire algorithm yourself, you typically still need several Newton steps anyway, so the gains aren't that significant. Most of the benefit of warm start comes in things like active set methods, which are mostly only used for LP.