This great SO answer points to a good sparse solver for Ax=b, but I've got constraints on x such that each element in x is >=0 an <=N.
Also, A is huge (around 2e6x2e6) but very sparse with <=4 elements per row.
Any ideas/recommendations? I'm looking for something like MATLAB's lsqlin but with huge sparse matrices.
I'm essentially trying to solve the large scale bounded variable least squares problem on sparse matrices:
EDIT:
In CVX:
cvx_begin
variable x(n)
minimize( norm(A*x-b) );
subject to
x <= N;
x >= 0;
cvx_end
You are trying to solve least squares with box constraints. Standard sparse least squares algorithms include LSQR and more recently, LSMR. These only require you to apply matrix-vector products. To add in the constraints, realize that if you are in the interior of the box (none of the constraints are "active"), then you proceed with whatever interior point method you chose. For all active constraints, the next iteration you perform will either deactivate the constraint, or constrain you to move along the constraint hyperplane. With some (conceptually relatively simple) suitable modifications to the algorithm you choose, you can implement these constraints.
Generally however, you can use any convex optimization package. I have personally solved this exact type of problem using the Matlab package CVX, which uses SDPT3/SeDuMi for a backend. CVX is merely a very convenient wrapper around these semidefinite program solvers.
Your problem is similar to a nonnegative least-squares problem (NNLS), which can be formulated as
$$\min_x ||Ax-b||_2^2 \text{ subject to } x \ge 0$$,
for which there seems to exist many algorithms.
Actually, you problem can be more or less converted into an NNLS problem, if, in addition to your original nonnegative variables $x$ you create additional variables $x'$ and link them with linear constraints $x_i+x_i'=N$. The problem with this approach is that these additional linear constraints might not be satisfied exactly in the least-squares solution - it might be appropriate then to try to weight them with a large number.
If you reformulate your model as:
min
subject to:
then it is a standard quadratic programming problem. This is a common type of model that can be easily solved with a commercial solver such as CPLEX or Gurobi. (Disclaimer: I currently work for Gurobi Optimization and formerly worked for ILOG, which provided CPLEX).
Your matrix A^T A is positive semi-definite, so your problem is convex; be sure to take advantage of that when setting up your solver.
Most go-to QP solvers are in Fortran and/or non-free; however I've heard good things about OOQP (http://www.mcs.anl.gov/research/projects/otc/Tools/OOQP/OoqpRequestForm.html), though it's a bit of a pain getting a copy.
How about CVXOPT? It works with sparse matrices, and it seems that some of the cone programming solvers may help:
http://abel.ee.ucla.edu/cvxopt/userguide/coneprog.html#quadratic-cone-programs
This is a simple modification of the code in the doc above, to solve your problem:
from cvxopt import matrix, solvers
A = matrix([ [ .3, -.4, -.2, -.4, 1.3 ],
[ .6, 1.2, -1.7, .3, -.3 ],
[-.3, .0, .6, -1.2, -2.0 ] ])
b = matrix([ 1.5, .0, -1.2, -.7, .0])
N = 2;
m, n = A.size
I = matrix(0.0, (n,n))
I[::n+1] = 1.0
G = matrix([-I, I])
h = matrix(n*[0.0] + n*[N])
print G
print h
dims = {'l': n, 'q': [n+1], 's': []}
x = solvers.coneqp(A.T*A, -A.T*b, G, h)['x']#, dims)['x']
print x
CVXOPT support sparse matrices, so it be useful for you.
If you have Matlab, this is something you can do with TFOCS. This is the syntax you would use to solve the problem:
x = tfocs( smooth_quad, { A, -b }, proj_box( 0, N ) );
You can pass A as a function handle if it's too big to fit into memory.
Related
Introduction
Let’s assume that I need the Jacobian matrix for the following set of ODE:
dxdt[ 0 ] = -90.0 * x[0] - 50.0 * x[1];
dxdt[ 1 ] = x[0] + 3*x[1];
dxdt[ 2 ] = x[1] + 50*x[2];
In Matlab/Octave this would be pretty easy:
syms x0 x1 x2;
f = [-90.0*x0-50.0*x1, x0+3*x1, x1+50*x2]
v=[x0, x1, x2]
fp = jacobian(f,v)
This would results with following output matrix:
[-90 -50 0 ]
[ 1 3 0 ]
[ 0 1 50]
What I need
Now I want to reproduce the same results in C++. I can’t compute the Jacobian before and hard-code it, as it will depend for example on user inputs and time. So my question is: How to do this? Usually for mathematics operations, I use the Boost library, however in this case I can’t find any solution. There’s only short note about this in implicit systems, but the following code doesn’t work:
sys.second( x , jacobi , t )
It also requests the time (t), so it probably doesn’t generate an analytic form of solution. Do I misunderstand the documentations? Or should I use another function? I would prefer to stay within Boost, as I need the Jacobian as ublas::matrix and I want to avoid conversion.
EDIT:
More specific I will use Jacobian inside rosenbrock4 ODE solver. Example here - lines 47-52. I need automatic generation of this structure as the ODE set may be changed later and I want to avoid manually rewriting Jacobian ever time. Also some variables inside ODE definitions are not constant in time.
I know this is long after the fact, but I have recently been wanting to do the same thing and have come across many auto differentiation (AD) libraries that do this pretty well. I have mostly been using Eigen's AD because I am already using Eigen everywhere. Here's an example of how you can use Eigen's AD to get the jacobian like you asked.
There's also a long list of c++ AD libraries on autodiff.org.
Hope this helps someone!
The Jacobian is based on derivatives of the function. If the function f is only known at run-time (and there are no constraints such as linearity), you have to automatise the differentiation. If you want this to happen exactly (as opposed to a numerical estimation), you need to use symbolic computation. Look for example here and here for libraries supporting this.
Note that the Jacobian usually depends on the state and the time, so it’s impossible to represent it as a constant matrix (such as in your example), unless your problem is so boring that you can solve it analytically anyway.
I have to find all basic solutions of some tiny linear-programming problems.
Here's an example (in lp_solve format):
max: x1 + x2;
x1 + x2 <= 1;
x1 <= 0.8;
x2 <= 0.8;
All 2 basic solutions:
x1 = 0.2, x2 = 0.8
x1 = 0.8, x2 = 0.2
Of course there is a way of finding alternative solutions, but I really prefer using existing libraries instead of crafting my own simplex code.
I'm using Python as my programming language, and hoping there's some method in lp_solve or GLPK's C API can do this.
Thanks.
There is no routine to do that with glpk; and IMHO it is very unlikely that any real-world solver implements something like that, since it is not very useful in practise and it is certainly not a simple problem.
What is indeed easy to find one other basic solution once you reached optimality with the simplex algorithm, which does not mean that it is easy to list them all.
Consider a LP whose domain has dimension n; the set S of the optimal solutions is a convex polyhedron whose dimension m can be anything from 0 to n-1.
You want a method to list all the basic solutions of the problem, that is all the vertices of S: as soon as m is greater than 2, you will need to carefully avoid cycling when you move from one basic solution to another.
However, there is (luckily!) no need to write your own simplex code: you can access the internals of the current basis with the glpk library, and probably with lpsolve too.
Edit: two possible solutions
The better way would be to use another library such as PPL for this.
Assume that you have a problem of the form:
min cx; subject to: Ax <= b
First solve your problem with glpk, this will give you the optimal value V of the problem. From this point, you can use PPL to get the description of the polyedron of optimal values:
cx = V and Ax <= b
as the convex hull of its extreme points, which correspond to the BFSs you are looking for.
You can (probably) use the glpk simplex routines. Once you get an optimal BFS, you can get the reduced cost associated with all non-basic columns using the routine glp_get_row_dual (the basis status of the variable can be obtained with glp_get_row_stat), so you can find a non-basic variables with a null reduced cost. Then, I think that you can use function glp_set_row_stat to change the basis status of this column in order to have it enter the basis.
(And then, you can iterate this process as long as you avoid cycling.)
Note that I did not try any of those solutions myself; I think that the first one is by far the best, although it will require that you learn the PPL API. If you wish to go for the second one, I strongly suggest that you send an e-mail to the glpk maintainer (or look at the source code), because I am really not sure it will work as is.
I need to compute the eigenvalues and eigenvectors of a big matrix (about 1000*1000 or even more). Matlab works very fast but it does not guaranty accuracy. I need this to be pretty accurate (about 1e-06 error is ok) and within a reasonable time (an hour or two is ok).
My matrix is symmetric and pretty sparse. The exact values are: ones on the diagonal, and on the diagonal below the main diagonal, and on the diagonal above it. Example:
How can I do this? C++ is the most convenient to me.
MATLAB does not guarrantee accuracy
I find this claim unreasonable. On what grounds do you say that you can find a (significantly) more accurate implementation than MATLAB's highly refined computational algorithms?
AND... using MATLAB's eig, the following is computed in less than half a second:
%// Generate the input matrix
X = ones(1000);
A = triu(X, -1) + tril(X, 1) - X;
%// Compute eigenvalues
v = eig(A);
It's fast alright!
I need this to be pretty accurate (about 1e-06 error is OK)
Remember that solving eigenvalues accurately is related to finding the roots of the characteristic polynomial. This specific 1000x1000 matrix is very ill-conditioned:
>> cond(A)
ans =
1.6551e+003
A general rule of thumb is that for a condition number of 10k, you may lose up to k digits of accuracy (on top of what would be lost to the numerical method due to loss of precision from arithmetic method).
So in your case, I'd expect the results to be accurate up to an approximate error of 10-3.
If you're not opposed to using a third party library, I've had great success using the Armadillo linear algebra libraries.
For the example below, arma is the namespace they like to use, vec is a vector, mat is a matrix.
arma::vec getEigenValues(arma::mat M) {
return arma::eig_sym(M);
}
You can also serialize the data directly into MATLAB and vice versa.
Your system is tridiagonal and a (symmetric) Toeplitz matrix. I'd guess that eigen and Matlab's eig have special cases to handle such matrices. There is a closed-form solution for the eigenvalues in this case (reference (PDF)). In Matlab for your matrix this is simply:
n = size(A,1);
k = (1:n).';
v = 1-2*cos(pi*k./(n+1));
This can be further optimized by noting that the eigenvalues are centered about 1 and thus only half of them need to be computed:
n = size(A,1);
if mod(n,2) == 0
k = (1:n/2).';
u = 2*cos(pi*k./(n+1));
v = 1+[u;-u];
else
k = (1:(n-1)/2).';
u = 2*cos(pi*k./(n+1));
v = 1+[u;0;-u];
end
I'm not sure how you're going to get more fast and accurate than that (other than performing a refinement step using the eigenvectors and optimization) with simple code. The above should be able to translated to C++ very easily (or use Matlab's codgen to generate C/C++ code that uses this or eig). However, your matrix is still ill-conditioned. Just remember that estimates of accuracy are worst case.
I am trying to solve two point boundary problem with odeint. My equation has the form of
y'' + a*y' + b*y + c = 0
It is pretty trivial when I have boundary conditions of y(x_1) = y_1 , y'(x_2) = y_2, but when boundary conditions are y(x_1) = y_1 , y(x_2) = y_2 I am lost. Does anybody know the way to deal with problems like this with odeint or other scientific library?
In this case you need a shooting method. odeint does not have such a method, it solved the initial value problem (IVP) which is your first case. I think in the Numerical Recipies this method is explained and you can use Boost.Odeint to do the time stepping.
An alternative and more efficient method to solve this type of problem is finite differences or finite elements method. For finite differences you can check Numerical Recipes. For finite elements I recommend dealii library.
Another approach is to use b-splines: Assuming you do know the initial x0 and final xfinal points of integration, then you can expand the solution y(x) in a b-spline basis, defined over (x0,xfinal), i.e.
y(x)= \sum_{i=1}^n A_i*B_i(x),
where A_i are constant coefficients to be determined, and B_i(x) are b-spline basis (well defined polynomial functions, that can be differentiated numerically). For scientific applications you can find an implementation of b-splines in GSL.
With this substitution the boundary value problem is reduced to a linear problem, since (am using Einstein summation for repeated indices):
A_i*[ B_i''(x) + a*B_i'(x) + b*B_i(x)] + c =0
You can choose a set of points x and create a linear system from the above equation. You can find information for this type of method in the following review paper "Applications of B-splines in Atomic and Molecular Physics" - H Bachau, E Cormier, P Decleva, J E Hansen and F Martín
http://iopscience.iop.org/0034-4885/64/12/205/
I do not know of any library solving directly this problem, but there are several libraries for B-splines (I recommend GSL for your needs), that will allow you to form the linear system. See this stackoverflow question:
Spline, B-Spline and NURBS C++ library
I have a somewhat complicated algorithm that requires the fitting of a quadric to a set of points. This quadric is given by its parametrization (u, v, f(u,v)), where f(u,v) = au^2+bv^2+cuv+du+ev+f.
The coefficients of the f(u,v) function need to be found since I have a set of exactly 6 constraints this function should obey. The problem is that this set of constraints, although yielding a problem like A*x = b, is not completely well behaved to guarantee a unique solution.
Thus, to cut it short, I'd like to use alglib's facilities to somehow either determine A's pseudoinverse or directly find the best fit for the x vector.
Apart from computing the SVD, is there a more direct algorithm implemented in this library that can solve a system in a least squares sense (again, apart from the SVD or from using the naive inv(transpose(A)*A)*transpose(A)*b formula for general least squares problems where A is not a square matrix?
Found the answer through some careful documentation browsing:
rmatrixsolvels( A, noRows, noCols, b, singularValueThreshold, info, solverReport, x)
The documentation states the the singular value threshold is a clamping threshold that sets any singular value from the SVD decomposition S matrix to 0 if that value is below it. Thus it should be a scalar between 0 and 1.
Hopefully, it will help someone else too.