is there any way to compute the matrix logarithm in OpenCV? I understand that it's not available as a library function, but, pointers to a good source (paper, textbook, etc) will be appreciated.
As a matter of fact, I'm in the process of programming the matrix logarithm in the Eigen library which is apparently used in some Willow Garage libraries; not sure about OpenCV. Higham's book (see answer by aix) is the best reference in my opinion and I'm implementing Algorithm 11.11 in his book. That is a rather complicated algorithm though.
Diagonalization (as in Alexandre's comment) is an easy-to-program method which works very well for symmetric positive definite matrices. It also works well for many general matrices. However, it is not accurate for matrices whose eigenvalues are close together, and it fails for matrices that are not diagonalizable.
If you want something more robust than diagonalization but less complicated than Higham's Algorithm 11.11 then I'd recommend to do a Schur decomposition followed by inverse scaling and squaring. This is Algorithm 11.10 in Higham's book, and described in the paper "Approximating the Logarithm of a Matrix to Specified Accuracy" (http://dx.doi.org/10.1137/S0895479899364015, preprint at http://eprints.ma.man.ac.uk/318/).
If you use OpenCV matrices, you can easily map them to Eigen3 matrices. See this post:
OpenCV CV::Mat and Eigen::Matrix
Then, Eigen3 library has a matrix logarithm function that you can use:
http://eigen.tuxfamily.org/dox/unsupported/group__MatrixFunctions__Module.html
it is under the unsupported module, but this is not an issue, this just means:
These modules are contributions from various users. They are provided
"as is", without any support.
-- http://eigen.tuxfamily.org/dox/unsupported/
Hope this is of help.
Related
Let A be a positive definite matrix, and let A=L*L' be its cholesky factorization, where L is lower triangular.
Let A2 = A + alpha*x*x' be a rank-1 update of matrix A, where x is a vector of appropriate dimension and alpha is a scalar.
The Cholesky factor update is a procedure for obtaining the factorization A2=L2*L2' without calculating A2 first, which is useful to speed up computations in the case of such low-rank matrix updates.
I am using BLAS/LAPACK libraries for elementary algebra manipulations. I can calculate the Cholesky factorization of a positive definite matrix with the routine spptrf. However, I have been looking around and I have not been able to find a BLAS/LAPACK function which performs Cholesky factor updates. May it be that there is not function doing so?
Additionally: In this old post, the addition of such routine was discussed. However, it is a very old post (2013) and I have not been able of finding anything more recent.
There is no such function. You can look at this discussion we had on SciPy. I've made up a Python script that does the update with the relevant paper. You can make use of those information.
https://github.com/scipy/scipy/issues/8188
If you feel competitive and actually write the Fortran code for this, I would really appreciate if you can submit it to LAPACK repo as a PR https://github.com/Reference-LAPACK/lapack
The BLAS libraries on Netlib as you pointed out however I doubt it is on the site. If you are looking for code simply there is code here.
If you want it fast I'd just turn that code into Julia. There is a book I've never checked out which may have these in it. Also, note you cited a paper that the author wrote the code for. You could have simply contacted the author of the paper. His website appears to be here. There is a problem with that link though.
I am used to Eigen for almost all my mathematical linear algebra work.
Recently, I have discovered that Boost also provides a C++ template class library that provides Basic Linear Algebra Library (Boost::uBLAS). This got me wondering if I can get all my work based only on boost as it is already a major library for my code.
A closer look at both didn't really got me a clearer distinction between them:
Boost::uBLAS :
uBLAS provides templated C++ classes for dense, unit and sparse vectors, dense, identity, triangular, banded, symmetric, hermitian and sparse matrices. Views into vectors and matrices can be constructed via ranges, slices, adaptor classes and indirect arrays. The library covers the usual basic linear algebra operations on vectors and matrices: reductions like different norms, addition and subtraction of vectors and matrices and multiplication with a scalar, inner and outer products of vectors, matrix vector and matrix matrix products and triangular solver.
...
Eigen :
It supports all matrix sizes, from small fixed-size matrices to arbitrarily large dense matrices, and even sparse matrices.
It supports all standard numeric types, including std::complex, integers, and is easily extensible to custom numeric types.
It supports various matrix decompositions and geometry features.
Its ecosystem of unsupported modules provides many specialized features such as non-linear optimization, matrix functions, a polynomial solver, FFT, and much more.
...
Does anyone have a better idea about their key differences and on which basis can we choose between them?
I'm rewriting a substantial project from boost::uBLAS to Eigen. This is production code in a commercial environment. I was the one who chose uBLAS back in 2006 and now recommended the change to Eigen.
uBLAS results in very little actual vectorization performed by the compiler. I can look at the assembly output of big source files, compiled to amd64 architecture, with SSE, using the float type, and not find a single ***ps instruction (addps, mulps, subps, 4 way packed single-precision floating point instructions) and only ***ss instructions (addss, ..., scalar single-precision).
With Eigen, the library is written to make sure that vector instructions result.
Eigen is very feature complete. Has lots of matrix factorizations and solvers. In boost::uBLAS the LU factorization is an undocumented add-on, a piece of contributed code. Eigen has additions for 3D geometry, such as rotations and quaternions, not uBLAS.
uBLAS is slightly more complete on the most basic operations. Eigen lacks some things, such as projection (indexing a matrix using another matrix), while uBLAS has it. For features that both have, Eigen is more terse, resulting in expressions that are easier to read.
Then, uBLAS is completely stale. I can't understand how anyone considers it in 2016/2017. Read the FAQ:
Q: Should I use uBLAS for new projects?
A: At the time of writing (09/2012) there are a lot of good matrix libraries available, e.g., MTL4, armadillo, eigen. uBLAS offers a stable, well tested set of vector and matrix classes, the typical operations for linear algebra and solvers for triangular systems of equations. uBLAS offers dense, structured and sparse matrices - all using similar interfaces. And finally uBLAS offers good (but not outstanding) performance. On the other side, the last major improvement of uBLAS was in 2008 and no significant change was committed since 2009. So one should ask himself some questions to aid the decision: Availability? uBLAS is part of boost and thus available in many environments. Easy to use? uBLAS is easy to use for simple things, but needs decent C++ knowledge when you leave the path. Performance? There are faster alternatives. Cutting edge? uBLAS is more than 10 years old and missed all new stuff from C++11.
I just did a time complexity comparison between boost and eigen for fairly trivial matrix computations. These results, limited as they are, seem to denote that boost is a much better alternative.
I had an FEM code which does the pre-processing parts (setting up the element matrices and stitching them together). So naturally, this would involve a lot of memory allocations.
I wrote identical pieces of codes with Boost and Eigen on C++ (gcc 5.4.0, ubuntu 16.04, Intel i3 Quad Core, 2.40GHz, RAM : 4Gb) and ran them separately for varying node sizes (N) and calculated time using the linux cl-utility.
As far as I'm concerned, I have decided to proceed with my code in Boost.
Choose Eigen if you care the performance and performance gain introduced by expression templates, and choose uBlas if you only want to learn expression templates.
http://eigen.tuxfamily.org/index.php?title=Benchmark
I implemented a program on the GPU (CUDA) which only uses the host (in C++) to start new kernels. During the calculation on the device I need SVD and solving systems of 3x3 (dense) matrices, fixed size.
I've got my own SVD and solver implementation but it is not numerical stable (thus not usable). Due to me being rather new with C++ and CUDA I would prefer to use a library instead. (numerical stuff is very tricky)
Now I have trouble finding that library:
cuSOLVER is not callable from the device
cuLA is not callable form the device (and abandoned so it seems)
Eigen looks promising (should be callable from device?) but it is unclear what the status is on CUDA support (it says experimental). I find people saying it works, others got compile errors?
Preferable I would also being able to do general matrix operations with the library (transpose, inversion, sum, multiply, ...) as my own implementations will likely be less efficient and numerically stable for those.
Any ideas on how to achieve this?
UPDATE:
Seems like Eigen supports basic functions like *,+, transpose and even eigenvalues but SVD, inverse ect is not yet supported. This is at the time of writing.
According to the website, a subset of features works for fixed size matrices (3x3 in your case) from Eigen 3.3. The current stable release is 3.2.6 while 3.3 is in alpha. I don't know if specifically SVD is supported in CUDA. I would recommend trying a small MCVE to see if it works (as well as the other functions you require), and if so, implementing it in your project.
I'm having a similar problem; want to generate random vectors within a kernel function which requires performing cholesky/eigenvalue decompositions of NxN (N<=5) covariance matrices. Since, as you noted, the MAGMA and CULA libraries are not available from the device, and there seems to be no cuSOLVER device API yet, I've resorted to implementing these myself following algorithms outlined in, for example, Numerical Recipes in C. As for solving linear systems, I'd suggest checking out the cuBLAS (level 2 functions), as it provides some basic functionality. If you want to invert matrices, I'd suggest cublasmatinvBatched(). I haven't used it myself, will give it a try during the weekend, but from the description it sounds promising. Hope others will chime into this thread with better solutions...
I have a sparse banded matrix A and I'd like to (direct) solve Ax=b. I have about 500 vectors b, so I'd like to solve for the corresponding 500 x's.
I'm brand new to CUDA, so I'm a little confused as to what options I have available.
cuSOLVER has a batch direct solver cuSolverSP for sparse A_i x_i = b_i using QR here. (I'd be fine with LU too since A is decently conditioned.) However, as far as I can tell, I can't exploit the fact that all my A_i's are the same.
Would an alternative option be to first determine a sparse LU (QR) factorization on the CPU or GPU then perform in parallel the backsubstitution (respectively, backsub and matrix mult) on the GPU? If cusolverSp< t >csrlsvlu() is for one b_i, is there a standard way to batch perform this operation for multiple b_i's?
Finally, since I don't have intuition for this, should I expect a speedup on a GPU for either of these options, given the necessary overhead? x has length ~10000-100000. Thanks.
I'm currently working on something similar myself. I decided to basically wrap the conjugate gradient and level-0 incomplete cholesky preconditioned conjugate gradient solvers utility samples that came with the CUDA SDK into a small class.
You can find them in your CUDA_HOME directory under the path:
samples/7_CUDALibraries/conjugateGradient and /Developer/NVIDIA/CUDA-samples/7_CUDALibraries/conjugateGradientPrecond
Basically, you would load the matrix into the device memory once (and for ICCG, compute the corresponding conditioner / matrix analysis) then call the solve kernel with different b vectors.
I don't know what you anticipate your matrix band structure to look like, but if it is symmetric and either diagonally dominant (off diagonal bands along each row and column are opposite sign of diagonal and their sum is less than the diagonal entry) or positive definite (no eigenvectors with an eigenvalue of 0.) then CG and ICCG should be useful. Alternately, the various multigrid algorithms are another option if you are willing to go through coding them up.
If your matrix is only positive semi-definite (e.g. has at least one eigenvector with an eigenvalue of zero) you can still ofteb get away with using CG or ICCG as long as you ensure that:
1) The right hand side (b vectors) are orthogonal to the null space (null space meaning eigenvectors with an eigenvalue of zero).
2) The solution you obtain is orthogonal to the null space.
It is interesting to note that if you do have a non-trivial null space, then different numeric solvers can give you different answers for the same exact system. The solutions will end up differing by a linear combination of the null space... That problem has caused me many many man hours of debugging and frustration before I finally caught on, so its good to be aware of it.
Lastly, if your matrix has a Circulant Band structure you might consider using a fast fourier transform (FFT) based solver. FFT based numerical solvers can often yield superior performance in cases where they are applicable.
is there a standard way to batch perform this operation for multiple b_i's?
One option is to use the batched refactorization module in CUDA's cuSOLVER, but I am not sure if it is standard.
Batched refactorization module in cuSOLVER provides an efficient method to solve batches of linear systems with fixed left-hand side sparse matrix (or matrices with fixed sparsity pattern but varying coefficients) and varying right-hand sides, based on LU decomposition. Only some partially completed code snippets can be found in the official documentation (as of CUDA 10.1) that are related to it. A complete example can be found here.
If you don't mind going with an open-source library, you could also check out CUSP:
CUSP Quick Start Page
It has a fairly decent suite of solvers, including a few preconditioned methods:
CUSP Preconditioner Examples
The smoothed aggregation preconditioner (a variant of algebraic multigrid) seems to work very well as long as your GPU has enough onboard memory for it.
Does boost have one?
Where A, y and x is a matrix (sparse and can be very large) and vectors respectively.
Either y or x can be unknown.
I can't seem to find it here:
http://www.boost.org/doc/libs/1_39_0/libs/numeric/ublas/doc/index.htm
yes, you can solve linear equations with boost's ublas library. Here is one short way using LU-factorize and back-substituting to get the inverse:
using namespace boost::ublas;
Ainv = identity_matrix<float>(A.size1());
permutation_matrix<size_t> pm(A.size1());
lu_factorize(A,pm)
lu_substitute(A, pm, Ainv);
So to solve a linear system Ax=y, you would solve the equation trans(A)Ax=trans(A)y by taking the inverse of (trans(A)A)^-1 to get x: x = (trans(A)A)^-1Ay.
Linear solvers are generally part of the LAPACK library which is a higher level extension of the BLAS library. If you are on Linux, the Intel MKL has some good solvers, optimized both for dense and sparse matrices. If you are on windows, MKL has a one month trial for free... and to be honest I haven't tried any of the other ones out there. I know the Atlas package has a free LAPACK implementation but not sure how hard it is to get running on windows.
Anyways, search around for a LAPACK library which works on your system.
One of the best solvers for Ax = b, when A is sparse, is Tim Davis's UMFPACK
UMFPACK computes a sparse LU decomposition of A. It is the algorithm that
gets used behind the scenes in Matlab when you type x=A\b (and A is sparse
and square). UMFPACK is free software (GPL)
Also note if y=Ax, and x is known, but y is not, you compute y by performing a sparse matrix vector multiply, not by solving a linear system.
Reading the boost documentation, it does not seem like solving w.r.t x is implemented. Solving in y is only a matter of matrix-vector product, which seems implemented in ublas.
One thing to keep in mind is that blas only implement 'easy' operations like addition, multiplication, etc... of vector and matrix types. Anything more advanced (linear problem solving, like your "solve in x y = A x", eigen vectors and co) is part of LAPACK, which built on top of BLAS. I don't know what boost provides in that respect.
Boost's linear algebra package's tuning focused on "dense matrices".
As far as I know, Boost's package do not have any linear-system-solver.
How about use source code in "Numerical Recipe in C (http://www.nr.com/oldverswitcher.html)" ?
Note. There can be subtle index bug in the source code (some code uses array index start from 1)
Take a look at JAMA/TNT. I've only used it for non-sparse matrices (you probably want the QR or LU factorizations, both of which have solver utility methods), but it apparently has some facilities for sparse matrices.