Armadillo (C++ linear algebra library : http://arma.sourceforge.net/) support a preliminary version of Sparse Matrix (in CSC storing model). As I read in the documentation and the code, the armadillo sparse matrix implementation does not differentiate between zero values and "unset" values.
The documentation says that all the value stored are non-zero, by deduction all non-stored values are zero. Instead what I need is to distinguish "unset" from zero.
I currently port a Julia project to C++/Armadillo where I need to manipulate some sparses matrices with -1, 0, 1 and "unset" values. In contrast to armadillo, Julia distinguishes zero and "unset" in its sparse matrix implementation.
My first idea is maybe to use complex sparse matrix (arma::sp_cx_mat, arma::sp_cx_imat) with little tricks to manage simili-zero values (exploit the imaginary as dummy non-zero part). But this is really non-elegant and surely impact the code performance.
Do you think there's a way to bypass the Armadillo limitation without write my own matrix class?
Thank you very much for your answer.
Related
UPDATE: I ended up not using Eigen and implementing my own GF(2) matrix representation where each row is an array of integers, and each bit of the integer represents a single entry. I then use a modified Gaussian Elimination with bit operations to obtain the desired vectors
I currently have a (large) rectangular sparse matrix that I'm storing using Eigen3 that I want to find the (right) null space over GF(2). I researched around and found some possible approaches to this:
(Modified) Gaussian Elimination
This means simply using some form of Gaussian Elimination to find a reduced form of the matrix that preserves the nullspace then extract the nullspace off of that. Though I know how I would do this by hand, I'm quite clueless as to how I would actually implement this.
SVD Decomposition
QR Decomposition
I'm not familiar with these, but I from my understanding the (orthonormal) basis vectors of the nullspace can be extracted from the decomposed form of the matrix.
Now my question is: Which approach should I use in my case (i.e. rectangular sparse matrix over GF(2)) that doesn't involve converting into a dense matrix? And if there are many approaches, what would recommended in terms of performance and ease of implementation?
I'm also open to using other libraries besides Eigen as well.
For context, I'm trying to find combine equivalence relations for factoring algorithms (e.g. as in Quadratic Sieve). Also, if possible, I would like to look into parallelising these algorithms in the future, so if there exists an approach that would allow this, that would be great!
Let's call the matrix in question M. Then (please correct me if I'm wrong):
GF(2) implies that M is a equivalent to a matrix of bits - each element can have one of two values.
Arithmetic on GF(2) is just like integer arithmetic on non-negative numbers, but done modulo 2, so addition is a bitwise XOR, and multiplication is a bitwise AND. It won't matter what exact elements the GF(2) has - they are all equivalent to bits.
Vectors in GF(2) are linearly independent as long as they are not equal, or as long as they differ by at least on bit, or v_1 + v_2 ≠ 0 (since addition in GF(2) is boolean XOR).
By definition, the (right) nullspace spans basis vectors that the matrix transforms to 0. A vector v would be in the nullspace if one multiplies each j-th column of M with the j-th bit of v, sum them, and the result is zero.
I see at least two ways of going about it.
Do dense Gaussian elimination in terms of bit operations, and organize the data and write the loops so that the compiler vectorizes everything and operates on 512-bit data types. You could use Compiler Explorer on godbolt.org to easily check that the vectorization takes place and e.g. AVX512 instructions are used. Linear gains will eventually lose out with the squared scaling of the problem, of course, but the performance increase over naive bool-based implementation will be massive and may be sufficient for your needs. The sparsity adds a possible complication: if the matrix won't comfortably fit in memory in a dense representation, then a suitable representation has to be devised that makes Gaussian elimination perform well. More is need to be known about the matrices you work on. Generally speaking, row operations will be performed at memory bandwidth if the implementation is correct, on the order of 1E10 elements/s, so a 1E3x1E3 M should process in about a second at most.
Since the problem is equivalent to a set of boolean equations, use a SAT solver (Boolean satisfiability problem solver) to incrementally generate the nullspace. The initial equation set is M × v = 0 and v ≠ 0, where v is a bit vector. Run the SAT until it finds some v, let's call it v_i. Then add a constraint v ≠ v_i, and run SAT again - adding the constraints in each iteration. That is, k-th iteration has constraints v ≠ 0, v ≠ v1, ... v ≠ v(k-1).
Since all bit vectors that are different are also linearly independent, the inequality constraints will force incremental generation of nullspace basis vectors.
Modern SAT excels at sparse problems with more boolean equations than variables, so I imagine this would work very well - the sparser the matrix, the better. The problem should be pre-processed to remove all zero columns in M to minimize the combinatorial explosion. Open source SAT solvers can easily deal with 1M variable problems - so, for a sparse problem, you could be realistically solving with 100k-1M columns in M, and about 10 "ones" in each row. So a 1Mx1M sparse matrix with 10 "ones" in each row on average would be a reasonable task for common SAT solvers, and I imagine that state of the art could deal with 10Mx10M matrices and beyond.
Furthermore, your application is ideal for incremental solvers: you find one solution, stop, add a constraint, resume, and so on. So I imagine you may get very good results, and there are several good open source solvers to choose from.
Since you use Eigen already, the problem would at least fit into the SparseMatrix representation with byte-sized elements, so it's not a very big problem as far as SAT is concerned.
I wonder whether this nullspace basis finding is a case of a cover problem, possibly relaxed. There are some nice algorithms for those, but it's always a question of whether the specialized algorithm will work better than just throwing SAT at it and waiting it out, so to speak.
Updated answer - thanks to harold: QR decomposition is not applicable in general for your case.
See for instance
https://math.stackexchange.com/questions/1346664/how-to-find-orthogonal-vectors-in-gf2
I wrongly assumed, QR is applicable here, but it's not by theory.
If you are still interested in details about QR-algorithms, please open a new thread.
I have been trying to develop a program to solve a system Ax=b for a square matrix A using LU-Decomposition. However, I realized that this decomposition does not always exist (one way to tell is if a row exchange operation is not required, then exists). However, I see from many sources that this is an excellent method in computing the solutions to Ax=b.
My question is: how often is it that one comes across a matrix that does not have an LU-decomposition? If one does encounter such a matrix, how should he handle it? Should he create a separate method such as Gaussian Elimination just in case?
Please provide me with some insight on this. Thanks in advance.
Note: I am trying to use this information to solve A^TAx=A^Tb, i.e. finding a mathematical model using least squares.
Taken from wikipedia in its most concise form
Any square matrix $A$ admits an LUP factorization. If $A$ is invertible, then it admits an LU (or LDU) factorization if and only if all its leading principal minors are non-zero. If $A$ is a singular matrix of rank $k$, then it admits an LU factorization if the first $k$ leading principal minors are non-zero, although the converse is not true.
I don't have the implementation fully written, but this looks involved. I would think depending on your matrix, there exists simpler numerical schemes that reduces your solution down.
As for often how does one come across such? Well no one has any idea what you do, so that is impossible to answer. If you encounter such, switch to another scheme.
One that I have used often in practice is Gauss-Seidel. Actually wikipedia has a completely written scheme.
The LU decompositions exists if and only if all leading principal minors of the matrix are non-zero.
From your actual question, you are solving:
A^TAx=A^T
A^TA is a square symmetric matrix. We can diagonalize the matrix as: A = R^-1 D R and you can always rearrange it to find x. You need non-zero eigen values for this to work.
A (square) matrix is invertible if and only if it does not have a zero eigenvalue.
I think inverting it via Gaussian elimination might be the best solution.
I am using the Armadillo library to manually port a piece of Matlab code. The matlab code uses the eigs() function to find a small number (~3) of eigen vectors of a relative large(200x200) covariance matrix R. The code looks like this:
[E,D] = eigs(R,3,"lm");
In Armadillo there are two functions eigs_sym() and eigs_gen() however the former only support real symmetric matrix and the latter requires ARPACK (I'm building the code for Android). Is there a reason eigs_sym doesn't support complex matrices? Is there any other way to find the eigenvectors of a complex symmetric matrix?
The eigs_sym() and eigs_gen() functions (where the s in eigs stands for sparse) in Armadillo are for large sparse matrices. A "large" size in this context is roughly 5000x5000 or larger.
Your R matrix has a size of 200x200. This is very small by current standards. It would be much faster to simply use the dense eigendecomposition eig_sym() or eig_gen() functions to get all the eigenvalues / eigenvectors, followed by extracting a subset of them using submatrix operations like .tail_cols()
Have you tested constructing a 400x400 real symmetric matrix by replacing each complex value, a+bi, by a 2x2 matrix [a,b;-b,a] (alternatively using a block variant of this)?
This should construct a real symmetric matrix that in some way correspond to the complex one.
There will be a slow-down due to the larger size, and all eigenvalues will be duplicated (which may slow down the algorithm), but it seems straightforward to test.
Introduction
I am developing a code in Fortran solving an MHD problem with preconditioning of a linear operator. The sparse matrix to be inverted can be considered as a matrix of the following hierarchical structure. The original matrix (say, A_1) is a band matrix of blocks. Each block of A_1 is a sparse matrix (say, A_2) of the same structure (i.e. a block banded matrix). Each block of A_2 is again a block banded matrix of the same sparsity structure, A_3. Each block of A_3 is, finally, a dense matrix 5 by 5, A_4. I find this hierarchical representation is very convenient to initialize elements of the matrix.
Question
I wonder if there exists a library (in Fortran) permitting to handle such a structure and convert it in one of the standard sparse matrix formats (CSR, CSC, BSR,...), since Sparse BLAS or MKL Pardiso will be used to invert it. Let me stress that my intention is to use the hierarchical structure only to initialize elements of the matrix. Of course, the hierarchical structure can be disregarded and the matrix could be hard-coded in the CSR format, but I find this is too time consuming to implement and test.
Comments
I don't expect a linear solver to use the hierarchical structure, although in S. Pissanetsky " Sparse matrix technology", 1984, Academmic Press, page 27 (available online here) such storage schemes are mentioned, namely, the "hypermatrix" and "supersparse" storage schemes, and were used in Gauss elimination. I have not found available implementations of these schemes yet.
Block compressed sparse row (BSR) format (supported by MKL) can be used to handle two levels of the matrix, A_3(sparse) + A_4(dense), not more.
I want to use the Sparse Blas in Fortran95 just for the creation of the matrices and I am using the point entry construction. After creation of the matrix using the command
call duscr_begin(n,n,a,istat)
here a is the handle to the matrix n by n. After inserting value in it, how can I see the final matrix using its handles a ? As I want to use the matrix for some other operation, so I want to see the matrix in three vectors (sparse) form (row_index, Col_index, Value).
detail about this Sparse Blas is given in Chapter 3 and can be seen here
http://www.netlib.org/blas/blast-forum/
actually what i have asked is before 16 days and it is not just writing of a variable to thee screen. I was using some library known as Sparse Blas for creation of the Sparse matrices. Later on by digging in to the library i found the solution to my problem that using the handles how can we get the three vectors row, col and Val. The commands are something like
call accessdata_dsp(mat,a_handle,ierr)
call get_infoa(mat%INFOA,'n',nnz,ierr)
allocate(K0_row(nnz),K0_col(nnz),K0_A(nnz))
K0_row=mat%IA1; K0_col=mat%IA2; K0_A=mat%A
so here nnz are the non zeros entries in the sparse matrix while K0_row, K0_col and K0_A are our required three vectors, which can be used in further calculation.