I am implementing Latent Dirichlet Allocation (LDA) in Rcpp. In LDA, we need to deal with a huge sparse matrix (e.g. 50 x 3000).
I decided to use SparseMatrix in Eigen. However, since I need access to each cell, computationally expensive .coeffRef slows down my function a lot.
Is there any way to use SparseMatrix while keeping the speed?
What I want to do has four steps,
I know which cell (i,j) I want to access.
I want to know whether the cell (i,j) is 0 or not.
If the cell (i,j) is not 0, I want to know its value.
After doing some analysis with the value in step 2 and 3, I want to update the cell (i,j). In this step, I might need to update the cell (i,j) which originally has 0.
#include <iostream>
#include <Eigen/dense>
#include <Eigen/Sparse>
using namespace std;
using namespace Eigen;
typedef Eigen::Triplet<double> T;
int main(){
Eigen::SparseMatrix<double> spmat;
// Insert in spmat
vector<T> tripletList;
int value;
tripletList.push_back(T(0,1,1));
tripletList.push_back(T(0,3,2));
tripletList.push_back(T(1,5,3));
tripletList.push_back(T(2,4,4));
tripletList.push_back(T(4,1,5));
tripletList.push_back(T(4,5,6));
spmat.resize(5,7); // define size
spmat.setFromTriplets(tripletList.begin(), tripletList.end());
for(int i=0; i<5; i++){ // I am accessing all cells just to clarify I need to access cell
for(int j=0; j<7; j++){
// Check if (i,j) is 0
if(spmat.coeffRef(i,j) != 0){
// Some analysis
value = spmat.coeffRef(i,j)*2; // just an example, more complex in the model
}
spmat.coeffRef(i,j) += value; // update (i,j)
}
}
cout << spmat << endl;
return 0;
}
Since the number of rows is much smaller than the columns, I considered accessing a column and then check the row value, but I couldn't handle SparseMatrix<double>::InnerIterator it(spmat, colid).
Related
I am currently trying to implement matrix multiplication methods using the Microsoft SEAL library. I have created a vector<vector<double>> as input matrix and encoded it with CKKSEncoder. However the encoder packs an entire vector into a single Plaintext so I just have a vector<Plaintext> which makes me lose the 2D structure (and then of course I'll have a vector<Ciphertext> after encryption). Having a 1D vector allows me to access only the rows entirely but not the columns.
I managed to transpose the matrices before encoding. This allowed me to multiply component-wise the rows of the first matrix and columns (rows in transposed form) of the second matrix but I am unable to sum the elements of the resulting vector together since it's packed into a single Ciphertext. I just need to figure out how to make the vector dot product work in SEAL to perform matrix multiplication. Am I missing something or is my method wrong?
It has been suggested by KyoohyungHan in the issue: https://github.com/microsoft/SEAL/issues/138 that it is possible to solve the problem with rotations by rotating the output vector and summing it up repeatedly.
For example:
// my_output_vector is the Ciphertext output
vector<Ciphertext> rotations_output(my_output_vector.size());
for(int steps = 0; steps < my_output_vector.size(); steps++)
{
evaluator.rotate_vector(my_output_vector, steps, galois_keys, rotations_output[steps]);
}
Ciphertext sum_output;
evaluator.add_many(rotations_output, sum_output);
vector of vectors is not the same as an array of arrays (2D, matrix).
While one-dimentional vector<double>.data() points to contiguous memory space (e.g., you can do memcpy on that), each of "subvectors" allocates own, separate memory buffer. Therefore vector<vector<double>>.data() makes no sense and cannot be used as a matrix.
In C++, two-dimensional array array2D[W][H] is stored in memory identically to array[W*H]. Therefore both can be processed by the same routines (when it makes sense). Consider the following example:
void fill_array(double *array, size_t size, double value) {
for (size_t i = 0; i < size; ++i) {
array[i] = value;
}
}
int main(int argc, char *argv[])
{
constexpr size_t W = 10;
constexpr size_t H = 5;
double matrix[W][H];
// using 2D array as 1D to fill all elements with 5.
fill_array(&matrix[0][0], W * H, 5);
for (const auto &row: matrix) {
for (const auto v : row) {
cout << v << '\t';
}
cout << '\n';
}
return 0;
}
In the above example, you can substitute double matrix[W][H]; with vector<double> matrix(W * H); and feed matrix.data() into fill_array(). However, you cannot declare vector(W) of vector(H).
P.S. There are plenty of C++ implementations of math vector and matrix. You can use one of those if you don't want to deal with C-style arrays.
I am trying to utilize sparse matrices in Armadillo, and am noticing a significant difference in access times with SpMat<int> compared to equivalent code using Mat<int>.
Description:
Below are two methods, which are identical in every respect except that Method_One uses regular matrices and Method_Two uses sparse matrices.
Both methods take following arguments:
WS, DS: Pointers to a NN dimensional array
WW: 13 K [max(WS)]
DD: 1.7 K [max(DS)]
NN: 2.3 M
TT: 50
I am using Visual Studio 2017 for compiling the code into a .mexw64 executable which can be called from Matlab.
Code:
void Method_One(int WW, int DD, int TT, int NN, double* WS, double* DS)
{
Mat<int> WP(WW, TT, fill::zeros); // (13000 x 50) matrix
Mat<int> DP(DD, TT, fill::zeros); // (1700 x 50) matrix
Col<int> ZZ(NN, fill::zeros); // 2,300,000 column vector
for (int n = 0; n < NN; n++)
{
int w_n = (int) WS[n] - 1;
int d_n = (int) DS[n] - 1;
int t_n = rand() % TT;
WP(w_n, t_n)++;
DP(d_n, t_n)++;
ZZ(n) = t_n + 1;
}
return;
}
void Method_Two(int WW, int DD, int TT, int NN, double* WS, double* DS)
{
SpMat<int> WP(WW, TT); // (13000 x 50) matrix
SpMat<int> DP(DD, TT); // (1700 x 50) matrix
Col<int> ZZ(NN, fill::zeros); // 2,300,000 column vector
for (int n = 0; n < NN; n++)
{
int w_n = (int) WS[n] - 1;
int d_n = (int) DS[n] - 1;
int t_n = rand() % TT;
WP(w_n, t_n)++;
DP(d_n, t_n)++;
ZZ(n) = t_n + 1;
}
return;
}
Timing:
I am timing both methods using wall_clock timer object in Armadillo. For example,
wall_clock timer;
timer.tic();
Method_One(WW, DD, TT, NN, WS, DS);
double t = timer.toc();
Results:
Timing elapsed for Method_One using Mat<int>: 0.091 sec
Timing elapsed for Method_Two using SpMat<int>: 30.227 sec (almost 300 times slower)
Any insights into this are highly appreciated!
UPDATE:
This issue has been fixed with newer version (8.100.1) of Armadillo.
Here are the new results:
Timing elapsed for Method_One using Mat<int>: 0.141 sec
Timing elapsed for Method_Two using SpMat<int>: 2.127 sec (15 times slower, which is acceptable!)
Thanks to Conrad and Ryan.
As hbrerkere already mentioned, the problem stems from the fact that the values of the matrix are stored in a packed format (CSC) that makes it time-consuming to
Find the index of an already existing entry: Depending on whether the column entries are sorted by their row index you need either linear or binary search.
Insert a value that was previously zero: Here you need to find the insertion point for your new value and move all elements after that, leading to Ω(n) worst case time for a single insertion!
All these operations are constant-time operations for dense matrices, which mostly explains the runtime difference.
My usual solution was to use a separate sparse matrix type for assembly (where you usually access an element multiple times) based on the coordinate format (storing triples (i, j, value)) that uses a map like std::map or std::unordered_map to store the triple index corresponding to a position (i,j) in the matrix.
Some similar approaches are also discussed in this question about matrix assembly
Example from my most recent use:
class DynamicSparseMatrix {
using Number = double;
using Index = std::size_t;
using Entry = std::pair<Index, Index>;
std::vector<Number> values;
std::vector<Index> rows;
std::vector<Index> cols;
std::map<Entry, Index> map; // unordered_map might be faster,
// but you need a suitable hash function
// like boost::hash<Entry> for this.
Index num_rows;
Index num_cols;
...
Number& value(Index row, Index col) {
// just to prevent misuse
assert(row >= 0 && row < num_rows);
assert(col >= 0 && col < num_cols);
// Find the entry in the matrix
Entry e{row, col};
auto it = map.find(e);
// If the entry hasn't previously been stored
if (it == map.end()) {
// Add a new entry by adding its value and coordinates
// to the end of the storage vectors.
it = map.insert(make_pair(e, values.size())).first;
rows.push_back(row);
cols.push_back(col);
values.push_back(0);
}
// Return the value
return values[(*it).second];
}
...
};
After assembly you can store all the values from rows, cols, values (which actually represent the matrix in Coordinate format), possibly sort them and do a batch insertion into your Armadillo matrix.
Sparse matrices are stored in a compressed format (CSC). Every time a non-zero element inserted into a sparse matrix, the entire internal representation has to be updated. This is time consuming.
It's much faster to construct the sparse matrix using batch constructors.
The issue is I need to create a random undirected graph to test the benchmark of Dijkstra's algorithm using an array and heap to store vertices. AFAIK a heap implementation shall be faster than an array when running on sparse and average graphs, however when it comes to dense graphs, the heap should became less efficient than an array.
I tried to write code that will produce a graph based on the input - number of vertices and total number of edges (maximum number of edges in undirected graph is n(n-1)/2).
On the entrance I divide the total number of edges by the number of vertices so that I have a const number of edges coming out from every single vertex. The graph is represented by an adjacency list. Here is what I came up with:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <list>
#include <set>
#define MAX 1000
#define MIN 1
class Vertex
{
public:
int Number;
int Distance;
Vertex(void);
Vertex(int, int);
~Vertex(void);
};
Vertex::Vertex(void)
{
Number = 0;
Distance = 0;
}
Vertex::Vertex(int C, int D)
{
Number = C;
Distance = D;
}
Vertex::~Vertex(void)
{
}
int main()
{
int VertexNumber, EdgeNumber;
while(scanf("%d %d", &VertexNumber, &EdgeNumber) > 0)
{
int EdgesFromVertex = (EdgeNumber/VertexNumber);
std::list<Vertex>* Graph = new std::list<Vertex> [VertexNumber];
srand(time(NULL));
int Distance, Neighbour;
bool Exist, First;
std::set<std::pair<int, int>> Added;
for(int i = 0; i < VertexNumber; i++)
{
for(int j = 0; j < EdgesFromVertex; j++)
{
First = true;
Exist = true;
while(First || Exist)
{
Neighbour = rand() % (VertexNumber - 1) + 0;
if(!Added.count(std::pair<int, int>(i, Neighbour)))
{
Added.insert(std::pair<int, int>(i, Neighbour));
Exist = false;
}
First = false;
}
}
First = true;
std::set<std::pair<int, int>>::iterator next = Added.begin();
for(std::set<std::pair<int, int>>::iterator it = Added.begin(); it != Added.end();)
{
if(!First)
Added.erase(next);
Distance = rand() % MAX + MIN;
Graph[it->first].push_back(Vertex(it->second, Distance));
Graph[it->second].push_back(Vertex(it->first, Distance));
std::set<std::pair<int, int>>::iterator next = it;
First = false;
}
}
// Dijkstra's implementation
}
return 0;
}
I get an error:
set iterator not dereferencable" when trying to create graph from set data.
I know it has something to do with erasing set elements on the fly, however I need to erase them asap to diminish memory usage.
Maybe there's a better way to create some undirectioned graph? Mine is pretty raw, but that's the best I came up with. I was thinking about making a directed graph which is easier task, but it doesn't ensure that every two vertices will be connected.
I would be grateful for any tips and solutions!
Piotry had basically the same idea I did, but he left off a step.
Only read half the matrix, and ignore you diagonal for writing values to. If you always want a node to have an edge to itself, add a one at the diagonal. If you always do not want a node to have an edge to itself, leave it as a zero.
You can read the other half of your matrix for a second graph for testing your implementation.
Look at the description of std::set::erase :
Iterator validity
Iterators, pointers and references referring to elements removed by
the function are invalidated.
All other iterators, pointers and
references keep their validity.
In your code, if next is equal to it, and you erase element of std::set by next, you can't use it. In this case you must (at least) change it and only after this keep using of it.
I am trying to sort pixel values of an image (example 80x20) from lowest to highest.
Below is the some code:
bool sortPixel(int first, int second)
{
return (first < second);
}
vector<int>vect_sortPixel;
for(int y=0; y<height; y++)
{
for(int x=0; x<width; x++)
{
vect_sortPixel.push_back(cvGetReal2D(srcImg, y, x));
sort(vect_sortPixel.begin(), vect_sortPixel.end(), sortPixel);
}
}
But it takes quite long time to compute. Any suggestion to reduce the processing time?
Thank you.
Don't use getReal2D. It's quite slow.
Convert image to cv::Mat or Mat. Use its data pointer to get the pixel values. Mat.data() will give you pointer to the original matrix. Use that.
And as far as sorting is concerned, I would advise you to first make an array of all the pixels, then sort it using Merge sort (time complexity O(n log n))
#include<opencv2/highgui/highgui.hpp>
#include<stdio.h>
using namespace cv;
using namespace std;
int main()
{
Mat img = imread("filename.jpg",CV_LOAD_IMAGE_COLOR);
unsigned char *input = (unsigned char*)(img.data);
int i,j,r,g,b;
for(int i = 0;i < img.cols;i++){
for(int j = 0;j < img.rows;j++){
b = input[img.cols * j + i] ;
g = input[img.cols * j+ i + 1];
r = input[img.cols *j + i +2];
}
}
return 0;
}
Using this you can access pixel values from the main matrix.
Warning: This is not how you compare it. I'm suggesting that by using something like this, you can access pixel values.
Mat.data() gives you pointer to the original matrix. This matrix is a 1 D matrix with all the given pixel values.
Image => (x,y,z),(x1,y1,z1), etc..
Mat(original matrix) => x,y,z,x1,y1,z1,...
If you still have some doubts regarding how to extract data from Mat, visit this link OpenCV get pixel channel value from Mat image
and here's a link regarding Merge Sort http://www.cplusplus.happycodings.com/Algorithms/code17.html
There are few problems in your code:
As Froyo already said you use cvGetReal2D which is actually not very fast. You have to convert your cvMat to cv::Mat. To do this there's cv::Mat constructor:
// converts old-style CvMat to the new matrix; the data is not copied by default
Mat(const CvMat* m, bool copyData=false);
And after this use direct pixels acces as mentioned in this SO question.
Another problem is that you use push_back which actually also not very fast. You know the size of array, so why don't you allocate needed memory at the beginning? Like this:
vector<int> vect_sortPixel(mat.cols*mat.rows);
And than just use vect_sortPixel[i] to get needed pixel.
Why do you call sort in the loop? You have to call it after loop, when array is already created! Default STL's sort should work fast:
Complexity
Approximately N*logN comparisons on average (where N is
last-first). In the worst case, up to N^2, depending on specific
sorting algorithm used by library implementation.
I have a ~3000x3000 covariance-alike matrix on which I compute the eigenvalue-eigenvector decomposition (it's a OpenCV matrix, and I use cv::eigen() to get the job done).
However, I actually only need the, say, first 30 eigenvalues/vectors, I don't care about the rest. Theoretically, this should allow to speed up the computation significantly, right? I mean, that means it has 2970 eigenvectors less that need to be computed.
Which C++ library will allow me to do that? Please note that OpenCV's eigen() method does have the parameters for that, but the documentation says they are ignored, and I tested it myself, they are indeed ignored :D
UPDATE:
I managed to do it with ARPACK. I managed to compile it for windows, and even to use it. The results look promising, an illustration can be seen in this toy example:
#include "ardsmat.h"
#include "ardssym.h"
int n = 3; // Dimension of the problem.
double* EigVal = NULL; // Eigenvalues.
double* EigVec = NULL; // Eigenvectors stored sequentially.
int lowerHalfElementCount = (n*n+n) / 2;
//whole matrix:
/*
2 3 8
3 9 -7
8 -7 19
*/
double* lower = new double[lowerHalfElementCount]; //lower half of the matrix
//to be filled with COLUMN major (i.e. one column after the other, always starting from the diagonal element)
lower[0] = 2; lower[1] = 3; lower[2] = 8; lower[3] = 9; lower[4] = -7; lower[5] = 19;
//params: dimensions (i.e. width/height), array with values of the lower or upper half (sequentially, row major), 'L' or 'U' for upper or lower
ARdsSymMatrix<double> mat(n, lower, 'L');
// Defining the eigenvalue problem.
int noOfEigVecValues = 2;
//int maxIterations = 50000000;
//ARluSymStdEig<double> dprob(noOfEigVecValues, mat, "LM", 0, 0.5, maxIterations);
ARluSymStdEig<double> dprob(noOfEigVecValues, mat);
// Finding eigenvalues and eigenvectors.
int converged = dprob.EigenValVectors(EigVec, EigVal);
for (int eigValIdx = 0; eigValIdx < noOfEigVecValues; eigValIdx++) {
std::cout << "Eigenvalue: " << EigVal[eigValIdx] << "\nEigenvector: ";
for (int i = 0; i < n; i++) {
int idx = n*eigValIdx+i;
std::cout << EigVec[idx] << " ";
}
std::cout << std::endl;
}
The results are:
9.4298, 24.24059
for the eigenvalues, and
-0.523207, -0.83446237, -0.17299346
0.273269, -0.356554, 0.893416
for the 2 eigenvectors respectively (one eigenvector per row)
The code fails to find 3 eigenvectors (it can only find 1-2 in this case, an assert() makes sure of that, but well, that's not a problem).
In this article, Simon Funk shows a simple, effective way to estimate a singular value decomposition (SVD) of a very large matrix. In his case, the matrix is sparse, with dimensions: 17,000 x 500,000.
Now, looking here, describes how eigenvalue decomposition closely related to SVD. Thus, you might benefit from considering a modified version of Simon Funk's approach, especially if your matrix is sparse. Furthermore, your matrix is not only square but also symmetric (if that is what you mean by covariance-like), which likely leads to additional simplification.
... Just an idea :)
It seems that Spectra will do the job with good performances.
Here is an example from their documentation to compute the 3 first eigen values of a dense symmetric matrix M (likewise your covariance matrix):
#include <Eigen/Core>
#include <Spectra/SymEigsSolver.h>
// <Spectra/MatOp/DenseSymMatProd.h> is implicitly included
#include <iostream>
using namespace Spectra;
int main()
{
// We are going to calculate the eigenvalues of M
Eigen::MatrixXd A = Eigen::MatrixXd::Random(10, 10);
Eigen::MatrixXd M = A + A.transpose();
// Construct matrix operation object using the wrapper class DenseSymMatProd
DenseSymMatProd<double> op(M);
// Construct eigen solver object, requesting the largest three eigenvalues
SymEigsSolver< double, LARGEST_ALGE, DenseSymMatProd<double> > eigs(&op, 3, 6);
// Initialize and compute
eigs.init();
int nconv = eigs.compute();
// Retrieve results
Eigen::VectorXd evalues;
if(eigs.info() == SUCCESSFUL)
evalues = eigs.eigenvalues();
std::cout << "Eigenvalues found:\n" << evalues << std::endl;
return 0;
}