Eigen Simplicial LDLT Performance for Sparse Linear Systems - c++

Question

I've been using Eigen for a while, albeit probably pretty poorly. The reason I say this is because I can't seem to achieve the performance that others are achieving from the library. I've scoured the forums and dark corners of the internet trying to find solutions to why I cannot seem to achieve great performance, so asking here and hoping that someone can help. Be aware that I don't have much experience with C++ and proper ways of including Eigen. Currently I have simply copied the Eigen source into my project directory and included the folder in my include paths. My project is in Visual Studio 2019.
But on to the problem. I have some reasonably sized sparse linear systems for which I'm trying to solve A*x = b, where A is sparse (typically 0.05% or far less) and can be anywhere from 10x10 to 400,000x400,000, or larger. 'A' is symmetric positive definite. The 'b' vector is typically fairly dense, but can be sparse on occasion.
As a benchmark, I have a problem that is approximately 100,000x100,000. I am pre-conditioning prior to solving potentially numerous 'b' vectors with the same 'A' matrix. Preconditioning takes approximately 6s. Solving takes another 0.5 seconds per 'b' vector. I'm seeing lots of people with problems just as large solving in 0.2 seconds (total, including preconditioning). So what am I doing wrong?
My code for preconditioning is:
SimplicialLDLT<SparseMatrix<double>>* SimplicialLDLT_Preconditioner(SparseMatrix<double>*ptr)
{
SimplicialLDLT<SparseMatrix<double>, Eigen::Lower, AMDOrdering<int>>* solver = new SimplicialLDLT<SparseMatrix<double>, Eigen::Lower, AMDOrdering<int>>();
solver->compute(*ptr);
if (solver->info() != Eigen::ComputationInfo::Success)
{
return nullptr;
}
return solver;
}
My code for solving is:
double* SimplicialLDLT_Solve(SimplicialLDLT<SparseMatrix<double>>* solverPtr, double* f, int numRows)
{
VectorXd f2(numRows);
for (int i = 0; i < numRows; i++)
{
f2[i] = f[i];
}
VectorXd x;
x = solverPtr->solve(f2).eval();
if (solverPtr->info() != Eigen::ComputationInfo::Success)
{
return nullptr;
}
int z = x.rows();
double* x_array = new double[z];
for (int i = 0; i < z; i++)
{
x_array[i] = x[i];
}
return x_array;
}
Based on all of the samples I've seen, I'm doing the same.
My compiler flags (in Microsoft Visual Studio 2019) are:
/Yu"stdafx.h" /GS /Qpar /GL /W3 /Gy /Zc:wchar_t /Zi /Gm- /O2 /Ob2 /Fd"obj\x64\Release\vc142.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "NDEBUG" /D "_USRDLL" /D "NATIVEEIGENWRAPPER_EXPORTS" /D "NOMINMAX" /D "_WINDLL" /D "_UNICODE" /D "UNICODE" /errorReport:prompt /WX- /Zc:forScope /arch:SSE2 /Gd /Oi /MD /openmp /FC /Fa"obj\x64\Release\" /EHsc /nologo /Fo"obj\x64\Release\" /Ot /Fp"obj\x64\Release\myProj.pch" /diagnostics:column
Any insights would be much appreciated. Thanks!
UPDATE #1:
Tried switching compilers to g++ to see if that would help, given that I've read that VS C++ compiler is not fantastic for optimizing. I used the following compiler flags:
-Wa,-mbig-obj -O2 -mavx2 -mfma -finline-limit=1000000 -ffp-contract=fast -march=native -funroll-loops -fopenmp -DNDEBUG
I received maybe a 5% increase in speed, but nothing that I would call significant.
Running another benchmark problem that is approximately 200,000x200,000. Takes approximately 3.5 minutes to precondition using SimplicialLDLT. Matrix is 0.048% filled (so quite sparse) and SPD.