Using Eigen to solve a dense, constrained least squares fit - c++

Question

I need to solve a classic problem of the form Ax = b for a vector x that is of size 4. A is on the order of ~500 data points and thus is a dense 500x4 matrix.
Currently I can solve this using the normal equations described here and it works fine however I would like to constrain one of my parameters in x to never be above a certain value.
Is there a good way to do this programmatically with Eigen?

Answer 1

You can try my quadradic programming solver based on Eigen there. You'll still have to form the normal equation.

Answer 2

Here is some python-demo (using numpy which is not that far away from Eigen) showing an accelerated projected-gradient algorithm for your kind of problem. Typically this approach is used for large-scale problems (where other algorithms incorporating second-order information might struggle), but it's also nice to implement.
This version, which is a small modification from some old code of mine is not the most simple approach which can be used, as we are using:
acceleration / momentum (faster iteration)
line-search (saves us some step-size tuning trouble)
You could remove the line-search and tune the step-size. Momentum is also not needed.
As i'm not doing much C++ right now, i don't think i will port this to Eigen. But i'm sure, that if you would wanto to port it, it's not that hard. Eigen should not be too different from numpy.
I did not measure the performance, but the results are calculated instantly (perceptually).
Edit: some non-scientific timings (more momentum; lesser tolerance than in following code):
A=(500,4):
Solve non-bounded with scipys lsq_linear
used: 0.004898870004675975
cost : 244.58267993
Solve bounded (0.001, 0.05) with scipys lsq_linear
used: 0.005605718416479959
cost : 246.990611225
Solve bounded (0.001, 0.05) with accelerated projected gradient
early-stopping # it: 3
used: 0.002282825315435914
cost: 246.990611225
A=(50000, 500):
Solve non-bounded with scipys lsq_linear
used: 4.118898701951786 secs
cost : 24843.7115776
Solve bounded (0.001, 0.05) with scipys lsq_linear
used: 14.727660030288007 secs
cost : 25025.0328661
Solve bounded (0.001, 0.05) with accelerated projected gradient
early-stopping # it: 14
used: 5.319953458329618 secs
cost: 25025.0330754
The basic idea is to use some gradient-descent-like algorithm, and project onto our constraints after each gradient-step. This approach is very powerful, if that projection can be done efficiently. Box-constraint-projections are simple!
Page 4 in this pdf shows you the box-constraint projection.
We just clip our solution-vector to lower_bound, upper_bound. Clipping in numpy is described as: Given an interval, values outside the interval are clipped to the interval edges. For example, if an interval of [0, 1] is specified, values smaller than 0 become 0, and values larger than 1 become 1.
It's an iterative-algorithm approximating the solution and i think every algorithm in use will be an iterated one.
Code
import numpy as np
from scipy.optimize import lsq_linear
np.random.seed(1)
A = np.random.normal(size=(500, 4))
b = np.random.normal(size=500)
""" Solve Ax=b
----------
"""
print('Solve non-bounded with scipys lsq_linear')
sol = lsq_linear(A, b)
print('Ax=b sol: ', sol['x'])
print('cost : ', sol['cost'])
print()
""" Solve Ax=b with box-constraints
-------------------------------
"""
print('Solve bounded (0.001, 0.05) with scipys lsq_linear')
sol = lsq_linear(A, b, bounds=(0.001, 0.05))
print('Ax=b constrained sol: ', sol['x'])
print('cost : ', sol['cost'])
print()
""" Solve Ax=b with box-constraints using a projected gradient algorithm
--------------------------------------------------------------------
"""
def solve_pg(A, b, bounds=(-np.inf, np.inf), momentum=0.9, maxiter=1000):
""" remarks:
algorithm: accelerated projected gradient
projection: proj onto box constraints
line-search: armijo-rule along projection-arc (Bertsekas book)
stopping-criterion: naive
gradient-calculation: precomputes AtA
"""
lb = np.empty(A.shape[1])
ub = np.empty(A.shape[1])
if len(bounds) == 2:
# apply lb & ub to all variables
lb = bounds[0]
ub = bounds[1]
else:
# assume dimensions are ok
lb = np.array(bounds[0])
ub = np.array(bounds[1])
M, N = A.shape
x = np.zeros(N)
AtA = A.T.dot(A)
Atb = A.T.dot(b)
stop_count = 0
def gradient(x):
return AtA.dot(x) - Atb
def obj(x):
return 0.5 * np.linalg.norm(A.dot(x) - b)**2
it = 0
while True:
grad = gradient(x)
# line search
alpha = 1
beta = 0.5
sigma=1e-2
old_obj = obj(x)
while True:
new_x = x - alpha * grad
new_obj = obj(new_x)
if old_obj - new_obj >= sigma * grad.dot(x - new_x):
break
else:
alpha *= beta
x_old = x[:]
x = x - alpha*grad
# projection
np.clip(x, lb, ub, out=x) # Projection onto box constraints
# see SO-text
# in-place clipping
y = x + momentum * (x - x_old)
if np.abs(old_obj - obj(x)) < 1e-2:
stop_count += 1
else:
stop_count = 0
if stop_count == 3:
print('early-stopping # it: ', it)
return x
it += 1
if it == maxiter:
return x
print('Solve bounded (0.001, 0.05) with accelerated projected gradient')
sol = solve_pg(A, b, bounds=(0.001, 0.05))
print(sol)
print('cost: ', 0.5 * (np.square(np.linalg.norm(A.dot(sol) - b))))
Output
Solve non-bounded with scipys lsq_linear
Ax=b sol: [ 0.06627173 -0.06104991 -0.07010355 0.04024075]
cost : 244.58267993
Solve bounded (0.001, 0.05) with scipys lsq_linear
Ax=b constrained sol: [ 0.05 0.001 0.001 0.03902291]
cost : 246.990611225
Solve bounded (0.001, 0.05) with accelerated projected gradient
early-stopping # it: 3
[ 0.05 0.001 0.001 0.03902229]
cost: 246.990611225