Suppose I want to implement a gradient descent based optimizer of some function f(x),
which is a combination of simple functions: +,*,sin,cos (leaving out / for simplicity)
Is there a way, using templates to symbolically calculate the derivative and make a function f'(x)
in C++, then use the function and its gradient at runtime to optimize it.
I'm comfortable with symbolic mathematics, so that isn't the focus of the question.
I could write a parser and input the function as a string, expanding it dynamically at runtime, but especially for more complex functions, this is liable to be slow.
If there's a way to produce the function at compile time, that would be awesome.
Related
Consider the following goal:
Create a program that solves: minimize f(x) for an arbitrary f and x supplied as input.
How could one design a C++ program that could receive a description of f and x and process it efficiently?
If the program was actually a C++ library then one could explicitly write the code for f and x (probably inheriting from some base function class for f and state class for x).
However, what should one do if the program is for example a service, and the user is sending the description of f and x in some high level representation, e.g. a JSON object?
Ideas that come to mind
1- Convert f into an internal function representation (e.g. a list of basic operations). Apply those whenever f is evaluated.
Problems: inefficient unless each operation is a batch operation (e.g. if we are doing vector or matrix operations with large vectors / matrices).
2- Somehow generate C++ code and compile the code for representing x and computing f. Is there a way to restrict compilation so that only that code needs to be compiled, but the rest of the code is 'pre-compiled' already?
The usual approach used by the mp library and others is to create an expression tree (or DAG) and use some kind of a nonlinear optimization method that normally relies on derivative information which can be computed using automatic or numeric differentiation.
An expression tree can be efficiently traversed for evaluation using a generic visitor pattern. Using JIT might be an overkill unless the time taken for evaluating a function takes substantial fraction of the optimization time.
I have a cyclic program in C++ which includes composing of a function (every time it is different) and further minimization of it. Composing of a function is implemented with GiNaC package (symbolic expressions).
I tried to minimize functions using Matlab fmincon function but it ate all the memory while converting string to lambda function (functions are rather complicated). And I couldn't manage to export function from C++ to Matlab in any way but as a string.
Is there any way to compose a complicated function (3 variables, sin-cos-square root etc.) and minimize it without determing gradient by myself because I don't know how functions look before running the program?
I also looked at NLopt and as I understood it requires gradients to be writte by programmer.
Most optimization algorithms do require the gradient. However, if it's impossible to 'know' it directly, you may evaluate it considering a small increment of every coordinate. If your F function depends on coordinates of x vector, you may approximate the i's component of you gradient vector G as
x1 = x;
x1[i] += dx;
G[i] = (F(x1) - F(x))/dx;
where dx is some small increment. Although such a calculation is approximate it's usually absolutely good for a minimum finding provided that dx is small enough.
I am studying numerical analysis and also solving algorithms which is described in book. My problem is about Newton's method. In general, if some function is given and we have to find root, how can we determine derivative of function in code? or even limit? because as you know Newton's method involves derivative and makes iteration like this.
Suppose some function f(x) and initial guess,p0, then p(n)=p(n-1)+f(p(n-1))/f'(p(n-1)) here f' denotes derivative of f.
How can I approximate it in code? Thanks a lot.
If you want to use Newton's Method, you will need to know the derivative of the function and code it in.
Otherwise, you can go with the Secant Method which doesn't require knowing the derivative. But it converges at a slower rate.
Depending on how the function is given, you can do a couple of things
symbolic differentiation, if you have a symbolic representation of your function
Numerical differentiation, if you only have point-value pairs
Interpolate with a polynomial and differentiate that (symobolically of course)
All Options are viable. Which of these is most suited to your problem depends on the function and also the time you want to invest in coding and/or reading up on how to do it.
Edit: If you already know the function before execution time, then compute the differential by hand and implement it as a function. You should also already have implemented your f(x) as a function like this
float f (float x) {
// ...
}
And thus:
float df_dx (float x) {
// ...
}
Libraries such as intel-MKL or amd-ACML provide easier interface to SIMD operations on vectors, but I want to chain several functions together. Are there readily available libraries where I can register a parse tree for an expression like
log( tanh(x) + exp(x) )
and then evaluate it on all members of an array ? What I want to avoid is to make a temporary arrays of tanh(x), exp(x) and tanh(x) + exp(x) by calling the mkl or acml functions for tanh(), exp() and +.
I can unroll the loop by hand and use the sse instructions directly, but was wondering if there are C++ libraries which does this for you, i.e.
1. Handles SIMD/SSE functions
2. Allows building of parse trees out of SIMD/SSE functions.
I am very much a newbie and have never used SSE or MKL/ACML before, just venturing out into new territory.
It may not do exactly what you want, but I suggest you take a look at macstl. It's a SIMD valarray implementation which uses template metaprogramming, and which can combine expressions into a single loop. You may be able to use this as is or perhaps as a basis for something closer to what you need.
Have a look at Intel ABB. It uses a just in time compilation approach IIRC. It can use vector instructions and multithreading depending on the sizes of the vectors you act upon.
Let say I have a snippet of code like this:
typedef double My_fp_t;
My_fp_t my_fun( My_fp_t input )
{
// some fp computation, it uses operator+, operator- and so on for type My_fp_t
}
My_fp_t input = 0.;
My_fp_t output = my_fun( input );
Is it possible to retrofit my existing code with a floating point arbitrary precision C++ library?
I would like to simple add #include <cpp_arbitrary_precision_fp>, change my typedef double My_fp_t; into typedef arbitrary_double_t My_fp_t; and let the operator overloading of C++ doing its job...
My main problem is that actually my code do NOT have the typedef :-( and so maybe my plan is doomed to failure.
Assuming that my code had the typedef, what other problems would I face?
This might be tough. I used a template approach in my PhD thesis code do deal with different numerical types. You might want to take a look at it to see the problems I encountered.
The thing is you are fine if all you do with your numbers is use the standard arithmetic operators. However, as soon as you use a square root or some other non operator function you need to create helper objects to detect your object's type (at compile time as it is too slow to do this at run time; see the boost metaprogramming library for help on that) and then call the correct function and return it as the correct type. It is all totally doable, but is likely to take longer than you think and will add considerably to the complexity of your code.
In my experience, (I was using GMP which must be the fastest arbitrary precision library available for C++) after all of the effort and complexity I had introduced, I found that GMP was just too slow for the sorts of computation that I was doing; so it was academically interesting, but practically useless. Before you start on this do some speed tests to see whether your library will still be usable if you use arbitrary precision arithmetic.
If the library defines a type that correctly overloads the operators you use, I don't see any problem...