We're working on a machine learning project in which we'd like to see the influence of certain online sample embedding methods on SVMs.
In the process we've tried interfacing with Pegasos and dlib as well as designing (and attempting to write) our own SVM implementation.
dlib seems promising as it allows interfacing with user written kernels.
Yet kernels don't give us the desired "online" behavior (unless that assumption is wrong).
Therefor, if you know about an SVM library which supports online embedding and custom written embedders, it would be of great help.
Just to be clear about "online".
It is crucial that the embedding process will happen online in order to avoid heavy memory usage.
We basically want to do the following within Stochastic subGradient Decent(in very general pseudo code):
w = 0 vector
for t=1:T
i = random integer from [1,n]
embed(sample_xi)
// sample_xi is sent to sub gradient loss i as a parameter
w = w - (alpha/t)*(sub_gradient(loss_i))
end
I think in your case you might want to consider the Budgeted Stochastic Gradient Descent for Large-Scale SVM Training (BSGD) [1] by Wang, Crammer, Vucetic
This is because, as specified in the paper about the "Curse of Kernelization" you might want to explore this option instead what you have indicated in the pseudocode in your question.
The Shark Machine Learning Library implements BSGD. Check a quick tutorial here
Maybe you want to use something like dlib's empirical kernel map. You can read it's documentation and particularly the example program for the gory details of what it does, but basically it lets you project a sample into the span of some basis set in a kernel feature space. There are even algorithms in dlib that iteratively build the basis set, which is maybe what you are asking about.
Related
Has anybody tried developing a SLAM system that uses deep learned features instead of the classical AKAZE/ORB/SURF features?
Scanning recent Computer Vision conferences, there seem to be quite a few reports of successful usage of neural nets to extract features and descriptors, and benchmarks indicate that they may be more robust than their classical computer vision equivalent. I suspect that extraction speed is an issue, but assuming one has a decent GPU (e.g. NVidia 1050), is it even feasible to build a real-time SLAM system running say at 30FPS on 640x480 grayscale images with deep-learned features?
This was a bit too long for a comment, so that's why I'm posting it as an answer.
I think it is feasible, but I don't see how this would be useful. Here is why (please correct me if I'm wrong):
In most SLAM pipelines, precision is more important than long-term robustness. You obviously need your feature detections/matchings to be precise to get reliable triangulation/bundle (or whatever equivalent scheme you might use). However, the high level of robustness that neural networks provide is only required with systems that do relocalization/loop closure on long time intervals (e.g. need to do relocalization in different seasons etc). Even in such scenarios, since you already have a GPU, I think it would be better to use a photometric (or even just geometric) model of the scene for localization.
We don't have any reliable noise models for the features that are detected by the neural networks. I know there have been a few interesting works (Gal, Kendall, etc...) for propagating uncertainties in deep networks, but these methods seem a bit immature for deployment ins SLAM systems.
Deep learning methods are usually good for initializing a system, and the solution they provide needs to be refined. Their results depend too much on the training dataset, and tend to be "hit and miss" in practice. So I think that you could trust them to get an initial guess, or some constraints (e.g. like in the case of pose estimation: if you have a geometric algorithm that drifts in time, then you can use the results of a neural network to constrain them. But I think that the absence of a noise model as mentioned previously will make the fusion a bit difficult here...).
So yes, I think that it is feasible and that you can probably, with careful engineering and tuning produce a few interesting demos, but I wouldn't trust it in real life.
I work on palmprint recognition using feature2D with Open_CV library, and I use algorithms such as SIFT, SURF, ORB... to detect features and extract/match descriptors. My test include (1 vs 1) palmprint and also (1 vs Data Base) of palmprint.
Ones I get the result, I need to evaluate the algorithm, and for this I know that there are some rates or scores (like EER, rank-1 identification, recall and accuracy) which gives an estimation about how much this method was successful. Now I need to know if any of those rates are implemented in Open_CV, and how to use them. If they aren't, what are the different formulas used in the literary.
As far as I know there is little implemented in OpenCV. A common way is to store the results (e.g. in JSON) and process those with other programs such as Matlab or Python. This also allows you to change the evaluation without the need to recompute the algorithms.
There is no overall best method to show the results. It always depends on what you want to show. In my opinion ROC is the best way to express your output. It is also very widely used in research.
If you insist on doing it in C++, then you could use:
Roceasy or
DLIB
I'm writing an application that uses an SVM to do classification on some images (specifically these). My Matlab implementation works really well. Using a SIFT bag-of-words approach, I'm able to get near 100% accuracy with a linear kernel.
I need to implement this in C++ for speed/portability reasons, and so I've tried using both libsvm and dlib. I've tried multiple SVM types (c_svm, nu_svm, one_class) and multiple kernels (linear, polynomial, rbf). The best I've been able to achieve is around 50% accuracy - even on the same samples that I've trained on. I've confirmed that my feature generators are working, because when I export my c++-generated features to Matlab and train on those, I'm able to get near-perfect results again.
Is there something magical about Matlab's SVM implementation? Are there any common pitfalls or areas that I might look into that would explain the behavior I'm seeing? I know this is a little vague, but part of the problem is that I don't know where to go. Please let me know in the comments if there is other info I can provide that would be helpful.
There is nothing magical about the Matlab version of the libraries, other that it runs in Matlab which makes it harder to shoot yourself on the foot.
A check list:
Are you normalizing your data, making all values lie between 0 and 1
(or between -1 and 1), either linearly or using the mean and the
standard deviation?
Are you parameter searching for a good value of C (or C and gamma in
the case of an RBF kernel)? Doing cross validation or on a hold out set?
Are you sure that your're handling NaN, and all other floating point
nastiness? Matlab is very good at hiding this from you, C++ not so
much.
Could it be that you're loading your data incorrectly, reading a
"%s" into a double or something that is adding noise to your input
data?
Could it be that libsvm/dlib expects the data in row major order and
your're sending it in in column major (or the other way around)? Again Matlab makes this almost impossible, C++ not so much.
32-64 bit nastiness one version of the library, executable compiled
with the other?
Some other things:
Could it be that in Matlab you're somehow leaking the class (y) into
the preprocessing? no one does this on purpose, but I've seen it happen.
If you make almost any f(y) a feature, you'll get almost 100%
everytime.
Sometimes it helps to verify that everything is numerically
identical by printing to file before training both in C++ and
Matlab.
i'm very happy with libsvm using the rbf kernel. carlosdc pointed out the most common errors in the correct order :-). for libsvm - did you use the python tools shipped with libsvm? if not i recommend to do so. write your feature vectors to a file (from matlab and/or c++) and do a metatraining for the rbf kernel with easy.py. you get the parameters and a prediction for the generated model. if this prediction is ok continue with c++. from training you also get a scaled feature file (min/max transformed to -1.0/1.0 for every feature). compare these to your c++ implementation as well.
some libsvm issues: a nasty habit is (if i remember correctly) that values scaling to 0 (zero) are omitted in the scaled file. in grid.py is a parameter "nr_local_worker" which is defining the mumber of threads. you might wish to increase it.
I am newbie for integer linear programming.
I plan to use a integer linear programming solver to solve my combinatorial optimization problem.
I am more familiar with C++/object oriented programming on an IDE.
Now I am using NetBeans with Cygwin to write my applications most of time.
May I ask if there is an easy use ILP solver for me?
Or it depends on the problem I want to solve ? I am trying to do some resources mapping optimization. Please let me know if any further information is required.
Thank you very much, Cassie.
If what you want is linear mixed integer programming, then I would point to Coin-OR (and specifically to the module CBC). It's Free software (as speech)
You can either use it with a specific language, or use C++.
Use C++ if you data requires lots of preprocessing, or if you want to put your hands into the solver (choosing pivot points, column generation, adding cuts and so on...).
Use the integrated language if you want to use the solver as a black box (you're just interested in the result and the problem is easy or classic enough to be solved without tweaking).
But in the tags you mention genetic algorithms and graphs algorithms. Maybe you should start by better defing your problem...
For graphs I like a lot Boost::Graph
I have used lp_solve ( http://lpsolve.sourceforge.net/5.5/ ) on a couple of occasions with success. It is mature, feature rich and is extremely well documented with lots of good advice if your linear programming skills are rusty. The integer linear programming is not a just an add on but is strongly emphasized with this package.
Just noticed that you say you are a 'newbie' at this. Well, then I strongly recommend this package since the documentation is full of examples and gentle tutorials. Other packages I have tried tend to assume a lot of the user.
For large problems, you might look at AMPL, which is an optimization interpreter with many backend solvers available. It runs as a separate process; C++ would be used to write out the input data.
Then you could try various state-of-the-art solvers.
Look into GLPK. Comes with a few examples, and works with a subset of AMPL, although IMHO works best when you stick to C/C++ for model setup. Copes with pretty big models too.
Linear Programming from Wikipedia covers a few different algorithms that you could do some digging into to see which may work best for you. Does that help or were you wanting something more specific?
i am trying to implement discrete curve evolution algorithm in c++ do any one help me with psudo code or c code or
some simple steps of your understanding
Discrete Curve Evolution is an algorithm to compute an everywhere convex curve from one that is concave. It moves concave sections of the curve outward along their normal in discrete steps until all concavities are eliminated. It is not a genetic algorithm, the term evolution refers to 'evolving' the position of the curve over time.
Having searched on this for quite some time the best source on the internet is here:
https://cis.temple.edu/~latecki/Software/Evo.zip
This is matlab code so it's not quite what you are looking for but you have three good options:
Port it to C++ (usually not to hard with matlab as long as it doesn't use matrix prims.)
Wrap the matlab code so you can call it from C (matlab provides libraries to do this)
Compile it to an executable and call that from C (matlab also allows this)
Option 2 would require anyone that want's to run it to have a copy of the matlab dynamic library on their computer which may be undesirable. I'm guessing option 3 would require this too, but I only have experience with options 1 and 2. Porting matlab to c++ is usually not that bad; it depends on how much the code utilizes matrix primitives and matrix operations which are easy to use in matlab and hard to use in C++ (because they aren't built-in). Still, I'd recommend giving it the old college try!
If you're just looking for DCE, check out the file evolution.m. That's the function that implements DCE. The full skeleton pruning algorithm this comes from can only be described simply at a high level. The individual steps and parts are QUITE complicated and DCE is only a small piece of that.
Hope this helps! I will be working with this code myself so if I do end up using it in C++ in some way that might help you I will let you know.
I'm not exactly sure what you mean by Discrete Curve evolutionary algorithm, but if you mean a Symbolic regression algorithm, you can start by reading about symbolic regression (or genetic programming in general):
http://en.wikipedia.org/wiki/Symbolic_Regression
There's also some nice existing programs. The Eureqa one has an open API:
http://code.google.com/p/eureqa-api/