I am new to deep learning as well as using Theano and tensorflow. I was able to understand how the tensor flow example code of a convolution neural network to recognize MNIST benchmark works. But since all inbuilt libraries are used I was not able to get a full grip of it.
I would like to write a fast training algorithm for a convolution neural network that recognizes MNIST benchmark but without tensorflow or Theano. I have already coded the network and I can access the output of each layer as well as the weights and bias used in each layer. I need to use these outputs to update the weights and bias of each layer but I’m not sure how.
I would really appreciate if someone could guide me in this regard.
The network that I have used is the network specified in the tensorflow example. It has two convolution pool layers and two fully connected layers.
Thanks
Related
I was trying to change my activation function of my neural net from sigmoid to RELU (or more specifically SELU). Since I got a lot of exploding gradients with that change, I tried to use the batch normalization. I calculated the gradients of my error function w.r.t the learning parameters \beta and \gamma, but it seems that they are a bit different from the ones I saw in several (sadly only Python) examples.
Here, for example, the code example on the bottom of the page says dbeta = np.sum(dout, axis=0) and I wonder what exactly this dout is.
My derivatives look like this:
Derivation of error function w.r.t \beta
What am I doing wrong in this derivation?
Thank you a lot for your help.
I try to add batchnorm2d layer in a small CNN testet on MNIST with Libtorch C++ code with or without GPU use
Here
https://github.com/ollewelin/libtorch-GPU-CNN-test-MNIST-with-Batchnorm
And the precision increase a little then.
Search for
”bn1”
Or
”bn2”
In this code you find.
Installation at Ubuntu with GPU and Libtorch + OpenCV for C++ here:
https://github.com/ollewelin/torchlib-opencv-gpu
It seems that dlib needs a loss layer that dictates how the layers most distant to our input layer are treated. I cannot find any documentation towards the loss layers but it seems that there is no way to have just some summation layer.
Summing up all the values of the last layer would be exactly what I need for the regression, though (see also: https://deeplearning4j.org/linear-regression)
I was thinking along the lines of writing a custom loss layer but could not find information about this, either.
So, have I overseen some corresponding layer here or is there a possibility to have what I need?
The loss layers in dlib are listed in the menu on dlib's machine learning page. Look for the words "loss layers". There is lots of documentation.
The current released version of dlib doesn't include a regression loss. However, if you get the current code from github you can use the new loss_mean_squared layer to do regression. See: https://github.com/davisking/dlib/blob/master/dlib/dnn/loss_abstract.h
I have designed a Convolutional Neural Network in Keras for image classification with several convolution/max-pooling layers, one densely connected hidden layer and softmax activation on the final layer. I want to replace softmax with an SVM or Random Forest in the final layer to see if that yields a better accuracy. Is there any way to do it in Keras?
In order to have (kind of) SVM simply use a hinge loss instead of log loss. Putting RF does not make sense, as you need a differentiable model to be a part of neural net (unless all you want to do is to train a network, and later chop off its final part and use it as a feature detector which is just fed into RF, but this is not a valid approach in general).
After obtaining the image dataset, the feature database is constructed for all images which is a vector based on mean and sd of RGB color model and HSV color model for a portion of the image. How can I use a svm to retieve related images from the database once the query image is given.
Also how to use unsupervised learning for the above problem
Assuming the query images are unlabeled, applying SVM would require a way of knowing the labels for dataset images since SVM is a form of supervised learning, which seeks to correctly determine class labels for unlabeled data. You would need another method for generating class labels, such as unsupervised learning, so this approach does not seem relevant if you only have feature vectors but no class labels.
A neural network allows for unsupervised learning with unlabeled data, but is a rather complex approach and is the subject of academic research. You may want to consider a simpler machine learning approach such as k-Nearest Neighbors, which allows you to obtain the k closest training samples that are similar in your feature space. This algorithm is simple to implement and is found in many machine learning libraries. For example in Python you can use scikit learn.
I am unsure what type of images you are working with, but you might also want to explore using feature detector algorithms such as SIFT rather than just pixel intensities.
I was given a project on vehicle type identification with neural network and that is how I came to know the awesomeness of neural technology.
I am a beginner with this field, but I have sufficient materials to learn it. I just want to know some good places to start for this project specifically, as my biggest problem is that I don't have very much time. I would really appreciate any help. Most importantly, I want to learn how to match patterns with images (in my case, vehicles).
I'd also like to know if python is a good language to start this in, as I'm most comfortable with it.
I am having some images of cars as input and I need to classify those cars by there model number.
Eg: Audi A4,Audi A6,Audi A8,etc
You didn't say whether you can use an existing framework or need to implement the solution from scratch, but either way Python is excellent language for coding neural networks.
If you can use a framework, check out Theano, which is written in Python and is the most complete neural network framework available in any language:
http://www.deeplearning.net/software/theano/
If you need to write your implementation from scratch, look at the book 'Machine Learning, An Algorithmic Perspective' by Stephen Marsland. It contains example Python code for implementing a basic multilayered neural network.
As for how to proceed, you'll want to convert your images into 1-D input vectors. Don't worry about losing the 2-D information, the network will learn 'receptive fields' on its own that extract 2-D features. Normalize the pixel intensities to a -1 to 1 range (or better yet, 0 mean with a standard deviation of 1). If the images are already centered and normalized to roughly the same size than a simple feed-forward network should be sufficient. If the cars vary wildly in angle or distance from the camera, you may need to use a convolutional neural network, but that's much more complex to implement (there are examples in the Theano documentation). For a basic feed-forward network try using two hidden layers and anywhere from 0.5 to 1.5 x the number of pixels in each layer.
Break your dataset into separate training, validation, and testing sets (perhaps with a 0.6, 0.2, 0.2 ratio respectively) and make sure each image only appears in one set. Train ONLY on the training set, and don't use any regularization until you're getting close to 100% of the training instances correct. You can use the validation set to monitor progress on instances that you're not training on. Performance should be worse on the validation set than the training set. Stop training when the performance on the validation set stops improving. Once you've accomplished this you can try different regularization constants and choose the one that results in the best validation set performance. The test set will tell you how well your final result is performing (but don't change anything based on test set results, or you risk overfitting to that too!).
If your car images are very complex and varied and you cannot get a basic feed-forward net to perform well, you might consider using 'deep learning'. That is, add more layers and pre-train them using unsupervised training. There's a detailed tutorial on how to do this here (though all the code examples are in MatLab/Octave):
http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial
Again, that adds a lot of complexity. Try it with a basic feed-forward NN first.