I am creating my neural network and I am using DFF architecture. I made a game about a car that drives on the road, and I want to train a neural network to drive a car. I want to use Q-learning, I create everything in c++, in fact, I didn't find any information on the Internet about how to set up the weights of a neural network using q-learning, basically they tell the main thing about this method, but how to change the weights is not said anywhere, I tried to implement it myself, but it turned out to be sheer nonsense.
I understand that it is necessary to use the Bellman equation, but I do not know exactly where to use it. I tried to use it to predict the reward, and then compare it with the reward received and, depending on the action and the difference between the expected and received reward, change the weights on the layers (I have 4 layers in the network), but this is of little use.
Related
Blockquote
i am working on this project asssigned by university as final project. But the issue is i am not getting any help from the internet so i thought may be asking here can solve issue. i had read many articles but they had no code or guidance and i am confused what to do. Basically it is an image processing work with machine learning. Data set can be found easily but issue is python python learning algorithm and code
Blockquote
I presume if it's your final project you have to create the program yourself rather than ripping it straight from the internet. If you want a good starting point which you can customise Tensor Flow from Google is very good. You'll want to understand how it works (i.e. how machine learning works) but as a first step there's a good example of image processing on the website in the form of number recognition (which is also the "Hello World" of machine learning).
https://www.tensorflow.org/get_started/mnist/beginners
This also provides a good intro to machine learning with neural nets: https://www.youtube.com/watch?v=uXt8qF2Zzfo
One note on Tensor Flow, you'll probably have to use Python 3.5+ as in my experience it can be difficult getting it on 2.7.
First of all I need to know what type of data are you using because depending on your data, if it is a MRI or PET scan or CT, there could be different suggestion for using machine learning in python for detection.
However, I suppose your main dataset consist of MR images, I am attaching an article which I found it a great overview of different methods>
This project compares four different machine learning algorithms: Decision Tree, Majority, Nearest Neighbors, and Best Z-Score (an algorithm of my own design that is a slight variant of the Na¨ıve Bayes algorithm)
https://users.soe.ucsc.edu/~karplus/abe/Science_Fair_2012_report.pdf
Here, breast cancer and colorectal cancer have been considered and the algorithms that performed best (Best Z-Score and Nearest Neighbors) used all features in classifying a sample. Decision Tree used only 13 features for classifying a sample and gave mediocre results. Majority did not look at any features and did worst. All algorithms except Decision Tree were fast to train and test. Decision Tree was slow, because it had to look at each feature in turn, calculating the information gain of every possible choice of cutpoint.
My Solution:-
Lung Image Database Consortium provides open access dataset for Lung Cancer Images.
Download it then apply any machine learning algorithm to classify images having tumor cells or not.
I attached a link for reference paper. They applied neural network to classify the images.
For coding part, use python "OpenCV" for image pre-processing and segmentation.
When it comes for classification part, use any machine learning libraries (tensorflow, keras, torch, scikit-learn... much more) as you are compatible to work with and perform classification using any better outperforming algorithms as you wish.
That's it..
Link for Reference Journal
I am facing a challenging problem. On the courtyard of company I am working is a camera trap which takes a photo of every movement. On some of these pictures there are different kinds of animals (mostly deep gray mice) that cause damages to our cable system. My idea is to use some application that could recognize if there is a gray mouse on the picture or not. Ideally in realtime. So far we have developed a solution that sends alarms for every movement but most of alarms are false. Could you provide me some info about possible ways how to solve the problem?
In technical parlance, what you describe above is often called event detection. I know of no ready-made approach to solve all of this at once, but with a little bit of programming you should be all set even if you don't want to code any computer vision algorithms or some such.
The high-level pipeline would be:
Making sure that your video is of sufficient quality. Gray mice sound kind of tough, plus the pictures are probably taken at night - so you should have sufficient infrared lighting etc. But if a human can make it out whether an alarm is false or true, you should be fine.
Deploying motion detection and taking snapshot images at the time of movements. It seems like you have this part already worked out, great! Detailing your setup could benefit others. You may also need to crop only the area in motion from the image, are you doing that?
Building an archive of images, including your decision of whether they are false or true alarm (labels in machine learning parlance). Try to gather at least a few tens of example images for both cases, and make them representative of real-world variations (do you have the problem during daytime as well? is there snowfall in your region?).
Classifying the images taken from the video stream snapshot to check whether it's a false alarm or contains bad critters eating cables. This sounds tough, but deep learning and machine learning is making advances by leaps; you can either:
deploy your own neural network built in a framework like caffe or Tensorflow (but you will likely need a lot of examples, at least tens of thousands I'd say)
use an image classification API that recognizes general objects, like Clarifai or Imagga - if you are lucky, it will notice that the snapshots show a mouse or a squirrel (do squirrels chew on cables?), but it is likely that on a specialized task like this one, these engines will get pretty confused!
use a custom image classification API service which is typically even more powerful than rolling your own neural network since it can use a lot of tricks to sort out these images even if you give it just a small number of examples for each image category (false / true alarm here); vize.it is a perfect example of that (anyone can contribute more such services?).
The real-time aspect is a bit open-ended, as the neural networks take some time to process an image — you also need to include data transfer etc. when using a public API, but if you roll out your own, you will need to spend a lot of effort to get low latency as the frameworks are by default optimized for throughput (batch prediction). Generally, if you are happy with ~1s latency and have a good internet uplink, you should be fine with any service.
Disclaimer: I'm one of the co-creators of vize.it.
How about getting a cat?
Also, you could train your own custom classifier using the IBM Watson Visual Recognition service. (demo: https://visual-recognition-demo.mybluemix.net/train ) It's free to try and you just need to supply example images for the different categories you want to identify. Overall, Petr's answer is excellent.
I've been experimenting with neural networks in C++ in implementing a network that plays and learns tic-tac-toe. A problem I have run into, and that I have been wondering about is, how do you keep a network with it's "memory" or learnt skills, intact, once you end the program/training? At the moment it learns as you keep playing, but once I close the program and restart it. It's stupid again. How do I get around this, and how do other large neural networks get around this problem?
The memory of a neural network is stored in the weights of its connections. If you want to prevent it from forgetting what it learnt you need to serialize these weights in a file or a database.
I'm trying to build a two-wheeled balancing robot for fun. I have all of the hardware built and put together, and I think I have it coded as well. I'm using an IMU with gyro and accelerometers to find my tilt angle with a complimentary filter for smoothing the signal. The input signal from the IMU seems pretty smooth, as in less than 0.7 variance + or - the actual tilt angle.
My IMU sampling rate is 50 Hz and I do a PID calculation at 50 Hz too, which I think should be fast enough.
Basically, I'm using the PID library found at PID Library .
When I set the P value to something low then the wheels go in the right direction.
When I set the P value to something large then I get an output like the graph.
From the graph it looks like your system is not stable.
I hope you have tested each subsystem of your robot before directly going for tuning. Which means that both sensors and actuators are responding properly and with acceptable error. Once each subsytem is calibrated properly for external error. You can start tuning.
Once this done is you can start with valid value of P may be (0.5) to first achieve proper response time, you will need to do some trials here, them increment I slowly to cut down steady state error if any and use D only when required(in case of oscillation).
I would suggest to handle P,I and D one by one instead of tweaking all at one time.
Also during the testing you will need to continuously monitor the your sensor and actuator data to see if they are in acceptable range.
As Praks Wrote, your system looks as if it is either unstable or at perhaps marginally stable.
Generally Two wheeled robots can be quite difficult to control as they are inherently unstable without a controller.
I would personally try A PD controller at first, and if you have problems with setpoint accuracy i would use a PID, but just remember that if you want to have a Differential gain in your controller (The D part) it is extremely important that you have a very smooth signal.
Also, the values of the controller greatly depends on your hardware setup (Weight and weight distribution of the robot, motor coefficients and voltage levels) and the units you use internally in your software for the control signals (eg. mV V, degrees/radians). This entails that it will almost be impossible for anybody to guess the correct parameters for you.
What a control engineer could do would be to make a mathematical model of the robot and analyse the pole/zero locations.
If you have any experience with control theory you can take a look at the following paper, and see if it makes sense to you.
http://dspace.mit.edu/bitstream/handle/1721.1/69500/775672333.pdf
There are many heuristic rules to PID tuning out there, but what most people fail to realize is that PID tuning should not be an heuristic process, but should based on math and science.
What #Sigurd V said is correct: "What a control engineer could do would be to make a mathematical model...", and this can get as complicated as you want. But now a days there are many software tools that can help you automate all the math stuff and get you your desired PID gains quite easily.
Assuming all your hardware is in good shape you can use a free online tool like PidTuner to input your data and get near to optimal PID gains. I have personally used it and achieved good results. Use these as an starting point and then tune manually if required.
If you haven't already, I'd suggest you do a search on the terms Arduino PID (obvious suggestion but lots of people have been down this road). I remember when that PID library was being written, the author posted quite a bit with tutorials, etc. (example). Also I came across this PIDAutotuneLibrary.
I wrote my own PID routines but also had a heck of a time tuning and never got it quite right.
I am really passionate about the machine learning,data mining and computer vision fields and I was thinking at taking things a little bit further.
I was thinking at buying a LEGO Mindstorms NXT 2.0 robot for trying to experiment machine learning/computer vision and robotics algorithms in order to try to understand better several existing concepts.
Would you encourage me into doing so? Do you recommend any other alternative for a practical approach in understanding these fields which is acceptably expensive like(nearly 200 - 250 pounds) ? Are there any mini robots which I can buy and experiment stuff with?
If your interests are machine learning, data mining and computer vision then I'd say a Lego mindstorms is not the best option for you. Not unless you are also interested in robotics/electronics.
Do do interesting machine learning you only need a computer and a problem to solve. Think ai-contest or mlcomp or similar.
Do do interesting data mining you need a computer, a lot of data and a question to answer. If you have an internet connection the amount of data you can get at is only limited by your bandwidth. Think netflix prize, try your hand at collecting and interpreting data from wherever. If you are learning, this is a nice place to start.
As for computer vision: All you need is a computer and images. Depending on the type of problem you find interesting you could do some processing of random webcam images, take all you holiday photo's and try to detect where all your travel companions are in them. If you have a webcam your options are endless.
Lego mindstorms allows you to combine machine learning and computer vision. I'm not sure where the datamining would come in, and you will spend (waste?) time on the robotics/electronics side of things, which you don't list as one of your passions.
Well, I would take a look at the irobot create... well within your budget, and very robust.
Depending on your age, you may not want to be seen with a "lego robot" if you are out of college :-)
Anyway, I buy the creates in batches for my lab. You can link to them with a hard cable(cheap) or put a blue tooth interface on it.
But a webcam on that puppy, hook it up to a multicore machine and you have an awesome working robot for the things you want to explore.
Also, the old roombas had a ttl level serial port (if that did not make sense to you , then skip it). I don't know about the new ones. So, it was possible to control any roomba vacuum from a laptop.
The Number One rule, and I cannot emphasize this enough: Have a reliable platform for experimentation. If you hand build something, just for basic functionality, you will spend all your time on minor issues and not get to the fun stuff.
Anyway. best of luck.