Here's my code snippet:
string trend()
{
double emaTrend0 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,0);
double emaTrend1 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,1);
double emaTrend2 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,2);
double emaTrend3 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,3);
double emaTrend4 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,4);
double emaTrend5 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,5);
double emaTrend6 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,6);
double emaTrend7 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,7);
double emaTrend8 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,8);
double emaTrend9 = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,9);
string signal;
double HighCandle0 = High[0];
double HighCandle1 = High[1];
double HighCandle2 = High[2];
double HighCandle3 = High[3];
double HighCandle4 = High[4];
double HighCandle5 = High[5];
double HighCandle6 = High[6];
double HighCandle7 = High[7];
double HighCandle8 = High[8];
double HighCandle9 = High[9];
double LowCandle0 = Low[0];
double LowCandle1 = Low[1];
double LowCandle2 = Low[2];
double LowCandle3 = Low[3];
double LowCandle4 = Low[4];
double LowCandle5 = Low[5];
double LowCandle6 = Low[6];
double LowCandle7 = Low[7];
double LowCandle8 = Low[8];
double LowCandle9 = Low[9];
if (emaTrend0 > HighCandle0 && emaTrend1 > HighCandle1 && emaTrend2 > HighCandle2 &&emaTrend3 > HighCandle3 &&
emaTrend4 > HighCandle4 &&emaTrend5 > HighCandle5 &&emaTrend6 > HighCandle6 &&emaTrend7 > HighCandle7 &&
emaTrend8 > HighCandle8 && emaTrend9 > HighCandle9 )
signal = "downtrend";
else if (emaTrend0 < LowCandle0 && emaTrend1 < LowCandle1 && emaTrend2 < LowCandle2 &&emaTrend3 < LowCandle3 &&
emaTrend4 < LowCandle4 &&emaTrend5 < LowCandle5 &&emaTrend6 < LowCandle6 &&emaTrend7 < LowCandle7 &&
emaTrend8 < LowCandle8 && emaTrend9 < LowCandle9 )
signal = "uptrend";
return signal;
}
What I wanted to happen is to loop the variable and every condition must be true before producing the signal such as downtrend or uptrend.
The code below is my attempt, but it was still no good. My attempt produced a signal of downtrend or uptrend when one of the conditions is true, but what I need is for everything to be true before producing a signal.
string trend() {
string signal = "";
for (int i = 0; i<=9 ; i++){
double emaTrend = iMA (NULL,0,200,0,MODE_EMA,PRICE_CLOSE,i);
double highCandle = High[i];
double lowCandle = Low[i];
if (emaTrend > highCandle){
signal = "downtrend";
}
else if (emaTrend < lowCandle){
signal = "uptrend";
}
}
return signal;
}
All replies would be deeply appreciated. Thank you!
Here's a possible solution by counting the number of down and up trends and then compare those counts (not tested):
std::string trend() {
const std::size_t max_count = 10;
std::size_t count_downtrend = 0;
std::size_t count_uptrend = 0;
for (std::size_t i = 0; i != max_count; ++i) {
const double emaTrend = iMA (NULL, 0, 200, 0, MODE_EMA, PRICE_CLOSE, i);
const double highCandle = High[i];
const double lowCandle = Low[i];
if (emaTrend > highCandle) {
++count_downtrend;
} else if (emaTrend < lowCandle) {
++count_uptrend;
}
}
std::string signal;
if (count_downtrend == max_count) {
signal = "downtrend";
} else if (count_uptrend == max_count) {
signal = "uptrend";
}
return signal;
}
This is very similar to the regular "linear search loop", with an early break if the trend is broken:
enum Direction
{
Up, Down, None
};
Direction direction(double value, double low, double high)
{
return value < low ? Down : (value > high ? Up : None);
}
double emaClose(int i)
{
return iMA(NULL,0,200,0,MODE_EMA,PRICE_CLOSE,i);
}
string trend()
{
const Direction d = direction(emaClose(0), Low[0], High[0]);
if (d == None)
{
return "";
}
for (int i = 1; i < 10 ; i++)
{
if (direction(emaClose(i), Low[i], High[i]) != d)
{
return "";
}
}
return d == Up ? "uptrend" : "downtrend";
}
This is going to be a long one. I am still very new to coding, started 3 months ago so I know my code is not perfect, any criticism beyond the question is more than welcome. I have specifically avoided using pointers because I do not fully understand them, I can use them but I dont trust that I will use them correctly in a program like this.
First things first, I have a version of this where there is only 1 hidden layer and the net works perfectly. I have started running into problems since I tried to expand the number of hidden layers.
Some info on the net:
-I am using softmax output activation as I have 3 output neurons.
-I am using tanh as my activation function on the rest of the net.
-The file being read for the input has a format of
"input: 0.56 0.76 0.23 0.67"
"output: 0.0 0.0 1.0" (this is the target)
-The weights for connecting layer 1 neuron to layer 2 neuron are stored in layer 1 one neuron.
-The bias's for each neuron are stored in that neuron.
-The target is 1.0 0.0 0.0 if the sum of the input numbers is below one, 0.0 1.0 0.0 if sum is between 1 and 2, 0.0 0.0 1.0 if sum is above 2.
-using L1 regularization.
Those problems specifically being:
The softmax output values do not move from an relatively equalised range ie:
(position 1 and 2 in the target vector have a roughly 50/50 occurance rate while position 3 less than 3% occurance rate. so by relatively equalised I mean the softmax output generally looks something like
"0.56.... 0.48.... 0.02..." even after 500 epochs.
The weights at the hidden layer closer to inputlayer dont change much at all, which is what i think vanishing gradients are. I might be wrong on this. But the weights at hiddenlayer closest to output are ending up at between -50 & 50 (which i think is okay?)
Things I have tried:
I have tried using Relu, parametric Relu, exponential Relu, but with all of these the softmax output value for neuron 3 keeps rising, the other 2 neurons values keep falling. these values continue their trajectory until either 500 epochs have been reached or they just turn into nans. (I think this is to do with the structure of my code rather than the Relu function itself).
If I set the number of hidden layers above 3 while using relu, it immediately spits out nans, within the first epoch.
The backprop function is pretty long, but this is specifically because I have deconstructed it many times over to try and figure out where I might be mismatching values or something. I do have it in a condensed version but I feel I have a higher chance of being completely off the mark there than I do if I have it deconstructed.
I have included the Relu function code that I used, it is the first time I use it so I might be wrong on that aswell but I dont think so, I have double checked multiple times. The Relu in the code is specifically "Elu" or exponential relu.
here is the code for the net:
#include <iostream>
#include <fstream>
#include <cmath>
#include <vector>
#include <sstream>
#include <random>
#include <string>
#include <iomanip>
double randomt(double x, double y)
{
std::random_device rd;
std::mt19937 mt(rd());
std::uniform_real_distribution<double> dist(x, y);
return dist(mt);
}
class InputN
{
public:
double val{};
std::vector <double> weights{};
};
class HiddenN
{
public:
double preactval{};
double actval{};
double actvalPD{};
double preactvalpd{};
std::vector <double> weights{};
double bias{};
};
class OutputN
{
public:
double preactval{};
double actval{};
double preactvalpd{};
double bias{};
};
class Net
{
public:
std::vector <InputN> inneurons{};
std::vector <std::vector <HiddenN>> hiddenneurons{};
std::vector <OutputN> outputneurons{};
double lambda{ 0.015 };
double alpha{ 0.02 };
};
double tanhderiv(double val)
{
return 1 - tanh(val) * tanh(val);
}
double Relu(double val)
{
if (val < 0) return 0.01 *(exp(val) - 1);
else return val;
}
double Reluderiv(double val)
{
if (val < 0) return Relu(val) + 0.01;
else return 1;
}
double regularizer(double weight)
{
double absval{};
if (weight < 0) absval = weight - weight - weight;
else if (weight > 0 || weight == 0) absval = weight;
else;
if (absval > 0) return 1;
else if (absval < 0) return -1;
else if (absval == 0) return 0;
else return 2;
}
void feedforward(Net& net)
{
double sum{};
int prevlayer{};
for (size_t Hsize = 0; Hsize < net.hiddenneurons.size(); Hsize++)
{
//std::cout << "in first loop" << '\n';
prevlayer = Hsize - 1;
for (size_t Hel = 0; Hel < net.hiddenneurons[Hsize].size(); Hel++)
{
//std::cout << "in second loop" << '\n';
if (Hsize == 0)
{
//std::cout << "in first if" << '\n';
for (size_t Isize = 0; Isize < net.inneurons.size(); Isize++)
{
//std::cout << "in fourth loop" << '\n';
sum += (net.inneurons[Isize].val * net.inneurons[Isize].weights[Hel]);
}
net.hiddenneurons[Hsize][Hel].preactval = net.hiddenneurons[Hsize][Hel].bias + sum;
net.hiddenneurons[Hsize][Hel].actval = tanh(sum);
sum = 0;
//std::cout << "first if done" << '\n';
}
else
{
//std::cout << "in else" << '\n';
for (size_t prs = 0; prs < net.hiddenneurons[prevlayer].size(); prs++)
{
//std::cout << "in fourth loop" << '\n';
sum += net.hiddenneurons[prevlayer][prs].actval * net.hiddenneurons[prevlayer][prs].weights[Hel];
}
//std::cout << "fourth loop done" << '\n';
net.hiddenneurons[Hsize][Hel].preactval = net.hiddenneurons[Hsize][Hel].bias + sum;
net.hiddenneurons[Hsize][Hel].actval = tanh(sum);
//std::cout << "else done" << '\n';
sum = 0;
}
}
}
//std::cout << "first loop done " << '\n';
int lasthid = net.hiddenneurons.size() - 1;
for (size_t Osize = 0; Osize < net.outputneurons.size(); Osize++)
{
for (size_t Hsize = 0; Hsize < net.hiddenneurons[lasthid].size(); Hsize++)
{
sum += (net.hiddenneurons[lasthid][Hsize].actval * net.hiddenneurons[lasthid][Hsize].weights[Osize]);
}
net.outputneurons[Osize].preactval = net.outputneurons[Osize].bias + sum;
}
}
void softmax(Net& net)
{
double sum{};
for (size_t Osize = 0; Osize < net.outputneurons.size(); Osize++)
{
sum += exp(net.outputneurons[Osize].preactval);
}
for (size_t Osize = 0; Osize < net.outputneurons.size(); Osize++)
{
net.outputneurons[Osize].actval = exp(net.outputneurons[Osize].preactval) / sum;
}
}
void lossfunc(Net& net, std::vector <double> target)
{
int pos{ -1 };
double val{};
for (size_t t = 0; t < target.size(); t++)
{
pos += 1;
if (target[t] > 0)
{
break;
}
}
for (size_t s = 0; net.outputneurons.size(); s++)
{
val = -log(net.outputneurons[pos].actval);
}
}
void backprop(Net& net, std::vector<double>& target)
{
for (size_t outI = 0; outI < net.outputneurons.size(); outI++)
{
double PD = target[outI] - net.outputneurons[outI].actval;
net.outputneurons[outI].preactvalpd = PD * -1;
}
size_t lasthid = net.hiddenneurons.size() - 1;
for (size_t LH = 0; LH < net.hiddenneurons[lasthid].size(); LH++)
{
for (size_t LHW = 0; LHW < net.hiddenneurons[lasthid][LH].weights.size(); LHW++)
{
double weight = net.hiddenneurons[lasthid][LH].weights[LHW];
double PD = net.outputneurons[LHW].preactvalpd * net.hiddenneurons[lasthid][LH].actval;
PD = PD * -1;
double delta = PD - (net.lambda * regularizer(weight));
weight = weight + (net.alpha * delta);
net.hiddenneurons[lasthid][LH].weights[LHW] = weight;
}
}
for (size_t OB = 0; OB < net.outputneurons.size(); OB++)
{
double bias = net.outputneurons[OB].bias;
double BPD = net.outputneurons[OB].preactvalpd;
BPD = BPD * -1;
double Delta = BPD;
bias = bias + (net.alpha * Delta);
}
for (size_t HPD = 0; HPD < net.hiddenneurons[lasthid].size(); HPD++)
{
double PD{};
for (size_t HW = 0; HW < net.outputneurons.size(); HW++)
{
PD += net.hiddenneurons[lasthid][HPD].weights[HW] * net.outputneurons[HW].preactvalpd;
}
net.hiddenneurons[lasthid][HPD].actvalPD = PD;
PD = 0;
}
for (size_t HPD = 0; HPD < net.hiddenneurons[lasthid].size(); HPD++)
{
net.hiddenneurons[lasthid][HPD].preactvalpd = net.hiddenneurons[lasthid][HPD].actvalPD * tanhderiv(net.hiddenneurons[lasthid][HPD].preactval);
}
for (size_t AllHid = net.hiddenneurons.size() - 2; AllHid > -1; AllHid--)
{
size_t uplayer = AllHid + 1;
for (size_t cl = 0; cl < net.hiddenneurons[AllHid].size(); cl++)
{
for (size_t clw = 0; clw < net.hiddenneurons[AllHid][cl].weights.size(); clw++)
{
double weight = net.hiddenneurons[AllHid][cl].weights[clw];
double PD = net.hiddenneurons[uplayer][clw].preactvalpd * net.hiddenneurons[AllHid][cl].actval;
PD = PD * -1;
double delta = PD - (net.lambda * regularizer(weight));
weight = weight + (net.alpha * delta);
net.hiddenneurons[AllHid][cl].weights[clw] = weight;
}
}
for (size_t up = 0; up < net.hiddenneurons[uplayer].size(); up++)
{
double bias = net.hiddenneurons[uplayer][up].bias;
double PD = net.hiddenneurons[uplayer][up].preactvalpd;
PD = PD * -1;
double delta = PD;
bias = bias + (net.alpha * delta);
}
for (size_t APD = 0; APD < net.hiddenneurons[AllHid].size(); APD++)
{
double PD{};
for (size_t APDW = 0; APDW < net.hiddenneurons[AllHid][APD].weights.size(); APDW++)
{
PD += net.hiddenneurons[AllHid][APD].weights[APDW] * net.hiddenneurons[uplayer][APDW].preactvalpd;
}
net.hiddenneurons[AllHid][APD].actvalPD = PD;
PD = 0;
}
for (size_t PPD = 0; PPD < net.hiddenneurons[AllHid].size(); PPD++)
{
double PD = net.hiddenneurons[AllHid][PPD].actvalPD * tanhderiv(net.hiddenneurons[AllHid][PPD].preactval);
net.hiddenneurons[AllHid][PPD].preactvalpd = PD;
}
}
for (size_t IN = 0; IN < net.inneurons.size(); IN++)
{
for (size_t INW = 0; INW < net.inneurons[IN].weights.size(); INW++)
{
double weight = net.inneurons[IN].weights[INW];
double PD = net.hiddenneurons[0][INW].preactvalpd * net.inneurons[IN].val;
PD = PD * -1;
double delta = PD - (net.lambda * regularizer(weight));
weight = weight + (net.alpha * delta);
net.inneurons[IN].weights[INW] = weight;
}
}
for (size_t hidB = 0; hidB < net.hiddenneurons[0].size(); hidB++)
{
double bias = net.hiddenneurons[0][hidB].bias;
double PD = net.hiddenneurons[0][hidB].preactvalpd;
PD = PD * -1;
double delta = PD;
bias = bias + (net.alpha * delta);
net.hiddenneurons[0][hidB].bias = bias;
}
}
int main()
{
std::vector <double> invals{ };
std::vector <double> target{ };
Net net;
InputN Ineuron;
HiddenN Hneuron;
OutputN Oneuron;
int IN = 4;
int HIDLAYERS = 4;
int HID = 8;
int OUT = 3;
for (int i = 0; i < IN; i++)
{
net.inneurons.push_back(Ineuron);
for (int m = 0; m < HID; m++)
{
net.inneurons.back().weights.push_back(randomt(0.0, 0.5));
}
}
//std::cout << "first loop done" << '\n';
for (int s = 0; s < HIDLAYERS; s++)
{
net.hiddenneurons.push_back(std::vector <HiddenN>());
if (s == HIDLAYERS - 1)
{
for (int i = 0; i < HID; i++)
{
net.hiddenneurons[s].push_back(Hneuron);
for (int m = 0; m < OUT; m++)
{
net.hiddenneurons[s].back().weights.push_back(randomt(0.0, 0.5));
}
net.hiddenneurons[s].back().bias = 1.0;
}
}
else
{
for (int i = 0; i < HID; i++)
{
net.hiddenneurons[s].push_back(Hneuron);
for (int m = 0; m < HID; m++)
{
net.hiddenneurons[s].back().weights.push_back(randomt(0.0, 0.5));
}
net.hiddenneurons[s].back().bias = 1.0;
}
}
}
//std::cout << "second loop done" << '\n';
for (int i = 0; i < OUT; i++)
{
net.outputneurons.push_back(Oneuron);
net.outputneurons.back().bias = randomt(0.0, 0.5);
}
//std::cout << "third loop done" << '\n';
int count{};
std::ifstream fileread("N.txt");
for (int epoch = 0; epoch < 500; epoch++)
{
count = 0;
if (epoch == 100 || epoch == 100 * 2 || epoch == 100 * 3 || epoch == 100 * 4 || epoch == 499)
{
printvals("no", net);
}
fileread.clear(); fileread.seekg(0, std::ios::beg);
while (fileread.is_open())
{
std::cout << '\n' << "epoch: " << epoch << '\n';
std::string fileline{};
fileread >> fileline;
if (fileline == "in:")
{
std::string input{};
double nums{};
std::getline(fileread, input);
std::stringstream ss(input);
while (ss >> nums)
{
invals.push_back(nums);
}
}
if (fileline == "out:")
{
std::string output{};
double num{};
std::getline(fileread, output);
std::stringstream ss(output);
while (ss >> num)
{
target.push_back(num);
}
}
count += 1;
if (count == 2)
{
for (size_t inv = 0; inv < invals.size(); inv++)
{
net.inneurons[inv].val = invals[inv];
}
//std::cout << "calling feedforward" << '\n';
feedforward(net);
//std::cout << "ff done" << '\n';
softmax(net);
printvals("output", net);
std::cout << "target: " << '\n';
for (auto element : target) std::cout << element << " / ";
std::cout << '\n';
backprop(net, target);
invals.clear();
target.clear();
count = 0;
}
if (fileread.eof()) break;
}
}
//std::cout << "fourth loop done" << '\n';
return 1;
}
Much aprecciated to anyone who actually made it through all that! :)
I'm trying to implement logistic regression in C++, but the predictions I'm getting are not even close to what I am expecting. I'm not sure if there is an error in my understanding of logistic regression or the code.
I have reviewed the algorithms and messed with the learning rate, but the results are very inconsistent.
double theta[4] = {0,0,0,0};
double x[2][3] = {
{1,1,1},
{9,9,9},
};
double y[2] = {0,1};
//prediction data
double test_x[1][3] = {
{9,9,9},
};
int test_m = sizeof(test_x) / sizeof(test_x[0]);
int m = sizeof(x) / sizeof(x[0]);
int n = sizeof(theta) / sizeof(theta[0]);
int xn = n - 1;
struct Logistic
{
double sigmoid(double total)
{
double e = 2.71828;
double sigmoid_x = 1 / (1 + pow(e, -total));
return sigmoid_x;
}
double h(int x_row)
{
double total = theta[0] * 1;
for(int c1 = 0; c1 < xn; ++c1)
{
total += theta[c1 + 1] * x[x_row][c1];
}
double final_total = sigmoid(total);
//cout << "final total: " << final_total;
return final_total;
}
double cost()
{
double hyp;
double temp_y;
double error;
for(int c1 = 0; c1 < m; ++c1)
{
//passes row of x to h to calculate sigmoid(xi * thetai)
hyp = h(c1);
temp_y = y[c1];
error += temp_y * log(hyp) + (1 - temp_y) * log(1 - hyp);
}// 1 / m
double final_error = -.5 * error;
return final_error;
}
void gradient_descent()
{
double alpha = .01;
for(int c1 = 0; c1 < n; ++c1)
{
double error = cost();
cout << "final error: " << error << "\n";
theta[c1] = theta[c1] - alpha * error;
cout << "theta: " << c1 << " " << theta[c1] << "\n";
}
}
void train()
{
for(int epoch = 0; epoch <= 10; ++epoch)
{
gradient_descent();
cout << "epoch: " << epoch << "\n";
}
}
vector<double> predict()
{
double temp_total;
double total;
vector<double> final_total;
//hypothesis equivalent function
temp_total = theta[0] * 1;
for(int c1 = 0; c1 < test_m; ++c1)
{
for(int c2 = 0; c2 < xn; ++c2)
{
temp_total += theta[c2 + 1] * test_x[c1][c2];
}
total = sigmoid(temp_total);
//cout << "final total: " << final_total;
final_total.push_back(total);
}
return final_total;
}
};
int main()
{
Logistic test;
test.train();
vector<double> prediction = test.predict();
for(int c1 = 0; c1 < test_m; ++c1)
{
cout << "prediction: " << prediction[c1] << "\n";
}
}
start with a very small learning rate wither larger iteration number at try. Haven`t tested ur code. But I guess the cost/error/energy jumps from hump to hump.
Somewhat unrelated to your question, but rather than computing e^-total using pow, use exp instead (it's a hell of a lot faster!). Also there is no need to make the sigmoid function a member func, make it static or just a normal C func (it doesn't require any member variable from your struct).
static double sigmoid(double total)
{
return 1.0 / (1.0 + exp(-total));
}
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Firstly, I'm a complete amateur so I may mix up some terminology.
I've been working on a Neural Network to play Connect 4 / Four In A Row.
The current design of the network model is 170 input values, 417 hidden neurons and 1 output neuron. The network is fully connected, i.e. every input is connected to every hidden neuron and every hidden neuron is connected to the output node.
Every connection has an independent weight, and every hidden node, and the single output node, have an additional bias node with a weight.
The input representation of 170 values for the game-state of Connect 4 is:
42 pairs of values (84 input variables) which denote whether a space is occupied by player 1, player 2 or is vacant.
0,0 means it's free
1,0 means it's player 1's position
0,1 means it's player 2's position
1,1 is not possible
Another 42 pairs of values (84 input variables) which denote whether adding a piece here will give player 1 or player 2 a "Connect 4"/"Four In a Row". The combination of values means the same as above.
2 final input variables to denote who's turn it is:
1,0 player 1's turn
0,1 player 2's turn
1,1 and 0,0 are not possible
I measured the average Mean Square Error of 100 games over 10,000 total games of various configurations to arrive at:
417 hidden neurons
Alpha and Beta learning rate of 0.1 at the start and dropping to 0.01 linearly across the total number of epochs
A lambda value of 0.5
90 out of 100 moves are random at the start and drop down to 10 out of every 100 after the first 50% of epochs. So at the midway point 10 out of 100 moves are random
The first 50% of epochs start with a random move
Sigmoid Activation Function used in every node
This image shows the results of the various configurations plotted with a logarithmic scale. This is how I determined which configuration to use.
I calculate this Mean Square Error by comparing the output of a board in a win-state against either -1 for a player 2 win and 1 for a player 1 win. I add these up every 100 games and divide the total by 100 to get 1000 values to plot in the above graph. I.e. the code snippet is:
if(board.InARowConnected(4) == Board<7,6,4>::Player1)
{
totalLoss += NN->BackPropagateFinal({1},previousNN,alpha,beta,lambda);
winState = true;
}
else if(board.InARowConnected(4) == Board<7,6,4>::Player2)
{
totalLoss += NN->BackPropagateFinal({-1},previousNN,alpha,beta,lambda);
winState = true;
}
else if(!board.IsThereAvailableMove())
{
totalLoss += NN->BackPropagateFinal({0},previousNN,alpha,beta,lambda);
winState = true;
}
...
if(gameNumber % 100 == 0 && gameNumber != 0)
{
totalLoss = totalLoss / gamesToOutput;
matchFile << std::fixed << std::setprecision(51) << totalLoss << std::endl;
totalLoss = 0.0;
}
The way I'm training the network is by having it play against itself over and over again. It's a feed-forward network and I'm using TD-Lambda to train it for every move (every move that wasn't randomly chosen).
The Board State that is given to the Neural Network is done through this:
template<std::size_t BoardWidth, std::size_t BoardHeight, std::size_t InARow>
void create_board_state(std::array<double,BoardWidth*BoardHeight*4+2>& gameState, const Board<BoardWidth,BoardHeight,InARow>& board,
const typename Board<BoardWidth,BoardHeight,InARow>::Player player)
{
using BoardType = Board<BoardWidth,BoardHeight,InARow>;
auto bb = board.GetBoard();
std::size_t stateIndex = 0;
for(std::size_t boardIndex = 0; boardIndex < BoardWidth*BoardHeight; ++boardIndex, stateIndex += 2)
{
if(bb[boardIndex] == BoardType::Free)
{
gameState[stateIndex] = 0;
gameState[stateIndex+1] = 0;
}
else if(bb[boardIndex] == BoardType::Player1)
{
gameState[stateIndex] = 1;
gameState[stateIndex+1] = 0;
}
else
{
gameState[stateIndex] = 0;
gameState[stateIndex+1] = 1;
}
}
for(std::size_t x = 0; x < BoardWidth; ++x)
{
for(std::size_t y = 0; y < BoardHeight; ++y)
{
auto testBoard1 = board;
auto testBoard2 = board;
testBoard1.SetBoardChecker(x,y,Board<BoardWidth,BoardHeight,InARow>::Player1);
testBoard2.SetBoardChecker(x,y,Board<BoardWidth,BoardHeight,InARow>::Player2);
// player 1's set
if(testBoard1.InARowConnected(4) == Board<7,6,4>::Player1)
gameState[stateIndex] = 1;
else
gameState[stateIndex] = 0;
// player 2's set
if(testBoard2.InARowConnected(4) == Board<7,6,4>::Player2)
gameState[stateIndex+1] = 1;
else
gameState[stateIndex+1] = 0;
stateIndex += 2;
}
}
if(player == Board<BoardWidth,BoardHeight,InARow>::Player1)
{
gameState[stateIndex] = 1;
gameState[stateIndex+1] = 0;
}
else
{
gameState[stateIndex] = 0;
gameState[stateIndex+1] = 1;
}
}
It's templated to make changing things later easier. I don't believe there's anything wrong in the above.
My Sigmoid activation function:
inline double sigmoid(const double x)
{
// return 1.0 / (1.0 + std::exp(-x));
return x / (1.0 + std::abs(x));
}
My Neuron Class
template<std::size_t NumInputs>
class Neuron
{
public:
Neuron()
{
for(auto& i : m_inputValues)
i = 9;
for(auto& e : m_eligibilityTraces)
e = 9;
for(auto& w : m_weights)
w = 9;
m_biasWeight = 9;
m_biasEligibilityTrace = 9;
m_outputValue = 9;
}
void SetInputValue(const std::size_t index, const double value)
{
m_inputValues[index] = value;
}
void SetWeight(const std::size_t index, const double weight)
{
if(std::isnan(weight))
throw std::runtime_error("Shit! this is a nan bread");
m_weights[index] = weight;
}
void SetBiasWeight(const double weight)
{
m_biasWeight = weight;
}
double GetInputValue(const std::size_t index) const
{
return m_inputValues[index];
}
double GetWeight(const std::size_t index) const
{
return m_weights[index];
}
double GetBiasWeight() const
{
return m_biasWeight;
}
double CalculateOutput()
{
m_outputValue = 0;
for(std::size_t i = 0; i < NumInputs; ++i)
{
m_outputValue += m_inputValues[i] * m_weights[i];
}
m_outputValue += 1.0 * m_biasWeight;
m_outputValue = sigmoid(m_outputValue);
return m_outputValue;
}
double GetOutput() const
{
return m_outputValue;
}
double GetEligibilityTrace(const std::size_t index) const
{
return m_eligibilityTraces[index];
}
void SetEligibilityTrace(const std::size_t index, const double eligibility)
{
m_eligibilityTraces[index] = eligibility;
}
void SetBiasEligibility(const double eligibility)
{
m_biasEligibilityTrace = eligibility;
}
double GetBiasEligibility() const
{
return m_biasEligibilityTrace;
}
void ResetEligibilityTraces()
{
for(auto& e : m_eligibilityTraces)
e = 0;
m_biasEligibilityTrace = 0;
}
private:
std::array<double,NumInputs> m_inputValues;
std::array<double,NumInputs> m_weights;
std::array<double,NumInputs> m_eligibilityTraces;
double m_biasWeight;
double m_biasEligibilityTrace;
double m_outputValue;
};
My Neural Network class
template
class NeuralNetwork
{
public:
void RandomiseWeights()
{
double inputToHiddenRange = 4.0 * std::sqrt(6.0 / (NumInputs+1+NumOutputs));
RandomGenerator inputToHidden(-inputToHiddenRange,inputToHiddenRange);
double hiddenToOutputRange = 4.0 * std::sqrt(6.0 / (NumHidden+1+1));
RandomGenerator hiddenToOutput(-hiddenToOutputRange,hiddenToOutputRange);
for(auto& hiddenNeuron : m_hiddenNeurons)
{
for(std::size_t i = 0; i < NumInputs; ++i)
hiddenNeuron.SetWeight(i, inputToHidden());
hiddenNeuron.SetBiasWeight(inputToHidden());
}
for(auto& outputNeuron : m_outputNeurons)
{
for(std::size_t h = 0; h < NumHidden; ++h)
outputNeuron.SetWeight(h, hiddenToOutput());
outputNeuron.SetBiasWeight(hiddenToOutput());
}
}
double GetOutput(const std::size_t index) const
{
return m_outputNeurons[index].GetOutput();
}
std::array<double,NumOutputs> GetOutputs()
{
std::array<double, NumOutputs> returnValue;
for(std::size_t o = 0; o < NumOutputs; ++o)
returnValue[o] = m_outputNeurons[o].GetOutput();
return returnValue;
}
void SetInputValue(const std::size_t index, const double value)
{
for(auto& hiddenNeuron : m_hiddenNeurons)
hiddenNeuron.SetInputValue(index, value);
}
std::array<double,NumOutputs> Calculate()
{
for(auto& h : m_hiddenNeurons)
h.CalculateOutput();
for(auto& o : m_outputNeurons)
o.CalculateOutput();
return GetOutputs();
}
std::array<double,NumOutputs> FeedForward(const std::array<double,NumInputs>& inputValues)
{
for(std::size_t h = 0; h < NumHidden; ++h)//auto& hiddenNeuron : m_hiddenNeurons)
{
for(std::size_t i = 0; i < NumInputs; ++i)
m_hiddenNeurons[h].SetInputValue(i,inputValues[i]);
m_hiddenNeurons[h].CalculateOutput();
}
std::array<double, NumOutputs> returnValue;
for(std::size_t h = 0; h < NumHidden; ++h)
{
auto hiddenOutput = m_hiddenNeurons[h].GetOutput();
for(std::size_t o = 0; o < NumOutputs; ++o)
m_outputNeurons[o].SetInputValue(h, hiddenOutput);
}
for(std::size_t o = 0; o < NumOutputs; ++o)
{
returnValue[o] = m_outputNeurons[o].CalculateOutput();
}
return returnValue;
}
double BackPropagateFinal(const std::array<double,NumOutputs>& actualValues, const NeuralNetwork<NumInputs,NumHidden,NumOutputs>* NN, const double alpha, const double beta, const double lambda)
{
for(std::size_t iO = 0; iO < NumOutputs; ++iO)
{
auto y = NN->m_outputNeurons[iO].GetOutput();
auto y1 = actualValues[iO];
for(std::size_t iH = 0; iH < NumHidden; ++iH)
{
auto e = NN->m_outputNeurons[iO].GetEligibilityTrace(iH);
auto h = NN->m_hiddenNeurons[iH].GetOutput();
auto w = NN->m_outputNeurons[iO].GetWeight(iH);
double e1 = lambda * e + (y * (1.0 - y) * h);
double w1 = w + beta * (y1 - y) * e1;
m_outputNeurons[iO].SetEligibilityTrace(iH,e1);
m_outputNeurons[iO].SetWeight(iH,w1);
}
auto e = NN->m_outputNeurons[iO].GetBiasEligibility();
auto h = 1.0;
auto w = NN->m_outputNeurons[iO].GetBiasWeight();
double e1 = lambda * e + (y * (1.0 - y) * h);
double w1 = w + beta * (y1 - y) * e1;
m_outputNeurons[iO].SetBiasEligibility(e1);
m_outputNeurons[iO].SetBiasWeight(w1);
}
for(std::size_t iH = 0; iH < NumHidden; ++iH)
{
auto h = NN->m_hiddenNeurons[iH].GetOutput();
for(std::size_t iI = 0; iI < NumInputs; ++iI)
{
auto e = NN->m_hiddenNeurons[iH].GetEligibilityTrace(iI);
auto x = NN->m_hiddenNeurons[iH].GetInputValue(iI);
auto u = NN->m_hiddenNeurons[iH].GetWeight(iI);
double sumError = 0;
for(std::size_t iO = 0; iO < NumOutputs; ++iO)
{
auto w = NN->m_outputNeurons[iO].GetWeight(iH);
auto y = NN->m_outputNeurons[iO].GetOutput();
auto y1 = actualValues[iO];
auto grad = y1 - y;
double e1 = lambda * e + (y * (1.0 - y) * w * h * (1.0 - h) * x);
sumError += grad * e1;
}
double u1 = u + alpha * sumError;
m_hiddenNeurons[iH].SetEligibilityTrace(iI,sumError);
m_hiddenNeurons[iH].SetWeight(iI,u1);
}
auto e = NN->m_hiddenNeurons[iH].GetBiasEligibility();
auto x = 1.0;
auto u = NN->m_hiddenNeurons[iH].GetBiasWeight();
double sumError = 0;
for(std::size_t iO = 0; iO < NumOutputs; ++iO)
{
auto w = NN->m_outputNeurons[iO].GetWeight(iH);
auto y = NN->m_outputNeurons[iO].GetOutput();
auto y1 = actualValues[iO];
auto grad = y1 - y;
double e1 = lambda * e + (y * (1.0 - y) * w * h * (1.0 - h) * x);
sumError += grad * e1;
}
double u1 = u + alpha * sumError;
m_hiddenNeurons[iH].SetBiasEligibility(sumError);
m_hiddenNeurons[iH].SetBiasWeight(u1);
}
double retVal = 0;
for(std::size_t o = 0; o < NumOutputs; ++o)
{
retVal += 0.5 * alpha * std::pow((NN->GetOutput(o) - GetOutput(0)),2);
}
return retVal / NumOutputs;
}
double BackPropagate(const NeuralNetwork<NumInputs,NumHidden,NumOutputs>* NN, const double alpha, const double beta, const double lambda)
{
for(std::size_t iO = 0; iO < NumOutputs; ++iO)
{
auto y = NN->m_outputNeurons[iO].GetOutput();
auto y1 = m_outputNeurons[iO].GetOutput();
for(std::size_t iH = 0; iH < NumHidden; ++iH)
{
auto e = NN->m_outputNeurons[iO].GetEligibilityTrace(iH);
auto h = NN->m_hiddenNeurons[iH].GetOutput();
auto w = NN->m_outputNeurons[iO].GetWeight(iH);
double e1 = lambda * e + (y * (1.0 - y) * h);
double w1 = w + beta * (y1 - y) * e1;
m_outputNeurons[iO].SetEligibilityTrace(iH,e1);
m_outputNeurons[iO].SetWeight(iH,w1);
}
auto e = NN->m_outputNeurons[iO].GetBiasEligibility();
auto h = 1.0;
auto w = NN->m_outputNeurons[iO].GetBiasWeight();
double e1 = lambda * e + (y * (1.0 - y) * h);
double w1 = w + beta * (y1 - y) * e1;
m_outputNeurons[iO].SetBiasEligibility(e1);
m_outputNeurons[iO].SetBiasWeight(w1);
}
for(std::size_t iH = 0; iH < NumHidden; ++iH)
{
auto h = NN->m_hiddenNeurons[iH].GetOutput();
for(std::size_t iI = 0; iI < NumInputs; ++iI)
{
auto e = NN->m_hiddenNeurons[iH].GetEligibilityTrace(iI);
auto x = NN->m_hiddenNeurons[iH].GetInputValue(iI);
auto u = NN->m_hiddenNeurons[iH].GetWeight(iI);
double sumError = 0;
for(std::size_t iO = 0; iO < NumOutputs; ++iO)
{
auto w = NN->m_outputNeurons[iO].GetWeight(iH);
auto y = NN->m_outputNeurons[iO].GetOutput();
auto y1 = m_outputNeurons[iO].GetOutput();
auto grad = y1 - y;
double e1 = lambda * e + (y * (1.0 - y) * w * h * (1.0 - h) * x);
sumError += grad * e1;
}
double u1 = u + alpha * sumError;
m_hiddenNeurons[iH].SetEligibilityTrace(iI,sumError);
m_hiddenNeurons[iH].SetWeight(iI,u1);
}
auto e = NN->m_hiddenNeurons[iH].GetBiasEligibility();
auto x = 1.0;
auto u = NN->m_hiddenNeurons[iH].GetBiasWeight();
double sumError = 0;
for(std::size_t iO = 0; iO < NumOutputs; ++iO)
{
auto w = NN->m_outputNeurons[iO].GetWeight(iH);
auto y = NN->m_outputNeurons[iO].GetOutput();
auto y1 = m_outputNeurons[iO].GetOutput();
auto grad = y1 - y;
double e1 = lambda * e + (y * (1.0 - y) * w * h * (1.0 - h) * x);
sumError += grad * e1;
}
double u1 = u + alpha * sumError;
m_hiddenNeurons[iH].SetBiasEligibility(sumError);
m_hiddenNeurons[iH].SetBiasWeight(u1);
}
double retVal = 0;
for(std::size_t o = 0; o < NumOutputs; ++o)
{
retVal += 0.5 * alpha * std::pow((NN->GetOutput(o) - GetOutput(0)),2);
}
return retVal / NumOutputs;
}
std::array<double,NumInputs*NumHidden+NumHidden+NumHidden*NumOutputs+NumOutputs> GetNetworkWeights() const
{
std::array<double,NumInputs*NumHidden+NumHidden+NumHidden*NumOutputs+NumOutputs> returnVal;
std::size_t weightPos = 0;
for(std::size_t h = 0; h < NumHidden; ++h)
{
for(std::size_t i = 0; i < NumInputs; ++i)
returnVal[weightPos++] = m_hiddenNeurons[h].GetWeight(i);
returnVal[weightPos++] = m_hiddenNeurons[h].GetBiasWeight();
}
for(std::size_t o = 0; o < NumOutputs; ++o)
{
for(std::size_t h = 0; h < NumHidden; ++h)
returnVal[weightPos++] = m_outputNeurons[o].GetWeight(h);
returnVal[weightPos++] = m_outputNeurons[o].GetBiasWeight();
}
return returnVal;
}
static constexpr std::size_t NumWeights = NumInputs*NumHidden+NumHidden+NumHidden*NumOutputs+NumOutputs;
void SetNetworkWeights(const std::array<double,NumInputs*NumHidden+NumHidden+NumHidden*NumOutputs+NumOutputs>& weights)
{
std::size_t weightPos = 0;
for(std::size_t h = 0; h < NumHidden; ++h)
{
for(std::size_t i = 0; i < NumInputs; ++i)
m_hiddenNeurons[h].SetWeight(i, weights[weightPos++]);
m_hiddenNeurons[h].SetBiasWeight(weights[weightPos++]);
}
for(std::size_t o = 0; o < NumOutputs; ++o)
{
for(std::size_t h = 0; h < NumHidden; ++h)
m_outputNeurons[o].SetWeight(h, weights[weightPos++]);
m_outputNeurons[o].SetBiasWeight(weights[weightPos++]);
}
}
void ResetEligibilityTraces()
{
for(auto& h : m_hiddenNeurons)
h.ResetEligibilityTraces();
for(auto& o : m_outputNeurons)
o.ResetEligibilityTraces();
}
private:
std::array<Neuron<NumInputs>,NumHidden> m_hiddenNeurons;
std::array<Neuron<NumHidden>,NumOutputs> m_outputNeurons;
};
I believe one of the places I may have an issue is the BackPropagate and BackPropagateFinal methods in the Neural Network class.
Here's my main function that is training the network:
int main()
{
std::ofstream matchFile("match.txt");
RandomGenerator randomPlayerStart(0,1);
RandomGenerator randomMove(0,100);
Board<7,6,4> board;
auto NN = new NeuralNetwork<7*6*4+2,417,1>();
auto previousNN = new NeuralNetwork<7*6*4+2,417,1>();
NN->RandomiseWeights();
const int numGames = 3000000;
double alpha = 0.1;
double beta = 0.1;
double lambda = 0.5;
double learningRateFloor = 0.01;
double decayRateAlpha = (alpha - learningRateFloor) / numGames;
double decayRateBeta = (beta - learningRateFloor) / numGames;
double randomChance = 90; // out of 100
double randomChangeFloor = 10;
double percentToReduceRandomOver = 0.5;
double randomChangeDecay = (randomChance-randomChangeFloor) / (numGames*percentToReduceRandomOver);
double percentOfGamesToRandomiseStart = 0.5;
int numGamesWonP1 = 0;
int numGamesWonP2 = 0;
int gamesToOutput = 100;
matchFile << "Num Games: " << numGames << "\t\ta,b,l: " << alpha << ", " << beta << ", " << lambda << std::endl;
Board<7,6,4>::Player playerStart = randomPlayerStart() > 0.5 ? Board<7,6,4>::Player1 : Board<7,6,4>::Player2;
double totalLoss = 0.0;
for(int gameNumber = 0; gameNumber < numGames; ++gameNumber)
{
bool winState = false;
Board<7,6,4>::Player playerWhoTurnItIs = playerStart;
playerStart = playerStart == Board<7,6,4>::Player1 ? Board<7,6,4>::Player2 : Board<7,6,4>::Player1;
board.ClearBoard();
int turnNumber = 0;
while(!winState)
{
Board<7,6,4>::Player playerWhoTurnItIsNot = playerWhoTurnItIs == Board<7,6,4>::Player1 ? Board<7,6,4>::Player2 : Board<7,6,4>::Player1;
bool wasRandomMove = false;
std::size_t selectedMove;
bool moveFound = false;
if(board.IsThereAvailableMove())
{
std::vector<std::size_t> availableMoves;
if((gameNumber <= numGames * percentOfGamesToRandomiseStart && turnNumber == 0) || randomMove() > 100.0-randomChance)
wasRandomMove = true;
std::size_t bestMove = 8;
double bestWorstResponse = playerWhoTurnItIs == Board<7,6,4>::Player1 ? std::numeric_limits<double>::min() : std::numeric_limits<double>::max();
for(std::size_t m = 0; m < 7; ++m)
{
Board<7,6,4> testBoard = board; // make a copy of the current board to run our tests
if(testBoard.AvailableMoveInColumn(m))
{
if(wasRandomMove)
{
availableMoves.push_back(m);
}
testBoard.AddChecker(m, playerWhoTurnItIs);
double worstResponse = playerWhoTurnItIs == Board<7,6,4>::Player1 ? std::numeric_limits<double>::max() : std::numeric_limits<double>::min();
std::size_t worstMove = 8;
for(std::size_t m2 = 0; m2 < 7; ++m2)
{
Board<7,6,4> testBoard2 = testBoard;
if(testBoard2.AvailableMoveInColumn(m2))
{
testBoard2.AddChecker(m,playerWhoTurnItIsNot);
StateType state;
create_board_state(state, testBoard2, playerWhoTurnItIs);
auto outputs = NN->FeedForward(state);
if(playerWhoTurnItIs == Board<7,6,4>::Player1 && (outputs[0] < worstResponse || worstMove == 8))
{
worstResponse = outputs[0];
worstMove = m2;
}
else if(playerWhoTurnItIs == Board<7,6,4>::Player2 && (outputs[0] > worstResponse || worstMove == 8))
{
worstResponse = outputs[0];
worstMove = m2;
}
}
}
if(playerWhoTurnItIs == Board<7,6,4>::Player1 && (worstResponse > bestWorstResponse || bestMove == 8))
{
bestWorstResponse = worstResponse;
bestMove = m;
}
else if(playerWhoTurnItIs == Board<7,6,4>::Player2 && (worstResponse < bestWorstResponse || bestMove == 8))
{
bestWorstResponse = worstResponse;
bestMove = m;
}
}
}
if(bestMove == 8)
{
std::cerr << "wasn't able to determine the best move to make" << std::endl;
return 0;
}
if(gameNumber <= numGames * percentOfGamesToRandomiseStart && turnNumber == 0)
{
std::size_t rSelection = int(randomMove()) % (availableMoves.size());
selectedMove = availableMoves[rSelection];
moveFound = true;
}
else if(wasRandomMove)
{
std::remove(availableMoves.begin(),availableMoves.end(),bestMove);
std::size_t rSelection = int(randomMove()) % (availableMoves.size());
selectedMove = availableMoves[rSelection];
moveFound = true;
}
else
{
selectedMove = bestMove;
moveFound = true;
}
}
StateType prevState;
create_board_state(prevState,board,playerWhoTurnItIs);
NN->FeedForward(prevState);
*previousNN = *NN;
// now that we have the move, add it to the board
StateType state;
board.AddChecker(selectedMove,playerWhoTurnItIs);
create_board_state(state,board,playerWhoTurnItIsNot);
auto outputs = NN->FeedForward(state);
if(board.InARowConnected(4) == Board<7,6,4>::Player1)
{
totalLoss += NN->BackPropagateFinal({1},previousNN,alpha,beta,lambda);
winState = true;
++numGamesWonP1;
}
else if(board.InARowConnected(4) == Board<7,6,4>::Player2)
{
totalLoss += NN->BackPropagateFinal({-1},previousNN,alpha,beta,lambda);
winState = true;
++numGamesWonP2;
}
else if(!board.IsThereAvailableMove())
{
totalLoss += NN->BackPropagateFinal({0},previousNN,alpha,beta,lambda);
winState = true;
}
else if(turnNumber > 0 && !wasRandomMove)
{
NN->BackPropagate(previousNN,alpha,beta,lambda);
}
if(!wasRandomMove)
{
outputs = NN->FeedForward(state);
}
++turnNumber;
playerWhoTurnItIs = playerWhoTurnItIsNot;
}
alpha -= decayRateAlpha;
beta -= decayRateBeta;
NN->ResetEligibilityTraces();
if(gameNumber > 0 && randomChance > randomChangeFloor && gameNumber <= numGames * percentToReduceRandomOver)
{
randomChance -= randomChangeDecay;
if(randomChance < randomChangeFloor)
randomChance = randomChangeFloor;
}
if(gameNumber % gamesToOutput == 0 && gameNumber != 0)
{
totalLoss = totalLoss / gamesToOutput;
matchFile << std::fixed << std::setprecision(51) << totalLoss << std::endl;
totalLoss = 0.0;
}
}
matchFile << std::endl << "Games won: " << numGamesWonP1 << " . " << numGamesWonP2 << std::endl;
auto weights = NN->GetNetworkWeights();
matchFile << std::endl;
matchFile << std::endl;
for(const auto& w : weights)
matchFile << std::fixed << std::setprecision(51) << w << ", \n";
matchFile << std::endl;
return 0;
}
One place I think I may have an issue is the minimax that's choosing the best move to make.
There's a few additional pieces that I don't think are too pertinent to the issues I'm having.
The Problems
It doesn't seem to matter whether I train 1000 games or 3000000 games, either Player 1 or Player 2 will win the vast majority of games. To the point of like 90 out of 100 games won by one player. If I output the actual individual game moves and outputs I can see that the games won by the other player are almost always the result of a lucky random move.
At the same time, I notice that the prediction outputs sort of "favour" a player. I.e. the outputs seem to be on the negative side of 0, so Player 1 is always making the best moves it can for example, but they all seem to be predicted toward Player 2 winning.
Sometimes it's Player 1 who wins majority, other times it's Player 2. I'm assuming that this is due to the random weights initialising
slight toward one player.
The first game or so doesn't favour one player over the other, but it very quickly starts to "lean" one way.
I've tried training now over 3000000 games, that took 3 days, but the network still doesn't seem to be able to make good decisions. I've tested the network by having it play other "bots" on riddles.io Connect 4 comp.
It fails to recognise that it needs to block the opponents 4 in a row
It, even after 3000000 games, doesn't play the centre column as the first move, which we know is the only starting move you can make that will guarantee a win.
Any help and direction would be greatly appreciated. Specifically, is my implementation of TD-Lambda back-propagation correct?