Neural network output is always 1 - c++

Question

I'm developing a neural network application, but I have a problem when I'm trying to print the output of the feedforward: the console shows this value:
The output is always 1 1 1 1 1
I'm using 3 layers consisting of 2500 input nodes, 1800 hidden nodes, and 5 output nodes.
I'm using sigmoid binary as the activation function, the weight for testing is fixed at 0.5.
When I'm trying to use a very simple architecture (2 input, 3 hidden, 1 output) it works perfectly, but right now with so many nodes I can't manually calculate the output so I don't know if the result is true or false.
Is my code wrong, or can't I give a fixed weight? I only want to do one feedforward, not back-propagation, and the input is random from 0 to 1.
This is network.cpp:
#include <stdio.h>
#include <iostream>
#include "network.h"
#include "neuron.h"
using namespace std;
layer::layer(int numberOfNeuron,int numberOfInput,int numberOfOutput):
neuron_layer(numberOfNeuron)
{
for(int i=0;i<numberOfNeuron;i++)
{
neuron_layer[i] = new neuron(numberOfInput,numberOfOutput);
}
}
layer::~layer()
{
}
int layer::get_number_of_neuron()
{
return neuron_layer.size();
}
network::network(int layerNumber,int hiddenNeuronNumber):
layer_network(layerNumber)
{
this->layer_numbers = layerNumber;
for(int i=0;i<layerNumber;i++)
{
if(i==0)
{
layer_network[i] = new layer(2500,5,hiddenNeuronNumber);
}
else if(i==1)
{
layer_network[i] = new layer(hiddenNeuronNumber,2500,5);
}
else if(i==2)
{
layer_network[i] = new layer(5,hiddenNeuronNumber,1);
}
}
cout<<endl<<"Input layer : "<<layer_network[0]->get_number_of_neuron()<<endl;
cout<<"Hidden layer : "<<layer_network[1]->get_number_of_neuron()<<endl;
cout<<"Output layer : "<<layer_network[2]->get_number_of_neuron()<<endl;
}
network::~network()
{
}
void network::init_input_layer(int inputNeuronNumber,int hiddenNeuronNumber)
{
for(int i=0;i<inputNeuronNumber;i++)
{
for(int j=0;j<hiddenNeuronNumber;j++)
{
layer_network[0]->neuron_layer[i]->outputs[j]->weights = 0.5f;
}
}
}
void network::init_hidden_layer(int inputNeuronNumber,int hiddenNeuronNumber,int outputNeuronNumber)
{
for(int i=0;i<hiddenNeuronNumber;i++)
{
for(int j=0;j<inputNeuronNumber;j++)
{
layer_network[1]->neuron_layer[i]->inputs[j]->weights = layer_network[0]->neuron_layer[j]->outputs[i]->weights;
}
}
for(int k=0;k<hiddenNeuronNumber;k++)
{
for(int l=0;l<outputNeuronNumber;l++)
{
layer_network[1]->neuron_layer[k]->outputs[l]->weights = 0.5f;
}
}
}
void network::init_ouput_layer(int hiddenNeuronNumber,int outputNeuronNumber)
{
for(int i=0;i<outputNeuronNumber;i++)
{
for(int j=0;j<hiddenNeuronNumber;j++)
{
layer_network[2]->neuron_layer[i]->inputs[j]->weights = layer_network[1]->neuron_layer[j]->inputs[i]->weights;
}
}
}
This is neuron.cpp:
#include "neuron.h"
#include <stdio.h>
#include <iostream>
using namespace std;
synapse::synapse()
{
}
synapse::~synapse()
{
}
neuron::neuron(int numberOfInput,int numberOfOutput):
inputs(numberOfInput),outputs(numberOfOutput)
{
for(int i=0;i<numberOfInput;i++)
{
inputs[i] = new synapse();
}
for(int i=0;i<numberOfOutput;i++)
{
outputs[i] = new synapse();
}
}
neuron::~neuron()
{
}
int neuron::get_input_size()
{
int input_length;
input_length=(int) inputs.size();
return input_length;
}
int neuron::get_output_size()
{
int output_length;
output_length=(int) outputs.size();
return output_length;
}
void neuron::input_fire()
{
output_value = inputs[0]->activation_values;
for(int i=0;i<get_output_size();i++)
{
outputs[i]->activation_values = output_value;
}
}
void neuron::fire()
{
output_value = 0.0f;
for(int i=0;i<get_input_size();i++)
{
output_value+=(inputs[i]->activation_values)*(inputs[i]->weights);
}
//cout<<endl<<"Before Sigmoid"<<output_value;
output_value = 1.0f / (1.0f+ exp(-output_value));
//cout<<" After Sigmoid"<<output_value;
for(int i=0;i<get_output_size();i++)
{
outputs[i]->activation_values = output_value;
}
}
I think I don't need to post neuron.h and network.h since both of them only consist of declarations. I define the functions in the .cpp. This is how I created the object:
srand (time(NULL));
float inputTest[2500];
network test(3,1800);
test.init_network(1800);
for(int i=0;i<2500;i++)
{
inputTest[i]=(float)rand()/(float)RAND_MAX;
}
test.feedforward(inputTest);

Answer 1

I didn't check your code but...
No, you can't use fixed weight like that. Each node in your hidden layer will get larger and larger values as you increase the amount of inputs. Sigmoid will scale the large values to 1.
Think about it:
Let's say that you have 100 inputs each having "random" input value of 0.1. For simplicity, let's just forget everything else. Since your weights are constant 0.5, all the nodes in the hidden layer will get a same value that consists of "sigmoided" sum of each input*weight, that is sigm(0.1*0.5*100) = sigm(5) -> ~1
So, the more you have positive inputs with constant positive weights the more close all the hidden layers' outputs will become to 1.