Exact repeatability of PyBrain experiments - python-2.7

What parameters in pybrain should be set to ensure exact replication of results when using the neural network modules as shown in the code below?
For each new run, the outputs differ although the random seed is set to the same value. The weights and biases are also the same for each run (due to np.random.seed(0)).
The weights and biases can also be set using net._setParameters(), but the results are also different when following this approach.
#!/usr/bin/python
#Python 2.7
from __future__ import division
import numpy as np
from pybrain.structure import SigmoidLayer, LinearLayer, TanhLayer
from pybrain.datasets import SupervisedDataSet
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.tools.shortcuts import buildNetwork
np.random.seed(0)
net = buildNetwork(1,2,1,hiddenclass=SigmoidLayer,outclass=LinearLayer,bias=True)
N = 10
t = np.arange(0,N,1)/N
x = np.cos(2*np.pi*0.1*t)
y = np.cos(2*np.pi*0.1*t)
ds = SupervisedDataSet(1, 1)
for c in range(N):
ds.addSample(x[c], y[c])
#net._setParameters(np.random.normal(0,1,(len(net.params))))
#net._setParameters(np.array([1.0]*len(net.params)))
trainer = BackpropTrainer(net, ds)
print 'NN parameters after setup:'
print net.params
for c in range(2):
e1 = trainer.train()
print 'Epoch %d Error: %f'%(c,e1)
print 'NN parameters after training:'
print net.params
p=np.zeros(N)
for c in range(N):
p[c] = net.activate([x[c]])
err = np.sum((y-p)**2)/N
print 'Prediction error = %2.4f'%err
The output of the code for two consecutive runs:
Run 1:
NN parameters after setup:
[ 1.76405235 0.40015721 0.97873798 2.2408932 1.86755799 -0.97727788
0.95008842]
Epoch 0 Error: 0.258780
Epoch 1 Error: 0.149163
NN parameters after training:
[ 1.63888191 0.40916677 0.97224621 2.24929727 1.8615028 -1.09298541
0.83265293]
Prediction error = 0.2179
Run 2:
NN parameters after setup:
[ 1.76405235 0.40015721 0.97873798 2.2408932 1.86755799 -0.97727788
0.95008842]
Epoch 0 Error: 0.258757
Epoch 1 Error: 0.148765
NN parameters after training:
[ 1.6384432 0.40916969 0.97225458 2.24931834 1.86149186 -1.09343599
0.83221073]
Prediction error = 0.2167
Clearly, the NN parameters before training are identical for both cases. After training, the NN parameters differ (and hence the predicted results and errors during training).

Related

ValueError: shapes (1,4) and (5,4) not aligned: 4 (dim 1) != 5 (dim 0), when adding my variables to my prediction machine

I am creating a prediction machine with four variables. When I add the variables it all messes up and gives me:
ValueError: shapes (1,4) and (5,4) not aligned: 4 (dim 1) != 5 (dim 0)
code
import pandas as pd
from pandas import DataFrame
from sklearn import linear_model
import tkinter as tk
import statsmodels.api as sm
# Approach 1: Import the data into Python
Stock_Market = pd.read_csv(r'Training_Nis_New2.csv')
df = DataFrame(Stock_Market,columns=['Month 1','Month 2','Month 3','Month
4','Month 5','Month 6','Month 7','Month 8',
'Month 9','Month 10','Month 11','Month
12','FSUTX','MMUKX','FUFRX','RYUIX','Interest R','Housing
Sale','Unemployement Rate','Conus Average Temperature
Rank','30FSUTX','30MMUKX','30FUFRX','30RYUIX'])
X = df[['Month 1','Interest R','Housing Sale','Unemployement Rate','Conus Average Temperature Rank']]
# here we have 2 variables for multiple regression. If you just want to use one variable for simple linear regression, then use X = df['Interest_Rate'] for example.Alternatively, you may add additional variables within the brackets
Y = df[['30FSUTX','30MMUKX','30FUFRX','30RYUIX']]
# with sklearn
regr = linear_model.LinearRegression()
regr.fit(X, Y)
print('Intercept: \n', regr.intercept_)
print('Coefficients: \n', regr.coef_)
# prediction with sklearn
# prediction with sklearn
HS=5.5
UR=6.7
CATR=8.9
New_Interest_R = 4.6
print('Predicted Stock Index Price: \n', regr.predict([[UR ,HS ,CATR
,New_Interest_R]]))
# with statsmodel
X = df[['Month 1','Interest R','Housing Sale','Unemployement Rate','Conus Average Temperature Rank']]
Y = df['30FSUTX']
print('\n\n*** Fund = FSUTX')
X = sm.add_constant(X) # adding a constant
model = sm.OLS(Y, X).fit()
predictions = model.predict(X)
print_model = model.summary()
print(print_model)

Working example of multi-stage model in Pyomo

This paper describes Pyomo's Differential and Algebraic Equations framework. It also mentions multi-stage problems; however, it does not show a complete example of such a problem. Does such an example exist somewhere?
The following demonstrates a complete minimum working example of a multi-stage optimization problem using Pyomo's DAE system:
#!/usr/bin/env python3
#http://www.gpops2.com/Examples/OrbitRaising.html
from pyomo.environ import *
from pyomo.dae import *
from pyomo.opt import SolverStatus, TerminationCondition
import random
import matplotlib.pyplot as plt
T = 10 #Maximum time for each stage of the model
STAGES = 3 #Number of stages
m = ConcreteModel() #Model
m.t = ContinuousSet(bounds=(0,T)) #Time variable
m.stages = RangeSet(0, STAGES) #Stages in the range [0,STAGES]. Can be thought of as an integer-valued set
m.a = Var(m.stages, m.t) #State variable defined for all stages and times
m.da = DerivativeVar(m.a, wrt=m.t) #First derivative of state variable with respect to time
m.u = Var(m.stages, m.t, bounds=(0,1)) #Control variable defined for all stages and times. Bounded to range [0,1]
#Setting the value of the derivative.
def eq_da(m,stage,t): #m argument supplied when function is called. `stage` and `t` are given values from m.stages and m.t (see below)
return m.da[stage,t] == m.u[stage,t] #Derivative is proportional to the control variable
m.eq_da = Constraint(m.stages, m.t, rule=eq_da) #Call constraint function eq_da for each unique value of m.stages and m.t
#We need to connect the different stages together...
def eq_stage_continuity(m,stage):
if stage==m.stages.last(): #The last stage doesn't connect to anything
return Constraint.Skip #So skip this constraint
else:
return m.a[stage,T]==m.a[stage+1,0] #Final time of each stage connects with the initial time of the following stage
m.eq_stage_continuity = Constraint(m.stages, rule=eq_stage_continuity)
#Boundary conditions
def _init(m):
yield m.a[0,0] == 0 #Initial value (at zeroth stage and zeroth time) of `a` is 0
yield ConstraintList.End
m.con_boundary = ConstraintList(rule=_init) #Repeatedly call `_init` until `ConstraintList.End` is returned
#Objective function: maximize `a` at the end of the final stage
m.obj = Objective(expr=m.a[STAGES,T], sense=maximize)
#Get a discretizer
discretizer = TransformationFactory('dae.collocation')
#Disrectize the model
#nfe (number of finite elements)
#ncp (number of collocation points within finite element)
discretizer.apply_to(m,nfe=30,ncp=6,scheme='LAGRANGE-RADAU')
#Get a solver
solver = SolverFactory('ipopt', keepfiles=True, log_file='/z/log', soln_file='/z/sol')
solver.options['max_iter'] = 100000
solver.options['print_level'] = 1
solver.options['linear_solver'] = 'ma27'
solver.options['halt_on_ampl_error'] = 'yes'
#Solve the model
results = solver.solve(m, tee=True)
print(results.solver.status)
print(results.solver.termination_condition)
#Retrieve the results in a pleasant format
r_t = [t for s in sorted(m.stages) for t in sorted(m.t)]
r_a = [value(m.a[s,t]) for s in sorted(m.stages) for t in sorted(m.t)]
r_u = [value(m.u[s,t]) for s in sorted(m.stages) for t in sorted(m.t)]
plt.plot(r_t, r_a, label="r_a")
plt.plot(r_t, r_u, label="r_u")
plt.legend()
plt.show()

tf.py_func , custom tensorflow function getting applied to only the first element in the tensor

I am new to tensorflow and was playing around with a deep learning network. I wanted to do a custom rounding off on all the weights after each iteration. As the round function in tensorflow library doesn't give you the option to round the values down to a certain number of decimal points.
So I wrote this
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
np_prec = lambda x: np.round(x,3).astype(np.float32)
def tf_prec(x,name=None):
with ops.name_scope( "d_spiky", name,[x]) as name:
y = tf.py_func(np_prec,
[x],
[tf.float32],
name=name,
stateful=False)
return y[0]
with tf.Session() as sess:
x = tf.constant([0.234567,0.712,1.2,1.7])
y = tf_prec(x)
y = tf_prec(x)
tf.global_variables_initializer
print(x.eval(), y.eval())
The output I got was this
[ 0.234567 0.71200001 1.20000005 1.70000005] [ 0.235 0.71200001 1.20000005 1.70000005]
So the custom rounding off worked only on the first item in the tensor and I am not sure about what I am doing wrong. Thanks in advance.
The error here because of the following line,
np_prec = lambda x: np.round(x,3).astype(np.float32)
you are casting the output to np.float32. You can verify the error by the following code,
print(np.round([0.234567,0.712,1.2,1.7], 3).astype(np.float32)) #prints [ 0.235 0.71200001 1.20000005 1.70000005]
The default output of np.round is float64. Moreover, you also have to change the Tout argument in tf.py_func to float64.
I have given the following code with the above fix and commented where necessary.
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
np_prec = lambda x: np.round(x,3)
def tf_prec(x,name=None):
with ops.name_scope( "d_spiky", name,[x]) as name:
y = tf.py_func(np_prec,
[x],
[tf.float64], #changed this line to tf.float64
name=name,
stateful=False)
return y[0]
with tf.Session() as sess:
x = tf.constant([0.234567,0.712,1.2,1.7],dtype=np.float64) #specify the input data type np.float64
y = tf_prec(x)
y = tf_prec(x)
tf.global_variables_initializer
print(x.eval(), y.eval())
Hope this helps.

SciPy curve_fit not working when one of the parameters to fit is a power

I'm trying to fit my data to a user defined function using SciPy curve_fit, which works when fitting to a function with a fixed power (func1). But curve_fit does not work when the function contains a power as a parameter to fit to (func2).
Curve_fit still does not work if I provide an initial guess for the parameters usins the keyword p0. I can not use the bounds keyword as the version of SciPy which I have does not have it.
This script illustrates the point:
import scipy
from scipy.optimize import curve_fit
import sys
print 'scipy version: ', scipy.__version__
print 'np.version: ', np.__version__
print sys.version_info
def func1(x,a):
return (x-a)**3.0
def func2(x,a,b):
return (x-a)**b
x_train = np.linspace(0, 12, 50)
y = func2(x_train, 0.5, 3.0)
y_train = y + np.random.normal(size=len(x_train))
print 'dtype of x_train: ', x_train.dtype
print 'dtype of y_train: ', y_train.dtype
popt1, pcov1 = curve_fit( func1, x_train, y_train, p0=[0.6] )
popt2, pcov2 = curve_fit( func2, x_train, y_train, p0=[0.6, 4.0] )
print 'Function 1: ', popt1, pcov1
print 'Function 2: ', popt2, pcov2
Which outputs the following:
scipy version: 0.14.0
np.version: 1.8.2
sys.version_info(major=2, minor=7, micro=6, releaselevel='final', serial=0)
dtype of x_train: float64
dtype of y_train: float64
stack_overflow.py:14: RuntimeWarning: invalid value encountered in power
return (x-a)**b
Function 1: [ 0.50138759] [[ 3.90044196e-07]]
Function 2: [ nan nan] [[ inf inf]
[ inf inf]]
(As #xnx first commented,) the problem with the second formulation (where the exponent b is unknown and considered to be real-valued) is that, in the process of testing potential values for a and b, quantities of the form z**p need to be evaluated, where z is a negative real number and p is a non-integer. This quantity is complex in general, hence the procedure fails. For example, for x=0 and test variables a=0.5, b=4.1, it holds (x-a)**b = (-0.5)**4.1 = 0.0555+0.018j.

Categorical Mixture Model in Pymc3

I'm new to Pymc3 and I'm trying to create the Categorical Mixture Model shown in https://en.wikipedia.org/wiki/Mixture_model#Categorical_mixture_model . I'm having difficulty hooking up the 'x' variable. I think it's because I have to make the z variable Deterministic, but I'm getting an error message at the line where 'x' is assigned : "ValueError: We expected 3 inputs but got 2.". It looks like the p function only accepts 2 inputs so I'm stuck. I've tried a bunch of different things, but haven't been able to get this to work yet.
import numpy as np
from pymc3 import *
import theano.tensor as t
K = 3 #NUMBER OF TOPICS
V = 20 #NUMBER OF WORDS
N = 15 #NUMBER OF DOCUMENTS
#GENERAETE RANDOM CATEGORICAL MIXTURES
data = np.ones([N,V])
#theano.compile.ops.as_op(itypes=[t.lscalar, t.dscalar, t.dscalar],otypes=[t.dvector])
def p(z=z, phi=phi):
return [phi[z[i,j]] for i in range(D) for j in range(W)]
model = Model()
with model:
alpha = np.ones(V)
beta = np.ones(K)
theta = [Dirichlet('theta_%i' % i, alpha, shape=V) for i in range(K)]
phi = Dirichlet('phi', beta, shape=K)
z = [Categorical('z_%i' % i, p = phi, shape=V) for i in range(N)]
x = [Categorical('x_%i_%i' % (i,j), p=p(z[i][j],phi), observed=data[i,j]) for i in range(N) for j in range(V)]
#x = [Categorical('x_%i_%i' % (i,j), p=theta[z[i][j]], observed=data[i,j]) for i in range(N) for j in range(V)]
print "Created model. Now begin sampling"
step = Slice()
trace = sample(n, step)
trace.get_values('phi')
For starters, in your example above, z and phi have no value which would allow them to be used as default values. We also don't have values for D and W.
As for the number of arguments, the function you define has 2 but your theano decorator above it has 3. I'd suggest
#theano.compile.ops.as_op(itypes=[t.lscalar, t.dvector],otypes=[t.dvector])
def p(z, phi):
return [phi[z[i,j]] for i,j in zip(range(D),range(W))]