I'm trying to run the optimization example with non-linear constraints shown here
https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html
>>> def cons_f(x):
... return [x[0]**2 + x[1], x[0]**2 - x[1]]
>>> def cons_J(x):
... return [[2*x[0], 1], [2*x[0], -1]]
>>> def cons_H(x, v):
... return v[0]*np.array([[2, 0], [0, 0]]) + v[1]*np.array([[2, 0], [0, 0]])
>>> from scipy.optimize import NonlinearConstraint
>>> nonlinear_constraint = NonlinearConstraint(cons_f, -np.inf, 1, jac=cons_J, hess=cons_H)
But when I try to import NonlinearConstraint this is what I get
ImportError: cannot import name NonlinearConstraint
I'm running scipy v.1.0.0
>>> import scipy
>>> print scipy.__version__
1.0.0
Any suggestions? Thanks in advance for your help
You will need scipy >= 1.1 or a master-branch based install!
As 1.1 was released recently (05.05.18), there are chances for binary-builds (depends a bit on how you use scipy).
Compare 1.1's optimize/init.py:
...
from ._lsq import least_squares, lsq_linear
from ._constraints import (NonlinearConstraint,
LinearConstraint,
Bounds)
from ._hessian_update_strategy import HessianUpdateStrategy, BFGS, SR1
__all__ = [s for s in dir() if not s.startswith('_')]
...
with 1.0.1's optimize/init.py:
...
from ._lsq import least_squares, lsq_linear
__all__ = [s for s in dir() if not s.startswith('_')]
...
More indications are available in the 1.1 release-text:
scipy.optimize improvements
The method trust-constr has been added to scipy.optimize.minimize. The
method switches between two implementations depending on the problem
definition. For equality constrained problems it is an implementation of
a trust-region sequential quadratic programming solver and, when
inequality constraints are imposed, it switches to a trust-region
interior point method. Both methods are appropriate for large scale
problems. Quasi-Newton options BFGS and SR1 were implemented and can be
used to approximate second order derivatives for this new method. Also,
finite-differences can be used to approximate either first-order or
second-order derivatives.
which is actually the solver introducing those abstractions.
Additionally, optimize/_constraints.py does not exist in 1.01.
Related
I have tried to write some example with keras,but some error happenError when checking target: expected dense_2 to have shape (2,) but got array with shape (1,)
I have tried to change the input_shape but it doesn't work
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
import numpy
print "hello"
input=[[1],[2],[3],[4],[5],[6],[7],[8]]
input=numpy.array(input, dtype="float")
# input=input.reshape(8,1)
output=[[1],[0],[1],[0],[1],[0],[1],[0]]
output=numpy.array(output, dtype="float")
(trainx,testx,trainy,testy)=train_test_split(input, output, test_size=0.25, random_state=42)
lb = LabelBinarizer()
trainy=lb.fit_transform(trainy)
testy=lb.transform(testy)
model=Sequential()
model.add(Dense(4,input_shape=(1,),activation="sigmoid"))
# model.add(Dense(4,activation="sigmoid"))
# print len(lb.classes_)
model.add(Dense(len(lb.classes_),activation="softmax",input_shape=(4,)))
INIT_LR = 0.01
EPOCHS = 20
print("[INFO] training network...")
opt = SGD(lr=INIT_LR)
model.compile(loss="categorical_crossentropy", optimizer=opt,metrics=["accuracy"])
H = model.fit(trainx, trainy, validation_data=(testx, testy),epochs=EPOCHS, batch_size=2)
Since you have two classes, you can have a single neuron in the final Dense layer and use sigmoid activation. Or if you want to use softmax, you need to create a one hot encoding of y like this.
(trainx,testx,trainy,testy)=train_test_split(input, output, test_size=0.25, random_state=42)
trainy = keras.utils.to_categorical(trainy, 2)
testy = keras.utils.to_categorical(testy, 2)
You should use "from tensorflow.python.keras.xx" instead of "from keras.xx". It prevents it from receiving the error like: "AttributeError: module 'tensorflow' has no attribute 'get_default_graph"
Below is the code for a simple Bayesian Linear regression. After I obtain the trace and the plots for the parameters, is there any way in which I can save the data that created the plots in a file so that if I need to plot it again I can simply plot it from the data in the file rather than running the whole simulation again?
import pymc3 as pm
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0,9,5)
y = 2*x + 5
yerr=np.random.rand(len(x))
def soln(x, p1, p2):
return p1+p2*x
with pm.Model() as model:
# Define priors
intercept = pm.Normal('Intercept', 15, sd=5)
slope = pm.Normal('Slope', 20, sd=5)
# Model solution
sol = soln(x, intercept, slope)
# Define likelihood
likelihood = pm.Normal('Y', mu=sol,
sd=yerr, observed=y)
# Sampling
trace = pm.sample(1000, nchains = 1)
pm.traceplot(trace)
print pm.summary(trace, ['Slope'])
print pm.summary(trace, ['Intercept'])
plt.show()
There are two easy ways of doing this:
Use a version after 3.4.1 (currently this means installing from master, with pip install git+https://github.com/pymc-devs/pymc3). There is a new feature that allows saving and loading traces efficiently. Note that you need access to the model that created the trace:
...
pm.save_trace(trace, 'linreg.trace')
# later
with model:
trace = pm.load_trace('linreg.trace')
Use cPickle (or pickle in python 3). Note that pickle is at least a little insecure, don't unpickle data from untrusted sources:
import cPickle as pickle # just `import pickle` on python 3
...
with open('trace.pkl', 'wb') as buff:
pickle.dump(trace, buff)
#later
with open('trace.pkl', 'rb') as buff:
trace = pickle.load(buff)
Update for someone like me who is still coming over to this question:
load_trace and save_trace functions were removed. Since version 4.0 even the deprecation waring for these functions were removed.
The way to do it is now to use arviz:
with model:
trace = pymc.sample(return_inferencedata=True)
trace.to_netcdf("filename.nc")
And it can be loaded with:
trace = arviz.from_netcdf("filename.nc")
This way works for me :
# saving trace
pm.save_trace(trace=trace_nb, directory=r"c:\Users\xxx\Documents\xxx\traces\trace_nb")
# loading saved traces
with model_nb:
t_nb = pm.load_trace(directory=r"c:\Users\xxx\Documents\xxx\traces\trace_nb")
I've read some answers on this question here and here, however I'm still a bit puzzled by tf.Variable being and/or not being a tf.Tensor.
The linked answers deal with a mutability of tf.Variable and mentioning that tf.Variables maintains their states (when instantiated with default parameter trainable=True).
What makes me still a bit confused is a test case I came across when writing simple unit tests using tf.test.TestCase
Consider the following code snippet. We have a simple class called Foo which has only one property, a tf.Variable initialized to w:
import tensorflow as tf
import numpy as np
class Foo:
def __init__(self, w):
self.w = tf.Variable(w)
Now, let's say you want to test that the instance of Foo has w initialized with tensor of the same dimension as passed in via w. The simplest test case could be written as follows:
import tensorflow as tf
import numpy as np
from foo import Foo
class TestFoo(tf.test.TestCase):
def test_init(self):
w = np.random.rand(3,2)
foo = Foo(w)
init = tf.global_variables_initializer()
with self.test_session() as sess:
sess.run(init)
self.assertShapeEqual(w, foo.w)
if __name__ == '__main__':
tf.test.main()
Now when you run the test you'll get the following error:
======================================================================
ERROR: test_init (__main__.TestFoo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_foo.py", line 12, in test_init
self.assertShapeEqual(w, foo.w)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/test_util.py", line 1100, in assertShapeEqual
raise TypeError("tf_tensor must be a Tensor")
TypeError: tf_tensor must be a Tensor
----------------------------------------------------------------------
Ran 2 tests in 0.027s
FAILED (errors=1)
You can "get around" this unit test error by doing something like this (i.e. note assertShapeEqual was replaced with assertEqual):
self.assertEqual(list(w.shape), foo.w.get_shape().as_list())
What I'm interested in, though, is the tf.Variable vs tf.Tensor relationship.
What the test error seems to be suggesting is that foo.w is NOT a tf.Tensor, meaning you probably can't use tf.Tensor API on it. Consider, however, the following interactive python session:
$ python3
Python 3.6.3 (default, Oct 4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> import numpy as np
>>> w = np.random.rand(3,2)
>>> var = tf.Variable(w)
>>> var.get_shape().as_list()
[3, 2]
>>> list(w.shape)
[3, 2]
>>>
In the session above, we create a variable and run the get_shape() method on it to retrieve its shape dimensions. Now, get_shape() method is a tf.Tensor API method as you can see here.
So to get back to my question, what parts of tf.Tensor API does tf.Variable implement. If the answer is ALL of them, why does the above test case fail?
self.assertShapeEqual(w, foo.w)
with
raise TypeError("tf_tensor must be a Tensor")
I'm pretty sure I'm missing something fundamental here or maybe it's a bug in assertShapeEqual ? I would appreciate if someone could shed some light on this.
I'm using following version of tensorflow on macOS with python3:
tensorflow (1.4.1)
That testing utility function is checking whether a variable implements tf.Tensor
>>> import tensorflow as tf
>>> v = tf.Variable('v')
>>> v
<tf.Variable 'Variable:0' shape=() dtype=string_ref>
>>> isinstance(v, tf.Tensor)
False
The answer appears to be 'no'.
Update:
According to the documentation that is correct:
https://www.tensorflow.org/programmers_guide/variables
Unlike tf.Tensor objects, a tf.Variable exists outside the context of
a single session.run call.
Although:
A tf.Variable represents a tensor whose value can be changed by
running ops on it.
(Not quite sure what 'represents a tensor' means - sounds like a design 'feature')
I saw a post from a few days ago by someone else: pymc3 likelihood math with non-theano function. Even though I think the problem at its core is the same, I thought I would ask with a simpler example:
Inside logp_wrap, I put some made up definition of a likelihood function. It depends on the rv and an observation. In this case I could do this with theano operations, but let's say that I want this function to be more complex and so I cannot use theano.
The problem comes when I try to define the likelihood both in terms of an RV and observations. From what I have seen, this format would work if I was specifying everything in 'logp_wrap' as theano operations.
I have searched around for a solution to this, but haven't found anything where this problem is fully addressed.
The problem in my attempt to do this is actually that the logp_ function is correctly decorated, but the logp_wrap function is only correctly decorated for its input, and not for its output, so I get the error
TypeError: 'TensorVariable' object is not callable.
Would be great if someone had a solution - don't think I am the only one with this problem.
The theano version of this that works (and uses the same function within a function definition) without the #as_op code is here: https://pymc-devs.github.io/pymc3/notebooks/lda-advi-aevb.html?highlight=densitydist (Specifically the sections: "Log-likelihood of documents for LDA" and "LDA model section")
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
"""
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import pymc3 as pm
from theano import as_op
import theano.tensor as T
from scipy.stats import norm
#Some data that we observed
g_observed = [0.0, 1.0, 2.0, 3.0]
#Define a function to calculate the logp without using theano
#This as_op is where the problem is - the input is an rv but the output is a
#function.
#as_op(itypes=[T.dscalar],otypes=[T.dscalar])
def logp_wrap(rv):
#We are not using theano so we wrap the function.
#as_op(itypes=[T.dvector],otypes=[T.dscalar])
def logp_(ob):
#Some made up likelihood -
#The key here is that lp depends on the rv input and the observations
lp = np.log(norm.pdf(rv + ob))
return lp
return logp_
hb1_model = pm.Model()
with hb1_model:
I_mean = pm.Normal('I_mean', mu=0.1, sd=0.05)
xs = pm.DensityDist('x', logp_wrap(I_mean),observed = g_observed)
with hb1_model:
step = pm.Metropolis()
trace = pm.sample(1000, step)
Is it possible to integrate any Ordinary Differential Equation backward in time
using scipy.integrate.odeint ?
If it is possible, could someone tell me what should be the arguement 'time' in 'odeint.
odeint handles negative values of the t argument. No special treatment is needed.
Here's an example:
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
def mysys(z, t):
"""A slightly damped oscillator."""
return [z[1] - 0.02*z[0], -z[0]]
if __name__ == "__main__":
# Note that t starts at 0 and goes "backwards"
t = np.linspace(0, -50, 501)
z0 = [1, 1]
sol = odeint(mysys, z0, t)
plt.plot(t, sol)
plt.xlabel('t')
plt.show()
The plot:
You can make a change of variables s = t_0 - t, and integrate the differential equation with respect to s. odeint doesn't do this for you.
It is not necessary to make a change of variables. Here an example:
import math
import numpy
import scipy
import pylab as p
from math import *
from numpy import *
from scipy.integrate import odeint
from scipy.interpolate import splrep
from scipy.interpolate import splev
g1=0.01
g2=0.01
w1=1
w2=1
b1=1.0/20.0
b2=1.0/20.0
b=1.0/20.0
c0=0
c1=0.2
wf=1
def wtime(t):
f=1+c0+c1*cos(2*wf*t)
return f
def dv(y,t):
return array([y[1], -(wtime(t)+g1*w1+g2*w2)*y[0]+w1*y[2]+w2*y[3], w1*y[2]-g1*w1*y[0], w2*y[3]-g2*w2*y[0]])
tv=linspace(100,0,1000)
v1zero=array([1,0,0,0])
v2zero=array([0,1,0,0])
v1s=odeint(dv,v1zero,tv)
v2s=odeint(dv,v2zero,tv)
p.plot(tv,v1s[:,0])
p.show()
I check the result with Wolfram Mathematica (that program can solve backward odes).