I was trying to find the optimal parameter order by using a loop:
d = 1
for p in range(3):
for q in range(3):
try:
order = (p, 0, q)
params = (p, d, q)
arima_mod = ARIMA(ts.dropna(), order).fit(method = 'css-mle', disp = 0)
arima_mod_aics[params] = arima_mod.aic
except:
pass
and I have received the message:
/usr/local/lib/python2.7/dist-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/base/model.py:466: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
"Check mle_retvals", ConvergenceWarning)
I would like to ignore this warning, what should I do? Any suggestion?
Thanks in advance.
To specifically ignore ConvergenceWarnings:
import warnings
from statsmodels.tools.sm_exceptions import ConvergenceWarning
warnings.simplefilter('ignore', ConvergenceWarning)
See this example from the statsmodels sourceforge, especially In [17]:.
import warnings
with warnings.catch_warnings():
warnings.filterwarnings("ignore")
# Line that is not converging
likev = mdf.profile_re(0, dist_low=0.1, dist_high=0.1)
A clean way to do it is to use the included warn_convergence. You can pass it to fit method like so:
arima.fit(method_kwargs={"warn_convergence": False})
Related
I followed the official tutorial of the tensorflow website: https://www.tensorflow.org/extend/adding_an_op
There is also described how to call the gradient of the example ZeroOut in the tutorial that I want to try in this short code snippet underneath.
I have found the code here: https://github.com/MatteoRagni/tf.ZeroOut.gpu
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import sparse_ops
zero_out_module = tf.load_op_library('./libzeroout.so')
#ops.RegisterGradient("ZeroOut")
def _zero_out_grad(op, grad):
to_zero = op.inputs[0]
shape = array_ops.shape(to_zero)
index = array_ops.zeros_like(shape)
first_grad = array_ops.reshape(grad, [-1])[0]
to_zero_grad = sparse_ops.sparse_to_dense([index], shape, first_grad, 0)
return [to_zero_grad] # List of one Tensor, since we have one input
t_in = tf.placeholder(tf.int32, [None,None])
ret = zero_out_module.zero_out(t_in)
grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)
with tf.Session(''):
feed_dict = {t_in: [[1, 2], [3, 4]]}
print "ret val: ", ret.eval(feed_dict=feed_dict)
print "grad: ", grad
print "grad: ", grad.eval(feed_dict=feed_dict)
I got this error ...
AttributeError: 'list' object has no attribute 'eval'
... but I can do ret.eval().
Why I cant call grad.eval()? I want to see these values inside the grad tensor. How to debug gradient?
Answer to old question
The implementation
def _zero_out_grad(op, *grads):
topdiff = grads[0]
bottom = op.inputs[0]
shape = array_ops.shape(bottom)
index = array_ops.zeros_like(shape)
first_grad = array_ops.reshape(topdiff, [-1])[0]
to_zero_grad = sparse_ops.sparse_to_dense([index], shape, first_grad, 0)
return to_zero_grad
works quite nicely here. Are you sure "#ops.RegisterGradient("ZeroOut")" is executed before the tf.Session()?
Usually the
zero_out_module = tf.load_op_library('./libzeroout.so')
#ops.RegisterGradient("ZeroOut")
def _zero_out_grad(op, grad):
# ...
is placed in a different file and just imported. A full working example even with the recent TensorFlow version is here.
Answer to completely changed question
Your gradient function returns a list and a Python list has no 'eval()'. Try either:
grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)[0]
Or follow best practice and use
grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)
with tf.Session() as sess:
sess.run(grad, feed_dict=feed_dict)
Please do not change your entire question
Really struggling with this one... Forgive the longish post.
I have an experiment that on each trial displays some stimulus, collects a response, and then moves on to the next trial.
I would like to incorporate an optimizer that runs in between trials and therefore must have a specific time-window designated by me to run, or it should be terminated. If it's terminated, I would like to return the last set of parameters it tried so that I can use it later.
Generally speaking, here's the order of events I'd like to happen:
In between trials:
Display stimulus ("+") for some number of seconds.
While this is happening, run the optimizer.
If the time for displaying the "+" has elapsed and the optimizer has
not finished, terminate the optimizer, return the most recent set of parameters it tried, and move on.
Here is some of the relevant code I'm working with so far:
do_bns() is the objective function. In it I include NLL['par'] = par or q.put(par)
from scipy.optimize import minimize
from multiprocessing import Process, Manager, Queue
from psychopy import core #for clock, and other functionality
clock = core.Clock()
def optim(par, NLL, q)::
a = minimize(do_bns, (par), method='L-BFGS-B', args=(NLL, q),
bounds=[(0.2, 1.5), (0.01, 0.8), (0.001, 0.3), (0.1, 0.4), (0.1, 1), (0.001, 0.1)],
options={"disp": False, 'maxiter': 1, 'maxfun': 1, "eps": 0.0002}, tol=0.00002)
if __name__ == '__main__':
print('starting optim')
max_time = 1.57
with Manager() as manager:
par = manager.list([1, 0.1, 0.1, 0.1, 0.1, 0.1])
NLL = manager.dict()
q = Queue()
p = Process(target=optim, args=(par, NLL, q))
p.start()
start = clock.getTime()
while clock.getTime() - start < max_time:
p.join(timeout=0)
if not p.is_alive():
break
if p.is_alive():
res = q.get()
p.terminate()
stop = clock.getTime()
print(NLL['a'])
print('killed after: ' + str(stop - start))
else:
res = q.get()
stop = clock.getTime()
print('terminated successfully after: ' + str(stop - start))
print(NLL)
print(res)
This code, on its own, seems to sort of do what I want. For example, the res = q.get() right above the p.terminate() actually takes something like 200ms so it will not terminate exactly at max_time if max_time < ~1.5s
If I wrap this code in a while-loop that checks to see if it's time to stop presenting the stimulus:
stim_start = clock.getTime()
stim_end = 5
print('showing stim')
textStim.setAutoDraw(True)
win.flip()
while clock.getTime() - stim_start < stim_end:
# insert the code above
print('out of loop')
I get weird behavior such as multiple iterations of the whole code from the beginning...
showing stim
starting optim
showing stim
out of loop
showing stim
out of loop
[1.0, 0.10000000000000001, 0.10000000000000001, 0.10000000000000001, 0.10000000000000001, 0.10000000000000001]
killed after: 2.81303440395
Note the multiple 'showing stim's' and 'out of loop's.
I'm open to any solution that accomplishes my goal :|
Help and thank you!
Ben
General remark
Your solution would give me nightmares! I don't see a reason to use multiprocessing here and i'm not even sure how you grab those updated results before termination. Maybe you got your reason for this approach, but i highly recommend something else (which has a limitation).
Callback-based approach
The general idea i would pursue is the following:
fire up your optimizer with some additional time-limit information and some callback enforcing this
the callback is called in each iteration of this optimizer
if time-limit reached: raise a customized Exception
The limits:
as the callback is only called once in each iteration, there is some limited sequence of points in time where the optimizer might get stopped
the potential difference is highly dependent on iteration-time for your problem! (numerical-differentiation, huge-data, slow function eval; all this matters)
if not exceeding some given time is of highest priority, this approach might be not right or you would need some kind of safeguarded interpolation to reason if one more iteration is possible in time
or: combine your kind of killing off workers with my approach of updating intermediate-results through some callback
Example code (bit hacky):
import time
import numpy as np
import scipy.sparse as sp
import scipy.optimize as opt
np.random.seed(1)
""" Fake task: sparse NNLS """
M, N, D = 2500, 2500, 0.1
A = sp.random(M, N, D)
b = np.random.random(size=M)
""" Optimization-setup """
class TimeOut(Exception):
"""Raise for my specific kind of exception"""
def opt_func(x, A, b):
return 0.5 * np.linalg.norm(A.dot(x) - b)**2
def opt_grad(x, A, b):
Ax = A.dot(x) - b
grad = A.T.dot(Ax)
return grad
def callback(x):
time_now = time.time() # probably not the right tool in general!
callback.result = [np.copy(x)] # better safe than sorry -> copy
if time_now - callback.time_start >= callback.time_max:
raise TimeOut("Time out")
def optimize(x0, A, b, time_max):
result = [np.copy(x0)] # hack: mutable type
time_start = time.time()
try:
""" Add additional info to callback (only takes x as param!) """
callback.time_start = time_start
callback.time_max = time_max
callback.result = result
res = opt.minimize(opt_func, x0, jac=opt_grad,
bounds=[(0, np.inf) for i in range(len(x0))], # NNLS
args=(A, b), callback=callback, options={'disp': False})
except TimeOut:
print('time out')
return result[0], opt_func(result[0], A, b)
return res.x, res.fun
print('experiment 1')
start_time = time.perf_counter()
x, res = optimize(np.zeros(len(b)), A, b, 0.1) # 0.1 seconds max!
end_time = time.perf_counter()
print(res)
print('used secs: ', end_time - start_time)
print('experiment 2')
start_time = time.perf_counter()
x_, res_ = optimize(np.zeros(len(b)), A, b, 5) # 5 seconds max!
end_time = time.perf_counter()
print(res_)
print('used secs: ', end_time - start_time)
Example output:
experiment 1
time out
422.392771467
used secs: 0.10226839151517493
experiment 2
72.8470708728
used secs: 0.3943936788825996
import tensorflow as tf
array = tf.Variable(tf.random_normal([10]))
i = tf.constant(0)
l = []
def cond(i,l):
return i < 10
def body(i,l):
temp = tf.gather(array,i)
l.append(temp)
return i+1,l
index,list_vals = tf.while_loop(cond, body, [i,l])
I want to process a tensor array in the similar way as described in the above code. In the body of the while loop I want to process the array by element by element basis to apply some function. For demonstration, I have given a small code snippet. However, it is giving an error message as follows.
ValueError: Number of inputs and outputs of body must match loop_vars: 1, 2
Any help in resolving this is appreciated.
Thanks
Citing the documentation:
loop_vars is a (possibly nested) tuple, namedtuple or list
of tensors that is passed to both cond and body
You cannot pass regular python array as a tensor. What you can do, is:
i = tf.constant(0)
l = tf.Variable([])
def body(i, l):
temp = tf.gather(array,i)
l = tf.concat([l, [temp]], 0)
return i+1, l
index, list_vals = tf.while_loop(cond, body, [i, l],
shape_invariants=[i.get_shape(),
tf.TensorShape([None])])
The shape invariants are there, because normally tf.while_loop expects the shapes of tensors inside while loop won't change.
sess = tf.Session()
sess.run(tf.global_variables_initializer())
sess.run(list_vals)
Out: array([-0.38367489, -1.76104736, 0.26266089, -2.74720812, 1.48196387,
-0.23357525, -1.07429159, -1.79547787, -0.74316853, 0.15982138],
dtype=float32)
TF offers a TensorArray to deal with such cases. From the doc,
Class wrapping dynamic-sized, per-time-step, write-once Tensor arrays.
This class is meant to be used with dynamic iteration primitives such as while_loop and map_fn. It supports gradient back-propagation via special "flow" control flow dependencies.
Here is an example,
import tensorflow as tf
array = tf.Variable(tf.random_normal([10]))
step = tf.constant(0)
output = tf.TensorArray(dtype=tf.float32, size=0, dynamic_size=True)
def cond(step, output):
return step < 10
def body(step, output):
output = output.write(step, tf.gather(array, step))
return step + 1, output
_, final_output = tf.while_loop(cond, body, loop_vars=[step, output])
final_output = final_output.stack()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print(sess.run(final_output))
def classify0(inX, dataSet, labels, k):
dataSetSize = dataset.shpe[0]
diffMat = tile(inX, (dataSetSize, 1)) - dataSet
sqDiffMat = diffMat ** 2
sqDistances = sqDiffMat.sum(axis = 1)
distances = sqDistances ** 0.5
sortedDistIndicies = distances.argsort()
classCount={}
for i in range(k):
voteIlabel = labels[sortedDistIndicies[i]]
classCount[voteIlabel] = classCount.get(voteIlabel, 0) + 1
sortedClassCount = sorted(classCount.interitems(),
key=operator.itemgetter(1), reverse=true)
return sortedClassCount[0][0]
above is my first coding using python. I imported it to terminal and it say 'IndentationError: unindent does not match any outer indentation level'. But I don't know how to fix it. please help me.
You code at:
sortedClassCount = sorted(classCount.interitems(),
key=operator.itemgetter(1), reverse=true)
return sortedClassCount[0][0]
seems to jump for no reason. Just remove all the extra spaces like so:
sortedClassCount = sorted(classCount.interitems(),
key=operator.itemgetter(1), reverse=true)
return sortedClassCount[0][0]
In python since there are no queerly brackets {} to signify starts and end of functions the interpreter uses indentation level to know the difference.
This is why your code doesn't work.
try deleting all indentation and putting it back because you might have mixed spaces with tabs or you have accidentally added another space to a place you shouldn't have one.
I have the following code:
import matplotlib.pyplot as plt
horas = [1,2,3,4]
diccionario = {(1,1,2,1):[2,3,4,5],
(1,2,2,2):[2,5,1,5],
(1,3,2,3):[2,5,5,5],
(1,4,2,4):[2,6,8,5],
(1,5,2,5):[2,7,5,5],
(1,6,2,6):[2,8,2,5],
(1,7,2,7):[2,9,6,5],
(1,8,2,8):[2,4,9,5]}
plt.figure()
i = 1
maximo = 0
keys = diccionario.keys()
for n in range(0,len(keys)-1,2):
gn, = plt.plot(horas,diccionario[keys[n]],'ro-')
gn1, = plt.plot(horas,diccionario[keys[n+1]],'g*-')
plt.subplot(len(keys)//2, 1,i)
plt.legend([gn,gn1], [keys[n],keys[n+1]])
i+=1
plt.show()
I expect to have 4 subplots with two lines each. I have them, but the last one is empty.
Could anyone explain why? I have tried many different ways without succeeding.
Put your subplot() before you plot gn and gn1. That will solve your problem.
for n in range(0, len(keys) - 1,2):
plt.subplot(len(keys)//2, 1, i)
gn, = plt.plot(horas, diccionario[keys[n]], 'ro-')
gn1, = plt.plot(horas, diccionario[keys[n+1]], 'g*-')
plt.legend([gn, gn1], [keys[n], keys[n+1]])
i+=1
By the way, I recommend to use tuple instead of dict. You may notice that the sequence of results is quite different from what you want.