I want to use compute_gradients and generate local gradients. These gradients are to be averaged with multiple local gradients from other machines after which apply_gradients will be called. I am using 2 session.runs with a feed_dict in the second one that accepts gradients. Since apply_gradients expects a list of tuples, I am looking for an efficient way to do this.
This is how I am generating the list of tuples placeholder :
grads = cifar10.train_part1(loss, global_step)
xx = [tf.placeholder(tf.float32, shape=grads[0][0].shape) for i in range(10)]
yy = [tf.placeholder(tf.float32, shape=grads[0][0].shape) for i in range(10)]
xyz = zip(xx,yy)
train_op = cifar10.train_part2(loss,global_step, xyz)
I get the following error :
NotImplementedError: ('Trying to optimize unsupported type ', tf.Tensor 'Placeholder_10:0' shape=(5, 5, 3, 64) dtype=float32)
Related
I have a snippet of python code as a function here. The purpose of the code is to use the combinations tool of itertools and turn a list of pairs into a list of triplets by matching items that correlate with each other. Due to the time consuming nature of the function, I have been trying to get it to work using a GPU through numba. Here is the function followed by me calling it:
#jit(target_backend='cuda')
def combinator(size, theuniquegenes, thetuplelist):
limiter = size+1
lengths = [l + 1 for l in range(len(theuniquegenes)) if (l + 1 > 2) and (l + 1 < limiter)]
new_combs = {c for l in lengths for c in combinations(theuniquegenes, l)}
correlations = {x for x in new_combs if all([c in thetuplelist for c in combinations(x, 2)])}
print(len(correlations))
tuplelist = thetuplelist + list(correlations)
print(len(tuplelist))
return tuplelist
tuplelist = combinator(3, uniquegenes, tuplelist)
Unfortunately, I keep encountering the following error message:
numba.core.errors.UnsupportedError: Failed in object mode pipeline (step: inline calls to locally defined closures)
Use of unsupported opcode (SET_ADD) found
new_combs = {c for l in lengths for c in combinations(theuniquegenes, l)}
^
How can I rewrite this line to work?
I need to interpolate two variables written in the 3D grid on another 3D grid. I tried the inverse distance method, but I get only two values that do not represent the distribution on the original grid, assigned to each point of the new grid. Here is an example of my code:
text=text[pstart:pend]
x=[]
y=[]
z=[]
for line in text:
coords=line.split()
x.append(float(coords[2])) #coordinates of the new grid
y.append(float(coords[1]))
z.append(float(coords[0]))
Xg=np.asarray([x,y,z])
# Gather mean flow data
xd=[]
yd=[]
zd=[]
cd=[]
rhod=[]
with open(meanflowdata,'rb') as csvfile:
spamreader=csv.reader(csvfile, delimiter=',')
for row in spamreader:
if len(row)>2:
xd.append(float(row[0])) #coordinates and values of the source file
yd.append(float(row[1]))
zd.append(float(row[2]))
cd.append(float(row[3]))
rhod.append(float(row[4]))
Xd=np.asarray([xd,yd,zd])
Zd=np.asarray([cd,rhod])
leafsize = 20
print "# setting up KDtree"
invdisttree = Invdisttree( Xd.T, Zd.T, leafsize=leafsize, stat=1 )
print "# Performing interpolation"
interpol = invdisttree( Xg.T )
c=interpol.T[0]
rho=interpol.T[1]
As far as I could check, the problem lies when I call the invdisttree function, which does not work properly. Does someone have an idea or an alternative method to suggest for the interpolation?
Where do interpol.T[0], interpol.T[1] come from,
where did your Invdisttree come from ?
This
on SO has
invdisttree = Invdisttree( X, z ) -- data points, values
interpol = invdisttree( q, nnear=3, eps=0, p=1, weights=None, stat=0 )
In your case X could be 100 x 3, z 100 x 2,
query points q 10 x 3 ⟶ interpol 10 x 2.
(invdisttree is a function, which you call to do the interpolation:
interpol = invdisttree( q ...) . Is that confusing ?)
Assume I have the following matrix:
X = np.array([[1,2,3], [4,5,6], [7,8,9], [70,80,90], [45,43,68], [112,87,245]])
I want to draw a batch of 2 random rows at each time loop, and send it to a function. For instance, a batch in iteration i can be batch = [[4,5,6], [70,80,90]]
I do the following:
X = np.array([[1,2,3], [4,5,6], [7,8,9], [70,80,90], [45,43,68], [112,87,245]])
def caclulate_batch(batch):
pass
for i in range(X.shape[0]/2):
batch = np.array([])
for _ in range(2):
r = random.randint(0, 5)
batch = np.append(batch, X[r])
caclulate_batch(batch)
There are two problems here: (1) It returns appended array (2) The random number can be repeated which can choose the same row many times. How can modify the code to fit my requirement.
r = np.random.randint(0, len(x), 2) should get you the indices. That lets you use fancy indexing to get the subset: batch = x[r, :].
If you want to accumulate arrays along a new dimension, as your loop does, use np.stack or np.block instead of np.append.
(1) You can use numpy.stack instead of append. EDIT: But this function would be called when you have all your batch in a list like:
list = ([1,2], [3,4])
numpy.stack(list)
# gives [[1,2],
# [3,4]]
(2) You can shuffle X array, loop through the results and extract two by two. Look at numpy.random.shuffle
It would look like that:
S = np.random.shuffle(X)
for i in range(S.shape[0]/2):
batch = S[i*2:i*2+1]
caclulate_batch(batch)
I am working with a list of points in python 2.7 and running some interpolations on the data. My list has over 5000 points and I have some repeating "x" values within my list. These repeating "x" values have different corresponding "y" values. I want to get rid of these repeating points so that my interpolation function will work, because if there are repeating "x" values with different "y" values it runs an error because it does not satisfy the criteria of a function. Here is a simple example of what I am trying to do:
Input:
x = [1,1,3,4,5]
y = [10,20,30,40,50]
Output:
xy = [(1,10),(3,30),(4,40),(5,50)]
The interpolation function I am using is InterpolatedUnivariateSpline(x, y)
have a variable where you store the previous X value, if it is the same as the current value then skip the current value.
For example (pseudo code, you do the python),
int previousX = -1
foreach X
{
if(x == previousX)
{/*skip*/}
else
{
InterpolatedUnivariateSpline(x, y)
previousX = x /*store the x value that will be "previous" in next iteration
}
}
i am assuming you are already iterating so you dont need the actualy python code.
A bit late but if anyone is interested, here's a solution with numpy and pandas:
import pandas as pd
import numpy as np
x = [1,1,3,4,5]
y = [10,20,30,40,50]
#convert list into numpy arrays:
array_x, array_y = np.array(x), np.array(y)
# sort x and y by x value
order = np.argsort(array_x)
xsort, ysort = array_x[order], array_y[order]
#create a dataframe and add 2 columns for your x and y data:
df = pd.DataFrame()
df['xsort'] = xsort
df['ysort'] = ysort
#create new dataframe (mean) with no duplicate x values and corresponding mean values in all other cols:
mean = df.groupby('xsort').mean()
df_x = mean.index
df_y = mean['ysort']
# poly1d to create a polynomial line from coefficient inputs:
trend = np.polyfit(df_x, df_y, 14)
trendpoly = np.poly1d(trend)
# plot polyfit line:
plt.plot(df_x, trendpoly(df_x), linestyle=':', dashes=(6, 5), linewidth='0.8',
color=colour, zorder=9, figure=[name of figure])
Also, if you just use argsort() on the values in order of x, the interpolation should work even without the having to delete the duplicate x values. Trying on my own dataset:
polyfit on its own
sorting data in order of x first, then polyfit
sorting data, delete duplicates, then polyfit
... I get the same result twice
I have a list like df_all (see below).
A = matrix( ceiling(10*runif(8)), nrow=4)
colnames(A) = c("country", "year_var")
dfa = data.frame(A)
df1 = dfa[1,]
df2 = dfa[2,]
df3 = dfa[3,]
df4 = dfa[4,]
df_all = list(df1, df2, df3, df4)
df_all
Now I want to combine the list of interest by using variable a.
a <- "2,3,4"
b <- strsplit(a, ",")[[1]]
To combine this lists, I use the folling loop:
for (i in 1:length(b)){
c<-b[i]
aa <- df_all[c:c]
print(aa)
}
Now my question is, How can I combine this result and save this as as variable?
Thanks!
Would this work for you:
basnum<-as.integer(b)
do.call(rbind, df_all[basnum])
Through df_all[basnum], a list with only the relevant data.frames is created.
do.call takes a function and a list as parameters (and some more but not relevant right now). The items of the list are then passed on as parameters to the function.
So in this case, the above is the equivalent to calling:
rbind(df_all[[2]], df_all[[3]], df_all[[4]])
And this produces one data.frame holding all the rows of interest.