Related
I have this following code and a text file with 5 (X and Y) values The Image of the text file is here. I need to iterate 1000 times for every X and Y value. How can I achieve this?
import pandas as pd
data = pd.read_csv("test.txt", delim_whitespace=True, skipinitialspace=True,)
for every line in the text document:
for i in range(1, 1001, 1):
z = data["X"] + data["Y"]
z = z + 10
print z
The text file is like
X Y
1 10
2 20
3 30
4 40
5 50
The output must be:
10011
10022
10033
10044
10055
You can select one row at the time using .loc. Please read this documentation to fully understand how this work. Here is your data:
import pandas as pd
df = pd.DataFrame({'X':['1','2','3','4','5'], 'Y': ['10','20','30','40','50']})
This code
print df.loc[0]
will give you the first row (with index=0) as a pandas series (pd.Series), which is essentially like a dataframe with one column only: a vector.
X 1
Y 10
Name: 0, dtype: object
If you want the second row then: df.loc[1] and so on...
If you want to iterate one row at the time, you can select each row in the first for loop and perform your operations 1000 times in the second for loop:
for ix in df.index: # df.index gives [0,1,2,3,4]
for i in xrange(0,1000):
ser = df.loc[ix]
print ser['X'] + ser['Y'] + '10'
Try this,
data = pd.DataFrame({'X': [1, 2, 3, 4, 5], 'Y': [10,20,30,40,50]})
for each_line in data.index:
z = data['X'].loc[each_line] + data['Y'].loc[each_line]
for i in range(1,1001,1):
z +=10
print(z)
Output
10011
10022
10033
10044
10055
if you want to add new column to dataFrame:
data["total"] = sorted(set([(data.loc[ix]['X'].astype(int) + data.loc[ix]['Y'].astype(int)).astype(str) +"10" for ix in data.index for i in range(1,1001)]))
if you want concatenate the 'X' and 'Y' +'10' then:
[data.loc[ix]['X'].astype(str) + data.loc[ix]['Y'].astype(str) +"10" for ix in data.index for i in range(1,1001)]
And if you want to sum of 'X' + 'Y' and concat + '10' then:
final_data = [(data.loc[ix]['X'].astype(int) + data.loc[ix]['Y'].astype(int)).astype(str) +"10" for ix in data.index for i in range(1,1001)]
I want to make the values I plot to be compatible with the colorbar values. How can I do this? See more details below.
y1, y2, y3 values are : [-1.7 -1.62 -1.53 -1.43 -1.32 -1.2 -1.09 -0.97 -0.85],
[-1.43 -1.28 -1.09 -0.88 -0.66 -0.44 -0.21 0.03 0.27], [-3.65 -3.58 -3.48 -3.38 -3.27 -3.16 -3.04 -2.92 -2.8 ]
import matplotlib.pyplot as plt
import numpy as np
#plot
fig = plt.figure(figsize=(9.6,6), dpi=300, linewidth=3.0)
ax = fig.add_subplot(311)
y1 = y.transpose() #y should be the first data I gave out in the beginning
gradient = [ y1,y1 ]
plt.imshow(gradient, aspect='auto', cmap=plt.get_cmap('hot'))
ax.set_xticklabels(data[list[1]])
ax2 = fig.add_subplot(312)
y2 = y.transpose() #y should be the second data I gave out in the beginning
gradient = [ y2,y2 ]
plt.imshow(gradient, aspect='auto', cmap=plt.get_cmap('hot'))
ax2.set_xticklabels(data[list[5]])
ax3 = fig.add_subplot(313)
y3 = y.transpose() #y should be the third data I gave out in the beginning
gradient = [ y3,y3 ]
plt.imshow(gradient, aspect='auto', cmap=plt.get_cmap('hot'))
ax3.set_xticklabels(data[list[9]])
sm = plt.cm.ScalarMappable(cmap=plt.get_cmap('hot'),norm=plt.Normalize(-6.39,9.29))
sm._A = []
plt.colorbar(sm,ax=[ax,ax2,ax3])
#fig.set_tight_layout(True) #how can I use this? If I use this it generates a warning and the plot overlaps
plt.savefig('CuxOxi.png',dpi=300,format='png', orientation='landscape')
As you can see from the graph, the colorbar ranges from -6.39 to 9.29. Each subplot ranges only a fraction of the complete colorbar range. How can I make for example -1.62 to -1.2 to have the color range as defined in the colorbar (which is mostly red)
In each plot, you can add the vmin and vmax options to the plt.imshow function in order to set the color scale min and max for that plot. You can define these to be the same for each plot so that they all have the same scale.
vmin = -6
vmax = 9
plt.imshow(gradient, aspect='auto', cmap=plt.get_cmap('hot'),vmin=vmin,vmax=vmax)
I have an expression given below and i was wondering if you can help me to formalize as an ILP constraint in order to solve by Gurobi optimizer (Python):
forall ( y in Y), forall (j in M), forall (x in X):
IF r[x][y] = 1 and c[y,j] = 1 THEN p[x,a] = 1 , forall (a in {U[j],...,W[j] - 1} )
Where:
r[x][y], c[y,j] and p[x,a] are 3 binary variables;
U[j] and W[j] are 2 positive integer variables, where U[j] + beta = W[j]
(beta is a positive constant)
I know that this constraint can be written as a logical implication in conjunctive normal form: x ∧ y → z
I have already tried this solution: z≥x+y−1 together with several other possibilities :(
But, i had an error with Gurobi solver
My Python code for this constraint is as follows:
for y in Y:
for j in M:
for x in X:
for a in range(int(U[j]),int(W[j])):
M1.addConstr(r[x][y] + c[y,j] - 1 <= p[x,a], 'TileRequirement_%s_%s_%s_%s'%(y,j,x,a))
I always get the error in this line: for a in range(int(U[j]),int(W[j])):, because both U[j] and W[j] are defined as positive integer variables
So, can someone help me ?
Thanks :)
Best regards
Khadija
You can't build constraints based on yet-to-optimize variables like in:
for a in range(int(U[j]),int(W[j])) # optimized value unknown # build-constr-time
Casting like that looks also dangerous and it solely depends on gurobipy, if that's possible in general (but not helping here).
Your question is hard to read and there is no information about the motivation for these constraints, but the general idea could be:
get rid of the range defined by U[j] and W[j]
formulate your constraint for the full-range
with one modification:
introduce one more activating-variable a:
(x^y)->z becomes: (a^x^y)->z == !a v !x v !y v z
as linear expression: (1-a) + (1-x) + (1-y) + z >= 1
now use the concept of indicator-variables to formulate your activating-variables
Yes, it's messy and because of this (and because information is sparse) i won't post a full solution.
# -*- coding: utf-8 -*-
#programmer: Khadija HS
#date: June 2017
#name: B-C-MCT PLNE Model (Khadija.HS,2017) <---> BCMCT1.py
"""
Solve the B-C-MCTP (fixed Z & min Delta) sub-pb of 3-PSDPP (K-HS et al.,2016), where:
X: list of input tiles (tile x)
Y: list of output tiles (tile y)
Ry: requirement relation between X & Y
<---> is a List of Y list, each sub List define the input tiles required by each y
<---> rxy: incidence matrix (0-1): Input Tiles/Output Tiles (Configuration of each output tile % input tile)
<---> Ry is a list of list where Row <--- x & Column <--- y
alpha: prefetches times (uniform)
beta: computations times (uniform)
Delta: the Total completion time (to be determined)
"""
from gurobipy import *
""" Find Yx: set of output tiles y that required the input tile x """
def OuputTileTe(Ry,X):
Yx=[]
for x in X:
Yx.append(OuputTileTex(Ry,x))
return Yx
""" Find B: List Ts for x """
def OuputTileTex(Ry,x):
B=[]
for y in range(len(Ry)):
if x in Ry[y]:
B.append(y)
return B
""" Find N: Max Value of N ( <---> sum(len(Ry),y in Y)) """
def NbPrefetchTile(S):
N=0
for k in range(0,len(S)):
N += len(S[k])
return N
""" BCMCT1 - Model"""
def BCMCT1(X,Y,Z,Ry,alpha,beta):
# DET VBLES: M,N,Z1,T,K,L
M=list(range(len(Y))) # List of Computation steps
nb=NbPrefetchTile(Ry) # Number of prefetches (Big Value of N)
N=range(nb) # List of Prefetches steps
ListZ=range(Z) # List of Buffers
T=range(alpha*len(X) + beta*len(Y)) # List of Start Date times (Computation+Prefetches)
K=range(alpha) # Interval Time of a prefetch step
L=range(beta) # Interval Time of a compute step
# DET VBLES: A,T1,B,Yx
A=alpha*nb + beta*len(Y) # Big Value of Total Completion Time
T1=range(A) # List of Start Date times (Computation+Prefetches)
minLen=min([len(elt) for elt in Ry]) #1,alpha+1
B=alpha*minLen + beta*len(Y) # Value of LB2
Yx=OuputTileTe(Ry,X) # List of output tile y, for x, x in X
# MODEL
M1=Model("BCMCT1")
# CONSTANT VARIABLES
r=[[0]*len(Y) for i in range(len(X))]
for x in X:
for y in Y:
if x in Ry[Y.index(y)]:
r[x][y]=1
# DECISION VARIABLES
c,p,q,U,W,a={},{},{},{},{},{}
for y in Y:
for j in M:
c[y,j]=M1.addVar(vtype=GRB.BINARY,name="c[%s,%s]"%(y,j)) #obj=beta,
for x in X:
for t in T:
p[x,t]=M1.addVar(vtype=GRB.BINARY,name="p[%s,%s]"%(x,t)) #obj=1,
for x in X:
for t in T:
q[x,t]=M1.addVar(vtype=GRB.BINARY,name="q[%s,%s]"%(x,t)) #obj=1,
for j in M:
U[j]=M1.addVar(vtype='I',name="U_%s"%j)
W[j]=M1.addVar(obj=1,vtype='I',name="W_%s"%j)
for j in M:
a[j]=M1.addVar(vtype=GRB.BINARY,name="a[%s]"%j)
# MODEL UPDATE
M1.update()
# OBJECTIVE
Obj=W[len(M)-1]
M1.setObjective(Obj, GRB.MINIMIZE)
# CONSTRAINTS
""" (1): Computation's Assignement Constraints """
""" (a) """
for j in M:
M1.addConstr(quicksum(c[y,j] for y in Y)==1,'ComputeAssign1_%s'%j)
""" (b) """
for y in Y:
M1.addConstr(quicksum(c[y,j] for j in M)==1,'ComputeAssign2_%s'%y)
""" (2): Buffer's Constraints """
for t in T:
M1.addConstr(quicksum(p[x,t] for x in X) <= Z,'BufferNb_%s'%t)
""" 3): Computation/Prefetch's Constraints """
""" (a) """
for t in T:
M1.addConstr(quicksum(q[x,t] for x in X) <= 1,'PrefetchTileA_%s'%t)
""" (b) """
for x in X:
for t in T[1:]:
for k in K:
M1.addConstr(p[x,t] - p[x,t-1] <= q[x,t-k],'PrefetchTileB_%s_%s_%s'%(x,t,k))
""" (c) """
for y in Y:
for j in M:
for x in X:
for t in T:
M1.addConstr(3 - r[x][y] - c[y,j] - a[j] + p[x,t] >= 1, 'TileRequirement_%s_%s_%s_%s'%(y,j,x,t))
""" (5): Computation Time's Constraint """
""" (a) """
for j in M:
M1.addConstr(W[j] == U[j] + beta,'ComputeTime1_%s'%j)
""" (b) """
for j in M[1:]:
M1.addConstr(W[j-1] <= U[j],'ComputeTime2_%s'%j)
# SOLUTION
M1.__data=c,p,q,U,W,a
return M1
Please find attached my detailed ILP
May be, it will be easier to understand my question about constraint number 17
where,
L = range(beta)
K=range(alpha)
\Lambda (Big M)=alpha*Z*Y+beta*Y
r[x][y]= 1 if x in Ry and 0 otherwise (forall x in X & forall y in Y) : incidence matrix given as input data
I will give you an example that is very simple to understand more my problem as follows:
let:
X=[X1,X2,X3,X4]
Y=[Y1,Y2,Y3]
Ry=[(X1,X2,X3), (X2,X4),(X1,X3,X4)]
Z=3
alpha=2, beta=4
The objective is to find a computation sequence for computing Y1,Y2 & Y3 in order to minimize Delta (total completion time)
An optimal solution is: Y2, Y3, Y1 (or Y2,Y1,Y3) with \Delta=17
ILP Formulation
I have a point cloud in 4 dimensions, where each point in the cloud has a location and a value (x,y,z,Value). In addition, I have a 'special' point, S0, within the 3d point cloud; I've used this example to find the closest 10 points in the cloud, relative to S0. Now, I have a numpy array for each of the 10 closest points and their values. How can I interpolate these 10 points, to find the interpolated value at point S0? Example code is shown below:
import numpy as np
import matplotlib.pyplot as plt
numpoints = 20
linexs = 320
lineys = 40
linezs = 60
linexe = 20
lineye = 20
lineze = 0
# Create vectors of points
xpts = np.linspace(linexs, linexe, numpoints)
ypts = np.linspace(lineys, lineye, numpoints)
zpts = np.linspace(linezs, lineze, numpoints)
lin = np.dstack((xpts,ypts,zpts))
# Image line of points
fig = plt.figure()
ax = fig.add_subplot(211, projection='3d')
ax.set_xlim(0,365); ax.set_ylim(-85, 85); ax.set_zlim(0, 100)
ax.plot_wireframe(xpts, ypts, zpts)
ax.view_init(elev=12, azim=78)
def randrange(n, vmin, vmax):
return (vmax - vmin)*np.random.rand(n) + vmin
n = 10
for n in range(21):
xs = randrange(n, 0, 350)
ys = randrange(n, -75, 75)
zs = randrange(n, 0, 100)
ax.scatter(xs, ys, zs)
dat = np.dstack((xs,ys,zs))
ax.set_xlabel('X Label')
ax.set_xlim(0,350)
ax.set_ylabel('Y Label')
ax.set_ylim(-75,75)
ax.set_zlabel('Z Label')
ax.set_zlim(0,100)
ax = fig.add_subplot(212, projection='3d')
ax.set_xlim(0,365); ax.set_ylim(-85, 85); ax.set_zlim(0, 100)
ax.plot_wireframe(xpts,ypts,zpts)
ax.view_init(elev=12, azim=78)
plt.show()
dist = []
# Calculate distance from first point to all other points in cloud
for l in range(len(xpts)):
aaa = lin[0][0]-dat
dist.append(np.sqrt(aaa[0][l][0]**2+aaa[0][l][1]**2+aaa[0][l][2]**2))
full = np.dstack((dat,dist))
aaa = full[0][full[0][:,3].argsort()]
print(aaa[0:10])
A basic example. Note that the meshgrid is not needed for the interpolation, but only to make a fast ufunc to generate an example function A=f(x,y,z), here A=x+y+z.
from scipy.interpolate import interpn
import numpy as np
#make up a regular 3d grid
X=np.linspace(-5,5,11)
Y=np.linspace(-5,5,11)
Z=np.linspace(-5,5,11)
xv,yv,zv = np.meshgrid(X,Y,Z)
# make up a function
# see http://docs.scipy.org/doc/numpy/reference/ufuncs.html
A = np.add(xv,np.add(yv,zv))
#this one is easy enough for us to know what to expect at (.5,.5,.5)
# usage : interpn(points, values, xi, method='linear', bounds_error=True, fill_value=nan)
interpn((X,Y,Z),A,[0.5,0.5,0.5])
Output:
array([ 1.5])
If you pass in an array of points of interest, it will give you multiple answers.
I have a pandas dataframe that resembles one generated as follows.
import numpy as np
import pandas as pd
x0 = pd.DataFrame(np.random.normal(size=(10, 4)))
x1 = pd.DataFrame({'x': [1,1,2,3,2,3,4,1,2,3]})
df = pd.concat((x0, x1), axis=1)
and a function:
def fun(df, n=100):
z = np.random.normal(size=n)
return np.dot(df[[0,1,2,3]], [0.5*z,-1*z,0.3*z,1.2*z])
I would like to:
use identical draws z for each unique value in x,
take the product of the output in the above step over items of unique x
Any suggestion?
Explanation:
Generate n=100 draws to get z such that len(z)=100
For each elem in z, evaluate the function fun,
For i in df.x.unique(), compute the product of the output in step (2) element-wise. I am expecting to get a DataFrame or array of dimension (len(df.x.unique(), n=100)
4.
It sounds like you want to group by 'x', taking one of its instances (let's assume we take the first one observed).
just call your function as follows:
f = fun(df.groupby('x').first())
>>> f.shape
Out[25]: (4, 100)
>>> len(df.x.unique()
Out[26]: 4