I want to iteratively add to a Pyomo objective function based upon rows in a pandas dataframe. I've created variables based upon the size of the dataframe:
model.A = Var(range(0,len(arc_decvars)), initialize=0,within=Integers,bounds=(0,5))
where arc_decvars is a list in the same order as the corresponding pandas dataframe series.
What I would like to do is create an objective function like this, but without having to specify the indices explicitly in the code:
model.ObjFcn = ( df.coefficient[0] * model.A[0] * ( 1- df.coefficient[0] * model.A[0]) ) + ( df.coefficient[1] * model.A[1] * ( 1- df.coefficient[1] * model.A[1]) +...+ ( df.coefficient[999] * model.A[999] * ( 1- df.coefficient[999] * model.A[999]) )
Any idea on how to do this?
---- Followup:
I am adding a series of constraints like this, one for each node:
def NodeA(model):
k = sum(
model.ArcVar[i] * node_arc_matrix[i,0]
for i in model.ArcVar
)
return k == 2*model.NodeVar[1]
model.NodeAConstraint=Constraint(rule=NodeA)
Where the 0-index in node_arc_matrix refers to the column corresponding to Node_A. Instead of creating a constriant for each node manually can I create constraints across both i (arcs) and j (nodes)?
Try using Python list comprehension into the summation formulation.
Considering df.coefficient and model.A have arc_decvars as the domain and that arc_decvars is in a Pyomo Set, you can create your objective function this way:
model.ObjFcn = sum(
df.coefficient[i] * model.A[i] * ( 1- df.coefficient[i] * model.A[i])
for i in model.arc_devcars
)
Related
i am new to pymc3. I am trying to implement an unpooled model with 3 random variables (b0,b1,b2 ~ Normal) instead of one (b) with shape= 3. The code is from the book Bayesian Modeling and Computation in Python. I think it will help me implement a more complex multilevel model.
customers = sales_df.loc[:, "customers"].values
sales_observed = sales_df.loc[:, "sales"].values
food_category = pd.Categorical(sales_df["Food_Category"])
with pm.Model() as model_sales_unpooled:
s = pm.HalfNormal("s", 20, shape=3)
b = pm.Normal("b", mu=10, sigma=10, shape=3)
m = pm.Deterministic("m", b[food_category.codes] *customers)
sales = pm.Normal("sales", mu=m, sigma=s[food_category.codes],
observed=sales_observed)
I created this function in python using pandas dataframe, and I'd like to use it also in spark.
What I'm doing with this function is :
converting the df column to a list ( t1 )
converting the unique values of the column to a list ( t2 )
creating a list for each unique value of each feature ( t ). this list takes value 1 when the unique value is present in t1, 0 otherwise.
at the end the result is a dictionary with the unique values of each feature as key and as argument a list with value 1 when the key (the unique value) appears and 0 otherwise.
feat_list is just a list with all the column names.
def binary_dict(pandas_df, feat_list):
dict_feature = dict()
for col in feat_list:
t1 = pandas_df[col].tolist()
t2 = pandas_df[col].unique().tolist()
for value in t2:
t = []
for i in range (0, len(t1)):
if value == t1[i]:
t.append(1)
else:
t.append(0)
cc = str(col)
vv = "_" + str(value)
cv = cc + vv
dict_feature[cv] = t
return dict_feature
I tried using
t1 = df.select("col_name").rdd.flatMap(list).collect()
for creating t1 but it took over 20 minutes to create the list for a single column. I got something like 100 columns. Is there a way to convert this function to spark efficiently?
Thanks everyone for the answers!
PS: I'm using synapse analytics by azure/microsoft, Python 3.8 and pyspark 3.1.
I have two tables: Kanji, and Vocabulary. Imagine the kanji table looks like this:
目
一
人
And the vocabulary table looks like this:
目的
一番目
一人
二人
人々
注目
目標
一匹
I want to generate a table that finds all vocabulary which contains the kanji in the kanji table and list them together. So the end result would look like this:
人 一人
二人
人々
一 一人
一番目
一匹
目 目的
一番目
注目
目標
I'm not sure how to go about this. If I have just one kanji, I can use the QUERY function to generate all of the vocabularies which contain that one kanji. But can I create a dynamic table which essentially inner joins the "kanji" and "vocabulary" tables, looking for every instance of "vocabulary" contains "kanji"?
I tried using a QUERY to combine the two tables, but it won't work because the tables are mismatched in size:
=QUERY({C1:C296,D1:D224}, "SELECT Col2 WHERE Col1 contains Col2")
In the above example, the C column / Col2 is vocabulary, the D column / Col1 is kanji.
Is there a way to do this using Google Sheets?
The simplest workaround is to get intersections by 1 value:
=FILTER(D:D;REGEXMATCH(D:D;"目"))
The picture shows how to use the copy of this function to get all intersections
The other approach is to use big array-formula like:
=TRANSPOSE(SPLIT(TEXTJOIN(",";1;TRANSPOSE(ARRAYFORMULA(IF(REGEXMATCH(A1:A8;TRANSPOSE(B1:B3));A1:A8;))));","))
=ARRAYFORMULA(SORT(TRIM(SPLIT(TRANSPOSE(SPLIT(QUERY(TRANSPOSE(QUERY(TRANSPOSE(
IF(IFERROR(REGEXEXTRACT(IFERROR(REGEXEXTRACT(C1:C, REPT("(.)", LEN(C1:C)))),
TEXTJOIN("|", 1, A1:A)))<>"", "♦"&IFERROR(REGEXEXTRACT(IFERROR(
REGEXEXTRACT(C1:C, REPT("(.)", LEN(C1:C)))), TEXTJOIN("|", 1, A1:A)))&"♣"&C1:C, ))
,,999^99)),,999^99), "♦")), "♣"))))
seriously, much better way, just add a custom function:
/**
* inner join on equality
*
* #param {Range} items1 Table 1
* #param {Range} items2 Table 2
* #param {Integer} ix1 0-based index of key in Table1
* #param {Integer} ix2 0-based index of key in Table2
*
* #customfunction
* #author marc meyer (marqueymarc)
*/
function innerJoinEQ(items1 = [[]], items2 = [[]], ix1= 0, ix2= 0) {
var res = [];
var item2ix = 0;
var map = new Map();
items1 = Array.isArray(items1)? items1: [[items1]];
items2 = Array.isArray(items2)? items2: [[items2]];
items2.forEach((item2) => {
let entries = map.get(item2[ix2]) || [];
entries.push(item2);
map.set(item2[ix2], entries);
});
items1.forEach((item1) => {
let entries = map.get(item1[ix1]) || [];
entries.forEach((rightPart) => {
let cp = rightPart.slice();
cp.splice(ix2, 1);
res.push([...item1, ...cp]);
})
});
return res;
}
I have a query that returns a dynamic number of columns. I need to dynamically add the same amount of custom columns. I have successfully gotten this far. I'm stuck creating the formulas for the custom columns. This is what I have so far. (This is not the actual query, this is simplified)
Here is the Code:
Test = List.Accumulate(MyList, Source,
(state, current) => Table.AddColumn(
state, "A Temp" & Number.ToText(current), each [A1])
)
For now, I just added [A1] as a place holder for the formula. I need the formula to accumulate as follows:
A Temp1 = [A1] / [TOTAL]
A Temp2 = [A2] / [TOTAL]
A Temp3 = [A3] / [TOTAL]
Above is not actual code. Just what I need the formulas to do for each custom column.
Is this possible? I have tried everything I could think of. I'm using power query in excel BTW.
This isn't exactly what you asked for, but I think it will help.
Test = List.Accumulate(
List.Select(Table.ColumnNames(Source), each _ <> "TOTAL"),
Source,
(state, current) => Table.AddColumn(state,
"Temp " & current,
each Record.Field(_, current) / [TOTAL]))
It's not exactly what you asked for as it gives column names like Temp A1 instead of A Temp1.
I'm trying to do a simple select query with a subquery in the SELECT clause and have simply not found a way to do it. I've tried with both DQL and with the QueryBuilder, neither work. The code follows, please don't say I could just use a join, this is a simplified example just to illustrate the problem, I have legitimate use cases for subqueries.
// With QueryBuilder
$query = $qb->select(array('a',
'(SELECT at.addresstypeName
FROM e:Addresstype at
WHERE at.addresstypeId = a.addresstypeId
) AS addresstypeName'))
->from('e:Address', 'a')
->where('a.addressId = :addressId')
->setParameter('addressId', 1);
// With DQL
$dql = "SELECT a,
(SELECT at.addresstypeName
FROM e:Addresstype at
WHERE at.addresstypeId = a.addresstypeId
) AS addresstypeName
FROM e:Address a
WHERE a.addressId = :addressId";
$query = $em->createQuery($dql)->setParameter(':addressId', 1);
The following relationship is defined on the Address table:
/**
* #ORM\ManyToOne(targetEntity="Addresstype")
* #ORM\JoinColumns({
* #ORM\JoinColumn(name="addresstype_id", referencedColumnName="addresstype_id")
* })
*/
protected $addresstype;
In native SQL, the query would look like this:
SELECT
a.*,
(
SELECT at.addresstype_name
FROM addresstype at
WHERE at.addresstype_id = a.addresstype_id
) AS addresstype_name
FROM address a
WHERE a.address_id = 1
Any ideas?
$query = $qb->select('a')
->addSelect('(SELECT at.addresstypeName
FROM e:Addresstype at
WHERE at.addresstypeId = a.addresstypeId) AS addresstypeName'
)
->from('e:Address', 'a')
->where('a.addressId = :addressId')
->setParameter('addressId', 1);
For me subquery with doctrine works with this query :
$qb->select('e.field')
->addSelect('(SELECT count(mv.nm)
FROM Clt\Bundle\MyBundle\Entity\MV mv
LEFT JOIN Clt\Bundle\MyBundle\Entity\M ma WITH mv.nm=ma.nm
WHERE mv.ne=e.ne and ma.nm is null
) AS nm'
)
->from($this->_entityName, 'e')
->leftJoin('e.m', 'm')
->where($qb->expr()->eq('t.id'.$typeModule, $idElementModule));
Note that in the left join you must use WITH instead of ON...
I know this is an old question, but if you want, you could have used another query builder as your subquery:
$qb->select("a")
->addSelect("(" . $qb2->select("at.addresstypeName")
->from("e:Addresstype", "at")
->where("at.addresstypeId = a.addresstypeId")
->getDQL() . ") AS addresstypeName"
)
->from('e:Address', 'a')
->where('a.addressId = :addressId')
->setParameter('addressId', 1);
In my scenario what I needed was to look into a join and find an Id and use it as boolean, found 1 otherwise 0, then applying this to orderBy. DQL expressions worked only when combined with Where clause, which wasn't my case. So, a DQL subselect saved me.
Adapted more or less to your scenario, it would look like this:
// With QueryBuilder
// In AddressRepository
// Where one address may belong to several addressTypes
public function getWithType($addressType){
$qb = $this->createQueryBuilder('a1');
$qb->addSelect('a1.someField', 'a1.otherField')
$qb->addSelect(
'(SELECT at.addressTypeName
FROM App\Entity\Address a2
JOIN a2.addressType at
WHERE at.id = '.$addressType.' AND a2.id = a1.id
) AS addressTypeName')
//The rest part of the query
}