I have an optimization problem that I need to solve. It's a binary linear programming problem, so all of the decision variables are equal to 0 or 1. I need certain combinations of these decision variables to add up to either 0 or 2+, they cannot sum to 1. I'm struggling with how to accomplish this in PROC OPTMODEL.
Something like this is what I need:
con sum_con: x+y+z~=1;
Unfortunately, this just throws a syntax error... Is there any way to accomplish this?
See below for a linear reformulation. However, you may not need it. In SAS 9.4m2 (SAS/OR 13.2), your expression works as written. You just need to invoke the (experimental) CLP solver:
proc optmodel;
/* In SAS/OR 13.2 you can use your code directly.
Just invoke the experimental CLP solver */
var x binary, y binary, z binary;
con sum_con: x+y+z~=1;
solve with clp / findall;
print {i in 1 .. _NSOL_} x.sol[i]
{i in 1 .. _NSOL_} y.sol[i]
{i in 1 .. _NSOL_} z.sol[i];
produces immediately:
[1] x.SOL y.SOL z.SOL
1 0 0 0
2 0 1 1
3 1 0 1
4 1 1 0
5 1 1 1
In older versions of SAS/OR, you can still call PROC CLP directly,
which is not experimental.
The syntax for your example will be very similar to PROC OPTMODEL's.
I am sure, however, that your model has other variables and constraints.
In that case, remember that no matter how you formulate this,
it is still a search space with a hole in the middle.
So it potentially can make the solver perform poorly.
How poorly is hard to predict. It depends on other features of your model.
If MILP is a better fit for the rest of your model,
you can reformulate your constraint as a valid MILP in two steps.
First, add a binary variable that is zero only when the expression is zero:
/* If solve with CLP is not available, you can linearize the disjunction: */
var IsGTZero binary; /* 1 if any variable in the expression is 1 */
con IsGTZeroBoundsExpression: 3 * IsGTZero >= x + y + z;
Then add another constraint that forces the expression to be
at least the constant you want (in this case 2) when it is nonzero.
num atLeast init 2;
con ZeroOrAtLeast: x + y + z >= atLeast * IsGTZero;
min f=0; /* Explicit objectives are unnecessary in 13.2 */
solve;
The following equation should work:
(x+y-z)*z + (y+z-x)*x + (x+z-y)*y > -1
It can be generalized to more than three variables and if you have some large number you should be able to use index expansions to make it easier.
Related
In my Stata data set, the "alternative" variable consists of 4 modes including pier, private, beach and charter.
I want to generate new variable y as follows:
We collapse the model to three alternatives and order the alternatives, with y = 0 if fishing from a pier or beach, y = 1 if fishing from a private boat and y = 2 if fishing from a charter.
I tried to do this by looking at thetas in this website:
stata tips but I can't solve it.
Note: I don't understand from the dataset. And I get error related to type of the variable while generating variable I download the dataset from the website https://www.stata-press.com/data/musr/musr.zip The data name is mus15data
The variables in the dataset is as follows:
Here, "mode" variable is alternatives.
If I understand correctly, this is
gen y = 0 if inlist(1, dbeach, dpier)
* gen y = 0 if dbeach == 1 | dpier == 1
replace y = 1 if dprivate == 1
replace y = 2 if dcharter == 1
Many other solutions are possible. Here is one more.
gen y = cond(inlist(1, dbeach, pier), 0, 2 * (dcharter == 1) + (dprivate == 1))
If all those variables are only ever 0 or 1 (and never missing) some simplifications are possible.
Go only with code you find clear and can explain to others.
I am assuming that pier, beach, private, charter are mutually exclusive. I've not checked with the dataset.
Goal: perform rolling window calculations on panel data in Stata with variables PanelVar, TimeVar, and Var1, where the window can change within a loop over different window sizes.
Problem: no access to SSC for the packages that would take care of this (like rangestat)
I know that
by PanelVar: gen Var1_1 = Var1[_n]
produces a copy of Var1 in Var1_1. So I thought it would make sense to try
by PanelVar: gen Var1SumLag = sum(Var1[(_n-3)/_n])
to produce a rolling window calculation for _n-3 to _n for the whole variable. But it fails to produce the results I want, it just produces zeros.
You could use sum(Var1) - sum(Var1[_n-3]), but I also want to be able to make the rolling window left justified (summing future observations) as well as right justified (summing past observations).
Essentially I would like to replicate Python's ".rolling().agg()" functionality.
In Stata _n is the index of the current observation. The expression (_n - 3) / _n yields -2 when _n is 1 and increases slowly with _n but is always less than 1. As a subscript applied to extract values from observations of a variable it always yields missing values given an extra rule that Stata rounds down expressions so supplied. Hence it reduces to -2, -1 or 0: in each case it yields missing values when given as a subscript. Experiment will show you that given any numeric variable say numvar references to numvar[-2] or numvar[-1] or numvar[0] all yield missing values. Otherwise put, you seem to be hoping that the / yields a set of subscripts that return a sequence you can sum over, but that is a long way from what Stata will do in that context: the / is just interpreted as division. (The running sum of missings is always returned as 0, which is an expression of missings being ignored in that calculation: just as 2 + 3 + . + 4 is returned as 9 so also . + . + . + . is returned as 0.)
A fairly general way to do what you want is to use time series operators, and this is strongly preferable to subscripts as (1) doing the right thing with gaps (2) automatically working for panels too. Thus after a tsset or xtset
L0.numvar + L1.numvar + L2.numvar + L3.numvar
yields the sum of the current value and the three previous and
L0.numvar + F1.numvar + F2.numvar + F3.numvar
yields the sum of the current value and the three next. If any of these terms is missing, the sum will be too; a work-around for that is to return say
cond(missing(L3.numvar), 0, L3.numvar)
More general code will require some kind of loop.
Given a desire to loop over lags (negative) and leads (positive) some code might look like this, given a range of subscripts as local macros i <= j
* example i and j
local i = -3
local j = 0
gen double wanted = 0
forval k = `i'/`j' {
if `k' < 0 {
local k1 = -(`k')
replace wanted = wanted + L`k1'.numvar
}
else replace wanted = wanted + F`k'.numvar
}
Alternatively, use Mata.
EDIT There's a simpler method, to use tssmooth ma to get moving averages and then multiply up by the number of terms.
tssmooth ma wanted1=numvar, w(3 1)
tssmooth ma wanted2=numvar, w(0 1 3)
replace wanted1 = 4 * wanted1
replace wanted2 = 4 * wanted2
Note that in contrast to the method above tssmooth ma uses whatever is available at the beginning and end of each panel. So, the first moving average, the average of the first value and the three previous, is returned as just the first value at the beginning of each panel (when the three previous values are unknown).
I need to program a nearest neighbor algorithm in stata from scratch because my dataset does not allow me to use any of the available solutions (as far as I am concerned).
To be pecise. I have a dataset that is of similar structure to that of the following (original has around 14k observations)
input id value treatment match
1 0.14 0 .
2 0.32 0 .
3 0.465 1 2
4 0.878 1 2
5 0.912 1 2
6 0.001 1 1
end
I want to generate a variable called match (already included in the example above). For each observation with treatment == 1 the variable match should store the id of another observation from within treatment == 0 whose value is closest to value of the considered observation (treatment == 1).
I am new to stata programming, so I am not yet familiar with the syntax. My first shot is the following however it does not produce any changes to the match variable. I am sure this is a novice question but I am hoping for some advice on how to make the code running.
EDIT: I have changed the code slightly and now it seems to work. Do you see any problems that may arise if I run it on a bigger dataset?
set more off
clear all
input id pscore treatment
1 0.14 0
2 0.32 0
3 0.465 1
4 0.878 1
5 0.912 1
6 0.001 1
end
gen match = .
forval i = 1/`= _N' {
if treatment[`i'] == 1 {
local dist 1
forvalues j = 1/`= _N' {
if (treatment[`j'] == 0) {
local current_dist (pscore[`i'] - pscore[`j'])^2
if `dist' > `current_dist' {
local dist `current_dist' // update smallest distance
replace match = id[`j'] in `i' // write match
}
}
}
}
}
Consider some simulated data: 1,000 observations, 200 of them untreated (treat == 0) and the rest treated (treat == 1). Then the code included below will be much more efficient than the originally posted. (Ties, like in your code, are not explicitly handled.)
clear
set more off
*----- example data -----
set obs 1000
set seed 32956
gen id = _n
gen pscore = runiform()
gen treat = cond(_n <= 200, 0, 1)
*----- new method -----
timer clear
timer on 1
// get id of last non-treated and first treated
// (data is sorted by treat and ids are consecutive)
bysort treat (id): gen firsttreat = id[1]
local firstt = first[_N]
local lastnt = `firstt' - 1
// start loop
gen match = .
gen dif = .
quietly forvalues i = `firstt'/`=_N' {
// compute distances
replace dif = (pscore[`i'] - pscore)^2
summarize dif in 1/`lastnt', meanonly
// identify id of minimum-distance observation
replace match = . in 1/`lastnt'
replace match = id in 1/`lastnt' if dif == r(min)
summarize match in 1/`lastnt', meanonly
// save the minimum-distance id
replace match = r(max) in `i'
}
// clean variable and drop
replace match = . in 1/`lastnt'
drop dif firsttreat
timer off 1
tempfile first
save `first'
*----- your method -----
drop match
timer on 2
gen match = .
quietly forval i = 1/`= _N' {
if treat[`i'] == 1 {
local dist 1
forvalues j = 1/`= _N' {
if (treat[`j'] == 0) {
local current_dist (pscore[`i'] - pscore[`j'])^2
if `dist' > `current_dist' {
local dist `current_dist' // update smallest distance
replace match = id[`j'] in `i' // write match
}
}
}
}
}
timer off 2
tempfile second
save `second'
// check for equality of results
cf _all using `first'
// check times
timer list
The results in seconds to finish execution:
. timer list
1: 0.19 / 1 = 0.1930
2: 10.79 / 1 = 10.7900
The difference is huge, specially considering this data set has only 1,000 observations.
An interesting thing to notice is that as the number of non-treated cases increases relative to the number of treated, then the original method improves, but never reaches the levels of efficiency of the new method. As an example, invert the number of cases, so there is now 800 untreated and 200 treated (change data setup to gen treat = cond(_n <= 800, 0, 1)). The result is
. timer list
1: 0.07 / 1 = 0.0720
2: 4.45 / 1 = 4.4470
You can see that the new method also improves and is still much faster. In fact, the relative difference is still the same.
Another way to do this is using joinby or cross. The problem is they temporarily expand (a lot) the size of your data base. In many cases, they are not feasible due to the hard limit Stata has on the number of possible observations (see help limits). You can find an example of joinby here: https://stackoverflow.com/a/19784222/2077064.
Edit
If there's a large number of treated relative to untreated, your code suffers
because you go through the whole first loop many more times (due to the first if).
Furthermore, going through
that whole loop once, implies going through another loop that
has itself two if conditions, _N more times.
The opposite case in which there are few treated observations means that you go through the whole
first loop only in a small number of occasions, speeding up your code substantially.
The reason my code can maintain its efficiency is due to the use of in. This always
offers speed gains over if. Stata will go directly to those observations with no
logical checking needed. Your problem provides an opportunity for that replacement
and it's wise to seize it.
If my code used if where in is in place, the results would be different.
Your code would be faster for the
case in which there's a large number of untreated relative to treated, and again, that
is because in your code there would not be the need to go through the complete loop,
requiring very little work;
the first loop is short-circuited with the first if. For the opposite case,
my code would still dominate.
The key is to "separate" treated from untreated and work on each group using in.
I am using an ordinal independent variable in an OLS regression as a categorical variable using the factor variable technique in Stata (i.e, i.ordinal). The variable can take on values of the integers from 0 to 9, with 0 being the base category. I am interested in testing if the coefficient of each variable is greater (or less) than that which succeeds it (i.e. _b[1.ordinal] >= _b[2.ordinal], _b[2.ordinal] >= _b[3.ordinal], etc.). I've started with the following pseudocode based on FAQ: One-sided t-tests for coefficients:
foreach i in 1 2 3 5 6 7 8 {
test _b[`i'.ordinal] - _b[`i+'.ordinal] = 0
gen sign_`i'`i+' = sign(_b[`i'.ordinal] - _b[`i+'.ordinal])
display "Ho: i <= i+ p-value = " ttail(r(df_r), sign_`i'`i+'*sqrt(r(F)))
display "Ho: i >= i+ p-value = " 1-ttail(r(df_r), sign_`i'`i+'*sqrt(r(F)))
}
where I want the ```i+' to mean the next value of i in the sequence (so if i is 3 then ``i+' is 5). Is this even possible to do? Of course, if you have any cleaner suggestions to test the coefficients in this manner, please advise.
Note: The model only uses a sub-sample of my dataset for which there are no observations for 4.ordinal, which is why I use foreach instead of forvalues. If you have suggestions for developing a general code that can be used regardless of missing variables, please advise.
There are various ways to do this. Note that there is little obvious point to creating a new variable just to hold one constant. Code not tested.
forval i = 1/8 {
local j = `i' + 1
capture test _b[`i'.ordinal] - _b[`j'.ordinal] = 0
if _rc == 0 {
local sign = sign(_b[`i'.ordinal] - _b[`j'.ordinal])
display "Ho: `i' <= `j' p-value = " ttail(r(df_r), `sign' * sqrt(r(F)))
display "Ho: `i' >= `j' p-value = " 1-ttail(r(df_r), `sign' * sqrt(r(F)))
}
}
The capture should eat errors.
I've got something like the following:
proc means data = ... missing;
class 1 2 3 4 5;
var a b;
output sum=;
run;
This does what I want it to do, except for the fact that it is very difficult to differentiate between a missing value that represents a total, and a missing value that represents a missing value. For example, the following would appear in my output:
1 2 3 4 5 type sumA sumB
. . . . . 0 num num
. . . . . 1 num num
Ways I can think of handling this:
1) Change missings to a cardinal value prior to proc means. This is definitely doable...but not exactly clean.
2) Format the missings to something else prior, and then use preloadfmt? This is a bit of a pain...I'd really rather not.
3) Somehow use the proc means-generated variable type to determine whether the missing is a missing or a total
4) Other??
I feel like this is clearly a common enough problem that there must be a clean, easy way, and I just don't know what it is.
Option 3, for sure . Type is simply a binary number with 1 for each class variable, in order, that is included in the current row and 0 for each one that is missing. You can use the CHARTYPE option to ask for it to be given explicitly as a string ('01101110' etc.), or work with math if that's more your style.
How exactly you use this depends on what you're trying to accomplish. Rows that have a missing value on them will have a type that suggests a class variable should exist, but doesn't. So for example:
data want;
set have; *post-proc means assuming used CHARTYPE option;
array classvars a b c d e; *whatever they are;
hasmissing=0;
do _t = 1 to dim(classvars);
if char(_type_,_t) = 1 and classvars[_t] = . then hasmissing=1;
end;
*or;
if cmiss(of classvars[*]) = countc(_type_,'0') then hasmissing=0;
else hasmissing=1; *number of 0s = number of missings = total row, otherwise not;
run;
That's a brute force application, of course. You may also be able to identify it based on the number of missings, if you have a small number of types requested. For example, let's say you have 3 class variables (so 0 to 7 values for type), and you only asked for the 3 way combination (7, '111') and the 3 two way combination 'totals' (6,5,3, ie, '110','101','011'). Then:
data want;
set have;
if (_type_=7 and cmiss(of a b c) = 0) or (cmiss(of a b c) = 1) then ... ; *either base row or total row, no missings;
else ... ; *has at least one missing;
run;
Depending on your data, NMISS may also work. That checks to see if the number of missings is appropriate for the type of data.
Joe's strategy, modified slightly for my exact problem, because it may be useful to somebody at some point in the future.
data want;
set have;
array classvars a b c d e;
do _t = 1 to dim(classvars);
if char(_type_,_t) = 1 and (strip(classvars[_t] = "") or strip(classvars[_t]) = ".") then classvars[_t] = "TOTAL";
end;
run;
The rationale for the changes is as follows:
1) I'm working with (mostly) character variables, not numeric.
2) I'm not interested in whether a row has any missing or not, as those are very frequent, and I want to keep them. Instead, I just want the output to differentiate between the missings and the totals, which I have accomplished by renaming the instances of non-missing to something that indicates total.