I am using Rapidminer Studio 6 and I want to replace values in dataset (or results, or series), lets say we have an attribute with values between 1 and 10, so I want to apply an operator which will replace values 1 to 4 and 8 to 10 by 0 so the new values will be 0's and numbers from 4 to 8. Say we have
2 4 1 5 7 9 -op-> 0 4 0 5 7 0.
Can someone tell me which operator to use, or subprocess?
(copied from original answer)
You can use the Generate Attributes operator for this with if in the parameters section.
If your attribute is called a2 and you want to change it to zero if its value is below 3 and above 5, the parameters to the Generate Attributes operator would look like this.
attribute name: a2
function expressions: if(a2<3,0,if(a2>5,0,a2))
Related
I have a data set of flows between locations, say they are 50 locations, but the number of pairs is not even because some locations do not have flows. I would like to create ids for each pair of observation (w_id and h_id)
Thank you.
Desired output
w_code h_code w_id h_id
295101011001003 291892204451015 1 1
295101011001003 295101011001003 1 2
295101011001003 291892202003011 1 3
295101011001025 295101021003001 2 1
295101011001025 295101011001025 2 2
295101011001026 291879507003038 3 1
295101011001026 190130007001013 3 2
295101011001026 295101105001027 3 3
295101011001026 291892126002008 3 4
295101011001026 291892126001005 3 5
295101011001029 291892199006006 4 1
295101011002007 295101011002015 5 1
295101011002014 295101011002016 6 1
295101011002014 295101011001003 6 2
295101011002016 295101011001007 7 1
295101011002030 295101255001008 8 1
Documentation accessible through Stata includes this paper on composite categorical variables and this paper on handling dyadic data. The Stata command search would have led to these papers, except that the art to finding as well as searching is thinking of the right keywords.
In your case the natural question arises whether for example the pair (1, 2) is really the same as (2, 1) and for flows my guess is No. In mathematics, abstraction is often the key to solving a problem; in statistical computing some concreteness may make a problem clearer. Perhaps h means husband and w means wife, and perhaps not. Assuming that (1, 2) and (2, 1) are quite different, a joint identifier is immediately obtained by
egen newid = group(w_id h_id)
and for a small number of identifiers -- you mention 50 -- there is no pain in asking for values to be labelled, so that with
egen newid = group(w_id h_id), label
the pair (1, 1) would be mapped to the value 1 and the value label 1 1.
As this solution was not immediately obvious, it is likely that a study of help egen will reveal a bunch of tools likely to be useful in data management; some are directly statistical.
For pairs of identifiers where Billy, Bob is to be treated like Bob, Billy see the second paper linked above. Whether this is true for the OP is a little unclear, but it is likely to be true for some others reading this in the future.
I got stuck with a specific question in R around concatenating columns of a data frame by using a wildcard. Perhaps I am searching wrongly. However I could not find a matching answer yet.
Here is my question:
I have a data frame df where each column represents a user (U1, U2, U3), e.g.:
> df <-data.frame(U1=1:3, U2=4:6, U3=7:9)
> df
> U1 U2 U3
1 1 4 7
2 2 5 8
3 3 6 9
I would like to concatenate the values from all users into a single vector as one would do using the c() function, e.g.:
> c(df$U1, df$U2, df$U3)
[1] 1 2 3 4 5 6 7 8 9
However, my number of users is large and varies over time. So, I look for an elegant dynamic way of concatenating the columns such as
> c(df$U*)
Unfortunately this does not seem to work. I played around with grep and regular expressions but could not get it to work. For sure, I could use a for-loop and program my own cat function but I assume there is a better way. I just don't find it. Maybe I am just blind. Hope you can help.
sub_df <- df[, grep(pattern ='^U.*', names(df))]
stack(df)$values
Hope this works for you. You could first subset some columns according to your need.
Coerce the data frame to a matrix first:
as.vector(as.matrix(df))
Use the bracket [ to select columns whose names match a certain expression:
df[, grep("U.*", colnames(df)), drop = FALSE]
From this Stata FAQ, I know the answer to the first part of my question. But here I'd like to go a step further. Suppose I have the following data (already sorted by a variable not shown):
id v1
A 9
B 8
C 7
B 7
A 5
C 4
A 3
A 2
To calculate the minimum in this sequence, I do
generate minsofar = v1 if _n==1
replace minsofar = min(v1[_n-1], minsofar[_n-1]) if missing(minsofar)
To get
id v1 minsofar
A 9 9
B 8 9
C 7 8
B 7 7
A 5 7
C 4 5
A 3 4
A 2 3
Now I'd like to generate a variable, call it id_min that gives me the ID associated with minsofar, so something like
id v1 minsofar id_min
A 9 9 A
B 8 9 A
C 7 8 B
B 7 7 C
A 5 7 C
C 4 5 A
A 3 4 C
A 2 3 A
Note that C is associated with 7, because 7 is first associated with C in the current sorting. And just to be clear, my ID variable here shows as a string variable just for the sake of readability -- it's actually numeric.
Ideas?
EDIT:
I suppose
gen id_min = id if _n<=2
replace id_min = id[_n-1] if v1[_n-1]<minsofar[_n-1] & missing(id_min)
replace id_min = id_min[_n-1] if missing(id_min)
does the job at least for the data in this example. Don't know if it would work for more complex cases.
This works for your example. It uses the user-written command vlookup, which you can install running findit vlookup and following through the link that appears.
clear
set more off
input ///
str1 id v1
A 9
B 8
C 7
B 7
A 5
C 4
A 3
A 2
end
encode id, gen(id2)
order id2
drop id
list
*----- what you want -----
// your code
generate minsofar = v1 if _n==1
replace minsofar = min(v1[_n-1], minsofar[_n-1]) if missing(minsofar)
// save original sort
gen osort = _n
// group values of v1 but respecting original sort so values of
// id2 don't jump around
sort v1 osort
// set obs after first as missing so id2 is unique within v1
gen v2 = v1
by v1: replace v2 = . if _n > 1
// lookup
vlookup minsofar, gen(idmin) key(v2) value(id2)
// list
sort osort
drop osort v2
list, sep(0)
Your code has generate minsofar = v1 if _n==1 which is better coded as generate minsofar = v1 in 1, because it is more efficient.
Your minsofar variable is just a displaced copy of v1, so if this is always the case, there should be simpler ways of handling your problem. I suspect your problem is easier than you have acknowledged until now, and that has come through your post. Perhaps giving more context, expanded example data, etc. could get you better advice.
This is both easier and a little more challenging than implied so far. Given value (a little more evocative than the OP's v1) and a desire to keep track of minimum so far, that's for example
generate min_so_far = value[1]
replace min_so_far = value if value < min_so_far[_n-1] in 2/L
where the second statement exploits the unsurprising fact that Stata replaces in the current order of observations. [_n-1] is the index of the previous observation and in 2/L implies a loop over all observations from the second to the last.
Note that the OP's version is buggy: by always looking at the previous observation, the code never looks at the very last value and will overlook that if it is a new minimum. It may be that the OP really wants "minimum before now" but that is not what I understand by "minimum so far".
If we have missing values in value they will not enter the comparison in any malign way: missing is always regarded as arbitrarily large by Stata, so missings will be recorded if and only if no non-missings are present so far, which is as it should be.
The identifier of that minimum at first sight yields to the same logic
generate min_so_far = value[1]
gen id_min = id[1]
replace min_so_far = value if value < min_so_far[_n-1] in 2/L
replace id_min = id if value < min_so_far[_n-1] in 2/L
There are at least two twists that might bite. The OP mentions a possibility that the identifier might be missing so that we might have a new minimum but not know its identifier. The code just given will use a missing identifier, but if the desire is to keep separate track of the identifier of the minimum value with known identifiers, different code is needed.
A twist not mentioned to date is that observations with different identifier might all have the same minimum so far. The code above replaces the identifier only the first time a particular minimum is seen; if the desire is to record the identifier of the last occurrence the < in the last code line above should be replaced with <=. If the desire is to keep track of the all the identifiers of the minimum so far, then a string variable is needed to concatenate all the identifiers.
With a structure of panel or longitudinal data the whole thing is done under the aegis of by:.
I can't see a need to resort to user-written extensions here.
I'm trying to make a simple "fill in the blanks" type of exam in django and would like to know what is the best way to design the database.
Example: "9 is the sum of 4 and 5, or 3 and 6."
During the exam, the above sentence would appear as "__ is the sum of __ and _, or _ and __."
Obviously there are unlimited number of answers to this question, but assume that the above numbers are the only answers. But the catch is that you can switch the places of 4 and 5, or the places of 3 and 6 and still get the right answer. Besides, the number of blanks is not known, so it can be 1 or more.
I would go with something like. First define a Question table:
Question
--------------------------
Id Text
1 9 is the sum of 4 and 5, or 3 and 6
...
Then save the position of the hidden substrings, let's call them fields, in another table:
QuestionField
--------------------------
Id QuestionId StartsAt EndsAt Set
1 1 0 1 1
2 1 16 17 2
3 1 22 23 2 # NOTE: Is in the same set as QuestionField #2
...
This table lets you retrieve the actual value of the field by querying the Question table (e.g. entry one refers to the value '9' in the first question).
The "Set" column contains an identifier of the "set" in which this field is, where fields in the same set can be replaced by each other. When you populate it, you would have to ensure that all questions that can be replaced by each other are in the same set. The actual number of the set doesn't matter, as long as it's unique. But it makes sense to have it equal to the ID of one of the elements of the set.
I have a query that's basically "count all the items of type X, and return the items that exist more than once, along with their counts". Right now I have this:
Item.objects.annotate(type_count=models.Count("type")).filter(type_count__gt=1).order_by("-type_count")
but it returns nothing (the count is 1 for all items). What am I doing wrong?
Ideally, it should get the following:
Type
----
1
1
2
3
3
3
and return:
Type, Count
-----------
1 2
3 3
In order to count the number of occurrences of each type, you have to group by the type field. In Django this is done by using values to get just that field. So, this should work:
Item.objects.values('group').annotate(
type_count=models.Count("type")
).filter(type_count__gt=1).order_by("-type_count")
It's logical error ;)
type_count__gt=1 means type_count > 1 so if the count == 1 it won't be displayed :)
use type_count__gte=1 instead - it means type_count >= 1 :)