I got stuck with a specific question in R around concatenating columns of a data frame by using a wildcard. Perhaps I am searching wrongly. However I could not find a matching answer yet.
Here is my question:
I have a data frame df where each column represents a user (U1, U2, U3), e.g.:
> df <-data.frame(U1=1:3, U2=4:6, U3=7:9)
> df
> U1 U2 U3
1 1 4 7
2 2 5 8
3 3 6 9
I would like to concatenate the values from all users into a single vector as one would do using the c() function, e.g.:
> c(df$U1, df$U2, df$U3)
[1] 1 2 3 4 5 6 7 8 9
However, my number of users is large and varies over time. So, I look for an elegant dynamic way of concatenating the columns such as
> c(df$U*)
Unfortunately this does not seem to work. I played around with grep and regular expressions but could not get it to work. For sure, I could use a for-loop and program my own cat function but I assume there is a better way. I just don't find it. Maybe I am just blind. Hope you can help.
sub_df <- df[, grep(pattern ='^U.*', names(df))]
stack(df)$values
Hope this works for you. You could first subset some columns according to your need.
Coerce the data frame to a matrix first:
as.vector(as.matrix(df))
Use the bracket [ to select columns whose names match a certain expression:
df[, grep("U.*", colnames(df)), drop = FALSE]
Related
I have a large data set in Stata.
There are several item batteries in this data set.
One item battery consists of 8 items (v1 - v8), each scaled from 1 to 7.
I want to code all items that take the value 1 in all items as missing values.
If v1 to v8 have the value "1", all rows to which this applies are to be replaced with missings.
I know how to code missing values with the if qualifier, but the selection with the complex condition causes me difficulties.
The code for R would probably solve this via rowSums, but I need the solution for Stata.
(I assume in R it would work like this:
df[rowSums(df[,c("v1", ... "v8")]!=1)==0, c("v1", .... "v8")] <- NA
But I need a solution for Stata.
If I understood this correctly, you want
egen rowall = concat(v1-v8)
mvdecode v1-v8 if rowall == 8 * "1", mv(1)
That is, all instances in v1-v8 of 1 are recoded as missing if and only if the values of those variables are all 1 in any observation.
I haven't been able to find an answer to this anywhere, so here it goes:
I have a pandas dataframe df like this:
X Y name
0 3 1 foo
1 5 2 fa
2 1 3 hoo
3 2 4 ha
I can easily find information in df by making conditions (for example df['X' >=3]), which is great. However, I want to make a more generic solution, where I can create a long condition in string form (e.g. '(X >= 3) & (name == foo)'), which could be split into an actual condition, that can be used in a pandas dataframe.
Can anyone suggest a smart solution (if something like this is possible) or redirect me to a similar discussion on the discussion board, where this topic has been debated?
It seems you need query for filtering:
df = df.query("X >= 3 & name == 'foo'")
I have got a data frame like this:
ID A B
1 x5.11 2,34
2 x5.57 5,36
3 x6,13 0,45
I would like to remove the 'x' of all values of the column A. How might I best accomplish this in R.
Thanks!
I have found a very easy way:
data.frama$A <- gsub("x", "", data.frame$A)
From this Stata FAQ, I know the answer to the first part of my question. But here I'd like to go a step further. Suppose I have the following data (already sorted by a variable not shown):
id v1
A 9
B 8
C 7
B 7
A 5
C 4
A 3
A 2
To calculate the minimum in this sequence, I do
generate minsofar = v1 if _n==1
replace minsofar = min(v1[_n-1], minsofar[_n-1]) if missing(minsofar)
To get
id v1 minsofar
A 9 9
B 8 9
C 7 8
B 7 7
A 5 7
C 4 5
A 3 4
A 2 3
Now I'd like to generate a variable, call it id_min that gives me the ID associated with minsofar, so something like
id v1 minsofar id_min
A 9 9 A
B 8 9 A
C 7 8 B
B 7 7 C
A 5 7 C
C 4 5 A
A 3 4 C
A 2 3 A
Note that C is associated with 7, because 7 is first associated with C in the current sorting. And just to be clear, my ID variable here shows as a string variable just for the sake of readability -- it's actually numeric.
Ideas?
EDIT:
I suppose
gen id_min = id if _n<=2
replace id_min = id[_n-1] if v1[_n-1]<minsofar[_n-1] & missing(id_min)
replace id_min = id_min[_n-1] if missing(id_min)
does the job at least for the data in this example. Don't know if it would work for more complex cases.
This works for your example. It uses the user-written command vlookup, which you can install running findit vlookup and following through the link that appears.
clear
set more off
input ///
str1 id v1
A 9
B 8
C 7
B 7
A 5
C 4
A 3
A 2
end
encode id, gen(id2)
order id2
drop id
list
*----- what you want -----
// your code
generate minsofar = v1 if _n==1
replace minsofar = min(v1[_n-1], minsofar[_n-1]) if missing(minsofar)
// save original sort
gen osort = _n
// group values of v1 but respecting original sort so values of
// id2 don't jump around
sort v1 osort
// set obs after first as missing so id2 is unique within v1
gen v2 = v1
by v1: replace v2 = . if _n > 1
// lookup
vlookup minsofar, gen(idmin) key(v2) value(id2)
// list
sort osort
drop osort v2
list, sep(0)
Your code has generate minsofar = v1 if _n==1 which is better coded as generate minsofar = v1 in 1, because it is more efficient.
Your minsofar variable is just a displaced copy of v1, so if this is always the case, there should be simpler ways of handling your problem. I suspect your problem is easier than you have acknowledged until now, and that has come through your post. Perhaps giving more context, expanded example data, etc. could get you better advice.
This is both easier and a little more challenging than implied so far. Given value (a little more evocative than the OP's v1) and a desire to keep track of minimum so far, that's for example
generate min_so_far = value[1]
replace min_so_far = value if value < min_so_far[_n-1] in 2/L
where the second statement exploits the unsurprising fact that Stata replaces in the current order of observations. [_n-1] is the index of the previous observation and in 2/L implies a loop over all observations from the second to the last.
Note that the OP's version is buggy: by always looking at the previous observation, the code never looks at the very last value and will overlook that if it is a new minimum. It may be that the OP really wants "minimum before now" but that is not what I understand by "minimum so far".
If we have missing values in value they will not enter the comparison in any malign way: missing is always regarded as arbitrarily large by Stata, so missings will be recorded if and only if no non-missings are present so far, which is as it should be.
The identifier of that minimum at first sight yields to the same logic
generate min_so_far = value[1]
gen id_min = id[1]
replace min_so_far = value if value < min_so_far[_n-1] in 2/L
replace id_min = id if value < min_so_far[_n-1] in 2/L
There are at least two twists that might bite. The OP mentions a possibility that the identifier might be missing so that we might have a new minimum but not know its identifier. The code just given will use a missing identifier, but if the desire is to keep separate track of the identifier of the minimum value with known identifiers, different code is needed.
A twist not mentioned to date is that observations with different identifier might all have the same minimum so far. The code above replaces the identifier only the first time a particular minimum is seen; if the desire is to record the identifier of the last occurrence the < in the last code line above should be replaced with <=. If the desire is to keep track of the all the identifiers of the minimum so far, then a string variable is needed to concatenate all the identifiers.
With a structure of panel or longitudinal data the whole thing is done under the aegis of by:.
I can't see a need to resort to user-written extensions here.
I just imported data from Excel. However, due to the collection process, certain entries are empty. For example, for Variable 1, my observations might be like:
1
[empty]
[empty]
2
[empty]
3
[empty]
[empty]
[empty]
I now want to fill these empty entries with the non-empty before it. Thus, using the example above, I want to produce something like:
1
1
1
2
2
3
3
3
3
As you can see, I want to fill the empty slots with the non-empty slot preceding it. I tried this block of code:
gen Judge1p = Judge1
forvalues x = 1/1395 {
if Judge1[`x'] != "" replace Judge1p[`x'] = Judge1[`x']
else replace Judge1p[`x'] = Judge1p[`x'-1]
}
However, Stata shows the error of "weights not allowed" and I did a little search online. It says that Stata doesn't allow users to put [`x'] before an equal sign. I'm a bit stuck here. What's the best way to solve the problem?
It's not clear what [empty] means. I assume they are missing.
Below is an example; no need for a loop.
clear all
set more off
*----- example data -----
input ///
var1
1
1
.
.
.
6
.
.
.
4
5
.
.
end
list
*----- what you want -----
gen var2 = var1
replace var2 = cond(missing(var2), var2[_n-1], var2)
list
Many things can be done in Stata without explicit loops. A reference is
Cox, Nicholas J. “Speaking Stata: How to Repeat Yourself without Going Mad.” Stata Journal 1, no. 1 (2001): 86–97.
Missings in Stata are represented as . for numerical type and (blank) for string type. If you need something more complicated, you'll have to make it clear in your post.
See help <command> for anything that is not clear.
Imputation of missings is a big topic in statistical analysis, so make sure you know what you're doing.