I am working with the CES diary data from 2006. I have a file which for each household has an entry for each item bought during a week long period. I have the following variables
newid id of household
cost dollar cost of item
ucc a code denoting the type of item
I am interested in restaurant expenditures which is covered by ucc 190111, 190112, ... . I want to collapse my data so for each newid I have the sum of restaurant expenditures for the household during the week. I used the command
collapse (sum) cost if ucc=="190111".... , by (newid)
However, I would like to have a zero when there are no restaurant expenditures and Stata simply removes those entries.
You need an intermediate variable with some zeros for non-restaurant expenditures:
gen rest_exp = cond(inlist(ucc,"190111","190112"),cost,0)
collapse (sum) rest_exp, by(newid)
One caveat is that inlist() has a constraint of 9 possible values for strings, but you probably have fewer than that or should destring, in which case the limit is 254. You can also hitch a few inlist()s together with |.
Related
How can I identify groups with few observations in panel-data models?
I estimated using xtlogit several random effects models. On average I have 26 obs per group but some groups only record 1 observation. I want to identify them and exclude them from the models... any suggestion how?
My panel data is set using: xtset countrycode year
Let's suppose your magic number for a big enough panel is 7 and that you fit a first model.
bysort countrycode : egen n_used = total(e(sample))
then gives you a count of how many observations were available and can be used, after which your criterion for a later model is if n_used >= 7
You could just go
bysort countrycode : gen n_available = _N
regardless of a model fit.
The differences are two-fold:
That last statement would disregard any missing values in the variables used in a model fit.
If you also used if and/or in to restrict model fit to particular subsets of observations, then e(sample) knows about that, but the last statement does not.
For non-longitudinal analysis using long-formatted data, when subjects have multiple visits or records, I will typically hunt down a record within each subject using bysort ID, and set a temporary variable to hold the integer or real value that I found, and then egen max() to find the max value for all records found, then set a final value in record _n==1 for that subject. This is so I can have the values I want from different visits percolate to a single record for each subject. Each single record per subject will then be used during analysis (but not longitudinal, maybe cross-sectional or regression, ANOVA, etc.)
Let's say I want the highest cholesterol (ldl) value for the 3rd year of a trial, where ldl is measured quarterly (every 3 months) for all subjects, which can be accomplished using the code below:
cap drop ldl3tmp
cap drop ldl3max
cap drop ldl3
bysort id (visitdate): gen ldl3tmp = ldl if trialyear==3
bysort id (visitdate): egen ldl3max = max(ldl3tmp)
bysort id (visitdate): gen ldl3 = ldl3max if _n==1
Suppose there are initials for the lab technician or phlebotomist that did the blood draw. How can I percolate a string value to record _n==1 that's associated with the greatest ldl value among the subset of records for the 3rd year of the trial? String values can't be sorted, so I am guessing the answer might be to eliminate records for which ldl is not the greatest value in year 3, then the string will be in that record?
In this case, how can I find out what _n is for the maximum value? If I know that, I could use
bysort id (visitdate): drop if _n!=6 //if _n==6 has the max value of ldl
Here is how to find the record number associated with the greatest ldl value within 4 quarterly ldl values in year 3 of a trial. The result is a variable called recmax, which will only be filled in for the specific record where the greatest value was found (among all records for each subject).
cap drop tmpldl3
cap drop maxldl3
cap drop recmax
cap drop visitdate
gen long visitdate = date(dateofvisit, "MDY") //You have to convert date ("MM/DD/YYYY") to a long integer format - based on #days since Jan 1, 1960
bysort id (visitdate): gen tmpldl3 = ldl if trialyear ==3
bysort id (visitdate): egen maxldl3 = max(tmpldl3)
bysort id (visitdate): gen recmax = _n if tmpldl3==maxldl3 & tmpldl3!=. & maxldl3!=.
You can then analyze all the other data (such as string data) in that record cross-sectionally (ANOVA, correlation, regression) by specifying if recmax!=. in the trailing if statement for any analysis command. If you are careful, you could also drop all other records with extraneous ldl values not of interest by using the command drop if recmax!=. providing you realize you dropped data and if you save, save to a filename with "_reduced" or "_dropped" in it.
Suppose I make the following chart showing the weight of 9 pigs over time:
webuse pig
tw line weight week if inrange(id,1,9), by(id) subtitle(, nospan)
Is it possible to reorder the panels by another variable while retaining the original label? I can imagine defining another variable that is sorted the right way and then labeling it with the right id, but curious if there is a less clunky way of achieving that.
I think you are right: you need a new ordering variable. Positively, you can order on any criterion of choice. Watch out for ties on the variable used to order, which can always broken by referring to the original identifier. Here we sort on final weights, by default smallest first. (For largest first, negate the weight variable.)
webuse pig, clear
keep if id <= 9
bysort id (week) : gen last = weight[_N]
egen newid = group(last id)
bysort newid : gen toshow = strofreal(id) + " (" + strofreal(last, "%2.1f") + ")"
* search labmask for download links
labmask newid , values(toshow)
set scheme s1color
line weight week, by(newid, note("")) sort xla(1/9)
Short papers discussing the principles here are already in train for publication in the Stata Journal in 2021.
To say that a dataset is (person, year) level means that each row of that dataset has different (person, year) like this:
person year wage
Mike 2000 10
Mike 2010 30
Jack 1990 20
How can I make Stata display exactly those (person, year) variable sets that uniquely define each row?
I want to make a log file to record
person year
only, but not display any individual information (displaying individuals' information in a log file is against the rules set by the data provider).
How could I do this?
What I thought about is using bysort in some way
bysort person year: gen num=_n
and if every num is 1, then it means (person, year) defines each row.
But if a dataset is extremely large, then checking whether every num is 1 is too tedious. Is there any smarter way?
The command isid checks whether the variables you supply do jointly specify observations uniquely. Here is an example you can try:
. webuse grunfeld, clear
. isid company
variable company does not uniquely identify the observations
r(459);
. isid company year
Note the principle: no news is good news.
Another way to check for problems is through duplicates. For example, try duplicates list person year. In your case, you don't want that in the log. But what you can do first is anonymise your persons through
egen id = group(person)
and then check for duplicates on id year.
See also this FAQ.
Cross-posting:
german: http://www.stata-forum.de/post1716.html#p1716
english: http://www.talkstats.com/showthread.php/47299-sales-growth-rate-with-multiple-groups-conditions
I want to calculate the annual sales growth rate of different firm-groups in Stata. The firms are grouped by variables country and industry.
I summed sales for each group (called it sales_total: sales of all firms in a group with equal country, industry and year):
bysort country year industry: egen sales_total = sum(sales)
I have a much bigger sample, but I tried to calculate the growth-rate with a smaller sample.
I tried multiple combinations such as:
egen group = group(year country industry)
xtset group year, yearly
bys group: g salesgrowth = log(D.sales_total)
or
bysort group: gen salesgrowth=(sales[_n]-sales[_n-1])/sales[_n-1]*
also with tsset.
and tried everything from this answer:
Generate percent change between annual observations in Stata?
but I always get error messages such as
repeated time values within panel
or
repeated time values within sample
due to the repetition of the number in a variable such as group.
Can you help me to find the yearly growth rate from each group (firms from same country & industry)?
update
here again an example of my observations (which normally have 10,000 firms over 10 years). There are also missing values (for sales, industry, year, country)
firms -- country -- year -- industry -- sales
-a --------usa-------1----------1----------300
-a---------usa-------2----------1--------4000
-b---------ger-------1----------1--------200
-b---------ger-------2----------1--------400
-c---------usa------1----------1----------100
-c---------usa------2----------1----------300
-d---------usa------1----------1----------400
-d---------usa------2----------1----------200
-e---------usa------1----------1----------7000
-e---------usa------2----------1----------900
-f----------ger------1----------2----------100
-f---------ger------2----------2----------700
-h---------ger------1----------2----------700
-h---------ger------2----------2----------600
-.................etc.....................................
I tried the programing you mentioned, but I got a couple of variables that need to be used in the same row and not in the same column (which I would probably need). Is there a possibility to keep the data without reshaping, keeping them in a row, for example grouping the observations:
egen group=group(industry year country)
and then try
xtset group year
bysort group: sales_growth = log(D.sales)
or
bysort group: gen sales_growth = (sales[_n]-sales[_n-1])/sales[_n-1]
Thank you!
The strategy here is trying to work at the wrong level of resolution. You should
collapse (sum) sales, by(country year industry)
and then work with that reduced dataset. Depending on what you want precisely, you will probably need to restructure that data with reshape so that different industries give different variables. Then
xtset country year
and growth rates will then be easier to calculate.