I am currently trying to create a loop to deseasonalize several hundred time series which consist of search queries, economic-related words taken from dictionaries. The (working) command my deseasonalization is based on looks like this
sum ACCRUE, meanonly
local mACCRUE =r(mean)
reg ACCRUE January February March April May June July August September October November December, nocons
predict double ACCRUESA, residual
replace ACCRUESA=ACCRUESA+`mACCRUE'
So in the the end I created a new deseasonalized time series called ACCRUESA from the base time series ACCRUE.
In the next step I want to automatize the command for the rest of the queries. I tried the following
foreach var of varlist a-z {
sum `var', meanonly
local mu =r(mean)
reg `var' January February March April May June July August September October November December, nocons
predict double `var'SA, residual
replace `var'SA=`var'SA+`mu'
}
I use a-z to loop through the queries, but maybe this is the wrong approach. My goal was to exclude the monthly dummies. Anyway, after excecuting I get an error that a variable is unknown. You will observe that I tried to create a new variable using the `var' and a suffix, but I am not sure if this approach is feasible.
Does someone have an idea how to improve my command?
The problem seems to be the variable list you give for the loop: a-z. I initially suggested you use _all, instead.
#NickCox correctly pointed out that _all would include undesired variables in the <varlist> (i.e. the months). You can remove those from the <varlist>. Below an example.
clear all
set more off
*------------------- Create example data -----------------------
sysuse auto
foreach var in `=c(Months)' {
gen `var' = 0
}
*------------------ Remove some variables ----------------------
* All variables
ds
local allvars = r(varlist)
display "`allvars'"
* Strings to remove
local removethis = c(Months)
* modified local (no months)
local myvars: list allvars - removethis
display "`myvars'"
*-------------------------- Process ----------------------------
foreach var of varlist `myvars' {
display "`var'"
sum `var', meanonly
display r(mean)
}
This involves using macro lists. Type help macrolists for details.
Related
I want to match treated firms to control firms by industry and year considering firms that are the closest in terms of profitability (roa). I want a 1:1 match. I am using a distance measure (mahalanobis).
I have 530,000 firm-year observations in my sample, namely 267,000 treated observations and 263,000 control observations approximatively. Here is my code:
gen neighbor1 = .
gen idobs = .
levelsof industry
local a = r(levels)
levelsof year
local b = r(levels)
foreach i in `a' {
foreach j in `b'{
capture noisily psmatch2 treat if industry == `i' & year == `j', mahalanobis(roa)
capture noisily replace neighbor1 = _n1 if industry == `i' & year == `j'
capture noisily replace idobs = _id if industry == `i' & year == `j'
drop _treated _support _weight _id _n1 _nn
}
}
Treat is my treatment variable. It takes the value of 1 for treated observations and 0 for non-treated observations.
The command psmatch2 creates the variable _n1 and _id among others. _n1 is the id number of the matched observation (closest neighbor) and _id is an id number (1 - 530,000) that is unique to each observation.
The code 'works', i.e. I get no error message. My variable neighbor1 has 290,724 non-missing observations.
However, these 290,724 observations vary between 1 and 933 which is odd. The variable neighbor1 should provide me the observation id number of the matched observation, which can vary between 1 and 530,000.
It seems that the code erases or ignores the result of the matching process in different subgroups. What am I doing wrong?
Edit:
I found a public dataset and adapted my previous code so that you can run my code with this dataset and see more clearly what the problem could be.
I am using Vella and Verbeek (1998) panel data on 545 men worked every year from 1980-1987 from this website: https://www.stata.com/texts/eacsap/
Let's say that I want to match treated observations, i.e. people, to control observations by marriage status (married) and year considering people that worked a similar number of hours (hours), i.e. the shortest distance.
I create a random treatment variable (treat) for the sake of this example.
use http://www.stata.com/data/jwooldridge/eacsap/wagepan.dta
gen treat = round(runiform())
gen neighbor1 = .
gen idobs = .
levelsof married
local a = r(levels)
levelsof year
local b = r(levels)
foreach i in `a' {
foreach j in `b'{
capture noisily psmatch2 treat if married == `i' & year == `j', mahalanobis(hours)
capture noisily replace neighbor1 = _n1 if married == `i' & year == `j'
capture noisily replace idobs = _id if married == `i' & year == `j'
drop _treated _support _weight _id _n1 _nn
}
}
What this code should do is to look at each subgroup of observations: 444 observations in 1980 that are not married, 101 observations in 1980 that are married, ..., and 335 observations in 1987 that are married. In each of these subgroups, I would like to match a treated observation to a control observation considering the shortest distance in the number of hours worked.
There are two problems that I see after running the code.
First, the variable idobs should take a unique number between 1 and 4360 because there are 4360 observations in this dataset. It is just an ID number. It is not the case. A few observations can have an ID number 1, 2 and so on.
Second, neighbor1 varies between 1 and 204 meaning that the matched observations have only ID numbers varying from 1 to 204.
What is the problem with my code?
Here is a solution using the command iematch, installed through the package ietoolkit -> ssc install ietoolkit. For disclosure, I wrote this command. psmatch2 is great if you want the ATT. But if all you want is to match observations across two groups using nearest neighbor, then iematch is cleaner.
In both commands you need to make each industry-year match in a subset, then combine that information. In both commands the matched group ID will restart from 1 in each subset.
Using your example data, this creates one matchID var for each subset, then you will have to find a way to combine these to a single matchID without conflicts across the data set.
* use data set and keep only vars required for simplicity
use http://www.stata.com/data/jwooldridge/eacsap/wagepan.dta, clear
keep year married hour
* Set seed for replicability. NEVER use the 123456 seed in production, randomize a new seed
set seed 123456
*Generate mock treatment
gen treat = round(runiform())
*generate vars to store results
gen matchResult = .
gen matchDiff = .
gen matchCount = .
*Create locals for loops
levelsof married
local married_statuses = r(levels)
levelsof year
local years = r(levels)
*Loop over each subgroup
foreach year of local years {
foreach married_status of local married_statuses {
*This command is similar to psmatch2, but a simplified version for
* when you are not looking for the ATT.
* This command is only about matching.
iematch if married == `married_status' & year == `year', grp(treat) match(hour) seedok m1 maxmatch(1)
*These variables list meta info about the match. See helpfile for docs,
*but this copy info from each subset in this loop to single vars for
*the full data set. Then the loop specfic vars are dropped
replace matchResult = _matchResult if married == `married_status' & year == `year'
replace matchDiff = _matchDiff if married == `married_status' & year == `year'
replace matchCount = _matchCount if married == `married_status' & year == `year'
drop _matchResult _matchDiff _matchCount
*For each loop you will get a match ID restarting at 1 for each loop.
*Therefore we need to save them in one var for each loop and combine afterwards.
rename _matchID matchID_`married_status'_`year'
}
}
I'm working on a dataset in Stata
The first column is the name of the firm. the second column is the start date of this firm and the third column is the expiration date of this firm. If the expdate is missing, this firm is still in business. I want to create a variable that will record the number of firms at a given time. (preferably to be a monthly variable)
I'm really lost here. Please help!
Next time, try using dataex (ssc install dataex) rather than a screen shot, this is recommended in the Stata tag wiki, and will help others help you!
Here is an example for how to count the number of firms that are alive in each period (I'll use years, but point out where you can switch to month). This example borrows from Nick Cox's Stata journal article on this topic.
First, load the data:
* Example generated by -dataex-. To install: ssc install dataex
clear
input long(firmID dt_start dt_end)
3923155 20080123 99991231
2913168 20070630 99991231
3079566 20000601 20030212
3103920 20020805 20070422
3357723 20041201 20170407
4536020 20120201 20170407
2365954 20070630 20190630
4334271 20110721 20191130
4334338 20110721 20170829
4334431 20110721 20190429
end
Note that my in my example data my dates are not in Stata format, so I'll convert them here:
tostring dt_start, replace
generate startdate=date(dt_start, "YMD")
tostring dt_end, replace
generate enddate=date(dt_end, "YMD")
format startdate enddate
Next make a variable with the time interval you'd like to count within:
generate startyear = year(startdate)
generate endyear = year(enddate)
In my dataset I have missing end dates that begin with '9999' while you have them as '.' I'll set these to the current year, the assumption being that the dataset is current. You'll have to decide whether this is appropriate in your data.
replace endyear = year(date("$S_DATE","DMY")) if endyear == 9999
Next create an observation for the first and last years (or months) that the firm is alive:
expand 2
by firmID, sort: generate year = cond(_n == 1, startyear, endyear)
keep firmID year
duplicates drop // keeps one observation for firms that die in the period they were born
Now expand the dataset to have an observation for every period between the start and end date. For this I use tsfill.
xtset firmID year
tsfill
Now I have one observation per existing firm in each period. All that remains is to count the observations by year:
egen entities = count(firmID), by(year)
drop firmID
duplicates drop
I want to generate a variable month that has the month and year together as 2013M01.
Below is a sample of my data:
clear
input expected_arrival_month year
1 2013
2 2014
3 2015
4 2016
5 2017
6 2018
end
I tried the following command:
generate month = .
replace month = 2013M01 if expected_arrival_month == 1 & year == 2013
However, I received the error:
2013M01 invalid name
r(198)
How can I get the desired output?
For essentially all Stata purposes a numeric monthly date variable is better than anything hand- or homemade (and certainly than dates held as string variables). You can get such variables to appear as you ask. You certainly do not need to calculate individual values directly. Although this code is for a minimal dataset it will apply to all values in numeric variables as you describe. See help datetime for invaluable (and unavoidable) information.
clear
set obs 1
generate year = 2013
generate arrival_month = 1
generate wanted = ym(year, arrival_month)
format wanted %tmCCYY!MNN
list
+---------------------------+
| year arriva~h wanted |
|---------------------------|
1. | 2013 1 2013M01 |
+---------------------------+
(As commented, you should provide example data directly and in a way that makes variable types clear. If one or both variables are really string, apply destring first or use monthly().)
The issue here is in dealing with string rather than numeric variables. Given that the variable you are generating is a string variable, the contents of the variable must be enclosed in quotation marks:
generate month = "2013M01" if expected_arrival_month == 1 & year == 2013
There would also be other more efficient ways to deal with this generation, for example using Stata's egen command (and concat), or datetime functions as indicated in another response.
I have a dataset of the top management teams of US banks from 2005 - 2015.
Now I want to generate a change-variable if a TMT composition changed between 2006 and 2009.
So first I used:
drop if Year > 2009
drop if Year < 2006
by id (id), sort: gen changed = (DirectorID[1] != DirectorID[_N])
and afterwards I used
by id (id), sort: gen changed = (DirectorID[1] != DirectorID[_N]) if Year < 2010 & Year > 2005
However there is a difference in output between two variables:
247 cases of "No change" and 853 cases of "Change" in the first and 116 cases of "No change" and the rest as "Changed" in the second variable
Could anyone clarify what the differences between these two commands are in Stata?
There are a couple reasons you may be seeing a different count of changes to the dataset. The data is most likely sorted differently for these two calls. The (id) parts have no effect here because you are already sorting by id. What you likely want to do is residually sort by year. So, bysort id (Year) - this way the dataset will be in the same order for each command you type. In the second command, the if clause is going to set the variable changed to missing for observations outside of the year range, but those observations are still being included in the calculation. You could create a new variable to flag the years of interest, and then add that new variable to the bysort call.
Lastly, you need to decide whether you only want to look at changes year-over-year (the value of the changed could vary by year within id), or have the value of changed reflect whether there were any changes in DirectorID over the entire time frame of interest (the value of changed would be constant within id).
Here's a toy example illustrating the difference. Essentially, when you drop the data, the last and the first observation could be the same, but in general you will have less data to compare the first and last observation since much of the data will be gone. When you use if, then the data is still there, even though the calculation is restricted to the middle observation by the if:
. clear
. input id year director_id
id year directo~d
1. 1 2016 10
2. 1 2017 20
3. 1 2018 30
4. end
.
. bys id (year): gen changed = (director_id[1] != director_id[_N]) if year < 2018 & year > 2016
(2 missing values generated)
. list, clean noobs
id year direct~d changed
1 2016 10 .
1 2017 20 1
1 2018 30 .
.
. drop if inlist(year, 2016,2018)
(2 observations deleted)
. bys id (year): gen changed2 = (director_id[1] != director_id[_N]) if year < 2018 & year > 2016
. list, clean noobs
id year direct~d changed changed2
1 2017 20 1 0
I added a sort by year since that seems in the spirit of your exercise.
I have a string variable in Stata called YEAR with format "aaaa" (e.g. 2011). I want to replace "aaaa" with "31decaaaa" and destring the obtained variable.
My feeling is that the best way to proceed could be firstly destringing the variable YEAR and then adding "31dec". To destring the variable YEAR I have tried the command date but it does not seem to work. Any suggestion?
It would be best to describe your eventual goal here, as use of destring just appears to be what you have in mind as the next step.
If your goal is, given a string variable year, to produce a daily date variable for 31 December in each year, then destring is not necessary. Here are three ways to do it:
gen date = daily("31 Dec" + year, "DMY")
gen date = date("31 Dec" + year, "DMY")
gen date = mdy(12, 31, real(year))
Incidentally, there is no likely gain for Stata use in daily dates 365 or 366 days apart, as they just create a time series that is mostly implicit gaps.
If your data are yearly, but just associated with the end of each calendar year, keep them as yearly and use a display format to show "31 Dec", or the equivalent, in output.
. di %ty!3!1_!D!e!c_CCYY 2015
31 Dec 2015
Detail. date() is a function, not a command, in Stata. We can't comment on "does not seem to work" as no details are given of what you tried or what happened. daily() is just a synonym for date().