Hi does anyone know how to calculate the standard deviation over the next four quarters for each quarter? Thanks :)
My attempt is below:
date1 is the sas date for the quarter in a year
Proc sql ; create table th.totalroll as
Select distinct permco, date1 ,
(select std(adjret) from th.returns1 where qtr between
intnx('quarter',qtr(date),0) and intnx('quarter', qtr(date),+3)) as
TOTALroll From th.returns1 group by permco ,date1;
QUIT;
It's hard to tell how close you are because I'm not entirely certain what your data looks like, but here's an example assuming you have more than one date in each quarter. Create sample data:
data have;
format date date9.;
do m = 1 to 128;
date = intnx('month','01JAN2008'd,m-1);
amount = round(ranuni(date)*10);
output;
end;
drop m;
run;
Using proc sql, create quarter variable (you might already have this variable?) and group by this variable. Use a having clause to restrict results to the first date of each quarter.
proc sql;
create table want as
select
yyq(year(t1.date),qtr(t1.date)) as quarter format=yyq.,
(select std(t2.amount)
from have t2
where t2.date >= yyq(year(t1.date),qtr(t1.date))
and t2.date < intnx('quarter',yyq(year(t1.date),qtr(t1.date)),4)) as stddev
from
have t1
group by
calculated quarter
having
t1.date = min(t1.date)
;
quit;
You should be able to adapt this to work for your data.
You can use proc expand if your dataset is already in quarterly. So something like this:
proc expand data=th.returns1
out=th.totalroll
from=quarter
to=quarter;
by permco date1;
id date;
convert adjret=TOTALroll / transformout=( MOVSTD 4 );
run;
Don't forget to sort you data first. And MOVSTD gives you backward moving standard deviation. You may need to shift the output stream back by 4 quarters if you want the forward moving STD.
Transformation Operations for proc expand:
http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_expand_sect026.htm
Related
I have the following data
Date value_idx
2002-01-31 .
2002-01-31 24.533
2002-01-31 26.50
2018-02-28 25.2124
2019-09-12 22.251
2019-01-31 24.214
2019-05-21 25.241
2019-05-21 .
2020-05-21 25.241
2020-05-21 23.232
I would need to calculate the average of value_idx of the last 3 years and 7 years.
I tried first to calculate it as follows:
proc sql;
create table table1 as
select date, avg(value_idx) as avg_value_idx
from table
group by date;
quit;
The problem is that I do not know how to calculate the average of value_idx not per each month but for the last two years. So I think I should extract the year, group by that, and then calculate the average.
I hope someone of you can help me with this.
You can use CASE to decide which records contribute to which MEAN. You need to clarify what you mean by last 2 or last 7 years. This code will find the value of the maximum date and then compare the year of that date to the year of the other dates.
select
mean(case when year(max_date)-year(date) < 2 then value_idx else . end) as mean_yr2
,mean(case when year(max_date)-year(date) < 7 then value_idx else . end) as mean_yr7
from have,(select max(date) as max_date from have)
;
Results
mean_yr2 mean_yr7
------------------
24.0358 24.2319
The best way to do this sort of thing in SAS is with native PROCs, as they have a lot of functionality related to grouping.
In this case, we use multilabel formats to control the grouping. I assume you mean 'Last Three Years' as in calendar 2018/2019/2020 and 'Last Seven Years' as calendar 2014-2020. Presumably you can see how to modify this for other time periods - so long as you aren't trying to make the time period relative to each data point.
We create a format that uses the MULTILABEL option (which allows data points to fall in multiple categories), and the NOTSORTED option (to allow us to force the ordering of the labels, otherwise SEVEN is earlier than THREE).
Then, we use it in PROC TABULATE, enabling it with MLF (MultiLabel Format) and preloadfmt order=data which again keeps the ordering correct. This produces a report with the two averages only.
data have;
informat date yymmdd10.;
input Date value_idx;
datalines;
2002-01-31 .
2002-01-31 24.533
2002-01-31 26.50
2017-02-28 25.2124
2017-09-12 22.251
2018-01-31 24.214
2018-05-21 25.241
2019-05-21 .
2020-05-21 25.241
2020-05-21 23.232
;;;;
run;
proc format;
value yeartabfmt (multilabel notsorted)
'01JAN2018'd-'31DEC2020'd = 'Last Three Years'
'01JAN2014'd-'31DEC2020'd = 'Last Seven Years'
other=' '
;
quit;
proc tabulate data=have;
class date/mlf preloadfmt order=data;
var value_idx;
format date yeartabfmt.;
tables date,value_idx*mean;
run;
I created below SAS code to pull the data for particular a date.
%let date =2016-12-31;
proc sql;
connect to teradata as tera ( user=testuser password=testpass );
create table new as select * from connection tera (select acct,org
from dw.act
where date= &date.);
disconnect from tera;
quit;
There are situation where that particular date may be missing in the dataset due to holiday.
I thinking how to query the previous date(non-holiday) if the mention date in the %let statement is holiday
Before running your query you have to do a lookup or data check on the date you are using. You have two options:
Use a Date Dimension table in order identify/lookup holidays.
Count how many records you have for that date, if you get 0 obs for this date, use date+1 in your query.
I recommend using the date dimension table option.
Teradata has Sys_Calendar.Calendar view. You can use that in query, it has all the information regarding weekdays and others.
if you want to SAS way use weekday function and use call symput as shown below. Teradata needs single quote around the date, so it is better to have single quotes around when creating macro variable
data _null_;
/* this is for intial date*/
date_int = input('2016-12-31', yymmdd10.);
/* create a new date variable depending on weekday*/
if weekday(date_int) = 7 then date =date_int-2; /*sunday -2 days to get
friday*/
else if weekday(date_int) = 6 then date =date_int-1;/*saturday -1 day to get
friday*/
else date =date_int;
format date date_int yymmdd10.;
call symputx('date', ''''||put(date,yymmdd10.)||'''');
run;
%put modfied date is &date;
modified date is '2016-12-29'
Now you can use this macro variable in your pass through.
I have monthly datasets in SAS Library for customers from Jan 2013 onwards with datasets name as CUST_JAN2013,CUST_FEB2013........CUST_OCT2017. These customers datasets have huge records of 2 million members for each month.This monthly datset has two columns (customer number and customer monthly expenses).
I have one input dataset Cust_Expense with customer number and month as columns. This Cust_Expense table has only 250,000 members and want to pull expense data for each member from SPECIFIC monthly SAS dataset by joining customer number.
Cust_Expense
------------
Customer_Number Month
111 FEB2014
987 APR2017
784 FEB2014
768 APR2017
.....
145 AUG2017
345 AUG2014
I have tried using call execute, but it tries to loop thru each 250,000 records of input dataset (Cust_Expense) and join with corresponding monthly SAS customer tables which takes too much of time.
Is there a way to read input tables (Cust_Expense) by month so that we read all customers for a specific month and then read the same monthly table ONCE to pull all the records from that month, so that it does not loop 250,000 times.
Depending on what you want the result to be, you can create one output per month by filtering on cust_expenses per month and joining with the corresponding monthly dataset
%macro want;
proc sql noprint;
select distinct month
into :months separated by ' '
from cust_expenses
;
quit;
proc sql;
%do i=1 %to %sysfunc(countw(&months));
%let month=%scan(&months,&i,%str( ));
create table want_&month. as
select *
from cust_expense(where=(month="&month.")) t1
inner join cust_&month. t2
on t1.customer_number=t2.customer_number
;
%end;
quit;
%mend;
%want;
Or you could have one output using one join by 'unioning' all those monthly datasets into one and dynamically adding a month column.
%macro want;
proc sql noprint;
select distinct month
into :months separated by ' '
from cust_expenses
;
quit;
proc sql;
create table want as
select *
from cust_expense t1
inner join (
%do i=1 %to %sysfunc(countw(&months));
%let month=%scan(&months,&i,%str( ));
%if &i>1 %then union;
select *, "&month." as month
from cust_&month
%end;
) t2
on t1.customer_number=t2.customer_number
and t1.month=t2.month
;
quit;
%mend;
%want;
In either case, I don't really see the point in joining those monthly datasets with the cust_expense dataset. The latter does not seem to hold any information that isn't already present in the monthly datasets.
Your first, best answer is to get rid of these monthly separate tables and make them into one large table with ID and month as key. Then you can simply join on this and go on your way. Having many separate tables like this where a data element determines what table they're in is never a good idea. Then index on month to make it faster.
If you can't do that, then try creating a view that is all of those tables unioned. It may be faster to do that; SAS might decide to materialize the view but maybe not (but if it's extremely slow, then look in your temp table space to see if that's what's happening).
Third option then is probably to make use of SAS formats. Turn the smaller table into a format, using the CNTLIN option. Then a single large datastep will allow you to perform the join.
data want;
set jan feb mar apr ... ;
where put(id,CUSTEXPF1.) = '1';
run;
That only makes one pass through the 250k table and one pass through the monthly tables, plus the very very fast format lookup which is undoubtedly zero cost in this data step (as the disk i/o will be slower).
I guess you could output your data in specific dataset like this example :
data test;
infile datalines dsd;
input ID : $2. MONTH $3. ;
datalines;
1,JAN
2,JAN
3,JAN
4,FEB
5,FEB
6,MAR
7,MAR
8,MAR
9,MAR
;
run;
data JAN FEB MAR;
set test;
if MONTH = "JAN" then output JAN;
if MONTH = "FEB" then output FEB;
if MONTH = "MAR" then output MAR;
run;
You will avoid to loop through all your ID (250000)
and you will use dataset statement from SAS
At the end you will get 12 DATASET containing the ID related.
If you case, FEB2014 , for example, you will use a substring fonction and the condition in your dataset will become :
...
set test;
...
if SUBSTR(MONTH,1,3)="FEB" then output FEB;
...
Regards
I have a question regarding moving average. I use Proc Expand (cmovave 3), but those three days can be non consecutive I suppose. I want to avoid missing data between days and use moving average for just those adjacent days.
Is there any way that I can do this? If I want to put it in another way 'how can I select a part of my data set where I have values for consecutive period (days)?'. I hope you give me some examples for this problem.
Use Expand to make sure you have all the values in the timeseries interval. Then use a data step to calculate the ma3 with the lagN() functions.
If you data already has the correct timeseries interval, then skip the PROC EXPAND step.
data test;
start = "01JAN2013"d;
format date date9.
value best.;
do i=1 to 365;
r = ranuni(1);
value = rannor(1);
date = intnx('weekday',start,i);
dummy=1;
if r > .33 then output;
end;
drop i start r;
run;
proc expand data=test out=test2 to=weekday ;
id date;
var dummy;
run;
data test(drop=dummy);
merge test2 test;
by date;
ma3 = (value + lag(value) + lag2(value))/3;
run;
I use the DUMMY variable so that EXPAND will convert the series to WEEKDAY. Then drop it afterwards.
The following works fine for me, as I want to select the YYYYQ value for a column to show the year and quarter:
proc sql;
select YYQ(year(datepart(betdate))
, QTR(datepart(betdate))) FORMAT=YYQN. as yearquarter
, QTR(datepart(betdate)) as semiyear
from &dsn;
quit;
How can I calculate the 'SEMIYEAR' instead of QTR? I can find refernces to it in the SAS documentation, but can't seem to get it to work. I want to show YYYYS, as it the year and the year 'half'.
Thanks.
There's not exactly a format or function for that, unfortunately. Do you need the date part of the value to persist, or just the value "20131"? (In your YYQ, for example, the underlying value is an actual date, which corresponds to the first date in the period of 2013Q1, so Jan 1; it's just displayed as 20131).
If you just want to display the value, you can do something pretty simple, like this:
proc sql;
select YYQ(year(datepart(betdate))
, QTR(datepart(betdate))) FORMAT=YYQN. as yearquarter
, floor(QTR(datepart(betdate))/2)+1 as semiyear
from test;
quit;
And append on the year if you want. However that does not maintain the actual first-day-of-the-period value. If you want to do that, you should use INTNX:
proc sql;
select YYQ(year(datepart(betdate))
, QTR(datepart(betdate))) FORMAT=YYQN. as yearquarter
, intnx('SEMIYEAR',datepart(betdate),0,'b') FORMAT=DATE9. as semiyear
from test;
quit;
That doesn't format it neatly, of course, so you would have to write your own format, unless I'm missing one that exists already; that's pretty easy though.
proc format;
value SEMIYEAR
'01JAN2013'd-'30JUN2013'd = '20131'
'01JUL2013'd-'31DEC2013'd = '20132'
;
quit;
Sadly you can't use picture formats to do this as far as I know - the documentation at least doesn't offer an option to display semiannual period. You can either do like I did above and just explicitly specify the time periods in the range, or you can write a function format; see http://support.sas.com/documentation/cdl/en/proc/65145/HTML/default/viewer.htm#n1eyyzaux0ze5ln1k03gl338avbj.htm for examples of how to do that.
Edit: Here's an example that mostly works.
proc fcmp outlib=work.functions.smd;
function sfmt(date) $;
length snum $5;
snum = put(year(date)*10+floor(QTR(date)/2)+1,5.);
return(snum);
endsub;
run;
options cmplib=(work.functions);
proc format;
value semiyear
other=[sfmt()];
quit;
data test2;
set test;
x=put(datepart(betdate),semiyear.);
put x=;
run;
proc sql;
select YYQ(year(datepart(betdate))
, QTR(datepart(betdate))) FORMAT=YYQN. as yearquarter
, intnx('SEMIYEAR',datepart(betdate),0,'b') FORMAT=SEMIYEAR5. as semiyear
from test;
quit;
However, for some odd reason in my session at least the PROC SQL returns goofy characters instead of 20131. The data step returns the correct answer in the log. Not sure if this is a bug or if i'm doing something very slightly wrong.