I have many datasets for many years from 2001 to 2014 which look like the following. Each year is stored in one file, yXXXX.sas7bdat,
ID Weight X1 X2 X3
1 100 1 2 4
2 300 4 3 4
and I need to create a dataset where for each year we have the (weighted) sums of each of the X columns.
X1 X2 X3 Year
10 20 30 2014
40 15 20 2013
I would be happy to implement this into a macro but I am unsure of a way to isolate column sums, and also an efficient way to attach results together (proc append?)
Edit: Including an attempt.
%macro final_dataset;
%do i = 2001 %to 2014;
/*Code here which enables me to get the column sums I am interested in.*/
proc means data = y&i;
weight = weight;
X1 = SUM X1;
X2 = SUM X2;
X3 = SUM X3;
OUTPUT OUT = sums&i;
run;
data final;
set final sums&i;
run;
%end;
%mend;
Edit: Another attempt.
%macro final_dataset;
%do i = 2001 %to 2014;
/*Code here which enables me to get the column sums I am interested in.*/
proc means data = y&i SUM;
weight = weight;
var X1 X2 X3;
OUTPUT OUT = sums&i;
run;
data final;
set final sums&i;
run;
%end;
%mend;
Edit: Final.
%macro final_dataset;
%do i = 2001 %to 2014;
/*Code here which enables me to get the column sums I am interested in.*/
proc means data = y&i SUM NOPRINT;
weight = weight;
var X1 X2 X3;
OUTPUT OUT = sums&i sum(X1 X2 X3) = X1 X2 X3;
run;
data final;
set final sums&i;
run;
%end;
%mend;
This is probably what I'd do, append all the data sets together and run one proc means. You didn't mention how big the data sets are, but I'm assuming smaller data.
data combined;
length source year $50.;
set y2001-y2014 indsname=source;
*you can tweak this variable so it looks how you want it to;
year=source;
run;
proc means data=combined noprint nway;
class year;
var x1 x2 x3;
output out=want sum= ;
run;
Related
I have a table which contains one key id and 100 variables (x1, x2, x3 ..... x100) and i need to check every variables if there are any values stored as -9999, -8888, -7777, -6666 in of them.
For one variable i use
proc sql;
select keyid, x1
from mytable
where x1 in(-9999,-8888,-7777,-6666);
quit;
This is the data i am trying to get but it is just for one variable.
I do not have time for copying and pasting all the variables (100 times) in this basic query.
I have searched the forum but the answers i have found are a bit far from what i actually need
and since i am new to SAS i can not write a macro.
Can you help me please?
Thanks.
Try this. Just made up some sample data that resembles what you describe :-)
data have;
do key = 1 to 1e5;
array x x1 - x100;
do over x;
x = rand('integer', -10000, -5000);
end;
output;
end;
run;
data want;
set have;
array x x1 - x100;
do over x;
if x in (-9999, -8888, -7777, -6666) then do;
output;
leave;
end;
end;
run;
Don't use SQL. Instead use normal SAS code so you can take advantage of SAS syntax like ARRAYs and variable lists.
So make an array containing the variable you want to look at. Then loop over the array. There is no need to keep looking once you find one.
data want;
set mytable;
array list var1 varb another_var x1-x10 Z: ;
found=0;
do index=1 to dim(list) until (found);
found = ( list[index] in (-9999 -8888 -7777 -6666) );
end;
if found;
run;
And if you want to search all of the numeric variables you can even use the special variable list _NUMERIC_ when defining the array:
array list _numeric_;
thank you for your help i have found a solution and wanted to share it with you.
It has some points that needs to be evaluated but it is fine for me now. (gets the job done)
`%LET LIB = 'LIBRARY';
%LET MEM = 'GIVENTABLE';
%PUT &LIB &MEM;
PROC SQL;
SELECT
NAME INTO :VARLIST SEPARATED BY ' '
FROM DICTIONARY.COLUMNS
WHERE
LIBNAME=&LIB
AND
MEMNAME=&MEM
AND
TYPE='num';
QUIT;
%PUT &VARLIST;
%MACRO COUNTS(INPUT);
%LOCAL i NEXT_VAR;
%DO i=1 %TO %SYSFUNC(COUNTW(&VARLIST));
%LET NEXT_VAR = %SCAN(&VARLIST, &i);
PROC SQL;
CREATE TABLE &NEXT_VAR AS
SELECT
COUNT(ID) AS NUMBEROFDESIREDVALUES
FROM &INPUT
WHERE
&NEXT_VAR IN (6666, 7777, 8888, 9999)
GROUP BY
&NEXT_VAR;
QUIT;
%END;
%MEND;
%COUNTS(GIVENTABLE);`
The answer you provided to your own question gives more insight to what you really wanted. However, the solution you offered while it works is not very efficient. The SQL statement runs 100 times for each variable in the source data. That means the source table is read 100 times. Another problem is that it creates 100 output tables. Why?
A better solution is to create 1 table that contains the counts for each of the 100 variables. Even better is to do it in 1 pass of the source data instead of 100.
data sum;
set have end=eof;
array x(*) x:;
array csum(100) _temporary_;
do i = 1 to dim(x);
x(i) = (x(i) in (-9999, -8888, -7777, -6666)); * flag (0 or 1) those meeting criteria;
csum(i) + x(i); * cumulative count;
if eof then do;
x(i) = csum(i); * move the final total to the orig variable;
end;
end;
if eof then output; * only output the final obs which has the totals;
drop key i;
run;
Partial result:
x1 x2 x3 x4 x5 x6 x7 x8 ...
90 84 88 85 81 83 59 71 ...
You can keep it in that form or you can transpose it.
proc transpose data=sum out=want (rename=(col1=counts))
name=variable;
run;
Partial result:
variable counts
x1 90
x2 84
x3 88
x4 85
x5 81
... ...
[I have this piece of code. However, the Macro in proc univariate generate too many separate dataset due to loop t from 1 to 310. How can I modify this code to include all proc univariate output into one dataset and then modify the rest of the code for a more efficient run?]
%let L=10; %* 10th percentile *;
%let H=%eval(100 - &L); %* 90th percentile*;
%let wlo=V1&L V2&L V3&L ;
%let whi=V1&H V2&H V3&H ;
%let wval=wV1 wV2 wV3 ;
%let val=V1 V2 V3;
%macro winsorise();
%do v=1 %to %sysfunc(countw(&val));
%do t=1 %to 310;
proc univariate data=regressors noprint;
var &val;
output out=_winsor&t._V&v pctlpts=&H &L
prtlpre=&val&t._V&v;
where time_count<=&t;run;
%end;
data regressors (drop=__:);
set regressors;
if _n_=1 then set _winsor&t._V&v;
&wval&t._V&v=min(max(&val&t._V&v,&wlo&t._V&v),&whi&t._V&v);
run;
%end;
%mend;
Thank you.
Presume you have data time_count, x1, x2, x3 with samples at every 0.5 time unit.
data regressors;
call streaminit(123);
do time_count = 0 to 310 by .5;
x1 = 2 ** (sin(time_count/6) * log(time_count+1));
x2 = log2 (time_count+1) + log(time_count/10+.1);
x3 = rand('normal',
output;
end;
format x: 7.3;
run;
Stack the data into groups based on integer time_count levels. The stack is constructed from a full outer join with a less than (<=) criteria. Each group is identified by the top time_count in the group.
proc sql;
create table stack as
select
a.time_count
, a.x1
, a.x2
, a.x3
, b.time_count as time_count_group /* save top value in group variable */
from regressors as a
full join regressors as b /* self full join */
on a.time_count <= b.time_count /* triangular criteria */
where
int(b.time_count)=b.time_count /* select integer top values */
order by
b.time_count, a.time_count
;
quit;
Now compute ALL your stats for ALL your variables for ALL your groups in one go. No macro, no muss, no fuss.
proc univariate data=stack noprint;
by time_count_group;
var x1 x2 x3;
output out=_winsor n=group_size pctlpts=90 10 pctlpre=x1_ x2_ x3_;
run;
I have a dataset like so
data test;
do i = 1 to 100;
x1 = ceil(ranuni(0) * 100);
x2 = floor(ranuni(0) * 1600);
x3 = ceil(ranuni(0) * 1500);
x4 = ceil(ranuni(0) * 1100);
x5 = floor(ranuni(0) * 10);
output;
end;
run;
data test_2;
set test;
if mod(x1,3) = 0 then x1 = .;
if mod(x2,13) = 0 then x2 = .;
if mod(x3,7) = 0 then x3 = .;
if mod(x4,6) = 0 then x4 = .;
if mod(x5,2) = 0 then x5 = .;
drop i;
run;
I plan to calculate a number of percentiles including two non-standard percentiles (2.5th and 97.5th). I do this using proc stdize as below
PROC STDIZE
DATA=test_2
OUT=_NULL_
NOMISS
PCTLMTD=ORD_STAT
pctldef=3
OUTSTAT=STDLONGPCTLS
pctlpts=(2.5 5 25 50 75 95 97.5);
VAR _NUMERIC_;
RUN;
Comparing to proc means
DATA TEST_MEANS;
SET TEST_2;
IF NOT MISSING(X1);
IF NOT MISSING(X2);
IF NOT MISSING(X3);
IF NOT MISSING(X4);
IF NOT MISSING(X5);
RUN;
PROC MEANS
DATA=TEST_MEANS NOPRINT;
VAR _NUMERIC_;
OUTPUT OUT=MEANSWIDEPCTLS P5= P25= P50= P75= P95= / AUTONAME;
RUN;
However, something to do with how SAS labels missing values as -inf, when I compare the results above, to the results produced in excel and proc means, they aren't aligned, can someone confirm which would be correct?
You are using pctldef=3 in PROC STDIZE but the default definition for PROC MEANS, and that is 5. I tested your code with PCTLDEF=3 using PROC MEANS and get matching results.
I am trying to run the Granger causality test for a list of variables and have the following macro to do that in SAS -
%MACRO GRANGER();
%DO I = &START. %TO &END. ;
%LET VAR1 = &&VAR1_&I.;
%PUT &INDEPVAR1. ;
PROC VARMAX DATA= COMB ;
MODEL Y1 &VAR1. / DFTEST P=1;
CAUSAL GROUP1 = (Y1) GROUP2 = ( &VAR1.);
OUTPUT OUT = Results&I.;
RUN;
%END; %MEND;
I want an output like this in a Sas dataset -
Group1 Group2 Pr > Chisq
Y1 Var1 <0.0001
Y1 Var2 0.5690
Y1 Var3 0.0134
.........
But when I use the Out statement in Proc Varmax, it does not output the significance level. Instead it gives me a series of residuals, predicted Y1 etc. How do I just pull out these fields?
Are you sure you don't want the OUTSTAT= option in the PROC VARMAX statement?
Does SAS has an equivalent of += or *= expression just like the C++ language?
For instance I have a list of variables X1-X10 and I would like to define a variable Final in the following form:
%let name= X1 X2 X3 X4 X5 X6 X7 X8 X9 X10
%Macro Total;
%Do i=1 %to %sysfunc(countw(&name., " "));
%Let K= %sysfunc(scan(&name., %i., " "));
Final *= K;
%End;
%Mend;
Not sure why you are trying to do it with macro code. If you want to manipulate data use a data step.
There is a the sum statement.
var + (expression) ;
For example if I wanted to calculate a cumulative sum of a variable I could do this:
data want;
set have ;
cum_X + X;
run;
But there is no similar statement to do aggregate products. If you wanted to to make a similar thing using multiplication would need to spell out the RETAIN statement and use normal assignment statement.
data want;
set have ;
retain prod_x 1 ;
prod_x = prod_x * x;
run;
If you actually just have a macro variable with a list of variable names and you ant to operate strictly on a single observation at a time then you can just use text manipulation to convert the list into the calculation you want.
%let namelist = vara varb vard varx vary varz ;
data want ;
set have ;
sum = %sysfunc(tranwrd(&namelist,%str( ),+)) ;
product = %sysfunc(tranwrd(&namelist,%str( ),*)) ;
run;
If you want more control then define an array and loop over it. Here is an example using the DO OVER syntax. (SAS is trying to get rid of DO OVER so you can also do the same thing using an index variable and typing more characters in your array references.)
data want;
set have ;
array x &namelist ;
sum=0;
product=1;
do over x ;
if not missing(x) then do ;
sum= sum+x ;
product = product*x ;
end;
end;
run;