I have a dataset like so
data test;
do i = 1 to 100;
x1 = ceil(ranuni(0) * 100);
x2 = floor(ranuni(0) * 1600);
x3 = ceil(ranuni(0) * 1500);
x4 = ceil(ranuni(0) * 1100);
x5 = floor(ranuni(0) * 10);
output;
end;
run;
data test_2;
set test;
if mod(x1,3) = 0 then x1 = .;
if mod(x2,13) = 0 then x2 = .;
if mod(x3,7) = 0 then x3 = .;
if mod(x4,6) = 0 then x4 = .;
if mod(x5,2) = 0 then x5 = .;
drop i;
run;
I plan to calculate a number of percentiles including two non-standard percentiles (2.5th and 97.5th). I do this using proc stdize as below
PROC STDIZE
DATA=test_2
OUT=_NULL_
NOMISS
PCTLMTD=ORD_STAT
pctldef=3
OUTSTAT=STDLONGPCTLS
pctlpts=(2.5 5 25 50 75 95 97.5);
VAR _NUMERIC_;
RUN;
Comparing to proc means
DATA TEST_MEANS;
SET TEST_2;
IF NOT MISSING(X1);
IF NOT MISSING(X2);
IF NOT MISSING(X3);
IF NOT MISSING(X4);
IF NOT MISSING(X5);
RUN;
PROC MEANS
DATA=TEST_MEANS NOPRINT;
VAR _NUMERIC_;
OUTPUT OUT=MEANSWIDEPCTLS P5= P25= P50= P75= P95= / AUTONAME;
RUN;
However, something to do with how SAS labels missing values as -inf, when I compare the results above, to the results produced in excel and proc means, they aren't aligned, can someone confirm which would be correct?
You are using pctldef=3 in PROC STDIZE but the default definition for PROC MEANS, and that is 5. I tested your code with PCTLDEF=3 using PROC MEANS and get matching results.
Related
I have a table which contains one key id and 100 variables (x1, x2, x3 ..... x100) and i need to check every variables if there are any values stored as -9999, -8888, -7777, -6666 in of them.
For one variable i use
proc sql;
select keyid, x1
from mytable
where x1 in(-9999,-8888,-7777,-6666);
quit;
This is the data i am trying to get but it is just for one variable.
I do not have time for copying and pasting all the variables (100 times) in this basic query.
I have searched the forum but the answers i have found are a bit far from what i actually need
and since i am new to SAS i can not write a macro.
Can you help me please?
Thanks.
Try this. Just made up some sample data that resembles what you describe :-)
data have;
do key = 1 to 1e5;
array x x1 - x100;
do over x;
x = rand('integer', -10000, -5000);
end;
output;
end;
run;
data want;
set have;
array x x1 - x100;
do over x;
if x in (-9999, -8888, -7777, -6666) then do;
output;
leave;
end;
end;
run;
Don't use SQL. Instead use normal SAS code so you can take advantage of SAS syntax like ARRAYs and variable lists.
So make an array containing the variable you want to look at. Then loop over the array. There is no need to keep looking once you find one.
data want;
set mytable;
array list var1 varb another_var x1-x10 Z: ;
found=0;
do index=1 to dim(list) until (found);
found = ( list[index] in (-9999 -8888 -7777 -6666) );
end;
if found;
run;
And if you want to search all of the numeric variables you can even use the special variable list _NUMERIC_ when defining the array:
array list _numeric_;
thank you for your help i have found a solution and wanted to share it with you.
It has some points that needs to be evaluated but it is fine for me now. (gets the job done)
`%LET LIB = 'LIBRARY';
%LET MEM = 'GIVENTABLE';
%PUT &LIB &MEM;
PROC SQL;
SELECT
NAME INTO :VARLIST SEPARATED BY ' '
FROM DICTIONARY.COLUMNS
WHERE
LIBNAME=&LIB
AND
MEMNAME=&MEM
AND
TYPE='num';
QUIT;
%PUT &VARLIST;
%MACRO COUNTS(INPUT);
%LOCAL i NEXT_VAR;
%DO i=1 %TO %SYSFUNC(COUNTW(&VARLIST));
%LET NEXT_VAR = %SCAN(&VARLIST, &i);
PROC SQL;
CREATE TABLE &NEXT_VAR AS
SELECT
COUNT(ID) AS NUMBEROFDESIREDVALUES
FROM &INPUT
WHERE
&NEXT_VAR IN (6666, 7777, 8888, 9999)
GROUP BY
&NEXT_VAR;
QUIT;
%END;
%MEND;
%COUNTS(GIVENTABLE);`
The answer you provided to your own question gives more insight to what you really wanted. However, the solution you offered while it works is not very efficient. The SQL statement runs 100 times for each variable in the source data. That means the source table is read 100 times. Another problem is that it creates 100 output tables. Why?
A better solution is to create 1 table that contains the counts for each of the 100 variables. Even better is to do it in 1 pass of the source data instead of 100.
data sum;
set have end=eof;
array x(*) x:;
array csum(100) _temporary_;
do i = 1 to dim(x);
x(i) = (x(i) in (-9999, -8888, -7777, -6666)); * flag (0 or 1) those meeting criteria;
csum(i) + x(i); * cumulative count;
if eof then do;
x(i) = csum(i); * move the final total to the orig variable;
end;
end;
if eof then output; * only output the final obs which has the totals;
drop key i;
run;
Partial result:
x1 x2 x3 x4 x5 x6 x7 x8 ...
90 84 88 85 81 83 59 71 ...
You can keep it in that form or you can transpose it.
proc transpose data=sum out=want (rename=(col1=counts))
name=variable;
run;
Partial result:
variable counts
x1 90
x2 84
x3 88
x4 85
x5 81
... ...
[I have this piece of code. However, the Macro in proc univariate generate too many separate dataset due to loop t from 1 to 310. How can I modify this code to include all proc univariate output into one dataset and then modify the rest of the code for a more efficient run?]
%let L=10; %* 10th percentile *;
%let H=%eval(100 - &L); %* 90th percentile*;
%let wlo=V1&L V2&L V3&L ;
%let whi=V1&H V2&H V3&H ;
%let wval=wV1 wV2 wV3 ;
%let val=V1 V2 V3;
%macro winsorise();
%do v=1 %to %sysfunc(countw(&val));
%do t=1 %to 310;
proc univariate data=regressors noprint;
var &val;
output out=_winsor&t._V&v pctlpts=&H &L
prtlpre=&val&t._V&v;
where time_count<=&t;run;
%end;
data regressors (drop=__:);
set regressors;
if _n_=1 then set _winsor&t._V&v;
&wval&t._V&v=min(max(&val&t._V&v,&wlo&t._V&v),&whi&t._V&v);
run;
%end;
%mend;
Thank you.
Presume you have data time_count, x1, x2, x3 with samples at every 0.5 time unit.
data regressors;
call streaminit(123);
do time_count = 0 to 310 by .5;
x1 = 2 ** (sin(time_count/6) * log(time_count+1));
x2 = log2 (time_count+1) + log(time_count/10+.1);
x3 = rand('normal',
output;
end;
format x: 7.3;
run;
Stack the data into groups based on integer time_count levels. The stack is constructed from a full outer join with a less than (<=) criteria. Each group is identified by the top time_count in the group.
proc sql;
create table stack as
select
a.time_count
, a.x1
, a.x2
, a.x3
, b.time_count as time_count_group /* save top value in group variable */
from regressors as a
full join regressors as b /* self full join */
on a.time_count <= b.time_count /* triangular criteria */
where
int(b.time_count)=b.time_count /* select integer top values */
order by
b.time_count, a.time_count
;
quit;
Now compute ALL your stats for ALL your variables for ALL your groups in one go. No macro, no muss, no fuss.
proc univariate data=stack noprint;
by time_count_group;
var x1 x2 x3;
output out=_winsor n=group_size pctlpts=90 10 pctlpre=x1_ x2_ x3_;
run;
I've got pretty big table where I want to replace rare values (for this example that have less than 10 occurancies but real case is more complicated- it might have 1000 levels while I want to have only 15). This list of possible levels might change so I don't want to hardcode anything.
My code is like:
%let var = Make;
proc sql;
create table stage1_ as
select &var.,
count(*) as count
from sashelp.cars
group by &var.
having count >= 10
order by count desc
;
quit;
/* Join table with table including only top obs to replace rare
values with "other" category */
proc sql;
create table stage2_ as
select t1.*,
case when t2.&var. is missing then "Other_&var." else t1.&var. end as &var._new
from sashelp.cars t1 left join
stage1_ t2 on t1.&var. = t2.&var.
;
quit;
/* Drop old variable and rename the new as old */
data result;
set stage2_(drop= &var.);
rename &var._new=&var.;
run;
It works, but unfortunately it is not very officient as it needs to make a join for each variable (in real case I am doing it in loop).
Is there a better way to do it? Maybe some smart replace function?
Thanks!!
You probably don't want to change the actual data values. Instead consider creating a custom format for each variable that will map the rare values to an 'Other' category.
The FREQ procedure ODS can be used to capture the counts and percentages of every variable listed into a single table. NOTE: Freq table/out= captures only the last listed variable. Those counts can be used to construct the format according to the 'othering' rules you want to implement.
data have;
do row = 1 to 1000;
array x x1-x10;
do over x;
if row < 600
then x = ceil(100*ranuni(123));
else x = ceil(150*ranuni(123));
end;
output;
end;
run;
ods output onewayfreqs=counts;
proc freq data=have ;
table x1-x10;
run;
data count_stack;
length name $32;
set counts;
array x x1-x10;
do over x;
name = vname(x);
value = x;
if value then output;
end;
keep name value frequency;
run;
proc sort data=count_stack;
by name descending frequency ;
run;
data cntlin;
do _n_ = 1 by 1 until (last.name);
set count_stack;
by name;
length fmtname $32;
fmtname = trim(name)||'top';
start = value;
label = cats(value);
if _n_ < 11 then output;
end;
hlo = 'O';
label = 'Other';
output;
run;
proc format cntlin=cntlin;
run;
ods html;
proc freq data=have;
table x1-x10;
format
x1 x1top.
x2 x2top.
x3 x3top.
x4 x4top.
x5 x5top.
x6 x6top.
x7 x7top.
x8 x8top.
x9 x9top.
x10 x10top.
;
run;
I am trying to run the Granger causality test for a list of variables and have the following macro to do that in SAS -
%MACRO GRANGER();
%DO I = &START. %TO &END. ;
%LET VAR1 = &&VAR1_&I.;
%PUT &INDEPVAR1. ;
PROC VARMAX DATA= COMB ;
MODEL Y1 &VAR1. / DFTEST P=1;
CAUSAL GROUP1 = (Y1) GROUP2 = ( &VAR1.);
OUTPUT OUT = Results&I.;
RUN;
%END; %MEND;
I want an output like this in a Sas dataset -
Group1 Group2 Pr > Chisq
Y1 Var1 <0.0001
Y1 Var2 0.5690
Y1 Var3 0.0134
.........
But when I use the Out statement in Proc Varmax, it does not output the significance level. Instead it gives me a series of residuals, predicted Y1 etc. How do I just pull out these fields?
Are you sure you don't want the OUTSTAT= option in the PROC VARMAX statement?
I have many datasets for many years from 2001 to 2014 which look like the following. Each year is stored in one file, yXXXX.sas7bdat,
ID Weight X1 X2 X3
1 100 1 2 4
2 300 4 3 4
and I need to create a dataset where for each year we have the (weighted) sums of each of the X columns.
X1 X2 X3 Year
10 20 30 2014
40 15 20 2013
I would be happy to implement this into a macro but I am unsure of a way to isolate column sums, and also an efficient way to attach results together (proc append?)
Edit: Including an attempt.
%macro final_dataset;
%do i = 2001 %to 2014;
/*Code here which enables me to get the column sums I am interested in.*/
proc means data = y&i;
weight = weight;
X1 = SUM X1;
X2 = SUM X2;
X3 = SUM X3;
OUTPUT OUT = sums&i;
run;
data final;
set final sums&i;
run;
%end;
%mend;
Edit: Another attempt.
%macro final_dataset;
%do i = 2001 %to 2014;
/*Code here which enables me to get the column sums I am interested in.*/
proc means data = y&i SUM;
weight = weight;
var X1 X2 X3;
OUTPUT OUT = sums&i;
run;
data final;
set final sums&i;
run;
%end;
%mend;
Edit: Final.
%macro final_dataset;
%do i = 2001 %to 2014;
/*Code here which enables me to get the column sums I am interested in.*/
proc means data = y&i SUM NOPRINT;
weight = weight;
var X1 X2 X3;
OUTPUT OUT = sums&i sum(X1 X2 X3) = X1 X2 X3;
run;
data final;
set final sums&i;
run;
%end;
%mend;
This is probably what I'd do, append all the data sets together and run one proc means. You didn't mention how big the data sets are, but I'm assuming smaller data.
data combined;
length source year $50.;
set y2001-y2014 indsname=source;
*you can tweak this variable so it looks how you want it to;
year=source;
run;
proc means data=combined noprint nway;
class year;
var x1 x2 x3;
output out=want sum= ;
run;