Extracting unique values from all the datasets in a SAS library - sas

Question

I need to extract all the unique/distinct values of some common variables across all the datasets in a SAS library. I tried following code, but is there a better way of having this on one dataset.
%macro dslist();
proc sql noprint;
select memname into :mylist separated by ' '
from dictionary.tables where libname= "VIEW" and upcase(memname) like "data_%"
;
quit;
%put &mylist;
data _null_;
datanum = countw("&mylist");
call symput('Dataset', put(datanum, 10.));
run;
%put #######&Dataset;
proc sql ;
%do i = 1 %to &Dataset ;
%let dataname=view.%scan(&mylist,&i,%str( ));
create table %scan(&mylist,&i,%str( )) as
select distinct id,visit
from &dataname
order by id,visit
;
%end;
quit;
%mend;
%dslist;
I use proc append after this step to set all the datasets and then remove duplicates.
Also, if someone knows Hash approach for better efficiency!
Thank you!

Answer 1

If the number of datasets is small you might just generate one SQL statement that will select and de-dup. But there is a limit on the number of tables that a single SQL statement can reference. Just like there is a limit on the number of dataset names that can fit into a the single macro variable your current code it generating.
So to make something that is more robust you could use a data step view to combine the data and PROC SORT to de-dup.
First get the list of datasets that have both ID and VISIT variables and meet your other criteria.
proc sql ;
create table dslist as
select catx('.',libname,nliteral(memname)) as dsname
from dictionary.columns
where libname= "VIEW"
and memname like %upcase("data_%")
and upcase(name) in ('ID' 'VISIT')
group by 1
having count(*)=2
;
quit;
Then use that list to define a data step view that combines just the ID and VISIT variables from all of them.
filename code temp;
data _null_;
set dslist end=eof;
file code lrecl=72;
if _n_=1 then put 'data id_visit_v / viwe=id_visit_v;' / ' set ' #;
put dsname '(keep=id visit) ' #;
if eof then put ';' / 'run;' ;
run;
%include code / source2;
Then use PROC SORT to get the set of distinct ID*VISIT combinations.
proc sort data=id_visit_v nodupkey out=id_visit ;
by id visit;
run;
Clean up.
proc delete data=id_visit_v (memtype=view);
run;

Answer 2

I wonder how your code is actually working with the following upcase(memname) like "data_%".
Creating fake data
libname view "/home/kermit/folder";
data view.data_A;
call streaminit(123);
array _{5} $ ('s', 't', 'a', 'c', 'k');
do i=1 to 100000;
id=rand("integer", 1, 1000);
j=rand('integer', 1, dim(_));
visit=_[j];
output;
end;
drop i j _:;
run;
data view.data_B;
call streaminit(123);
array _{5} $ ('s', 't', 'a', 'c', 'k');
do i=1 to 100000;
id=rand("integer", 1, 1000);
j=rand('integer', 1, dim(_));
visit=_[j];
output;
end;
drop i j _:;
run;
data view.data_C;
call streaminit(123);
array _{5} $ ('s', 't', 'a', 'c', 'k');
do i=1 to 100000;
id=rand("integer", 1, 1000);
j=rand('integer', 1, dim(_));
visit=_[j];
output;
end;
drop i j _:;
run;
Consolidate in one single table
proc sql noprint;
select cats(libname,'.',memname,"(keep= id visit)") into :mylist separated by ' '
from dictionary.tables where libname="VIEW" and upcase(memname) like "DATA_%"
;
quit;
data have;
set &mylist.;
run;
Extract all unique values of id and visit
proc sort data=have out=want nodupkey; by id visit; run;
NOTE: There were 300000 observations read from the data set WORK.HAVE.
NOTE: 295000 observations with duplicate key values were deleted.
NOTE: The data set WORK.WANT has 5000 observations and 2 variables.
NOTE: PROCEDURE SORT a utilisé (Durée totale du traitement) :
real time 0.08 seconds
user cpu time 0.14 seconds
system cpu time 0.02 seconds
memory 23404.76k
OS Memory 51740.00k