Efficient way to select unique records in SAS - sas

Question

The dataset looks like this:
Code Type Rating
0001 NULL 1
0002 NULL 1
0003 NULL 1
0003 PA 1 3
0004 NULL 1
0004 PB 1 2
0005 AC 1 3
0005 NULL 6
0006 AC 1 2
I want the output dataset looks like
Code Type Rating
0001 NULL 1
0002 NULL 1
0003 PA 1 4
0004 PB 1 3
0005 AC 1 9
0006 AC 1 2
For each Code, Type has at most two values. I want to select the unique Code by summing Rating. But the problem is, for Type, if it has only one value, the passes its value to output dataset. If is has two values (one has to be NULL), then passes the one not equals to NULL to output dataset.
The total number of observation N>100,000,000. So is there any tricky way to achieve this?

Answer 1

If the data is sorted as per your example, then you can achieve this in a single data step. I've assumed that the NULL values are actually missing, however if not then change [if missing(type)] to [if type='NULL']. All this does is sum the Rating values for each Code, then output the last record, keeping the non-null Type. If your data isn't sorted or indexed on Code then you'll need to do a sort first, which will obviously add quite a bit to the execution time.
/* create input file */
data have;
input Code Type $ Rating;
infile datalines dsd;
datalines;
0001,,1
0002,,1
0003,,1
0003,PA 1,3
0004,,1
0004,PB 1,2
0005,AC 1,3
0005,,6
0006,AC 1,2
;
run;
/* create summarised dataset */
data want;
set have;
by code;
retain _type; /* temporary variable */
if first.code then do;
_type = type;
_rating_sum = 0; /* reset sum */
end;
_rating_sum + rating; /* sum rating per Code */
if last.code then do;
if missing(type) then type = _type; /* pick non-null value */
rating = _rating_sum; /* insert sum */
output;
end;
run;

Answer 2

Given the comments, another possibility presents, the hash solution. This is memory-constrained, so it may or may not be able to work with the actual data (the hash table isn't very big, but 100M rows might imply 60 or 70M rows in the hash table, times 40 or 50 bytes would still be pretty big).
This is almost certainly inferior to the plain data step method if the dataset is sorted by code, so this should only be used on unsorted data.
The concepts:
Create hash table keyed on code
If incoming record is new, add to hash table
If incoming record is not a new code, take the retrieved value and sum the rating. Check to see if type needs to be replaced.
Output to dataset.
Code:
data _null_;
if _n_=1 then do;
if 0 then set have;
declare hash h(ordered:'a');
h.defineKey('code');
h.defineData('code','type','rating');
h.defineDone();
end;
set have(rename=(type=type_in rating=rating_in)) end=eof;
rc_1 = h.find();
if rc_1 eq 0 then do;
if type ne type_in and type='NULL' then type=type_in;
rating=sum(rating,rating_in);
h.replace();
end;
else do;
type=type_in;
rating=rating_in;
h.add();
end;
if eof then do;
h.output(dataset:'want');
end;
run;

Answer 3

It's pretty easy to do in one SQL step as well. Just use a CASE...WHEN...END to remove the NULLs and a MAX to then get the non-null value.
data have;
input #1 Code 4.
#9 Type $4.
#19 Rating 1.;
datalines;
0001 NULL 1
0002 NULL 1
0003 NULL 1
0003 PA 1 3
0004 NULL 1
0004 PB 1 2
0005 AC 1 3
0005 NULL 6
0006 AC 1 2
;;;;
run;
proc sql;
create table want as
select code,
max(case type when 'NULL' then '' else type end) as type,
sum(Rating) as rating
from have
group by code;
quit;
If you want the NULLs back, then you need to wrap the select in a select code, case type when ' ' then 'NULL' else type end as type, rating from ( ... );, though I would suggest leaving them blank.