I have this data
data have;
input cust_id pmt months;
datalines;
AA 100 0
AA 50 1
AA 200 2
AA 350 3
AA 150 4
AA 700 5
BB 500 0
BB 300 1
BB 1000 2
BB 800 3
run;
and I'd like to generate an output that looks like this
data want;
input cust_id pmt months i;
datalines;
AA 100 0 0
AA 50 0 1
AA 200 0 2
AA 350 0 3
AA 150 0 4
AA 700 0 5
AA 50 1 0
AA 200 1 1
AA 350 1 2
AA 150 1 3
AA 700 1 4
AA 200 2 0
AA 350 2 1
AA 150 2 2
AA 700 2 3
AA 350 3 0
AA 150 3 1
AA 700 3 2
AA 150 4 0
AA 700 4 1
AA 700 5 0
BB 500 0 0
BB 300 0 1
BB 1000 0 2
BB 800 0 3
BB 300 1 0
BB 1000 1 1
BB 800 1 2
BB 1000 2 0
BB 800 2 1
BB 800 3 0
run;
There are few thousand rows with different cust_ID and different months length. I tried joining tables but it couldn't get me the sequence of 100 50 200 350 150 700 (for cust_ID AA). I could only replicated 100 if my months are 0, 50 if months are 1 & so on. I created a maxval which is the maximum month value. My code is something like this
data temp1;
set have;
do i = 0 to maxval;
if (months <=maxval) then output;
end;
i thought of creating a uniquekey to join my have data and temp1 data but it could only give me
AA 100 0 0
AA 50 0 1
AA 200 0 2
AA 350 0 3
AA 150 0 4
AA 700 0 5
AA 100 1 0
AA 50 1 1
AA 200 1 2
AA 350 1 3
AA 150 1 4
AA 100 2 0
AA 50 2 1
AA 200 2 2
AA 350 2 3
AA 100 3 0
AA 50 3 1
AA 200 3 2
AA 100 4 0
AA 50 4 1
AA 100 5 0
Any thoughts or different approach on how to generate my want table? Thank you!
This problem is a little tricky because you have things going in three directions
The number of group repetitions descends from group count. Within each repetition:
The payments item start index ascends and terminates at group count
The months (as I) item start index is 1 and termination descends from group count
SQL
One SQL approach is a three-way reflexive join with-in group. The months values act as a within group index and must be monotonic by 1 from 0 for this to work.
proc sql;
create table want as
select X.cust_id, Z.pmt, X.months, Y.months as i
from have as X
join have as Y on X.cust_id = Y.cust_id
join have as Z on Y.cust_id = Z.cust_id
where
X.months + Y.months = Z.months
order by
X.cust_id, X.months, Z.months
;
quit;
DATA Step
A DOW loop is used to count the group size. 2-deep looping crosses the combinations and three point= values are computed (finagled) to retrieve the relevant values.
data want2;
if 0 then set have; * prep pdv to match have;
retain point_end ;
point_start = sum(point_end,0);
do group_count = 1 by 1 until (last.cust_id);
set have(keep=cust_id);
by cust_id;
end;
do index1 = 1 to group_count;
point1 = point_start + index1;
set have (keep=months) point = point1;
do index2 = 0 to group_count - index1 ;
point2 = point_start + index1 + index2;
set have (keep=pmt) point=point2;
point3 = point_start + index2 + 1;
set have (keep=months rename=months=i) point=point3;
output;
end;
end;
point_end = point1;
keep cust_id pmt months i;
run;
Try the following:
data want(drop = start_obs limit j);
retain start_obs 1;
/* read by cust_id group */
do until(last.cust_id);
set have end = last_obs;
by cust_id;
end;
limit = months;
do j = 0 to limit;
i = 0;
do obs_num = start_obs + j to start_obs + limit;
/* read specific observations using direct access */
set have point = obs_num;
months = j;
output;
i = i + 1;
end;
end;
/* prepare for next direct access read */
start_obs = limit + 2;
if last_obs then
stop;
run;
Related
I have a big panel dataset that looks somewhat like this:
data have;
input id t a b ;
datalines;
1 1 0 0
1 2 0 0
1 3 1 0
1 4 0 0
1 5 0 1
1 6 1 0
1 7 0 0
1 8 0 0
1 9 0 0
1 10 0 1
2 1 0 0
2 2 1 0
2 3 0 0
2 4 0 0
2 5 0 1
2 6 0 1
2 7 0 1
2 8 0 1
2 9 1 0
2 10 0 1
3 1 0 0
3 2 0 0
3 3 0 0
3 4 0 0
3 5 0 0
3 6 0 0
3 7 1 0
3 8 0 0
3 9 0 0
3 10 0 0
;
run;
For every ID I want to record all 'trigger' events, namely when a=1 and then I need to how long it takes to the next occurrence of b=1. The final output should then give me the following:
data want;
input id a_no a_t b_t diff ;
datalines;
1 1 3 5 2
1 2 6 10 4
2 1 2 5 3
2 2 9 10 1
3 1 7 . .
;
run;
It is of course no problem to get all a=1 and b=1 events, but as it is a very big dataset with a lot of both events for every ID I am searching for an elegant and straight-forward solution. Any ideas?
Here's a fairly simple SQL approach that gives more or less the desired output:
proc sql;
create table want
as select
t1.id,
t1.t as a_t,
t2.t as b_t,
t2.t - t1.t as diff
from
have(where = (a=1)) t1
left join
have(where = (b=1)) t2
on
t1.id = t2.id
and t2.t > t1.t
group by t1.id, t1.t
having diff = min(diff)
;
quit;
The only part missing is a_no. This sort of row-incrementing ID is quite a lot of work to generate consistently in SQL, but it's trivial with an extra data step:
data want;
set want;
by id;
if first.id then a_no = 0;
a_no + 1;
run;
An elegant DATA step way can use nested DOW loops. It's straight forward when you understand DOW loops.
data want(keep=id--diff);
length id a_no a_t b_t diff 8;
do until (last.id); * process each group;
do a_no = 1 by 1 until(last.id); * counter for each output;
do until ( output_condition or end); * process each triggering state change;
SET have end=end; * read data;
by id; * setup first. last. variables for group;
if a=1 then a_t = t; * detect and record start of trigger state;
output_condition = (b=1 and t > a_t > 0); * evaluate for proper end of trigger state;
end;
if output_condition then do;
b_t = t; * compute remaining info at output point;
diff = b_t - a_t;
OUTPUT;
a_t = .; * reset trigger state tracking variables;
b_t = .;
end;
else
OUTPUT; * end of data reached without triggered output;
end;
end;
run;
Note: A SQL way (not shown) can use self join within groups.
Here is the data I have, I use proc tabulate to present it how it is presented in excel, and to make the visualization easier. The goal is to make sure groups strictly below the diagonal (i know it's a rectangle, the (1,1) (2,2)...(7,7) "diagonal") to roll up the column until it hits the diagonal or makes a group size of at least 75.
1 2 3 4 5 6 7 (month variable)
(age)
1 80 90 100 110 122 141 88
2 80 90 100 110 56 14 88
3 80 90 87 45 12 41 88
4 24 90 100 110 22 141 88
5 0 1 0 0 0 0 2
6 0 1 0 0 0 0 6
7 0 1 0 0 0 0 2
8 0 1 0 0 0 0 11
Ive already used if/thens to regroup certain data values, but I need a general way to do it for other sets.
Thanks in advance
desired results
1 2 3 4 5 6 7 (month variable)
(age)
1 80 90 100 110 122 141 88
2 80 90 100 110 56 14 88
3 104 90 87 45 12 41 88
4 0 94 100 110 22 141 88
5 0 0 0 0 0 0 2
6 0 0 0 0 0 0 6
7 0 0 0 0 0 0 13
8 0 0 0 0 0 0 0
Mock up some categorical data for some patients who have to be counted
data mock;
do patient_id = 1 to 2500;
month = ceil(7*ranuni(123));
age = ceil(8*ranuni(123));
output;
end;
stop;
run;
Create a tabulation of counts (N) similar to the one shown in the question:
options missing='0';
proc tabulate data=mock;
class month age;
table age,month*n=''/nocellmerge;
run;
For each month get the sub-diagonal patient count
proc sql;
/* create table subdiagonal_column as */
select month, count(*) as subdiag_col_freq
from mock
where age > month
group by month;
For each row get the pre-diagonal patient count
/* create table prediagonal_row as */
select age, count(*) as prediag_row_freq
from mock
where age > month
group by age;
other sets can be tricky if the categorical values are not +1 monotonic. To do a similar process for non-montonic categorical values you will need to create surrogate variables that are +1 monotonic. For example:
data mock;
do item_id = 1 to 2500;
pet = scan ('cat dog snake rabbit hamster', ceil(5*ranuni(123)));
place = scan ('farm home condo apt tower wild', ceil(6*ranuni(123)));
output;
end;
run;
proc tabulate data=mock;
class pet place;
table pet,place*n=''/nocellmerge;
run;
proc sql;
create table unq_pets as select distinct pet from mock;
create table unq_places as select distinct place from mock;
data pets;
set unq_pets;
pet_num = _n_;
run;
data places;
set unq_places;
place_num = _n_;
run;
proc sql;
select distinct place_num, mock.place, count(*) as subdiag_col_freq
from mock
join pets on pets.pet = mock.pet
join places on places.place = mock.place
where pet_num > place_num
group by place_num
order by place_num
;
I need to find for every row the last 3hr usage (Usage is one of the columns in dataset) grouped by User and ID_option.
Every line(row) represent one record (within 3 min time interval). For example (including desired column sum_usage_3hr):
User ID_option time usage sum_usage_3hr
1 a1 12OCT2017:11:20:32 3 10
1 a1 12OCT2017:10:23:24 7 14
1 b1 12OCT2017:09:34:55 12 12
2 b1 12OCT2017:08:55:06 4 6
1 a1 12OCT2017:07:59:53 7 7
2 b1 12OCT2017:06:59:12 2 2
I have used code below for hash table:
data want;
if _n_=1 then do;
if 0 then set have(rename=(usage=_usage));
declare hash h(dataset:'have(rename=(usage=_usage))',hashexp:20);
h.definekey('user','id_option','time');
h.definedata('_usage');
h.definedone();
end;
set have;
sum_usage_3hr=0;
do i=time-3*3600 to time ;
if h.find(key:user,key:id_option,key:i)=0 then sum_usage_3hr+_usage;
end;
drop _usage i;
run;
But I got an error: Invalid DO loop control information, either the INITIAL or TO expression is missing or the BY expression is missing, zero, or invalid. If I add:
output;
end:
just above the "run;" it gives me an error: 'No matching DO/Select statement'.
Anybody know what causes the problem?
I have also the version with sorting the table firstly and gives me the same error.
Thank you
After implementing the for answer:
User ID_option time usage sum_usage_3hr col_i_got
1 a1 12OCT2017:11:22:32 3 12 3
1 a1 12OCT2017:11:20:24 0 9 3
1 a1 12OCT2017:10:34:55 2 9 2
1 a1 12OCT2017:09:55:06 0 7 2
1 a1 12OCT2017:09:43:45 0 7 0
1 a1 12OCT2017:08:59:53 7 7 7
1 a1 12OCT2017:06:59:12 0 0 7
Try this out:
Problem 1:
Input:
data have;
input User ID_option $ time usage ;
informat time datetime18.;
format time datetime18.;
cards;
1 a1 12OCT2017:11:20:32 3
1 a1 12OCT2017:10:23:24 7
1 b1 12OCT2017:09:34:55 12
2 b1 12OCT2017:08:55:06 4
1 a1 12OCT2017:07:59:53 7
2 b1 12OCT2017:06:59:12 2
;
run;
Code:
proc sort data=have out=have1;
by user id_option time;
quit;
data have2;
set have1;
by user id_option;
format previous_time datetime18.;
previous_time = lag(time);
previous_usage = lag(usage);
if first.ID_option then previous_time=.;
if previous_time ~= . and intnx("hour",time,-3,"s") <= previous_time <= time then sum_usage_3hr=usage+previous_usage;
else sum_usage_3hr = usage;
drop previous_time previous_usage;
run;
proc sort data=have2 out=want;
by descending time ;
quit;
Output:
User ID_option time usage sum_usage_3hr
1 a1 12Oct2017 11:20:32 3 10
1 a1 12Oct2017 10:23:24 7 14
1 b1 12Oct2017 9:34:55 12 12
2 b1 12Oct2017 8:55:06 4 6
1 a1 12Oct2017 7:59:53 7 7
2 b1 12Oct2017 6:59:12 2 2
Problem2:
Input:
data have;
input user1 ID_option $ time usage ;
informat time datetime18.;
format time datetime18.;
cards;
1 a1 12OCT2017:11:22:32 3
1 a1 12OCT2017:11:20:24 0
1 a1 12OCT2017:10:34:55 2
1 a1 12OCT2017:09:55:06 0
1 a1 12OCT2017:09:43:45 0
1 a1 12OCT2017:08:59:53 7
1 a1 12OCT2017:06:59:12 0
;
run;
Code:
proc sql;
create table want as
select user1,id_option,time,min(usage) as usage,sum(usage1) as sum_usage_3hr
from
(
select a.*,b.time as time1 ,b.usage as usage1
from
have a
left join
have b
on a.user1 = b.user1 and a.id_option = b.id_option and b.time <= a.time
where intck("hour",a.time ,b.time) >= -3
)
group by 1,2,3
order by time desc;
quit;
Output:
user1 ID_option time usage sum_usage_3hr
1 a1 12Oct2017 11:22:32 3 12
1 a1 12Oct2017 11:20:24 0 9
1 a1 12Oct2017 10:34:55 2 9
1 a1 12Oct2017 9:55:06 0 7
1 a1 12Oct2017 9:43:45 0 7
1 a1 12Oct2017 8:59:53 7 7
1 a1 12Oct2017 6:59:12 0 0
Let me know in case of any queries.
I have the following two sas datasets:
data have ;
input a b;
cards;
1 15
2 10
3 40
4 200
1 25
2 15
3 10
4 75
1 1
2 99
3 30
4 100
;
data ref ;
input x y;
cards;
1 10
2 20
3 30
4 100
;
I would like to have the following dataset:
data want ;
input a b outcome ;
cards;
1 15 0
2 10 1
3 40 0
4 200 0
1 25 0
2 15 1
3 10 1
4 75 1
1 1 1
2 99 0
3 30 1
4 100 1
;
I would like to create a variable 'outcome' which is produced by an if statement upon conditions of variables a, b, x and y. As in reality the 'have' dataset is extremely large I would like to avoid a sort and merging the two datasets together (where a = x).
I am trying to use macro variables with the following code:
data _null_ ;
set ref ;
call symput('listx', x) ;
call symput('listy', y) ;
run ;
data want ;
set have ;
if a=&listx and b le &listy then outcome = 1 ; else outcome = 0 ;
run ;
which does not however produce the desired result:
data want ;
input a b outcome ;
cards;
1 15 0
2 10 1
3 40 0
4 200 0
1 25 0
2 15 1
3 10 1
4 75 1
1 1 1
2 99 0
3 30 1
4 100 1
;
redone my solution using hash tables. Below my approach
data ref2(rename=(x=a));
set ref ;
run;
data want;
declare Hash Plan ();
rc = plan.DefineKey ('a'); /*x originally*/
rc = plan.DefineData('a', 'y');
rc = plan.DefineDone();
do until (eof1);
set ref2 end=eof1;
rc = plan.add(); /*add each record from ref2 to plan (hash table)*/
end;
do until (eof2);
set have end=eof2;
call missing(y);
rc = plan.find();
outcome = (rc =0 and b<y);
output;
end;
stop;
run;
hope it helps
I was wondering, if anybody has a quick and easy way to collapse transactional data into one observation for easier modelling processing.
For example, let's say we look at a negotiation with a customer, every record is a quote for a certain car model with options A, B and C (all nominal indicators). The last record indicates a sale.
DATA TEMPSET;
INPUT CUST_ID $ A $ B $ C $;
DATALINES;
01 1 0 3
01 1 1 0
01 1 1 3
01 0 1 3
02 0 0 2
02 1 0 2
02 1 1 2
02 1 2 2
02 0 2 2
;
RUN;
To make things easier I would love to have one the resulting dataset to look like:
CUST_ID A B C A-1 B-1 C-1 A-2 B-2 C-2 A-3 B-3 C-3 A-4 B-4 C-4
01 1 0 3 1 1 0 1 1 3 0 1 3 . . .
02 0 0 2 1 0 2 1 1 2 1 2 2 0 2 2
My approach was a two dimensional array to create the variables. But then I could not combine it with a DO loop, trying to assign assign the values since it has multiple obs. I also tried using macro variables with SYMPUT/SYMGET and then LAST.CUst_ID = 1 to trigger the output, still with the problems of not having always the same length of quote history as well as requireing a hardcoding for each variable, which is practical for three variables, but not with the number increases. Any suggestions are welcome, probably possible with PROC SQL in a much simpler fashion?
Thanks!
PROC TRANSPOSE is your friend here. Make a vertical dataset with the wanted name and value, and transpose.
DATA TEMPSET;
INPUT CUST_ID $ A $ B $ C $;
DATALINES;
01 1 0 3
01 1 1 0
01 1 1 3
01 0 1 3
02 0 0 2
02 1 0 2
02 1 1 2
02 1 2 2
02 0 2 2
;
RUN;
data tempset_i;
set tempset;
by cust_id;
if first.cust_id then row=0;
row+1;
array vars a b c;
do _i = 1 to dim(vars);
varname = cats(vname(vars[_i]),row);
value = vars[_i];
output;
end;
keep cust_id varname value;
run;
proc transpose data=tempset_i out=tempset_t(drop=_name_);
by cust_ID;
id varname;
var value;
run;