in a datastep of this kind
ID VAR_1 VAR_2 VAR_3 ...
1 a1 b1 mv ...
2 a2 b2 mv ...
3 a3 b3 c3 ...
4 a4 mv mv ...
5 a5 b5 mv ...
6 a6 b6 mv ...
where the number of the variables are not known (i want to generalize as more as possible my code) I want to obtain a dataset like this (something like an inverted proc transpose):
ID VAR
1 a1
1 b1
2 a2
2 b2
3 a3
3 b3
3 c3
....
So i'm splitting the dataset in a nonfixed number of temp datasets, which one contains ID and only one column, trashing observation with missing values, then I'll merge all these temporary datasets obtaining my result. And this works.
But the call execute has a very high computational complexity, I mean, if I try to do this operation in a dataset with only one column (dropping missing values) my garbage computer takes 0.1 secs, while using a call execute in a dataset with 6 columns it won't take 0.1*6=0.6 secs, It will take some minutes. This because it won't work in column but in row, and this is SAS and I must get over it. But I'm asking myself (and now I'm asking to you) if there are some other ways for obtaining my results without this computational time. Here a focus on the code:
data _null_;
set old;
array try[*] VAR: ;
do i=1 to DIM(try);
call execute(catt("data var",i,"; set old; if var_",i," = ' ' then delete; allvarnew= col",i,"; ` `drop COL:; run;" ));
end;
run;
columns are char $1 (ID is char $4).
columns are the result of a proc transpose.
thanks.
I'm not sure of the efficiency of this, but it requires only one data-step as opposed to the multiple data-steps in the call execute approach described:
data new (drop=var_: i);
set test;
array try[*] VAR_: ;
do i=1 to DIM(try);
var=try[i]; output;
end;
run;
Related
I'd like to make the dataset like the below. I got it, but it’s a long program.
I think it would become more simple. If you have a good idea, please give me some advice.
This is the data.
data test;
input ID $ NO DAT1 $ TIM1 $ DAT2 $ TIM2 $;
cards;
1 1 2020/8/4 8:30 2020/8/5 8:30
1 2 2020/8/18 8:30 2020/8/19 8:30
1 3 2020/9/1 8:30 2020/9/2 8:30
1 4 2020/9/15 8:30 2020/9/16 8:30
2 1 2020/8/4 8:34 2020/8/5 8:34
2 2 2020/8/18 8:34 2020/8/19 8:34
2 3 2020/9/1 8:34 2020/9/2 8:34
2 4 2020/9/15 8:34 2020/9/16 8:34
3 1 2020/8/4 8:46 2020/8/5 8:46
3 2 2020/8/18 8:46 2020/8/19 8:46
3 3 2020/9/1 8:46 2020/9/2 8:46
3 4 2020/9/15 8:46 2020/9/16 8:46
;
run;
This is my program.
data
t1(keep = ID A1 A2 A3 A4)
t2(keep = ID B1 B2 B3 B4)
t3(keep = ID C1 C2 C3 C4)
t4(keep = ID D1 D2 D3 D4);
set test;
if NO = 1 then do;
A1 = DAT1;
A2 = TIM1;
A3 = DAT2;
A4 = TIM2;
end;
*--- cut (NO = 2, 3, 4 are same as NO = 1)--- ;
end;
if NO = 1 then output t1;
if NO = 2 then output t2;
if NO = 3 then output t3;
if NO = 4 then output t4;
run;
proc sort data = t1;by ID; run;
proc sort data = t2;by ID; run;
proc sort data = t3;by ID; run;
proc sort data = t4;by ID; run;
data test2;
merge t1 t2 t3 t4;
by ID;
run;
Since the result looks like a report use a reporting tool.
proc report data=test ;
column id no,(dat1 tim1 dat2 tim2 n) ;
define id / group width=5;
define no / across ' ' ;
define n / noprint;
run;
Tall to very wide data transformations are typically
sketchy, you put data into metadata (column names or labels) or lose referential context, or
a reporting layout for human consumption
Presuming your "as dataset like below" is accurate and you want to pivot your data in such a manner.
Way 1 - self merging subsets with renaming
You should see that the NO field is a sequence number that can be used as a BY variable when merging data sets.
Consider this example code as a template that could be the source code generation of a macro:
NO is changed name to seq for better clarity
data want;
merge
have (where=(seq=1) rename=(dat1=A1 tim1=B1 dat2=C1 tim2=D1)
have (where=(seq=2) rename=(dat1=A2 tim1=B2 dat2=C2 tim2=D2)
have (where=(seq=3) rename=(dat1=A3 tim1=B3 dat2=C3 tim2=D3)
have (where=(seq=4) rename=(dat1=A4 tim1=B4 dat2=C4 tim2=D4)
;
by id;
run;
For unknown data sets organized like the above pattern, the code generation requirements should be obvious; determine maximum seq and have the names of variables to pivot be specified (as macro parameters, in which loop over the names occurs).
Way 2 - multiple transposes
Caution, all pivoted columns will be character type and contain the formatted result of original values.
proc transpose data=have(rename=(dat1=A tim1=B dat2=C tim2=D)) out=stage1;
by id seq;
var a b c d;
run;
proc transpose data=stage1 out=want;
by id;
var col1;
id _name_ seq;
run;
Way 3 - Use array and DOW loop
* presume SEQ is indeed a unit monotonic sequence value;
data want (keep=id a1--d4);
do until (last.id);
array wide A1-A4 B1-B4 C1-C4 D1-D4;
wide [ (seq-1)*4 + 1 ] = dat1;
wide [ (seq-1)*4 + 2 ] = tim1;
wide [ (seq-1)*4 + 3 ] = dat2;
wide [ (seq-1)*4 + 4 ] = tim2;
end;
keep id A1--D4;
* format A1 A3 B1 B3 C1 C3 D1 D3 your-date-format;
* format A2 A4 ................. your-time-format;
Way 4 - change your data values to datetime
I'll leave this to esteemed others
Let's say I have data that look like this:
DATA temp;
INPUT id a1 b2 d1 f8;
DATALINES;
1 2.3 2.1 4.2 1.2
2 5.3 2.3 1.5 3.2
3 1.2 5.4 6.6 6.6
;
run;
What I want to do is use the data and set statements to say that if the values in a1 and f8 are less than the means of a1 and f8 (respectively), then those values are missing. So the resulting dataset would look like:
id a1 b2 d1 f8
1 . 2.1 4.2 .
2 5.3 2.3 1.5 .
3 . 5.4 6.6 6.6
Any tips for how I would start on this? I'm new to SAS and the examples in the manuals have not been very helpful. I had been thinking of something like this (but it doesn't work):
DATA temp2;
SET temp;
IF a1 < mean(a1) THEN a1=.;
IF f8 < mean(f8) THEN f8=.;
RUN;
The SAS implementation of SQL can do automatic application of group or data wise aggregates against a result set.
Proc SQL;
create table want as
select
case when (a1 < mean(a1)) then . else a1 as a1,
b2,
d1,
case when (f8 < mean(f8)) then . else f8 as f8
from have;
A solution that uses DATA step will need to precompute data set statistics, commonly with a procedure such as MEANS, SUMMARY or UNIVARIATE.
proc means noprint data=have;
output out=have_means mean(a1 f8)= / autoname;
run;
data want;
if _n_ = 1 then do;
set have_means(keep=a1_mean f8_mean);
end;
set have;
if a1 < a1_mean then a1 = .;
if f8 < f8_mean then f8 = .;
drop a1_mean f8_mean;
run;
Other techniques can update a data set in place and would use SQL UPDATE or DATA step MODIFY
mean function is applied across the row and not within the column in datastep, that is why you are not getting the results. #Richard answer is perfect. to do in datastep to get mean, you need to use DOW loop and then append with main dataset. It is much easier to use proc summary as #Richard explains.
data temp2_intial(keep= mean_a1 mean_f8);
do until(eof);
set temp end =eof;
tot_a1 = sum(tot_a1, a1);
cnt_a1=sum(cnt_a1,1);
mean_a1 = tot_a1/cnt_a1;
tot_f8 = sum(tot_f8, f8);
cnt_f8=sum(cnt_f8,1);
mean_f8 = tot_f8/cnt_f8;
end;
run;
data temp2(drop= mean_a1 mean_f8);
set temp ;
if _n_ =1 then set temp2_intial;
IF a1 < mean_a1 THEN a1=. ;
IF f8 < mean_f8 THEN f8=.;
run;
I just started using SAS and I'm trying to combine columns.
I've got table mainData
A1 A2 A3 A4
1 4 7 10
2 5 8 11
3 6 9 12
I want to create a new table rearrangedData
Type Value
A1 1
A1 2
A1 3
A2 4
A2 5
A2 6
A3 7
A3 8
A3 9
A4 10
A4 11
A4 12
There must be a simple solution to this I just can't figure this out. I'm thinking of writing do loop, but what if I don't know size of a table or amount of lines in a specific column. I can't figure how I would get such information in SAS.
This somewhat unusual transformation can be done via a transpose and some array logic:
data have;
input A1 A2 A3 A4;
cards;
1 4 7 10
2 5 8 11
3 6 9 12
;
run;
proc transpose data = have out = tr name=type prefix = r;
run;
data want;
set tr;
array r{*} r:;
do i = 1 to dim(r);
value = r[i];
output;
end;
drop i r:;
run;
Also, this preserves the original order without requiring a sort.
Make a dummy variable, then transpose data.
data have;
set have;
id=_n_;
run;
proc transpose data=have out=temp;
by id;
var A1-A4;
run;
proc sort data=temp out=want(rename=(_name_=type col1=value) drop=id);
by _name_;
run;
If you want to preserve the original order then you could use the POINT= option on the SET statement to loop over the data set once per variable (column).
So this data set will read the first observations just to get the variables defined. Then define the array VALUES so that we can use DIM(VALUES) to know how many columns. Then it uses the POINT= and NOBS= options on the SET statement to control the other loop. It uses the VNAME() function to find the name of the current variable in the array.
data want ;
set have ;
array values _numeric_;
do col=1 to dim(values);
length type $32 value 8;
type=vname(values(col));
do row=1 to nobs ;
set have point=row nobs=nobs ;
value=values(col);
output;
keep type value;
end;
end;
stop;
run;
For example, i have a data set like this (the value a1 a2 a3 b1 b2 b3 are numeric):
A B
a1 b1
a2 b2
a3 b3
I want to compare the average of 2 class A and B using proc ttest. But it seems that i have to change my data set in order to use this proc. I read lots of tutorials about the proc ttest and all of them use the data sets in this form below:
class value
A a1
A a2
A a3
B b1
B b2
B b3
So my question is: Does it exist a method to do the proc ttest without changing my data set?
Thank you and sorry for my bad english :D
The short answer is no, you can't run a ttest in SAS that compares multiple columns. proc ttest, when used for 2 samples, relies on the variable in the class statement to compare the groups. Only one variable can be entered and it must have 2 levels, therefore the structure of your data is not compatible with this.
You will therefore need to change the data layout, although you could do this in a view so that you don't create a new physical dataset. Here's one way to do that.
/* create dummy data */
data have;
input A B;
datalines;
10 11
15 14
20 21
25 24
;
run;
/* create a view that turns vars A and B into a single variable */
data have_trans / view=have_trans;
set have;
array vals{2} A B;
length grouping $2;
do i = 1 to 2;
grouping = vname(vals{i}); /* extracts the current variable name (A or B) */
value = vals{i}; /* extracts the current value */
output;
end;
drop A B i; /* drop unwanted variables */
run;
/* perform ttest */
proc ttest data=have_trans;
class grouping;
var value;
run;
data a1
col1 col2 flag
a 2 .
b 3 .
a 4 .
c 1 .
For data a1, flag is always missing. I want to update multiple rows using a2.
data a2
col1 flag
a 1
Ideal output:
col1 col2 flag
a 2 1
b 3 .
a 4 1
c 1 .
But this doesn't update all the records in by statement.
data a1;
modify a1 a2;
by col1;
run;
Question edited
Actually a1 is a very large data set on server. Hence I prefer to modify it (if possible) instead of creating a new one. Otherwise I have to drop previous a1 first and copy a new a1 from local to server, which will take much more time.
If you want to do this with MODIFY, you have to loop over the modify dataset in some fashion or it will only replace the first row (because the other dataset will then run out of records - normally this behaves like merge, where once it finds a match it advances to next record). Here's one option - there are others.
data a1(index=(col1));
input col1 $ col2 flag;
datalines;
a 2 .
b 3 .
a 4 .
c 1 .
;;;;
run;
data a2(index=(col1));
col1='a';
flag=1;
run;
data a1;
set a2(rename=flag=flag2);
do _n_ = 1 to nobs_a1;
modify a1 key=col1 nobs=nobs_a1;
if _iorc_=0 then do;
flag=flag2;
replace;
end;
end;
if _iorc_=%sysrc(_DSENOM) then _error_=0;
run;
If you're not using Merge statement for the sorting problem, you can simply change your merging approach.
If flag in A1 is always missing, you can drop it, otherwise you should temporary rename it for not losing those informations.
Here I will merge A1 and A2 using hash objects, this approach doesn't require any prior sorting on datasets.
data final_merged(drop = finder);
length flag 8.; /*please change length with the real one, use $ if char*/
if _N_ = 1 then do;
declare hash merger(dataset:'A2');
merger.definekey('col1');
merger.DefineData ('flag');
merger.definedone();
end;
set A1(drop=flag);
finder = merger.find();
if finder ne 0 then flag = .;
/*then flag='' or then flag='unknown' as you want if flag is a character var*/
run;
Please, let me know if this will help.
You could do the following but SQL sorts the observations so not sure how useful this would be for you? (you could always preprocess with ordvar=_n_; and then sort the SQL statement on it if that helps):
Data:
data a1 ;
input col1 $ col2 flag ;
cards ;
a 2 .
b 3 .
a 4 .
c 1 .
;run ;
data a2 ;
input col1 $ flag ;
cards ;
a 1
;run ;
Merge:
proc sql ;
create table output as
select a.col1, a.col2, b.flag
from a1 a
left join
a2 b
on a.col1=b.col1
;quit ;
To try and do it in one pass, how about creating two macros variables containing the mapping from a2?
proc sql ;
select distinct col1, flag
into :colvals separated by '', :flagvals separated by ''
from a2
;quit ;
Set flag to the corresponding character position between the two macro variables:
data a1 ;
set a1 ;
if findc("&colvals",col1) then
flag=input(substr("&flagvals", findc("&colvals",col1),1),8.) ;
run ;