Why these many observations are created in the following program? - sas

data test;
infile cards dsd dlm=', .';
input stmt : $ ##;
cards;
T
;run;
/*-----------------------------------------------*/
data test;
infile cards dsd dlm=', .';
input stmt : $ ##;
cards;
Th
;run;
/*-----------------------------------------------*/
data test;
infile cards dsd dlm=', .';
input stmt : $ ##;
cards;
This is SAS.
;run;
When first program is run, 80 observations are created
When second program is run, 79 observations are created
When third program is run, 72 observations are created
I know these program has worst programming style. Wrong options are set for wrong technique. DSD option is set, double trailing operator ## (line holder), Colon modifier (:) are used and more than 1 delimeter is used which is worst SAS programming ever.
Aside from this I want to know why so many observations are created, why 80? 79? how program is executed? I think DSD option & 2 delimeters have major impact. Can anyone explain?

The reason you get more records than you expect is because CARDS are fixed length records. The reason you get a difference number of records is because there is a different number of null fields left after reading the non-null field(s). You can see this by adding the COL option to the INFILE statement to show you where the column pointer is after reading each field. Col=3, 4 , 13
data test;
infile cards dsd dlm=', .' col=c;
input stmt : $ ##;
col=c;
cards;
T
;run;
proc print data=test(obs=5);
/*-----------------------------------------------*/
data test;
infile cards dsd dlm=', .' col=c;
input stmt : $ ##;
col=c;
cards;
Th
;run;
proc print data=test(obs=5);
/*-----------------------------------------------*/
data test;
infile cards dsd dlm=', .' col=c;
input stmt : $ ##;
col=c;
cards;
This is SAS.
;run;
proc print data=test(obs=5);
run;

Related

SAS importing messy data

I tried formatting the data so that U is only numeric while all descriptions should be under T. May i know what could i possibly do to fix it?
DATA data;
infile '....csv'
dlm=',' firstobs=2 dsd;
format A$ B$ C$ D$ E$ F$ G$ H$ I$ J$ K$ L$ M$ N$ O$ P$ Q$ R$ S$ T$ U V W$ X$ Y$ Z$ AA$ AB$ AC$ AD$ AE$ AF$ AG$ AH$ AI$ AJ$ AK$ AL$ AM$ AN$ AO$ AP$ AQ$ AR$ AS;
input A B C D E F G H I J K L M N O P Q R S T#;
do _n_=1 to 24;
input U #;
description=catx(', ',T, U);
end;
input U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS;
RUN;
If you are talking about the data file in this Kaggle project then I would use a divide and conquer approach. Check each line in the file to see how many columns it contains. Then split the problem lines into separate file(s) and figure out how to handle whatever issue it is that causes them to be poorly formatted/parsed.
So get a list of the rows and number of columns in each row.
data maxcol;
infile "C:\downloads\archive.zip" zip member='Datafiniti_Mens_Shoe_Prices.csv'
dsd truncover column=cc length=ll lrecl=2000000
;
row+1;
input #;
do maxcol=1 by 1 while(cc<=ll); input dummy :$1. # +(-1) dummy $char1. #; end;
if dummy ne ',' then maxcol=maxcol-1;
keep row maxcol ;
run;
proc freq ;
tables maxcol;
run;
For example you could get the list of bad row numbers into a macro variable.
proc sql noprint;
select row into :rowlist separated by ' ' from maxcol where maxcol > 48 ;
quit;
Then use that macro variable in your code that reads the datafile.
data want;
infile "C:\downloads\archive.zip" zip member='Datafiniti_Mens_Shoe_Prices.csv' dsd
truncover lrecl=2000000
;
input #;
if _n_ in (1 &rowlist) then delete;
... rest of data step to read the "good" rows ...
For the other rows take a look at them and figure out where they are getting extra columns inserted. Possibly just fix them by hand. Or craft separate data steps to read each set separately using the same &ROWLIST trick.
If you are positive that
the extra columns are inserted between column 20 and 21
that column 21 always has a valid numeric string
none of the extra values are valid numeric strings
then you could use logic like this to generate a new delimited file (why not use | as the delimiter this time?).
data _null_;
infile "C:\downloads\archive.zip" zip member='Datafiniti_Mens_Shoe_Prices.csv' dsd
truncover lrecl=2000000
;
row+1;
length col1-col48 $32767;
input col1-col21 #;
if _N_>1 then do while(missing(input(col21,??32.)));
col20=catx(',',col20,col21);
input col21 #;
end;
input col22-col48;
file "c:\downloads\shoes.txt" dsd dlm='|' lrecl=2000000 ;
put col1-col48 ;
run;
Which you could even then try to read using PROC IMPORT to guess how to define the variables. (But watch out as PROC IMPORT might truncate some of the records by using LRECL=32767)
proc import datafile="c:\downloads\shoes.txt" dbms=csv out=want replace ;
delimiter='|';
guessingrows=max;
run;
Checking column 21:
The MEANS Procedure
Analysis Variable : prices_amountMin
N Mean Std Dev Minimum Maximum
---------------------------------------------------------------------
19387 111.8138820 276.7080893 0 16949.00
---------------------------------------------------------------------

SAS group by first digits

I have a variable in SAS with a lot of numbers, for example 11000, 30129, 11111, 30999. I want to group this by the first two digits so "11000 and 11111" and "30129 and 30999" will be in a own table.
It's quite simple,
You have to create a second column and extract the 2 first digit.
Then sort the dataset by this second columns.
data test;
infile datalines dsd ;
input a : 15. ;
datalines;
11000,
30129,
11111,
309999,
;
run;
data test_a;
length val_a $2;
set test;
val_a= SUBSTRN(a,1,2);
run;
proc sort data=test_a out=test_b;
by val_a;
run;
Result will be :
val_a a
11 11000
11 11111
30 30129
30 309999
And then you can create 2 dataset with selection on the val_a like this :
data want data_11 data_30;
set test_b;
if val_a = 11 then output data_11;
if val_a = 30 then output data_30;
run;
Regards,
I think I did like you, but my new column only shows with ".". But I think your answer can give me some help anyways, thank you!
data books;
infile "&path\Boken.csv" dlm=';' missover dsd firstobs=2;
input ISBN: $12.
Book: $quote150.;
run;
data test_a;
format val_ISBN 15.;
set books;
val_ISBN= SUBSTRN(ISBN,1,2);
run;
proc sort data=test_a out=test_b;
by val_ISBN;
run;
proc print data=test_b (obs=10) noobs ;
run;

SAS - datalines equivalent of MISSOVER / TRUNCOVER statement?

I know for the infile statement, you can add a missover and truncover statement for missing values in the input, but if I'm using datalines for data instead of infile, is there an equivalent statement I can use?
You can still use infile by following it with datalines and then missover or truncover etc.
data _null_;
infile datalines missover;
input a b c;
put _all_;
datalines;
1 2
3 4
;
run;

usage of missover statement in SAS

I have below dataset
data ab;
infile cards missover;
input m p c;
cards;
1,2,3
4,5,
6,7,
run;
The output of this query is
m p c
. . .
. . .
. . .
Why did i get the below output instead of error?
I havent specified any delimiter also.
Please explain.
Thanks in Advance,
Nikhila
You do get INVALID DATA messages. SAS is defaulting to space delimited fields, you need to specify the DSD INFILE statement option and or DLM=','. You don't actually need MISSOVER as you have the proper number of delimiters for three comma delimited fields, but I would probably go ahead and keep it.
24 data ab;
25 infile cards missover;
26 input m p c;
27 cards;
NOTE: Invalid data for m in line 28 1-5.
RULE: ----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+----0
28 1,2,3
m=. p=. c=. _ERROR_=1 _N_=1
NOTE: Invalid data for m in line 29 1-4.
29 4,5,
m=. p=. c=. _ERROR_=1 _N_=2
NOTE: Invalid data for m in line 30 1-4.
30 6,7,
m=. p=. c=. _ERROR_=1 _N_=3
NOTE: The data set WORK.AB has 3 observations and 3 variables.
24 data ab;
25 infile cards dsd missover;
26 input m p c;
27 cards;
NOTE: The data set WORK.AB has 3 observations and 3 variables.
The MISSOVER is what makes you have three observations instead just one. Without the MISSOVER then SAS will try to read each line as one value and you will end up with one observation of all missing values. It is easier to see if you change your variables to character instead of numeric since you can see where the values end up.
data ab;
infile cards missover;
input m $ p $ c $;
put (m p c) (=);
cards;
1,2,3
4,5,
6,7,
;
m=1,2,3 p= c=
m=4,5, p= c=
m=6,7, p= c=
data ab;
infile cards /*missover */;
input m $ p $ c $;
put (m p c) (=);
cards;
1,2,3
4,5,
6,7,
;
m=1,2,3 p=4,5, c=6,7,
NOTE: SAS went to a new line when INPUT statement reached past the end of a line.

SAS: dlm and dsd?

I am confused about what DSD actually does in terms of "moving the pointer" and reading in data. To better explain, look at the following code:
data one;
infile cards dlm=',' TRUNCOVER ; /*using dlm','*/
input cust_id date ddmmyy10. A $ B $ C $;
cards;
1,10/01/2015,5000,dr
;
run;
data two;
infile cards dsd TRUNCOVER ;
input cust_id date ddmmyy10. A $ B $ C $;
cards;
1,10/01/2015,5000,dr
;
run;
The dataset one contains values for A and B of 5000 and dr but the dataset two contains values of A as missing whereas B and C are 5000 and dr. I don't get why the dsd sets A to missing.
Thanks!
Your problem is not DLM or DSD it is "DATE DDMMYY10." that is inFORMATTED input which is not compatible with delimited input in any form DSD or NO.
You need INFORMAT statement or : informat modified.
date :DDMMYY10.