How to skip certain lines with random character sequences in file - sas

Question

I have a file in following format:
Name Salary Age
bob 10000 18
sally 5555 20
#not found 4fjfjhdfjfnvndf
#not found 4fjfjhdfjfnvndf
9/2-10/2
but then I have random points in the file where there are 4-6 lines of random characters. The files has 2 million rows. I was wondering if the infile statement automatically skips these random spurt of lines or do I have to go into the file and delete these lines automatically.

Answer 1

You probably have to deal with them in some fashion. If you have truncover or missover on the infile statement, it won't do any harm (you must have one, though, or it might cause your next lines to get shifted over). But you'll have a garbage line in your program that you need to deal with.
The quick and dirty method would be something like this:
data have;
infile "blah.txt" dlm=' ' dsd lrecl=32767 truncover;
input name $ salary age;
if missing(salary) and missing(age) then delete;
run;
If the garbage was likely to generate missing values for the numerics, that would work. However, your log probably has some warnings in it that aren't great, and this isn't perfect in what it finds, either, if the garbage might be numeric values. (If it's entirely numeric values, you could test if name is a number.)
The better method is to preprocess _infile_ - which is a bit more 'advanced' but certainly a good approach.
data have;
infile "blah.txt" dlm=' ' dsd lrecl=32767 truncover;
input #;
if countw(_infile_) ne 3 then delete; *if there are not exactly 3 "words" then delete it;
if notdigit(scan(_infile_,2)) or notdigit(scan(_infile_,3)) then delete; *if the 2nd or 3rd word contain non-digit values then delete;
input name $ salary age;
run;
Both approaches require some consistency with data to work, and probably require some tweaking - for example if salary and age are acceptable to be missing, both of these would delete rows you don't want deleted.