Do loop following filevar option in SAS - sas

data &state.&sheet.;
set di;
retain &header.;
infile in filevar= path end=done missover;
do until(done);
if _N_ =1 then
input &headerlength.;
input &allvar.;
output;
end;run;
variable path is in di data set.
I wanna read multiple txt files into one SAS data set. In each txt file the first row is header and I want to retain this header for each observation so I used if _N_ = 1 input header then input second row of other variables for analysis.
The output is very strange. only the first row contains header and other rows are not correct observations.
Could someone help me a little bit? Thank you so much.

I like Shenglin Chen's answer, but here's another option: reset the row counter to 1 each time the data step starts importing a new file.
data &state.&sheet.;
set di;
retain &header.;
infile in filevar= path end=done missover;
do _N_ = 1 by 1 until(done);
if _N_ = 1 then input &headerlength.;
input &allvar.;
output;
end;
run;
This generalises more easily in case you ever want to do something different with every nth row within each file.

Try:
data &state.&sheet.;
set di;
retain &header.;
infile in filevar= path end=done missover dlm='09'x;
input &headerlength.;
do until(done);
input &allvar.;
output;
end;
run;

You should use WHILE (NOT DONE) instead of UNTIL (DONE) to prevent reading past the end of the file, and stopping the data step, when the file is empty. Or for some of the answers when the file only has the header row.

Related

Manually Reading in Data in SAS from CSV

So I have a large dataset that is rather oddly formatted and I want to read it in based on the header. It only has unique columns for each unique participant and each participant participated in multiple rounds of the study. The data is from some experiments and is formatted as having variables for each participant (e.g. "participant.code") then some session variables which I can drop and then the actual variables from the experiment. These are formatted as "study.[round number].player.[variable]"
Rather then repeating the variable for every round, I want to just take out the round number as a separate variable and have an observation for every round for each participant.
I want to read these in differently depending on the variable and pick it out. I would rather not have to manually mess with the source file since the experiment is going to be run multiple times.
If someone could just point me towards some relevant material or whatnot that would be great.
Thank you!
Edit: example of some of the raw data:
participant.id_in_session,participant.code,participant.label,participant._is_bot,participant._index_in_pages,participant._max_page_index,participant._current_app_name,participant._current_page_name,participant.time_started_utc,participant.visited,participant.mturk_worker_id,participant.mturk_assignment_id,participant.payoff,session.code,session.label,session.mturk_HITId,session.mturk_HITGroupId,session.comment,session.is_demo,session.config.real_world_currency_per_point,session.config.participation_fee,session.config.name,session.config.treatment,study.1.player.id_in_group,study.1.player.role,study.1.player.payoff,study.1.player.Seatfinal,study.1.player.finalpay,study.1.player.payroundpay,study.1.player.QCorrect,study.1.player.treatment,study.1.player.Q1a,study.1.player.Q1b,study.1.player.Q1c,study.1.player.Q2a,study.1.player.Q3,study.1.player.Q4,study.1.player.Q5,study.1.player.Q6,study.1.player.Q7,study.1.player.Q80,study.1.player.Q81,study.1.player.Q82,study.1.player.offer,study.1.player.OfferNum,study.1.player.OfferTaken,study.1.player.BuyerNumber,study.1.player.Seatnum2,study.1.player.Seatnum,study.1.player.pay,study.1.player.isoffertaken,study.1.player.hastakenoffer,study.1.player.consent,study.1.player.offerPrice,study.1.player.oprice,study.1.player.guess_num_seller,study.1.player.BoughtPrice,study.1.player.reward,study.1.player.guess_num_buyer,study.1.group.id_in_subsession,study.1.subsession.round_number,study.1.subsession.offersrem,study.1.subsession.game_finished,study.1.subsession.numbuyers,study.1.subsession.bnum,study.1.subsession.payround,study.2.player.id_in_group,study.2.player.role,study.2.player.payoff,study.2.player.Seatfinal,study.2.player.finalpay,study.2.player.payroundpay,study.2.player.QCorrect,study.2.player.treatment,study.2.player.Q1a,study.2.player.Q1b,study.2.player.Q1c,study.2.player.Q2a,study.2.player.Q3,study.2.player.Q4,study.2.player.Q5,study.2.player.Q6,study.2.player.Q7,study.2.player.Q80,study.2.player.Q81,study.2.player.Q82,study.2.player.offer,study.2.player.OfferNum,study.2.player.OfferTaken,study.2.player.BuyerNumber,study.2.player.Seatnum2,study.2.player.Seatnum,study.2.player.pay,study.2.player.isoffertaken,study.2.player.hastakenoffer,study.2.player.consent,study.2.player.offerPrice,study.2.player.oprice,study.2.player.guess_num_seller,study.2.player.BoughtPrice,study.2.player.reward,study.2.player.guess_num_buyer,study.2.group.id_in_subsession,study.2.subsession.round_number,study.2.subsession.offersrem,study.2.subsession.game_finished,study.2.subsession.numbuyers,study.2.subsession.bnum,study.2.subsession.payround,study.3.player.id_in_group,study.3.player.role,study.3.player.payoff,study.3.player.Seatfinal,study.3.player.finalpay,study.3.player.payroundpay,study.3.player.QCorrect,study.3.player.treatment,study.3.player.Q1a,study.3.player.Q1b,study.3.player.Q1c,study.3.player.Q2a,study.3.player.Q3,study.3.player.Q4,study.3.player.Q5,study.3.player.Q6,study.3.player.Q7,study.3.player.Q80,study.3.player.Q81,study.3.player.Q82,study.3.player.offer,study.3.player.OfferNum,study.3.player.OfferTaken,study.3.player.BuyerNumber,study.3.player.Seatnum2,study.3.player.Seatnum,study.3.player.pay,study.3.player.isoffertaken,study.3.player.hastakenoffer,study.3.player.consent,study.3.player.offerPrice,study.3.player.oprice,study.3.player.guess_num_seller,study.3.player.BoughtPrice,study.3.player.reward,study.3.player.guess_num_buyer,study.3.group.id_in_subsession,study.3.subsession.round_number,study.3.subsession.offersrem,study.3.subsession.game_finished,study.3.subsession.numbuyers,study.3.subsession.bnum,study.3.subsession.payround,study.4.player.id_in_group,study.4.player.role,study.4.player.payoff,study.4.player.Seatfinal,study.4.player.finalpay,study.4.player.payroundpay,study.4.player.QCorrect,study.4.player.treatment,study.4.player.Q1a,study.4.player.Q1b,study.4.player.Q1c,study.4.player.Q2a,study.4.player.Q3,study.4.player.Q4,study.4.player.Q5,study.4.player.Q6,study.4.player.Q7,study.4.player.Q80,study.4.player.Q81,study.4.player.Q82,study.4.player.offer,study.4.player.OfferNum,study.4.player.OfferTaken,study.4.player.BuyerNumber,study.4.player.Seatnum2,study.4.player.Seatnum,study.4.player.pay,study.4.player.isoffertaken,study.4.player.hastakenoffer,study.4.player.consent,study.4.player.offerPrice,study.4.player.oprice,study.4.player.guess_num_seller,study.4.player.BoughtPrice,study.4.player.reward,study.4.player.guess_num_buyer,study.4.group.id_in_subsession,study.4.subsession.round_number,study.4.subsession.offersrem,study.4.subsession.game_finished,study.4.subsession.numbuyers,study.4.subsession.bnum,study.4.subsession.payround,study.5.player.id_in_group,study.5.player.role,study.5.player.payoff,study.5.player.Seatfinal,study.5.player.finalpay,study.5.player.payroundpay,study.5.player.QCorrect,study.5.player.treatment,study.5.player.Q1a,study.5.player.Q1b,study.5.player.Q1c,study.5.player.Q2a,study.5.player.Q3,study.5.player.Q4,study.5.player.Q5,study.5.player.Q6,study.5.player.Q7,study.5.player.Q80,study.5.player.Q81,study.5.player.Q82,study.5.player.offer,study.5.player.OfferNum,study.5.player.OfferTaken,study.5.player.BuyerNumber,study.5.player.Seatnum2,study.5.player.Seatnum,study.5.player.pay,study.5.player.isoffertaken,study.5.player.hastakenoffer,study.5.player.consent,study.5.player.offerPrice,study.5.player.oprice,study.5.player.guess_num_seller,study.5.player.BoughtPrice,study.5.player.reward,study.5.player.guess_num_buyer,study.5.group.id_in_subsession,study.5.subsession.round_number,study.5.subsession.offersrem,study.5.subsession.game_finished,study.5.subsession.numbuyers,study.5.subsession.bnum,study.5.subsession.payround,study.6.player.id_in_group,study.6.player.role,study.6.player.payoff,study.6.player.Seatfinal,study.6.player.finalpay,study.6.player.payroundpay,study.6.player.QCorrect,study.6.player.treatment,study.6.player.Q1a,study.6.player.Q1b,study.6.player.Q1c,study.6.player.Q2a,study.6.player.Q3,study.6.player.Q4,study.6.player.Q5,study.6.player.Q6,study.6.player.Q7,study.6.player.Q80,study.6.player.Q81,study.6.player.Q82,study.6.player.offer,study.6.player.OfferNum,study.6.player.OfferTaken,study.6.player.BuyerNumber,study.6.player.Seatnum2,study.6.player.Seatnum,study.6.player.pay,study.6.player.isoffertaken,study.6.player.hastakenoffer,study.6.player.consent,study.6.player.offerPrice,study.6.player.oprice,study.6.player.guess_num_seller,study.6.player.BoughtPrice,study.6.player.reward,study.6.player.guess_num_buyer,study.6.group.id_in_subsession,study.6.subsession.round_number,study.6.subsession.offersrem,study.6.subsession.game_finished,study.6.subsession.numbuyers,study.6.subsession.bnum,study.6.subsession.payround,study.7.player.id_in_group,study.7.player.role,study.7.player.payoff,study.7.player.Seatfinal,study.7.player.finalpay,study.7.player.payroundpay,study.7.player.QCorrect,study.7.player.treatment,study.7.player.Q1a,study.7.player.Q1b,study.7.player.Q1c,study.7.player.Q2a,study.7.player.Q3,study.7.player.Q4,study.7.player.Q5,study.7.player.Q6,study.7.player.Q7,study.7.player.Q80,study.7.player.Q81,study.7.player.Q82,study.7.player.offer,study.7.player.OfferNum,study.7.player.OfferTaken,study.7.player.BuyerNumber,study.7.player.Seatnum2,study.7.player.Seatnum,study.7.player.pay,study.7.player.isoffertaken,study.7.player.hastakenoffer,study.7.player.consent,study.7.player.offerPrice,study.7.player.oprice,study.7.player.guess_num_seller,study.7.player.BoughtPrice,study.7.player.reward,study.7.player.guess_num_buyer,study.7.group.id_in_subsession,study.7.subsession.round_number,study.7.subsession.offersrem,study.7.subsession.game_finished,study.7.subsession.numbuyers,study.7.subsession.bnum,study.7.subsession.payround,study.8.player.id_in_group,study.8.player.role,study.8.player.payoff,study.8.player.Seatfinal,study.8.player.finalpay,study.8.player.payroundpay,study.8.player.QCorrect,study.8.player.treatment,study.8.player.Q1a,study.8.player.Q1b,study.8.player.Q1c,study.8.player.Q2a,study.8.player.Q3,study.8.player.Q4,study.8.player.Q5,study.8.player.Q6,study.8.player.Q7,study.8.player.Q80,study.8.player.Q81,study.8.player.Q82,study.8.player.offer,study.8.player.OfferNum,study.8.player.OfferTaken,study.8.player.BuyerNumber,study.8.player.Seatnum2,study.8.player.Seatnum,study.8.player.pay,study.8.player.isoffertaken,study.8.player.hastakenoffer,study.8.player.consent,study.8.player.offerPrice,study.8.player.oprice,study.8.player.guess_num_seller,study.8.player.BoughtPrice,study.8.player.reward,study.8.player.guess_num_buyer,study.8.group.id_in_subsession,study.8.subsession.round_number,study.8.subsession.offersrem,study.8.subsession.game_finished,study.8.subsession.numbuyers,study.8.subsession.bnum,study.8.subsession.payround,study.9.player.id_in_group,study.9.player.role,study.9.player.payoff,study.9.player.Seatfinal,study.9.player.finalpay,study.9.player.payroundpay,study.9.player.QCorrect,study.9.player.treatment,study.9.player.Q1a,study.9.player.Q1b,study.9.player.Q1c,study.9.player.Q2a,study.9.player.Q3,study.9.player.Q4,study.9.player.Q5,study.9.player.Q6,study.9.player.Q7,study.9.player.Q80,study.9.player.Q81,study.9.player.Q82,study.9.player.offer,study.9.player.OfferNum,study.9.player.OfferTaken,study.9.player.BuyerNumber,study.9.player.Seatnum2,study.9.player.Seatnum,study.9.player.pay,study.9.player.isoffertaken,study.9.player.hastakenoffer,study.9.player.consent,study.9.player.offerPrice,study.9.player.oprice,study.9.player.guess_num_seller,study.9.player.BoughtPrice,study.9.player.reward,study.9.player.guess_num_buyer,study.9.group.id_in_subsession,study.9.subsession.round_number,study.9.subsession.offersrem,study.9.subsession.game_finished,study.9.subsession.numbuyers,study.9.subsession.bnum,study.9.subsession.payround,study.10.player.id_in_group,study.10.player.role,study.10.player.payoff,study.10.player.Seatfinal,study.10.player.finalpay,study.10.player.payroundpay,study.10.player.QCorrect,study.10.player.treatment,study.10.player.Q1a,study.10.player.Q1b,study.10.player.Q1c,study.10.player.Q2a,study.10.player.Q3,study.10.player.Q4,study.10.player.Q5,study.10.player.Q6,study.10.player.Q7,study.10.player.Q80,study.10.player.Q81,study.10.player.Q82,study.10.player.offer,study.10.player.OfferNum,study.10.player.OfferTaken,study.10.player.BuyerNumber,study.10.player.Seatnum2,study.10.player.Seatnum,study.10.player.pay,study.10.player.isoffertaken,study.10.player.hastakenoffer,study.10.player.consent,study.10.player.offerPrice,study.10.player.oprice,study.10.player.guess_num_seller,study.10.player.BoughtPrice,study.10.player.reward,study.10.player.guess_num_buyer,study.10.group.id_in_subsession,study.10.subsession.round_number,study.10.subsession.offersrem,study.10.subsession.game_finished,study.10.subsession.numbuyers,study.10.subsession.bnum,study.10.subsession.payround,study.11.player.id_in_group,study.11.player.role,study.11.player.payoff,study.11.player.Seatfinal,study.11.player.finalpay,study.11.player.payroundpay,study.11.player.QCorrect,study.11.player.treatment,study.11.player.Q1a,study.11.player.Q1b,study.11.player.Q1c,study.11.player.Q2a,study.11.player.Q3,study.11.player.Q4,study.11.player.Q5,study.11.player.Q6,study.11.player.Q7,study.11.player.Q80,study.11.player.Q81,study.11.player.Q82,study.11.player.offer,study.11.player.OfferNum,study.11.player.OfferTaken,study.11.player.BuyerNumber,study.11.player.Seatnum2,study.11.player.Seatnum,study.11.player.pay,study.11.player.isoffertaken,study.11.player.hastakenoffer,study.11.player.consent,study.11.player.offerPrice,study.11.player.oprice,study.11.player.guess_num_seller,study.11.player.BoughtPrice,study.11.player.reward,study.11.player.guess_num_buyer,study.11.group.id_in_subsession,study.11.subsession.round_number,study.11.subsession.offersrem,study.11.subsession.game_finished,study.11.subsession.numbuyers,study.11.subsession.bnum,study.11.subsession.payround,study.12.player.id_in_group,study.12.player.role,study.12.player.payoff,study.12.player.Seatfinal,study.12.player.finalpay,study.12.player.payroundpay,study.12.player.QCorrect,study.12.player.treatment,study.12.player.Q1a,study.12.player.Q1b,study.12.player.Q1c,study.12.player.Q2a,study.12.player.Q3,study.12.player.Q4,study.12.player.Q5,study.12.player.Q6,study.12.player.Q7,study.12.player.Q80,study.12.player.Q81,study.12.player.Q82,study.12.player.offer,study.12.player.OfferNum,study.12.player.OfferTaken,study.12.player.BuyerNumber,study.12.player.Seatnum2,study.12.player.Seatnum,study.12.player.pay,study.12.player.isoffertaken,study.12.player.hastakenoffer,study.12.player.consent,study.12.player.offerPrice,study.12.player.oprice,study.12.player.guess_num_seller,study.12.player.BoughtPrice,study.12.player.reward,study.12.player.guess_num_buyer,study.12.group.id_in_subsession,study.12.subsession.round_number,study.12.subsession.offersrem,study.12.subsession.game_finished,study.12.subsession.numbuyers,study.12.subsession.bnum,study.12.subsession.payround,study.13.player.id_in_group,study.13.player.role,study.13.player.payoff,study.13.player.Seatfinal,study.13.player.finalpay,study.13.player.payroundpay,study.13.player.QCorrect,study.13.player.treatment,study.13.player.Q1a,study.13.player.Q1b,study.13.player.Q1c,study.13.player.Q2a,study.13.player.Q3,study.13.player.Q4,study.13.player.Q5,study.13.player.Q6,study.13.player.Q7,study.13.player.Q80,study.13.player.Q81,study.13.player.Q82,study.13.player.offer,study.13.player.OfferNum,study.13.player.OfferTaken,study.13.player.BuyerNumber,study.13.player.Seatnum2,study.13.player.Seatnum,study.13.player.pay,study.13.player.isoffertaken,study.13.player.hastakenoffer,study.13.player.consent,study.13.player.offerPrice,study.13.player.oprice,study.13.player.guess_num_seller,study.13.player.BoughtPrice,study.13.player.reward,study.13.player.guess_num_buyer,study.13.group.id_in_subsession,study.13.subsession.round_number,study.13.subsession.offersrem,study.13.subsession.game_finished,study.13.subsession.numbuyers,study.13.subsession.bnum,study.13.subsession.payround
1,kppf7hjb,,0,221,221,study,FinalPay,2022-04-16 22:08:18.471115,1,,,0.0,lew8kph3,,,,,0,1.0,0.0,externality_control,0,2,Seller,0.0,1,0,0,10,0,125,125,50,100,50,0,0,0,1,1,,,1,3,,0,1,1,100,0,0,,50.0,,,,,,1,1,6,1,5,6,4,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,1,,0,,,100,0,0,,45.0,,,,,,1,2,6,1,5,6,13,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,0,0,,0,,,100,0,0,,,,,,,,1,3,5,1,5,6,6,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,6,,0,,,138,1,0,,38.0,,,,,,1,4,6,1,5,6,3,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,2,,0,,,135,1,0,,35.0,,,,,,1,5,6,1,5,6,11,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,0,0,,0,,,100,0,0,,,,,,,,1,6,5,1,5,6,6,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,6,,0,,,132,1,0,,32.0,,,,,,1,7,6,1,5,6,4,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,5,,0,,,150,1,0,,50.0,,,,,,1,8,6,1,5,6,9,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,2,,0,,,100,0,0,,49.0,,,,,,1,9,6,1,5,6,10,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,5,,0,,,100,0,0,,39.0,,,,,,1,10,6,1,5,6,3,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,1,,0,,,132,1,0,,32.0,,,,,,1,11,6,1,5,6,10,2,Seller,0.0,,0,0,0,0,,,,,,,,,,,,,1,1,,0,,,130,1,0,,30.0,,,,,,1,12,6,1,5,6,8,2,Seller,0.0,1,192,132,10,0,,,,,,,,,,,,,1,2,,0,,,128,1,0,,28.0,,,,,,1,13,6,1,5,6,11
Your file is not really as complicated as it first seems. For example the bulk of the data is just 43 columns that repeat 13 times. The STUDY.1 columns, then STUDY.2 columns etc.
For this one just write a program to read it. There are 22 columns that are not "study" columns. Then 13 copies of the 43 study columns.
data want;
infile csv dsd truncover firstobs=2;
input var1 ..... var22 #;
do study=1 to 13;
input svar1 .... svar43 # ;
output;
end;
run;
So you turn each line into 13 observations (study=1 to study=13).
To complete the sketch of a data step above you just need figure out want names you want to use for the 65 (22 + 43) variables other than STUDY. And for each variable what type of variable it is, numeric or character, and when character what length it needs to store the longest possible value.
If you need to work with a lot of different variations of files in this style then it might be worth working on a program to analyze the headers and determine the role of the columns based on the pattern of the header name and perhaps generate the code to read the file.
You might start by building a dataset with just the header names.
data headers;
infile csv dsd obs=1 ;
length col 8 words 8 ;
col+1;
array header [4] $50 ;
input header1 :$50. ## ;
words=countw(header1,'.');
do _n_=words to 1 by -1;
header[_n_] = scan(header1,_n_,'.');
end;
run;
You can use that list of the headers to help you figure out what would be useful names for the variables.
If you want to let SAS guess how to define and name the variables you could try splitting the CSV file into two separate CSV files. One with the first 22 columns and one with the other 43. So first split the headers (perhaps removing the STUDY.N. prefix while you are at it). Then split the data. Add an ROW number to make it easy to join them later.
filename single temp;
filename multiple temp;
data _null_;
infile csv dsd obs=1 ;
input header :$50. ## ;
file single dsd ;
if _n_=1 then put 'ROW,' #;
if _n_<= 22 then put header #;
else do;
file multiple dsd;
if _n_=23 then put 'ROW,STUDY,'# ;
call scan(header,3,pos,len,'.');
header = substr(header,pos);
put header #;
end;
if _n_=22+43 then stop;
run;
data _null_;
infile csv dsd firstobs=2 truncover ;
row+1;
length s1-s43 $200 ;
input s1-s22 #;
file single dsd mod;
put row s1-s22 ;
file multiple dsd mod;
do study=1 to 13 ;
input s1-s43 # ;
put row study s1-s43 ;
end;
run;
Now you can use PROC IMPORT to GUESS how to read SINGLE and MULTIPLE and then you can join them back together.
proc import file=single dbms=csv out=single replace;
run;
proc import file=multiple dbms=csv out=multiple replace;
run;
data want;
merge single multiple;
by row;
run;

How to get last modified time of file while using infile statement?

I am going to analyze a batch of SAS program file and I am stucked in getting the last modified time of program files. I have thought about X command but it was too inefficient.
I just find when I use infile statement:
data test;
infile 'D:\test.txt' truncover;
input ;
run;
Log shows the last modified time:
NOTE: The infile 'D:\test.txt' is:
Filename=D:\test.txt,
RECFM=V,LRECL=32767,File Size (bytes)=7,
Last Modified=2021/1/26 15:25:48,
Create Time=2021/1/26 15:25:42
As you can see, log window shows the infomation of file as a NOTE. However, my wish output is a variable filled with Last Modified Time.
Is there some option to get it while using infile statement?
Surely, Other efficient ways are welcomed, too.
Use functions FOPEN and FINFO
Example:
Show all available information items and their value for a sample data file.
filename datafile 'c:\temp\datafile.txt';
data _null_;
file datafile;
put 'Me data';
run;
data _null_;
fid = fopen('datafile');
if fid then do;
do index = 1 to foptnum(fid);
info_name = foptname(fid,index);
info_value = finfo(fid, info_name);
put index= info_name= #40 info_value=;
end;
rc = fclose(fid);
end;
run;
Will log information such as
index=1 info_name=Filename info_value=c:\temp\datafile.txt
index=2 info_name=RECFM info_value=V
index=3 info_name=LRECL info_value=32767
index=4 info_name=File Size (bytes) info_value=9
index=5 info_name=Last Modified info_value=26Jan2021:06:29:47
index=6 info_name=Create Time info_value=26Jan2021:06:28:23

How do I import multiple unformatted data files into SAS while skipping multiple lines for each file?

I am trying to import multiple unformatted data files in a single folder into a SAS dataset using a '*.xle' wildcard while skipping the first 47 lines of each file. SAS will use the 'firstobs=48' for the first file but will ignore for each subsequent file and begin reading at line 1. I have set up the code using the eov=0 as suggested on multiple other Stackoverflow threads, but it still does not seem to work. Any help is much appreciated. Please see my code below:
data test;
infile "*.xle" eov=eov firstobs=48;
input #;
if eov then input;
input Date $ 19-28 / Time $ 19-26 // Data 18-24 / Temp 18-22 //;
eov=0;
run;
You are very close you need to input 47 times when you start a new file EOV=1.
Alternatively you could use FILEVAR and FIRSTOBS would work for each file but that would require generating a list of filenames to use to drive the data step. Six vs. half dozen so to speak.
filename FT15F001 '.\a.xle';
parmcards;
a line 1
a line 2
a line 3
a line 4
;;;;
filename FT15F001 '.\b.xle';
parmcards;
b line 1
b line 2
b line 3
b line 4
;;;;
filename FT15F001 '.\c.xle';
parmcards;
c line 1
c line 2
c line 3
c line 4
;;;;
data test;
infile "*.xle" eov=eov firstobs=3 length=l;
input #;
if eov then do;
do _n_ = 1 to 2; input; end;
eov=0;
end;
input line $varying40. l;
list;
run;
proc print;
run;
It is possible to use the variable created by the EOV= option, but I have found it easier to just use FILENAME= instead and then use LAG() function to detect when a new file starts.
To skip 48 lines you could execute multiple INPUT statements or add multiple / characters to one INPUT statement.
data test;
length fname $256 ;
infile "*.xle" filename=fname ;
input #;
if fname ne lag(fname) then do;
input %sysfunc(repeat(/,48-1));
end;
input Date $ 19-28 / Time $ 19-26 // Data 18-24 / Temp 18-22 //;
run;
Note that if any of the files is actually shorter than expected then you will need to be more careful in both the skipping step and the reading step. Otherwise when you read multiples lines in one INPUT statement you could read past the end of one file and start reading lines from the next file.
It might be better to get the list of files first and use that with the FILEVAR= option to drive the process. Then INFILE executes separately for each file and you can use the FIRSTOBS= option. You will then need to add a loop to read and output the observations from the text file(s). This way each iteration of the data step will process one whole file.
data files;
infile "ls *.xle" pipe truncover ;
input filename $256.;
run;
data test;
set files ;
fname=filename ;
infile dummy filevar=fname firstobs=48 end=eof;
do while (not eof);
input Date $ 19-28 / Time $ 19-26 // Data 18-24 / Temp 18-22 //;
output;
end;
run;
But again reading multiple lines in one INPUT statement is dangerous and you should change the code that reads the lines to read them one by one and check that you have not read past the end of the file yet. Remember that SAS will stop the whole data step if INPUT statement (or SET statement) reads past the end of the input stream.

Read in Files with pattern match into one SAS dataset

I would like to read a number of .csv files into a single SAS dataset using a pattern match. For example if in the directory /home/datasets there are 5 files:
/home/datasets
~/output_group1a.csv
~/output_group1b.csv
~/output_group1c.csv
~/output_group2a.csv
~/output_group2b.csv
All with known and identical structures and data types. I would like to read in only those files corresponding to group 1 without having to explicitly specify the filenames.
You can use a wildcard in your infile statement. If you have headers in each file you'll need to account for that. Here's a bit more of an example.
https://gist.github.com/statgeek/4c27ea9a7ed6d3528835
data try01;
length filename txt_file_name $256;
retain txt_file_name;
infile "Path\*.txt" eov=eov filename=filename truncover;
input#;
if _n_ eq 1 or eov then do;
txt_file_name = scan(filename, -2, ".\");
eov=0;
end;
else input
*Place input code here;
;
run;

Get the current observation count in SAS

I have a file in which the first line is a header line containing some meta-data information.
How can I get the current observation number(say =1 for the first observation) that the SAS processor is dealing with so that I can put in a IF clause to handle such special data line.
Follow up: I want to process the first line and keep one of the column values in a local variable for further processing. I don't want to keep this line in my final output. is this possible?
The automatic variable _N_ returns the current iteration number of the SAS data step loop. For a traditional data step, ie:
data something;
set something;
(code);
run;
_N_ is equivalent to the row number (since one row is retrieved for each iteration of the data step loop).
So if you wanted to only do something once, on the first iteration, this would accomplish that:
data something;
set something;
if _n_ = 1 then do;
(code);
end;
(more code);
run;
For your follow up, you want something like this:
data want;
set have;
retain _temp;
if _n_ = 1 then do;
_temp = x;
end;
... more code ...
drop _temp;
run;
DROP and RETAIN statements can appear anywhere in the code and have the same effect, I placed them in their human-logical locations. RETAIN says to not reset the variable to missing each time through the data step loop, so you can access it further down.
if you are reading a particularly large text file, you may want to avoid having to execute the (if _n_=1 then) condition for every iteration. You can do this by reading the file twice - once to extract the header row, and again to read in the file, as follows:
data _null_; /* create dummy file for demo purposes */
file "c:\myfile.txt";
put 'blah'; output;
put 'blah blah blah 666'; output;
data _null_; /* read in header info */
infile "c:\myfile.txt";
input myvar:$10.; /* or wherever the info is that you need */
call symput('myvar',myvar);/* create macro variable with relevant info */
stop; /* no further processing at this point */
data test; /* read in data FROM SECOND LINE */
infile "c:\myfile.txt" firstobs=2 ; /* note the FIRSTOBS option */
input my $ regular $ input $ statement ;
remember="&myvar";
run;
For short / simple stuff though, Joe's answer is better as it's more readable.. (and may be more efficient for small files).