sas macro to read multiple multsheet excel in folder - sas

this is my code to read multiple multisheet excel in sas its giving an error which i am attaching in last.i am only reading first sheet named summary of every excel in that particular folder
%macro sks2sas01(input=d:\excels,out=work.tt);
/* read files in directory */
%let dir=%str(%'dir %")&input.%str(\%" /A-D/B/ON%');
filename myfiles pipe %unquote(&dir);
data list1; length fname $256.;
infile myfiles truncover;
input myfiles $100.;
/* put _infile_;*/
fname=quote(upcase(cats("&input",'\',myfiles)));
out="&out";
drop myfiles;
call execute('
PROC IMPORT DBMS=EXCEL2002 OUT= _1
DATAFILE= '||fname||' REPLACE ;
sheet="summary";
RUN;
proc append data=_1 base='||out||' force; run;
proc delete data=_1; run;
');
run;
filename myfiles clear;
%mend sks2sas01;
%sks2sas01(input=c:\sasupload\excels,out=work.tt);
hereby i am attaching error i am getting:
GOPTIONS ACCESSIBLE;
%macro sks2sas01(input=d:\excels,out=work.tt);
/* read files in directory */
%let dir=%str(%'dir %")&input.%str(\%" /A-D/B/ON%');
filename myfiles pipe %unquote(&dir);
data list1; length fname $256.;
infile myfiles truncover;
input myfiles $100.;
/* put _infile_;*/
fname=quote(upcase(cats("&input",'\',myfiles)));
out="&out";
drop myfiles;
call execute('
PROC IMPORT DBMS=EXCEL2002 OUT= _1
DATAFILE= '||fname||' REPLACE ;
sheet="summary";
RUN;
proc append data=_1 base='||out||' force; run;
proc delete data=_1; run;
');
run;
filename myfiles clear;
%mend sks2sas01;
%sks2sas01(input=c:\sasupload\excels,out=work.tt);
ERROR: Insufficient authorization to access PIPE.
ERROR: Error in the FILENAME statement.

I had this exact same problem the other day. I had no authorization for the pipe command, and dir using sftp wasn't working for me either. This alternative solution worked great for me. In a nutshell, you're going to use some oldschool SAS directory commands to read every file within that directory, and save only the ones that end in .xlsx.
You can consider the name of your Excel files .-delimited, scan the filename of each one backwards, and look at just the first word to obtain only those Excel files. For example:
File name.xlsx
Backwards:
Delimiter
v
xlsx.name File
^^^^
First word
Step 1: Read all XLSX files in the directory, and create a dataset of them
filename dir 'C:\MyDirectory';
data XLSX_Files;
DirID = dopen("dir");
do i = 1 to dnum(DirID);
file_name = dread(DirID, i);
if(upcase(scan(file_name, 1, '.', 'b') ) = 'XLSX') then output;
end;
rc=dclose(DirID);
drop i rc DirID;
run;
Step 2: Read all of those names into a pipe-delimited macro variable
proc sql noprint;
select file_name, count(file_name)
into :XLSX_Files separated by '|',
:tot_files
from XLSX_Files;
quit;
Step 3: Import them all in a macro loop
%macro import_all;
%do i = 1 %to &tot_files;
proc import file="C:\MyDirectory\%scan(&XLSX_Files,&i,|)"
out=XLSX_&i
dbms=xlsx replace;
run;
%end;
%mend;
%import_all;
You can then stack or merge them as you need.
data Master_File;
set XLSX_1-XLSX_&tot_files;
run;
OR
data Master_File;
merge XLSX_1-XLSX_&tot_files;
by key;
run;

Related

Missing the last few column names when exporting to CSV file

I am trying to export a dataset in my Library/Work. It shows normal in SAS. However when I export the data as CSV or txt file (either from right click -> export, or use SAS code), the last few column names were missing (showing empty in CSV), while the values were kept. The column names missing are all in the format of "Log_xxx" but some the same-format columns were exported correctly. There're around 4000+ columns in my dataset.
The code I've tried is like:
proc export data=logdata
outfile="path.csv"
dbms=csv
replace;
run;
I've exported many datasets before, but it's the first time I have this kind of problem. I've tried to restart SAS and it's still not working.
I simply wanted to export the whole dataset completely with all column names and values.
Do you have any ideas?
I don't think it is PROC EXPORT that is the issue. You have to tell SAS that you want to write lines that are longer then 32,767 bytes (the default setting for the LRECL option).
This code works:
data test;
array longname [3500] ;
run;
filename csv temp lrecl=1000000 ;
proc export data=test dbms=csv file=csv ;
run;
So change your code to set the LRECL long enough for all of the variable names.
filename csv "path.csv" lrecl=1000000 ;
proc export data=logdata
outfile=csv
dbms=csv
replace
;
run;
Based on this post, your header is likely exceeding 32k characters, which causes the issues.
Solution is to manually create the file without proc export, or proc export to XLSX doesn't appear to have the issue.
*Create demo data;
data class;
set sashelp.class;
label age='Age, Years' weight = 'Weight(lbs)' height='Height, inches';
run;
proc sql noprint;
create table temp as
select name as _name_, label as _label_
from dictionary.columns
where libname="WORK" and upcase(memname)="CLASS";
select nliteral(name) into :varList separated by ' '
from dictionary.columns
where libname="WORK" and upcase(memname)="CLASS";
quit;
data _null_;
file "&sasforum.\datasets\TwoLinesHeader.csv" dsd lrecl = 40000;
set class;
if _n_ = 1 then do;
do until(eof);
set temp end=eof;
put _name_ #;
end;
put;
end;
put (&varList) (:);
run;

How to create a MACRO to create a JSON file from dataset

I have some problems when I'm trying to generate a JSON file from dataset on SAS GUIDE. I generated a TEST.JSON:
{"TP_SMS":"1" "NM_REMETENTESMS":"00000159"},
{"TP_SMS":"2" "NM_REMETENTESMS":"00000159"},
{"TP_SMS":"3" "NM_REMETENTESMS":"00000159"},
{"TP_SMS":"4" "NM_REMETENTESMS":"00000159"},
{"TP_SMS":"5" "NM_REMETENTESMS":"00000159"},
.
.
.
{"TP_SMS":"9" "NM_REMETENTESMS":"00000159"},
The field TP_SMS is filled correct, but the second field is wrong - they are considering just the last position from my table.
Below there is my code white macro:
data teste30;
set MATABLES.EXIT_DATA;
RESP=cat(CD_CLIENTE,"|",ANWER_DATA);
ID=_N_;
call symputx('ID',ID);
call symputx('CD_CLIENTE',CD_CLIENTE);
call symputx('NM_PRIMNOMECLIENTE',NM_PRIMNOMECLIENTE);
call symputx('RESP',RESP);
call symputx('msgtext',msgtext);
run;
%macro MontaJSON(ID);
WRITE OPEN OBJECT;
WRITE VALUES "TP_SMS" "&ID";
WRITE VALUES "NM_REMETENTESMS" "&CD_CLIENTE";
WRITE CLOSE;
%mend MontaJSON(ID);
%macro SMSRecords;
%do i = 1 %to &dim_IDs;
%MontaJSON(&&&ID_&i);
%end;
%mend SMSRecords;
proc sql;
select id, CD_CLIENTE into :ID_1 - :ID_&SysMaxLong from work.teste30;
%let dim_IDs = &sqlObs;
quit;
proc json out="C:\TEMP\TEST.json" pretty nokeys nosastags;
write open array; /* container for all the data */
%SMSRecords;
write close; /* container for all the data */
run;
I expect this macro get all datas on sequence, as TP_SMS code:
{"TP_SMS":"1" "NM_REMETENTESMS":"00014578"},
{"TP_SMS":"2" "NM_REMETENTESMS":"21323445"},
{"TP_SMS":"3" "NM_REMETENTESMS":"23456753"},
{"TP_SMS":"4" "NM_REMETENTESMS":"00457663"},
{"TP_SMS":"5" "NM_REMETENTESMS":"00014795"},
{"TP_SMS":"6" "NM_REMETENTESMS":"00014566"},
{"TP_SMS":"7" "NM_REMETENTESMS":"00014578"},
{"TP_SMS":"8" "NM_REMETENTESMS":"00000122"},
{"TP_SMS":"9" "NM_REMETENTESMS":"00000159"}
Does anyone has some idea to solve it?
Tks
I would avoid generating all of those macro variables that are confusing your code.
Instead for that simple format you can just write the file directly instead of using PROC JSON.
data _null_;
set MATABLES.EXIT_DATA end=eof;
file "C:\TEMP\TEST.json" ;
if _n_=1 then put '[';
if not eof then delim=',';
put '{"TP_SMS":' id :$quote. ' "NM_REMETENTESMS":' CD_CLIENTE :$quote. '}' delim ;
if eof then put ']';
run;
Or if you really find that PROC JSON helps then use a similar data step to write the lines of code and use %INCLUDE to run the generated code.
filename code temp;
data _null_;
set MATABLES.EXIT_DATA ;
file code ;
put 'WRITE OPEN OBJECT;'
/ 'WRITE VALUES "TP_SMS" ' ID :$quote. ';'
/ 'WRITE VALUES "NM_REMETENTESMS" ' CD_CLIENTE :$quote. ';'
/ 'WRITE CLOSE;'
;
run;
proc json out="C:\TEMP\TEST.json" pretty nokeys nosastags;
write open array; /* container for all the data */
%include code;
write close; /* container for all the data */
run;
This line is your problem, it will only hold the last data point.
call symputx('CD_CLIENTE',CD_CLIENTE);
Instead, create a value for each ID, similar to how you created the IDs.
call symputx(catx('_', 'CD_CLIENTE', put(i, 8.-l)), CD_CLIENTE);
Then use it later on as &&&CD_CLIENTE&i

SAS Export data to create standard and comma-delimited raw data files

i m new to sas and studying different ways to do subject line task.
Here is two ways i knew at the moment
Method1: file statement in data step
*DATA _NULL_ / FILE / PUT ;
data _null_;
set engappeal;
file 'C:\Users\1502911\Desktop\exportdata.txt' dlm=',';
put id $ name $ semester scoreEng;
run;
Method2: Proc Export
proc export
data = engappeal
outfile = 'C:\Users\1502911\Desktop\exportdata2.txt'
dbms = dlm;
delimiter = ',';
run;
Question:
1, Is there any alternative way to export raw data files
2, Is it possible to export the header also using the data step method 1
You can also make use of ODS
ods listing file="C:\Users\1502911\Desktop\exportdata3.txt";
proc print data=engappeal noobs;
run;
ods listing close;
You need to use the DSD option on the FILE statement to make sure that delimiters are properly quoted and missing values are not represented by spaces. Make sure you set your record length long enough, including delimiters and inserted quotes. Don't worry about setting it too long as the lines are variable length.
You can use CALL VNEXT to find and output the names. The LINK statement is so the loop is later in the data step to prevent __NAME__ from being included in the (_ALL_) variable list.
data _null_;
set sashelp.class ;
file 'class.csv' dsd dlm=',' lrecl=1000000 ;
if _n_ eq 1 then link names;
put (_all_) (:);
return;
names:
length __name__ $32;
do while(1);
call vnext(__name__);
if upcase(__name__) eq '__NAME__' then leave;
put __name__ #;
end;
put;
return;
run;

How to write a sas code to read file in different folders?

I am wondering is it possible to write a macro to read file
in different folders.
Like for example, I have 100 folders and within each folder,
it has another folder, within the sub folder It may contain a txt file that I
want to read with SAS. New folder can be created to include new file.
Folder1 folder2 file
A1 aa1 file1
A2 aa2 (nofile)
… .. …
A100 aa100 file100
Folder2 is within folder1 and file is stored in folder2.
Thanks
Chen Xu
The following macro might do the job;
%macro procesFilesInRoot(rootFolder,fileExt,procesMacro);
%put "dir ""%unquote(&rootFolder)"" /s /b";
filename pipeTree pipe "dir ""%unquote(&rootFolder)"" /s /b" lrecl=32767;
data fullNames;
infile pipeTree truncover;
input fullName $char1000.;
ext = upcase(scan(fullName,countw(fullName, '.'), '.'));
if ext = "%upcase(&fileExt)";
run;
filename pipeTree clear;
title 'files found';
proc print;
run;
proc sql noprint;
select count(*) into :nrFiles from fullNames;
%let nrFiles = &nrFiles; /** to strip the leading blanks **/
%if &nrFiles %then %do;
select fullName into :fullName1-:fullName&nrfiles from fullNames;
%end;
quit;
%do fileNr = 1 %to &nrFiles.;
%&procesMacro(&&fullName&fileNr);
%end;
%mend;
Before using it, you need to write a macro that processes a single input file;
%macro import1file(fullName);
%let fileName = %scan(&fullName,%sysfunc(countw(&fullName, '\')), '\');
%let fileExt = %scan(&fileName,%sysfunc(countw(&fullName, '.')), '.');
%let dataName = %substr(&fileName,1, %length(&fileName)-%length(&fileExt)-1);
proc import datafile = "&fullName" out=&dataName;
run;
%mend;
Then you can use it as follows;
%procesFilesInRoot(D:\Horsten,txt,import1file);

Need to repeate data reading from multiple files using sas and run freqs on separate dataset created from separate files

I am new to SAS and facing few difficulties while creating following program.
My requirement is to pass the filename generated dynamically and read it so that don't have to write code five times to read data from 5 different files and then run freqs on the datasets.
I have provided the code below and have to write this code for more than 50 files:
Code
filename inp1 '/chshttp/prod/clients/coms/raw/coms_coms_relg_f1102_t1102_c10216_vEL5535.raw';
filename inp2 '/chshttp/prod/clients/coms/raw/coms_coms_relg_f1103_t1103_c10317_vEL8312.raw';
filename inp3 '/chshttp/prod/clients/coms/raw/coms_coms_relg_f1104_t1104_c10420_vEL11614.raw';
filename inp4 '/chshttp/prod/clients/coms/raw/coms_coms_relg_f1105_t1105_c10510_vEL13913.raw';
filename inp5 '/chshttp/prod/clients/coms/raw/coms_coms_relg_f1106_t1106_c10628_vEL17663.raw';
data test;
Do i = 1 to 5;
infile_name = 'inp' || i;
infile infile_name recfm = v lrecl=1800 end=eof truncover;
INPUT
#1 E_CUSTDEF1_CLIENT_ID $CHAR5.
#1235 E_MED_PLAN_CODE $CHAR20.
#1090 MED_INS_ELIG_COVERAGE_IND $CHAR20.
#1064 MED_COVERAGE_BEGIN_DATE $CHAR8.
#1072 MED_COVERAGE_TERM_DATE $CHAR8.
;
if E_CUSTDEF1_CLIENT_ID ='00002' then
output test;
end;
run;
proc freq data = test;
tables E_CUSTDEF1_CLIENT_ID*E_MED_PLAN_CODE / list missing;
run;
Please help!!
Here's an example you can adapt. There are different ways to do this, but this is one- depending no how you want the frequencies.
Step 1: Create a dataset, 'my_filenames', that stores the filename you want to read in, one per line, in a variable FILE_NAME.
Step 2: Read in the files.
data my_data;
set my_filenames;
infile a filevar=file_name <the rest of your options>;
<your input statement>;
run;
proc freq data=mydata;
by file_name;
<your table statements>;
run;
This is simple, data driven code that doesn't require macros or storing large amounts of data in things that shouldn't have data in them (macro variables, filenames, etc.)
To directly answer your question, here is a SAS macro to read each file and run PROC FREQ:
%macro freqme(dsn);
data test;
infile "&dsn" recfm = v lrecl=1800 end=eof truncover;
INPUT #1 E_CUSTDEF1_CLIENT_ID $CHAR5.
#1235 E_MED_PLAN_CODE $CHAR20.
#1090 MED_INS_ELIG_COVERAGE_IND $CHAR20.
#1064 MED_COVERAGE_BEGIN_DATE $CHAR8.
#1072 MED_COVERAGE_TERM_DATE $CHAR8.
;
if E_CUSTDEF1_CLIENT_ID = '00002';
run;
proc freq data=test;
tables E_CUSTDEF1_CLIENT_ID*E_MED_PLAN_CODE / list missing;
run;
proc delete data=test;
run;
%mend;
%freqme(/chshttp/prod/clients/coms/raw/coms_coms_relg_f1102_t1102_c10216_vEL5535.raw);
%freqme(/chshttp/prod/clients/coms/raw/coms_coms_relg_f1103_t1103_c10317_vEL8312.raw);
%freqme(/chshttp/prod/clients/coms/raw/coms_coms_relg_f1104_t1104_c10420_vEL11614.raw);
%freqme(/chshttp/prod/clients/coms/raw/coms_coms_relg_f1105_t1105_c10510_vEL13913.raw);
%freqme(/chshttp/prod/clients/coms/raw/coms_coms_relg_f1106_t1106_c10628_vEL17663.raw);
Note that I added a PROC DELETE step to delete the SAS data set after creating the report. I did that more for illustration, since you don't say you need the file as a SAS data set for subsequent processing.
You can use this as a template for other macro programming.