I have a DataSet that I created with
filename tmp pipe 'dir "C:\Temp1\*.txt*" /b /s';
data Full;
infile tmp dlm="\";
length Path $2000 Path2 $2000 Path3 $2000 Path4 $2000 FileName $2000;
input Path Path2 Path3 Path4 FileName;
run;
the Results look like:
C:\Temp1\test - Copy (12).txt
C:\Temp1\test - Copy.txt
C:\Temp1\test.txt
C:\Temp1\test - Copy (2).txt
C:\Temp1\test - Copy (3).txt
C:\Temp1\test - Copy (4).txt
C:\Temp1\test - Copy (5).txt
C:\Temp1\test - Copy (6).txt
C:\Temp1\test - Copy (7).txt
C:\Temp1\test - Copy (8).txt
C:\Temp1\test - Copy (9).txt
C:\Temp1\test - Copy (10).txt
C:\Temp1\test - Copy (11).txt
I am using the Code Below
options noxwait;
data _null_;
set Full nobs=nobs;
i = 1;
do while (i < nobs);
set Full point=i;
%let oldfile = <<<THIS IS WHERE I WANT THE DATA TO BE PULLED TO>>>;
%let newfile = C:\Temp2\;
rc= system("move &oldfile &newfile ");
i = i + 1;
put rc=;
end;
run;
basically I want to loop through my "Full" data set and put each row set to oldfile.
what do I need to fix in my code?
DATA step is already an implicit loop over all the rows in the data set. You do not need to have other external agency controlling that. Also, if just starting out, try not to mix data step scope and macro scope variables.
untested example:
data Full;
infile tmp;
input; * read whole line into _infile_ buffer;
filename = _infile_; * transfer buffer to variable;
run;
data _null_;
set full;
command = "move " || quote(trim(filename)) || " " || quote(trim("C:\Temp2\"));
rc = system(command);
run;
Related
I'm trying to create a custom text report from my sas code, below is the code
data have ;
ncandidates=1; ngames=3; controlppt=1; controlgame=2;
ppt1='Abc'; ppt2='Bcd';
infile cards dsd dlm='|';
input (var1-var21) ($);
cards;
1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b
1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b
;
filename report 'myreport.txt';
data _null_;
file report dsd dlm='|' LRECL=8614;
a='';
put
83*'#'
/ '##### Number of ppts'
/ 83*'#'
/ 'input.Name=' #
;
eof = 0;
do until(eof);
set have end=eof;
If not missing(var1) then
put var1-var10 ## ;
end;
put a
// 83*'#'
/ '##### Output Data'
/ 83* '#'
// 'output.Name=' #;
eof=0;
do until(eof);
set have ;
If not missing(var11) then
put var11-var20 ## ;
end;
put '1';
run;
Everything gets printed to the file except for the last put '1';
Nothing after the second do until block gets executed;
Also, if I add end=eof to the last do until block then everything gets printed twice.
Do we have a solution around this?
I am not sure about the cause of the issue. But sometimes SAS behaves weird if a dataset is read several times as you do it. But using a different variable for second set have end=eof2; resolves the problem:
data have ;
ncandidates=1; ngames=3; controlppt=1; controlgame=2;
ppt1='Abc'; ppt2='Bcd';
infile cards dsd dlm='|';
input (var1-var21) ($);
cards;
1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b
1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b1|2|a|1|3|b
;
filename report '~/myreport.txt';
data _null_;
file report dsd dlm='|' LRECL=8614;
a='';
put
83*'#'
/ '##### Number of ppts'
/ 83*'#'
/ 'input.Name=' #
;
eof = 0;
do until(eof);
set have end=eof;
If not missing(var1) then
put var1-var10 ## ;
end;
put a
// 83*'#'
/ '##### Output Data'
/ 83* '#'
// 'output.Name=' #;
eof2=0;
do until(eof2);
set have end=eof2;
If not missing(var11) then
put var11-var20 ## ;
end;
put '1';
stop;
run;
A 'plain' DATA step stops when a read is attempted after the last record of a set has been read. This typically happens during the implicit loop that is inherent in the magic of a DATA step. When you loop over a set explicitly with end of data checks, the read attempt beyond does not occur, and thus does not implicitly end the step.
The eof flag is changed only when the end of data is reached. It is not set to 0 when not at end of data -- the eof flag is simply what it is at the start of the loop. Thus the flag needs to be reset if reused for a subsequent loop.
* 'top' is logged twice;
* the data step ends when the second implicit iteration tries to read past eof of the first set;
data _null_;
put 'top';
do until (eof);
set sashelp.class(obs=2) end=eof;
put name=;
end;
eof = 0; * reset flag;
do until (eof);
set sashelp.class(where=(name=:'J')) end=eof;
put name=;
end;
run;
* 'top' is logged once;
* the data step ends when the stop is reached at the bottom;
data _null_;
put 'top';
do until (eof);
set sashelp.class(obs=2) end=eof;
put name=;
end;
eof = 0;
do until (eof);
set sashelp.class(where=(name=:'J')) end=eof;
put name=;
end;
run;
%let dirname = C:\Users\data;
filename DIRLIST pipe 'dir/B &dirname\*.dbf';
/* Create a data set with one observation for each file name */
data dirlist;
length fname $8.;
infile dirlist length=reclen;
input fname $8.;
run;
data all_text (drop=fname);
set dirlist;
filepath = "&dirname\"||fname||".dbf";
infile dummy filevar = filepath length=reclen end=done missover;
do while(not done);
INPUT
F1 : 2.
F2 : 2.
F3 : 2.
F4 : 10.
F5 : 4.;
output;
end;
run;
The problem is that it is only reading the first line of each files and not the whole file before moving to the next. Also variable F1 are shown as missing.
Suggestions are welcome
So a standard proc import would be:
proc import out=sample1 datafile="path to dbf file.dbf" dbms=DBF replace;
run;
The problem now, is how to generate this set of code for every file in your file list. Using the CALL EXECUTE statement from #Tom is your best bet. You call also create a small macro and call it for each filename, using CALL EXECUTE. If you're new to SAS this can be easier to understand.
*Create a macro that imports the DBF
%macro import_dbf(input= , output=);
proc import out=&out datafile="&output" dbms=DBF replace;
run;
%mend;
Then call macro from dataset. I'm naming the datasets DBF001, DBF0002 etc.
%let dirname=C:\_localdata;
data dirlist;
informat fname $20.;
input fname;
cards;
data1.dbf
data2.dbf
data3.dbf
data4.dbf
;
run;
data out;
set dirlist;
str=catt('%import_dbf(input="', "&dirname", '\', fname, '", output=dbf',
put(_n_, z4.), ');');
run;
proc print data=out;
run;
Import them one by one and then combine them.
%let dirname = C:\Users\data;
data filelist ;
infile "dir /b &dirname\*.dbf" pipe truncover end=eof;
fileno + 1;
input fname $256. ;
tempname = 'temp'||put(fileno,z4.);
call execute(catx(' ','proc import replace dbms=dbf'
,'out=',tempname,'datafile=',quote(trim(fname)),';run;'
));
if eof then call symputx('lastname',tempname);
run;
data want ;
set temp0001-&lastname;
run;
I'm attempting to build a loop in SAS to upload several files, and am running into a few issues to work through. Current code:
%Macro Weatherupload(File=, output=);
proc import datafile = &File;
out = &output;
dbms=dlm replace;
delimiter= ",";
getnames=yes;
guessingrows = 1000;
run;
%Mend Weatherupload;
%Macro WeatherPrepare(input=, output=);
data &output (keep=Wban_Number _YearMonthDay DewPoint Temp _Avg_Dew_Pt _Avg_Temp year month day);
set &input;
DewPoint = Input(compress(_Avg_Dew_Pt,"*"), 3.);
Temp = Input(compress(_Avg_Temp,"*"), 3.);
year = (_yearmonthday - mod(_yearmonthday, 10000))/10000;
month = ((_yearmonthday - mod(_yearmonthday, 100)) - (_yearmonthday - mod(_yearmonthday,10000)))/100;
day = mod(_yearmonthday, 100);
drop _Avg_Dew_Pt _Avg_Temp _YearMonthDay;
run;
%Mend WeatherPrepare;
data temperatures;
do i = 1999 to 2015;
do j = 1 to 12;
name = 'C:\Users\DILLON.SAXE\Documents\'||i||j||'.tar'||' \'||i||j||'daily.txt';
output = i||j||'weather';
final = i||j||'final';
%Weatherupload(File=name, output=output)
%WeatherPrepare(input=output, output=final)
end;
end;
run;
The goal is to run through several files, in several folders, listed in month + day + rest of title, and (at the moment) upload two variables of data from them. Later I will want to add in merging the files, and doing some more data work, but for the moment it's the macro issues and uploading that are holding it up.
Is there a way to either use proc upload in a loop, or use another data step in the loop?
I get the error "more positional variables than (something)" (I forget exact error, but it lists positional variables). I've tried adding and removing commas in the macros, but have not been able to get rid of this error. Any ideas?
I don't think you can call macro's like you have in your data step. I think you're intending to use Call Execute.
data temperatures;
do i = 1999 to 2015;
do j = 1 to 12;
name = 'C:\Users\DILLON.SAXE\Documents\'||i||j||'.tar'||' \'||i||j||'daily.txt';
output = i||j||'weather';
final = i||j||'final';
call execute('%Weatherupload(File='||name||', output='||output||')');
call execute('%WeatherPrepare(input='||output||', output='||final||')');
end;
end;
run;
Alternatively, assuming you're trying to read all files in a folder, I think you should be creating a list of file names in a data set, use a data step with the filename option to input all files at once instead. Here's a brief method on how to do it if all where in a single folder: https://communities.sas.com/docs/DOC-10426
Here is a page that has code to get a list of files into a data set
http://www.sascommunity.org/wiki/Making_Lists
since your macros have neither conditionals (%if) nor loops (%do)
then I suggest you use them as parameterized %incudes
Here is a tool to read the list-of-files data set and call a program
http://www.sascommunity.org/wiki/Call_Execute_Parameterized_Include
note: in proc import always set guessingrows to the max value;
in v9.3 that is 2147483647;
Got it sorted out, based on the first answer. Eventual code:
%Macro Weatherupload(File=, output=);
proc import datafile = "&File"
out = &output
dbms=dlm replace;
delimiter= ",";
getnames=yes;
guessingrows = 1000;
run;
%Mend Weatherupload;
%Macro WeatherPrepare(input=, output=);
data &output;
set &input;
DewPoint = Input(compress(_Avg_Dew_Pt,"*"), 3.);
Temp = Input(compress(_Avg_Temp,"*"), 3.);
year = (_yearmonthday - mod(_yearmonthday, 10000))/10000;
month = ((_yearmonthday - mod(_yearmonthday, 100)) - (_yearmonthday - mod(_yearmonthday,10000)))/100;
day = mod(_yearmonthday, 100);
keep Wban_Number DewPoint Temp year month day;
run;
%Mend WeatherPrepare;
%Macro WeatherPrepare2(input=, output=);
data &output;
set &input;
DewPoint = Input(DewPoint, 3.);
Temp = Input(compress(_Avg_Temp,"*"), 3.);
year = (_yearmonthday - mod(_yearmonthday, 10000))/10000;
month = ((_yearmonthday - mod(_yearmonthday, 100)) - (_yearmonthday - mod(_yearmonthday,10000)))/100;
day = mod(_yearmonthday, 100);
Wban_Number = Wban;
keep Wban_Number DewPoint Temp year month day;
run;
%Mend WeatherPrepare;
%Macro Append(merge=);
data temperatures;
set temperatures &merge;
%Mend Append;
data temperatures;
do i = 1999 to 2015;
do j = 1 to 12;
jzero = put(j, z2.);
name = compress('C:\Users\DILLON.SAXE\Documents\'||i||jzero||'.tar'||'\'||i||jzero||'daily.txt');
name2 = compress('C:\Users\DILLON.SAXE\Documents\'||'QCLCD'||i||jzero||'\'||i||jzero||'daily.txt');
output = compress('weather'||i||j);
final = compress('final'||i||j);
if 1000*i+j < 200708 then
do;
call execute('%Weatherupload(File='||name||', output='||output||')');
call execute('%WeatherPrepare(input='||output||', output='||final||')');
end;
else
do;
call execute('%Weatherupload(File='||name2||', output='||output||')');
call execute('%WeatherPrepare2(input='||output||', output='||final||')');
end;
call execute('%Append(merge='||final||')');
end;
end;
drop i j jzero name name2 output final;
run;
I am trying to read the folder with zip files using Pipe Command. But I get error saying ls command not recognized. There are actually 2 zip files(ABC_*.zip) in the folder /PROD/
Can anybody help me in this?
%let extl_dir=/PROD/ ;
filename zl pipe "ls &extl_dir.ABC_*.zip";
data ziplist_a;
infile zl end=last;
length path $200 zipnm $50 filedt $15;
input path $;
zipnm=scan(path,-1,"/");
filedt=scan(scan(path,-1,"_"),1,".");
call symput('zip'||left(_n_), zipnm);
call symput('path'||left(_n_), path);
call symput('filedt'||left(_n_),filedt);
if last then call symput('num_zip',_n_);
*call symput('flenm',filenm);
run;
SAS has published a convenient macro to list files within a directory that does not rely upon running external commands. It can be found here. I prefer this approach as it does not introduce external sources of possible error such as user permissions, pipe permissions etc.
The macro uses datastep functions (through %sysfunc) and the commands can be called in the same manner from a datastep. Below is an example which extracts tile information.
%let dir = /some/folder;
%let fType = csv;
data want (drop = _:);
_rc = filename("dRef", "&dir.");
_id = dopen("dRef");
_n = dnum(_id);
do _i = 1 to _n;
name = dread(_id, _i);
if upcase(scan(name, -1, ".")) = upcase("&fType.") then do;
_rc = filename("fRef", "&dir./" || strip(name));
_fid = fopen("fRef");
size = finfo(_fid, "File Size (bytes)");
dateCreate = finfo(_fid, "Create Time");
dateModify = finfo(_fid, "Last Modified");
_rc = fclose(_fid);
output;
end;
end;
_rc = dclose(_id);
run;
I am reading a .txt file into SAS, that uses "|" as the delimiter. The issue is there is one column that is using "|" as a word separator as well instead of acting like delimiter, this needs to be in one column.
For example the txt file looks like:
apple|fruit|Healthy|choices|of|food|12|2012|chart
needs to look like this in the SAS dataset:
apple | fruit | Healthy choices of Food | 12 | 2012 | chart
How do I eliminate "|" between "Healthy choices of Food"?
I think this will do what you want:
data tmp1;
length tmp $100;
input tmp $;
cards;
apple|fruit|Healthy|choices|of|food|12|2012|chart
apple|fruit|Healthy|choices|of|food|and|lots|of|other|stuff|12|2012|chart
;
run;
data tmp2;
set tmp1;
num_delims=length(tmp)-length(compress(tmp,"|"));
expected_delims=5;
extra_delims=num_delims-expected_delims;
length new_var $100;
i=1;
do while(scan(tmp,i,"|") ne "");
if i<=2 or (extra_delims+2)<i<=num_delims then new_var=trim(new_var)||scan(tmp,i,"|")||"|";
else new_var=trim(new_var)||scan(tmp,i,"|")||"#";
i+1;
end;
new_var=left(tranwrd(new_var,"#"," "));
run;
This isn't particularly elegant, but it will work:
data tmp;
input tmp $50.;
cards;
apple|fruit|Healthy|choices|of|food|12|2012|chart
;
run;
data tmp;
set tmp;
var1 = scan(tmp,1,'|');
var2 = scan(tmp,2,'|');
var4 = scan(tmp,-3,'|');
var5 = scan(tmp,-2,'|');
var6 = scan(tmp,-1,'|');
var3 = tranwrd(tmp,trim(var1)||"|"||trim(var2),"");
var3 = tranwrd(var3,trim(var4)||"|"||trim(var5)||"|"||trim(var6),"");
var3 = tranwrd(var3,"|"," ");
run;
Expanding a little on Itzy's answer, here is another possible solution:
data want;
/* Define variables */
attrib item length=$10 label='Item';
attrib class length=$10 label='Family';
attrib desc length=$80 label='Item Description';
attrib count length=8 label='Some number';
attrib year length=$4 label='Year';
attrib somevar length=$10 label='Some variable';
length countc $8; /* A temp variable */
infile 'c:\temp\delimited_temp.txt' lrecl=1000 truncover;
input;
item = scan(_infile_,1,'|','mo');
class = scan(_infile_,2,'|','mo');
countc = scan(_infile_,-3,'|','mo'); /* Temp var for numeric field */
count = inputn(countc,'8.'); /* Re-read the numeric field */
year = scan(_infile_,-2,'|','mo');
somevar = scan(_infile_,-1,'|','mo');
desc = tranwrd(
substr(_infile_
,length(item)+length(class)+3
,length(_infile_)
- ( length(item)+length(class)+length(countc)
+length(year)+length(somevar)+5))
,'|',' ');
drop countc;
run;
The key in this case it to read your file directly and handle the delimiters yourself. This can be tricky and requires that your data file is exactly as described. A much better solution would be to go back to whoever gave this this data and ask them to deliver it to you in a more appropriate form. Good luck!
Another possible workaround.
data tmp;
infile '/path/to/textfile';
input tmp :$100.;
array varlst (*) $30 v1-v6;
a=countw(tmp,'|');
do i=1 to dim(varlst);
if i<=2 then
varlst(i) = scan(tmp,i,'|');
else if i>=4 then
varlst(i) = scan(tmp,a-(dim(varlst)-i),'|');
else do j=3 to a-(dim(varlst)-i)-1;
varlst(i)=catx(' ', varlst(i),scan(tmp,j,'|'));
end;
end;
drop tmp a i j;
run;