I have to read csv files every day which are kept in a folder with a date time stamp on it. eg: newfolder20150430023210
I have to extract these files everyday from the folder. But since the folder has got time stamp on it, I'm unable to create a macro which can read the latest file from the directory. I could create a macro of the current datetime but not the folder datetime.
eg: path:- D:\SAS\Data\Newfolder20150430023210\file.csv I need to read this path where 20150430023210 is dynamic.
thank you
I think you need to find the newest folder first, this code wil do that:
filename f pipe "dir /B/A:D/O:-D D:\SAS\Data\Newfolder*";
data _null_;
infile f;
input;
call symputx("lastDir",_infile_);
stop;
run;
Now, you have a macrovariable "lastDir" that has the name of the folder.
Related
I am using sas enterprise guide 8.3 to import a csv file into SAS.
I try to use import data wizard to do that; then copy the log of it to reuse it later(e.g. next week). the log looks like the following:
/*-------------------------------------------------
code generated by a SAS task
generated on Thursday, August 5,2021 at 9:59:13 PM
By task: import data wizard
Source File:
d:\test\test.csv
Server: Local File System
Output data: Work.temptable
Server: [servername]
-------------------------------------------------------------------*/
Data: Work.temptable;
Length:
col1
col2
Format:
col1
col2
Informat:
col
col2
Infile 'H:\Saswork\TD13012_[Server_name]\#LN_00032'
Input
col1
col2;
run;
My scenario is: I want to save the code above and rerun the code each week because there will be a new csv file each week, I need to import the new csv file each week.
in the comment part, the source file is correct; but in the code part, it seems that infile points to a temp file 'H:\Saswork\TD13012_[Server_name]#LN_00032'. I wonder if this temp file always exists because I need to run the code each week. I try to replace the infile value with the correct location in the comment part(local folder, d:\test\test.csv ), there is an error message.
so how can I handle this infile? thanks!
If the file is on the server where SAS is running then just run the data step to read it. If the file is on the server where EG is running then you will first need to add a step in your process to upload the file to the SAS server before your SAS code can see the file. EG should have "copy files" task you can use
Note that the Import wizard might be modifying the file in the upload process, so make sure your new code works with the original file.
PS You can write a much better data step than what Enterprise Guide generates. For example most variables do not need to have either an INFORMAT or a FORMAT attached to them. SAS already knows how to read and write both character strings and numbers without any need to give it special instructions.
I have a PowerShell program that searches a folder on my PC for several text files. If the file is not in the folder, it writes the filename to another text file. When the procedure finishes, I have a text file with a list of files (one column) that are missing from the folder.
Next I would like PC SAS to read the list from the text file and launch the corresponding SAS program that I have already written that retrieves each file from our FTP server.
I am not sure how to go about having SAS read the filenames and launch my FTP programs. Any suggestions on how to accomplish this task?
Sounds like the first problem is you need to modify the SAS program to use a dataset with the list of filenames to process. One easy way to do that is to create a macro that downloads one file and takes the filename as an input parameter. So convert your code that downloads a file to a macro.
%macro mf_download(file);
* Code that downloads &FILE from mainframe. ;
%mend ;
Then it is easy to write a program that reads the names from a file and calls the macro for each name in the file. So say the file with the list of names if named filelist.txt then that part of the program might look like this:
data names;
infile 'filelist.txt' truncover ;
input filename :$256. ;
call execute(cats('%nrstr(%mf_download)(',filename,')'));
run;
I have a sas7bdat format file, but it's zipped.
I could unzip the file and work on it, but this makes me lose hard disk space and time.
So I tried this code on SAS :
filename myfile ZIP 'C:\...\data.zip' member="data.sas7bdat" ;
data yoyo;
infile myfile (data.sas7bdat);
input;
put _infile_;
run;
But I get an empty yoyo table in the WORK library.
How can I successfully import the data.sas7bdat ?
Thank you,
You need to uncompress the dataset before SAS can use it. So you need to find a place that has enough space for the fully expanded file.
Note that your code is trying to specify the member name of the file within the ZIP file twice. You should only do that once. Either point the fileref to the aggregate location and use member name in the reference. Or point the fileref to the individual member and just use the fileref.
Here is a method to expand the file into your current WORK folder.
%let member=data.sas7bdat;
filename in zip 'C:\...\data.zip' member="&member" recfm=n;
filename out "%sysfunc(pathname(work))/&member" recfm=n;
data _null_;
rc=fcopy('in','out');
run;
You can now work with the file using the name WORK.DATA.
proc print data=work.data(obs=1); run;
If you want to read data from a ZIP file directly then it either needs to be raw (text) data or in a streaming format, like a SAS V5 XPORT file.
I am using sas DI studio and I am trying to get a date value written within the name of the input file,to put the same date also in the output file name. So first I need to get the filename and the extract the date and then put the date back. What is the easiest approach?
The best way to automize your SAS code is bringing the date information from the system like:
data _null_;
call symput("PERDATE", Compress(put(intnx('month',date(),-1,'end'), YYMMDDn8.)));
run;
From the line above you get Perdate Macro variable in YYYYMMDD format for the previous month (-1). Then you can substract from this variable however you want.
%let ayear = %Substr(&PERDATE,1,4); /* ayear is the year in YYYY format */
%let amonth = %Substr(&PERDATE,5,2); /* amonth is the month in MM format */
%let aday = %Substr(&PERDATE,7,2); /* aday is the day in DD format */
%let adate=&ayear&amonth; /* adate is if you want in format YYYYMM */
%put &ayear&amonth;
%put &amonth;
%put &perdate;
Then you can read your input file daily or monthly basis automatically without changing the INPUT and OUTPUT file names. For example, if you get Sales report monthly basis in a format like Sales_YYYYMM then you can just write in your code like:
infile sales_&ayear.&amonth.; /* Put Dot after each Macro statement */
Sales_&ayear.&amonth. will give you Sales_201506. You can describe your output file in a same matter like Out=result_&ayear.&amonth.;
data result_&ayear.&amonth.;
set Sales_&ayear.&amonth.;
run;
If you are not getting your file regularly, if you need to enter the date manually, then you can just write create a macro variable for date at the beginning of your code, then you always use that macro variable in your code:
%let mydate=201506;
%put &mydate;
So in each run, you just change the variable mydate in yoru code. You don't need to change anything else...
The trick is setup the EXTERNAL FILE Metadata for input and output file correctly. Here's how to do that. Try creating EXTERNAL FILE metadata using an already existing file so that metadata of the columns are created correctly with format and informat. After EXTERNAL FILE metadata is created, edit the EXTERNAL FILE metadata by right clicking the FILE metadata and goto Properties -> File Location tab. In the file name section instead of file that already exist replace the date part with the macro variable. For example, if file name is : c:\Sushil_20150701.txt enter c:\Sushil_&mydatevar..txt . Also in FILE NAME QUOTING section, select "Double quotes around file name"
The same goes for Output file. Since we are going to use the &mydatevar macro variable on the input file name in the output file name, you can put the mydatevar similarly as done above.
After all this is done, we are good to use the file metadata with File Reader and File Writer transformation to read and write the files respectively.
Hope this helps!
I have a process in SAS that creates a .csv. I have another process in Python that waits until the file exists and then does something with that file. I want to make sure that the Python process doesn't start doing its thing until SAS is done writing the file.
I thought I would just have SAS create the .csv and then rename the file after it was done. Can I do the renaming in SAS? Then the python process could just wait until the renamed file existed.
EDIT: I accidentally posted this question twice and then accidentally deleted both versions. Sorry about that.
I think better than renaming would be to write a shell script that executes the SAS program, then only executes the Python script once the SAS program has exited without errors. The syntax for this varies somewhat depending on your OS, but wouldn't be too difficult.
With verion 9.2 and later SAS has a rename function that should work just the way you would like.
You could generate your output in a SAS dataset and then write to the .csv file only when you're finished with it. Not sure how you're creating the csv, but I usually just do this:
data _null_;
file './ouptut_file.csv' dlm=',';
set dataset_name;
put var1 var2;
run;
If you do this at the very end of your SAS code, no csv file will be generated until you're finished manipulating the data.
There are more than one way of doing this.
Does the python script monitor for .csv files only? Or it will be triggered when any new file is created in that directory?
If it will trigger only to .csv, then you could simply output to, say, a .txt file, then rename it using the x statement (OS command), or the rename function as #cmjohns suggested.
Another option is, you could output the .csv file to a different directory, then just move it to the directory that python is monitoring. As long as the file is in the same disk, moving the file will be done in an instant.
You could also modify the python script to only look for .csv, then do the option 1.
You could even create a "flag" file for python to check when sas has finished with the .csv.
But if possible, I think I would go with #jgunnink.
How about having the watching program watch for a separate small file instead of the large CSV file? That would be written in one write operation by SAS and reduce the risk of triggering the read before the write is done.
data _null_;
file 'bigfile.csv' dsd ;
set bigdata;
put (_all_) (+0);
run;
data _null_;
file 'bigfile.txt' ;
put 'bigfile.csv written';
run;