I have a process in SAS that creates a .csv. I have another process in Python that waits until the file exists and then does something with that file. I want to make sure that the Python process doesn't start doing its thing until SAS is done writing the file.
I thought I would just have SAS create the .csv and then rename the file after it was done. Can I do the renaming in SAS? Then the python process could just wait until the renamed file existed.
EDIT: I accidentally posted this question twice and then accidentally deleted both versions. Sorry about that.
I think better than renaming would be to write a shell script that executes the SAS program, then only executes the Python script once the SAS program has exited without errors. The syntax for this varies somewhat depending on your OS, but wouldn't be too difficult.
With verion 9.2 and later SAS has a rename function that should work just the way you would like.
You could generate your output in a SAS dataset and then write to the .csv file only when you're finished with it. Not sure how you're creating the csv, but I usually just do this:
data _null_;
file './ouptut_file.csv' dlm=',';
set dataset_name;
put var1 var2;
run;
If you do this at the very end of your SAS code, no csv file will be generated until you're finished manipulating the data.
There are more than one way of doing this.
Does the python script monitor for .csv files only? Or it will be triggered when any new file is created in that directory?
If it will trigger only to .csv, then you could simply output to, say, a .txt file, then rename it using the x statement (OS command), or the rename function as #cmjohns suggested.
Another option is, you could output the .csv file to a different directory, then just move it to the directory that python is monitoring. As long as the file is in the same disk, moving the file will be done in an instant.
You could also modify the python script to only look for .csv, then do the option 1.
You could even create a "flag" file for python to check when sas has finished with the .csv.
But if possible, I think I would go with #jgunnink.
How about having the watching program watch for a separate small file instead of the large CSV file? That would be written in one write operation by SAS and reduce the risk of triggering the read before the write is done.
data _null_;
file 'bigfile.csv' dsd ;
set bigdata;
put (_all_) (+0);
run;
data _null_;
file 'bigfile.txt' ;
put 'bigfile.csv written';
run;
Related
I have a PowerShell program that searches a folder on my PC for several text files. If the file is not in the folder, it writes the filename to another text file. When the procedure finishes, I have a text file with a list of files (one column) that are missing from the folder.
Next I would like PC SAS to read the list from the text file and launch the corresponding SAS program that I have already written that retrieves each file from our FTP server.
I am not sure how to go about having SAS read the filenames and launch my FTP programs. Any suggestions on how to accomplish this task?
Sounds like the first problem is you need to modify the SAS program to use a dataset with the list of filenames to process. One easy way to do that is to create a macro that downloads one file and takes the filename as an input parameter. So convert your code that downloads a file to a macro.
%macro mf_download(file);
* Code that downloads &FILE from mainframe. ;
%mend ;
Then it is easy to write a program that reads the names from a file and calls the macro for each name in the file. So say the file with the list of names if named filelist.txt then that part of the program might look like this:
data names;
infile 'filelist.txt' truncover ;
input filename :$256. ;
call execute(cats('%nrstr(%mf_download)(',filename,')'));
run;
I've large txt file stored in sas enterprise guide(sas is connected to Winscp this is where th txt file is stored). How to read it and convert it to sas data as output.
When I check in SAS community I've get the code sample to read txt file (see bellow) is it same to read txt stored in sas?
proc import datafile='path'
out=NAME
dbms=dlm
replace;
datarow=5;
delimiter='09'x;
run;
There is another method I see also which use infile.
Which method shoul I use for me case?
I’ve not tried any method yet. Because I do not understand parameters. the path should it be the one in sas (in server) or in winscp?
Proc IMPORT works only on operating file 'references' that deliver the file directly.
WinSCP is a ftp client, so you two options:
Use WinSCP to copy the remote file to the local operating system, then you can use IMPORT or DATA step with INFILE
Use filename FTP access method and DATA step that reads data lines retrieved by SAS FTP engine
filename offsite ftp 'remote-filename` user=… pass=… host=… cd=… ;
data gotit;
infile offsite;
input var1 var2 var3 etc … ;
run;
The specific input statement might need informats and pointer control options, all dependent on the data file layout. Other infile options might be needed depending on field delimiters and content.
I am trying to read numeric data from the file. But I am to read it properly the output that I am getting from my SAS program is attached. I recently started learning SAS programming.
I am using SAS University Edition on the windows machine. I already tried by reading data in character as well as numeric formate.
data ds;
infile '/folders/myshortcuts/my_folder/exrate.sas7bdat';
input s ;
run;
I am expecting the same table to be as an output result.
Data File
Output
There are 3 things you need to do:
Make sure that the folder you put the dataset in is accessible to the SAS University Edition VM. Did you follow the setup guide in full? Usually that directs you to set up a folder that becomes available within the VM as /folders/myfolders
Assign a library pointing to the folder using a libname statement.
Use a set statement to access the dataset, not an infile statement. The latter is for reading raw data like csv files.
The sas code should look like this:
libname mylib "/folders/myfolders"; /*Change this to point to your folder path if you're sure you've got the right one*/
data ds;
set mylib.exrate(keep = s);
run;
I have a program that starts with:
%let filename = file1.csv
The program then imports the file into a sas dataset and then moves some of the data to a sql table based on some rules.
I want to have the program loop through processing whatever files are in a folder..I guess by redefining filename each time it gets to the end of the program and going back to the top until all of the csv files in the folder have been processed?
This is an example of a process that might benefit from creating a macro. Sounds like your existing code is close to being ready to become a macro. Just wrap the existing code into a macro definition that takes FILENAME as a parameter (remove your %let statement).
Then your existing program can become something like this. Where the last line is the one that actually runs the steps defined in the macro definition.
%macro loadfile(filename=);
... existing code ....
%mend loadfile;
%loadfile(filename=file1.csv);
To extend it to loading all files in a directory you just need to generate the list of files and use the list to generate a series of calls to the macro. So something like this might work on a Windows machine. So it will call the Windows command DIR to get the list of files and read the result into a variable and for each file found generate a call to the macro. The commands pushed by CALL EXECUTE will run after the data step finishes.
data _null_;
infile 'dir /b *.csv' pipe truncover ;
input filename $256. ;
call execute(cats('%nrstr(loadfile)(filename=',filename,')'));
run;
We are evaluating the time taken for two set of codes in SAS. Is there a way we can write/ tabulate option fullstimer results in a SAS dataset, without copying the entire log file into a notepad?
I would go about it like this.
Create separate SAS program files containing your code for each approach. Include options fullstimer at the top of both.
Batch submit your programs and write the logs to permanent files using the -log command line option.
Create a simple program that reads in both logs and compares the results.
The last step can be accomplished by using data steps with the INFILE statement and restricting the input records to those which are standard output from FULLSTIMER. Then you can compare the created datasets however you wish, e.g. via PROC COMPARE.
SAS has provided a log parsing macro that looks as though it should do the sort of thing that you want. It's available here:
http://support.sas.com/kb/34/301.html