I have 1000+ .sas files that I am trying to run in batch (all the code in each file is on one line), but SAS truncates the line to 256 characters and the code fails. Running each file individually (outside the batch) works fine.
Is there a way around this that would not force me to open each file and manually change the length of each line to something SAS can handle?
Prior to SAS 9.2, you can't extend the line limit in a batch file. You can extend the line limit in an %include, though, using the LRECL option; one option is to do so and have your batch submission all in one file with a bunch of %includes.
If you are running SAS 9.2 or later, you have the LRECL system option, which allows up to 32767 characters per line. See this page for more information.
Related
i have many .csv files which are stored into gcs and i want to load data from.csv to BigQuery using below commands:
bq load 'datasate.table' gs://path.csv json_schema
i have tried but giving errors, same error is giving for many file.
error screenshot
how can i remove unwanted values from .csv files before importing into table.
Suggest me to load file in easiest way
The answer depends on what do you want to do with this junk rows. If you look at the documentation, you have several options
Number of errors allowed. By default, it's set to 0 and that why the load job fails at the first line. If you know the total number of rom, set this value to the Number of errors allowed and all the errors will be ignored in the Load Job
Ignore unknown values. If your errors are made because some line contains more column as defined in the schema, this option keep the line in error and only the known column, the others are ignore
Allow jagged rows. If your errors are made by too short line (and it is in your message) and you still want to keep the first columns (because the last ones are optional and/or not relevant), you can check this option
For more advanced and specific filters, you have to perform pre or post processing. If it's the case, let me know to add this part to my answer.
Having some files on disk. The files have fixed size lines with the following format:
98969547,1236548896,1236547899,0a234505,1478889565
which 0a234505 is an IP Address in hex format.
I should open a file, read on line of the file and found the IP address. Then, create a directory on disk (if not exists) with the same name as IP Address and create a file witch holds the line under that directory.
The file name is today date e.g. 2017-02-09. If the directory and the and its file is created previously, simply append the corresponding line to the end of the file.
My files contains too much lines e.g. 100000 or greater, so this steps must be repeated for all lines.
My requirement is to process one files with 100000 lines in one second.
so what i need to understand is what is the maximum number of file open-wirte-close operations per second in ubuntu 16.04?
if the answer does not satisfy my requirement, How should I properly do this?
so its better to say if the OS limitation does not allow me to do such a huge amount of open-write-close operations, is there any second way to do this?
Programming language: c++
OS: ubuntu-16.04 4.4.0-62-generic
I am running a huge program in SAS and am using proc printto in order to save the log elsewhere. This works for about ten hours, and then SAS switches itself back to the normal log window. It was printing to a notepad file, which only maxed out at 6718KB.
Anyone know why SAS reverts to printing in the log window? Is it just me?
Thanks!!
One workaround for this would be to execute your program in batch. In batch mode, there are no log limits, as the log is automatically printed out to a file (by default, a file that is in the same location as your .sas program, with the same name but .log). Listing similarly will go to the same location .lst.
This would avoid log length limitations. If you want your log printed to a specific place, you can use the -altlog startup option.
I have a script that generates command line invocations of rrdtool xport based on input provided in a domain specific language. This works well, until the number of DEFs in the command line exceed a certain number - it seems to be around 50. At that point the command simply returns without any output or error information.
Is there a limit on the number of DEFs in rrdtool export? If so, then can it be raised or circumvented?
The issue turned out to be the character limit on the command line sent to the shell via Python's os.system method call. The issue can be worked around by creating a temporary executable script, writing the command line to the script and executing it.
In Enterprise Guide 4.2, is there a way to refresh your view of a file short of deleting it from the Process Flow then reopening it?
My Google-fu has failed to provide an answer (one way or the other) and my SAS admin has said he's not aware of a way (but to let him know if I find one).
A definitive "no" (from documentation) or a "yes" with example would be much appreciated.
I have a log file that's updated when I run my SAS program from the command line (outside of EG). I edit my code within EG, and I'd like to peek at the log file to see the results. Currently I have to delete the log file from my Process Flow then reopen it to see the updated log.
From your last comment on your question, it sounds like you are running a non-interactive SAS program on a server (from a PuTTY session) and looking at the log file with your EG client, is that correct? If so, there are much easier ways to watch the log file.
When you mention PuTTY, I'll assume your server is UNIX. If so, use the tail command with the -f option. For example, if your SAS program is named "myprog.sas", it will create a log file named "myprog.log", so try this command at your UNIX prompt:
tail -f myprog.log
The -f option means to continue writing output to your terminal window as lines are written to the log. When you get tired of watching (or your see the SAS "end of job" message), type the letter "q" to quit.
EG in intended to be the application that you use to actually execute your SAS program. Running things from the UNIX prompt is outside the design (and you lose all those cool EG features), as well as miss out any site features that have been set up for you in the metadata environment.
If I'm completely off-base, please clarify your question.
When using SAS EG or SAS Studio in a SAS platform were the compute nodes are hosted in Linux machines, I always use code to see the contents of an output file created by SAS; The only requirements are that you know the fullpath of the file you want to browse and that you have the privileges to read from it.
The simple idea is to use a generic DATA step to:
Access the file
Read line by line
Filter the data, by contents, position, line number, etc.
Print the data to the SAS log so you can see it there
Here is a simple example to get you going:
First I create a file for the test. You already have it!, so use yours.
/* create a test file */
data _null_;
file '/folders/myshortcuts/test/file'; /* Note I'm using fullpath */
put "The begining, :)";
put "line 2 in my file is shy and likes to hide.";
put "line 3, all good so far.";
put "line 4 in my file is to remain private.";
put "Finally the last line in my file!";
run;
Then, here is the code to read its data
data _null_;
/*--------
references which input file will be read
setting variable 'theEnd'=1 when
reaches end-of-file
Note I'm using fullpath
--------*/
infile '/folders/myshortcuts/test/file' end=theEnd;
/*--------
reads one line at a time from input file
storing it in variable _infile_
--------*/
input;
/*--------
contents of file will be writen in the log, to keep it readable,
mark where the contents of the file will follow
--------*/
if _n_=1 then
put "-----start file ----";
/*--------
filter out shy, private or unwanted data
--------*/
if _n_ ne 4; /* continue only if row number is not 4 */
if indexw(_infile_,"shy") le 0; /* continue only if data does not contains 'shy' */
/*--------
write the data you want, complete line read in this case
--------*/
put _N_= "->" _infile_;
/*--------
mark where the data in the file has ended
--------*/
if theEnd then put "-----end file ----";
run;
Your SAS log will look like this:
NOTE: The infile '/folders/myshortcuts/test/file' is:
Filename=/folders/myshortcuts/test/file,
Owner Name=sasdemo,Group Name=sas,
Access Permission=-rw-rw-r--,
Last Modified=11Jan2017:22:42:56,
File Size (bytes)=160
-----start file ----
_N_=1 ->The begining, :)
_N_=3 ->line 3, all good so far.
_N_=5 ->Finally the last line in my file!
-----end file ----
NOTE: 5 records were read from the infile '/folders/myshortcuts/test/file'.
The minimum record length was 16.
The maximum record length was 43.
NOTE: DATA statement used (Total process time):
real time 0.02 seconds
cpu time 0.03 seconds