I am trying to import a file 'filename.csv.gz' into SAS.
Currently, I am trying to gunzip
FILENAME in PIPE "gunzip -dc filename.csv.gz" LRECL=80 ;
I am getting these errors:
ERROR: Insufficient authorization to access PIPE.
ERROR: Error in the FILENAME statement.
I am trying to figure out a way around the PIPE and I cannot manually do this.
Sounds like your SAS admin has disabled the ability to run operating system commands from SAS, so the PIPE engine does not work.
But you don't need the PIPE engine to read a gzipped file. Use the ZIP engine instead with the GZIP option.
FILENAME in zip "filename.csv.gz" gzip LRECL=80 ;
Sounds like you need OPTIONS XCMD; which may not be permitted. Can you run your program in BATCH?
https://support.sas.com/kb/15/179.html
Related
I am trying to extract encoded information like this from the VVC compression-log. Is there any setting, config in EncodeApp source for writing output values to file, I am using VTM Encoder Version 8.2. Thanks
lick here to view
Your question is pretty unrelated to the topic of video compression.
But anyway... What you may do is to read the console output and parse it. To do so, you can first write it into a text file by adding > log.txt at the end of your command line. Then, parse the text file and write it to a CSV file.
You could save the screen context as *.txt ( for instance:EncoderApp.exe -c xxx.cfg>Enc.txt, and convert Enc.txt to Enc.csv( Implemented in Python )
Hi I am writing some program that will write in some files (with more processes at the time) like:
with gzip.open('filename.gz', 'a') as f:
f.write(json.dumps(some dictionary) + '\n')
f.flush()
After writing finishes I upload files with:
s3.meta.client(filename, bucket, destination, filename without .gz)
Than I want to query data from Athena, after MSCK REPAIR everything seems fine but when I try to select data my rows are empty. Does anyone know what am I doing wrong?
EDIT: My mistake. I have forgot to add ContentType parameter to 'text/plain'
Athena detects the file compression format with the appropriate file extension.
So if you upload a GZIP file, but remove the '.gz' part (as I would guess from your "s3.meta.client(filename, bucket, destination, filename without .gz)" statement), the SerDe is not able to read the information.
If you rename your files to filename.gz, Athena should be able to read your files.
I have fixed the problem by first saving bigger chunks of files locally and than gzip them. I repeat the process but with appending to gziped file. Read that it is better to add bigger chunks of text than just line by line
For the upload I used boto3.transfet.upload_file with extra_args={'ContentEncoding': 'gzip', 'ContentType': 'text/plain'}
I forgot to add ContetType first time so the s3 saved them differently and Athena gave me errors that said my JSON is not formatted right.
I suggest you break the problem into several parts.
First, create a single JSON file that is not gzipped. Store it in Amazon S3, then use Athena to query it.
Once that works, manually gzip the file from the command-line (rather than programmatically), put the file in S3 and use Athena to query it.
If that works, use your code to programmatically gzip it, and try it again.
If that works with a single file, try it with multiple files.
All of the above can be tested with the same command in Athena -- you're simply substituting the source file.
This way, you'll know which part of the process is upsetting Athena without compounding the potential causes.
I want to compress all logs which matches the pattern entered by the user for which i am using the below code but getting an error.
import subprocess
input=input("Enter log details to compress")
subprocess.call(['gzip',input],shell=True)
Input given as : purato.log.2017-08*
Error : gzip: compressed data not written to a terminal. Use -f to force compression.
You are giving gzip the string purato.log.2017-08* in python, however in when you execute gzip purato.log.2017-08* in the shell, the wild card * substituted before gzip is invoked.
In order to replicate this behavior, enable shell on the call() function.
subprocess.call(['gzip purato.log.2017-08*'],shell=True)
I am just playing around with SAS, writing the following code
options nocenter nonumber;
data vag;
infile "C:\Users\Deborah\Desktop\School\STA 318\book\veggies.txt";
input Name $ Code $ Days Number Price;
CostPerSeed = Price / Number;
run;
The location of the file is correct, but the error I am getting
ERROR: Physical file does not exist, /opt/sasinside/SASConfig/Lev1/SASApp/C:\Users\Deborah\Desktop\School\STA 318\book\veggies.txt.
What can I do to fix this?
You are working on Unix server, not you local PC. While you are running a local client on your PC (like SAS EG), you can't actually access local resources on your local PC because these resources needs to be made available to Unix box itself.
You have four options:
1. Use an import Wizard in SAS EG. It will generate a data step and will use hidden SAS EG methods to convert your text file and move it as a data set to Unix server where you then can create variables that you want.
See if your file system is available to you in SAS EG. Expand your workspace server. You should see "files". Expand to the folder that you are allowed to use and drag&drop your text file into there.
Upload files to the unix server using whatever file transfer method is available. Usually you would have some sort of FTP server running.
Also, as Joe suggested, you can install this custom task http://blogs.sas.com/content/sasdummy/2012/12/06/copy-files-in-sas-eg/
It can do a lot better job that step 2 or 3 because it can be part of your process, resolve macro variables and fix file formatting issues between Windows and Unix. Thanks Joe!!
I have a user of a SAS server that wants to be able to "zip" a file they are outputting from a SAS script, by programming the file "zip" into the same script.
I'm hesitant to enable XCMD to allow this to be done natively on the Linux server on which SAS runs. Does anyone know of an alternative way of doing this without enabling XCMD?
Depends on what they're doing.
If they're producing a .sas7bdat (SAS Dataset), then options compress=yes; (or =char) will compress the SAS dataset, some. It won't compress nearly as much as normal gzip compression.
If they're writing out a text file, there is a method of writing a zip file. In 9.4, this is production; see this doc page for more information. Basically you can write it like this:
filename myfile zip '//mydir/myfile.z';
and then write to it like a normal file. You likely could write other kinds of files out to this as well (HTML, for example), if you read them in as text files and write them back out line by line to a zip fileref; but there's not much information available on this at the moment as it is brand new.
Prior to 9.4, there was the SASZIPAM filename engine, which is officially unsupported and not suitable for production use. You may be able to use it similarly.
If they're writing a PDF or similar file, then probably there is no suitable method for doing so, unless you can use the zip filename engine by reading the pdf in with recfmt=n and writing back out to the zip that way; that seems very dangerous, though.
The best solution from my point of view if you aren't comfortable enabling XCMD would be to have SAS write out a batch file (so a .sh file in Linux I suppose) containing the zip instructions, and have the process that runs SAS execute that .sh file when it's done.
If your output goes to an ODS destination you can create an ODS package and include all output in zip file.
ods package open;
ods html package;
proc print data=sashelp.class;
run;
ods html close;
ods package publish archive properties(archive_name="class.zip" archive_path="\");
ods package close;
I don't have a UNIX install so I cannot test this with -NOXCMD.
In Unix, you can use SAS to write through a pipe like you would any other Unix process.