I'm trying to parse out from Popen. Output looks like a table:
SceenShotAttached
How do I print just the col c?
I could do it easily using awk in a shell script but I'm depending on python solely now.
Thank you!
Related
I have an SSH step that executes a command which outputs CSV. This CSV should be parsed for further processing, however in Spoon I only found steps for parsing CSV data in files. How can I parse the stdOut field of the SSH step as CSV without writing it to a file first?
If you are using MAC / Linux or windows 10 WSL try using AWK before parsing it via Kettle.
It can parse CSV very easily.
Example : Code like this will parse the csv filter the rows and extract necessary columns which are needed.
$ awk -F ',' '{if($4 == "Online" && $5 =="L") {print $1,$2,$4}}' sales_100.csv
More info :
https://medium.com/analytics-vidhya/use-awk-to-save-time-and-money-in-data-science-eb4ea0b7523f
In a python script that is called by the test case, I writed a print.
Where the output of a the function print is printed ?
Note : I use Squish 6.2.0 with python 2.7, but any answer to the question will be welcome.
To see the text printed by the python print, click on 'Show View', then 'Runner/Server Log'.
I need to "extract" certain columns from a CSV file. The list of columns to extract is long and their indices do not follow a regular pattern. So far I've come up with a regular expression for a comma-separated value but I find it frustrating that in the RHS side of sed's substitute command I cannot reference more than 9 saved strings. Any ideas around this?
Note that comma-separated values that contain a comma must be quoted so that the comma is not mistaken for a field delimiter. I'd appreciate a solution that can handle such values properly. Also, you can assume that no value contains a new line character.
With GNU awk:
$ cat file
a,"b,c",d,e
$ awk -vFPAT='([^,]*)|("[^"]+")' '{print $2}' file
"b,c"
$ awk -vFPAT='([^,]*)|("[^"]+")' '{print $3}' file
d
$ cat file
a,"b,c",d,e,"f,g,h",i,j
$ awk -vFPAT='([^,]*)|("[^"]+")' -vOFS=, -vcols="1,5,7,2" 'BEGIN{n=split(cols,a,/,/)} {for (i=1;i<=n;i++) printf "%s%s", $(a[i]), (i<n?OFS:ORS)}' file
a,"f,g,h",j,"b,c"
See http://www.gnu.org/software/gawk/manual/gawk.html#Splitting-By-Content for details. I doubt if it'd handle escaped double quotes embedded in a field, e.g. a,"b""c",d or a,"b\"c",d.
See also What's the most robust way to efficiently parse CSV using awk? for how to parse CSVs with awk in general.
CSV is not that easy to parse like it might look in the first place.
This is because there can be a plenty of different delimiters or fixed column widths to separate the data, and also the data may contain the delimiter itself (escaped).
Like I already told here I would use a programming language which supports a CVS library for that.
Use
Python
Perl
Ruby
PHP
or even C.
Fully fledged CSV parsers such as Perl's Text::CSV_XS are purpose-built to handle that kind of weirdness.
I provided sample code within my answer here: parse csv file using gawk
There is command-line csvtool available - https://colin.maudry.com/csvtool-manual-page/
# apt-get install csvtool
Background: I have a fairly simple bash script that I'm using to generate a CSV log file. As part of that bash script I poll other devices on my network using netcat. The netcat command returns a stream of information that I can pipe that into a grep command to get to certain values I need in the CSV file. I save that return value from grep into a bash variable and then at the end of the script, I write out all saved bash variables to a CSV file. (Simple enough.)
The change I'd like to make is the amount of netcat commands I have to issue for each piece of information I want to save off. With each issued netcat command I get ALL possible values returned (so each time returns the same data and is burdensome on the network). So, I'd like to only use netcat once and parse the return value as many times as I need to create the bash variables that can later be concatenated together into a single record in the CSV file I'm creating.
Specific Question: Using bash syntax if I pass the output of the netcat command to a file using > (versus the current grepmethod) I get a file with each entry on its own line (presumably separated with the \n as the EOL record separator -- easy for perl regex). However, if I save the output of netcat directly to a bash variable, and echo that variable, all of the data is jumbled together, so it is cumbersome to parse out (not so easy).
I have played with two options: First, I think a perl one-liner may be a good solution here, but I'm not sure how to best execute it. Pseudo code might be to save the netcat output to a a bash variable and then somehow figure out how to parse it with perl (not straight forward though).
The second option would be to use bash's > and send netcat's output to a file. This would be easy to process with perl and Regex given the \n EOL, but that would require opening an external file and passing it to a perl script for processing AND then somehow passing its return value back into the bash script as a bash variable for entry into the CSV file.
I know I'm missing something simple here. Is there a way I can force a newline entry into the bash variable from netcat and then repeatedly run a perl-one liner against that variable to create each of the CSV variables I need -- all within the same bash script? Sorry, for the long question.
The second option would be to use bash's > and send netcat's output to
a file. This would be easy to process with perl and Regex given the \n
EOL, but that would require opening an external file and passing it to
a perl script for processing AND then somehow passing its return value
back into the bash script as a bash variable for entry into the CSV
file.
This is actually a fairly common idiom: save the output from netcat in
a temporary file, then use grep or awk or perl or what-have-you as
many times as necessary to extract data from that file:
# create a temporary file and arrange to have it
# deleted when the script exists.
tmpfile=$(mktemp tmpXXXXXX)
trap "rm -f $tmpfile" EXIT
# dump data from netcat into the
# temporary file.
nc somehost someport > $tmpfile
# extract some information into variable `myvar`
myvar=$(awk '/something/ {print $4}' $tmpfile)
That last line demonstrates how to get the output of something (in this case, an awk script) into a variable. If you were using perl to extract some information you could do the same thing.
You could also just write the whole script in perl, which might make your life easier.
Is there a way to run a one-liner in sas, or do I have to create a file? I'm looking for something like the -e flag in perl.
My favourite is using the -stdio option
Either:
sas -stdio
Then start typing. Or ...
echo "proc options; run;" | sas -stdio
The Unix version of SAS was ported from MVS years ago and to make a long story short, the SAS executable does not import from STDIN. To make this work in Unix, merely alter slightly the previous suggestion into something like:
echo "your SAS code" > temp;sas -sysin temp
Hope this is helpful.
sas -initstmt '%put hello world ; endsas ;'
sas -initstmt 'proc print data=sashelp.class; run ;'
Off course this could also be:
sas -initstmt '%inc large_program.sas; endsas;'
Never having used sas, what I might try is something like:
echo <insert sas code here> | sas --execute-file -
Oftentimes applications will let you specify '-' as a file to have it read from STDIN. And 'echo' just prints its arguments out, and the | connects them together.
You could also use the -nodms option. This will give you a command line version of Base.