Finding columns with only white space in a text file and replace them with a unique separator - regex

I have a file like this:
aaa b b ccc 345
ddd fgt f u 3456
e r der der 5 674
As you can see the only way that we can separate the columns is by finding columns that have only one or more spaces. How can we identify these columns and replace them with a unique separator like ,.
aaa,b b,ccc,345
ddd,fgt,f u,3456
e r,der,der,5 674
Note:
If we find all continuous columns with one or more white spaces (nothing else) and replace them with , (all the column) the problem will be solved.
Better explanation of the question by josifoski :
Per block of matrix characters, if all are 'space' then all block should be replaced vertically with one , on every line.

$ cat tst.awk
BEGIN{ FS=OFS=""; ARGV[ARGC]=ARGV[ARGC-1]; ARGC++ }
NR==FNR {
for (i=1;i<=NF;i++) {
if ($i == " ") {
space[i]
}
else {
nonSpace[i]
}
}
next
}
FNR==1 {
for (i in nonSpace) {
delete space[i]
}
}
{
for (i in space) {
$i = ","
}
gsub(/,+/,",")
print
}
$ awk -f tst.awk file
aaa,b b,ccc,345
ddd,fgt,f u,3456
e r,der,der,5 674

Another in awk
awk 'BEGIN{OFS=FS=""} # Sets field separator to nothing so each character is a field
FNR==NR{for(i=1;i<=NF;i++)a[i]+=$i!=" ";next} #Increments array with key as character
#position based on whether a space is in that position.
#Skips all further commands for first file.
{ # In second file(same file but second time)
for(i=1;i<=NF;i++) #Loops through fields
if(!a[i]){ #If field is set
$i="," #Change field to ","
x=i #Set x to field number
while(!a[++x]){ # Whilst incrementing x and it is not set
$x="" # Change field to nothing
i=x # Set i to x so it doesnt do those fields again
}
}
}1' test{,} #PRint and use the same file twice

Since you have also tagged this r, here is a possible solution using the R package readr. It looks like you want to read a fix width file and convert it to a comma-seperated file. You can use read_fwf to read the fix width file and write_csv to write the comma-seperated file.
# required package
require(readr)
# read data
df <- read_fwf(path_to_input, fwf_empty(path_to_input))
# write data
write_csv(df, path = path_to_output, col_names = FALSE)

Related

Matching strings across non-consecutive rows with AWK

I have been working with an AWK one-liner that does a good job of identifying string matches on previous rows, i.e. comparing field x on row n with field y on row (n+1). E.g., say input file consists of rows, 3 fields each:
A B C
B B B
C C C
D B D
The one-liner is:
awk "$2==a[2] { print a[1],a[2],a[3] } { for (i=1;i<=NF;i++) a[i]=$i }"
So this example prints out all three fields of any immediately previous row that matches on field 2, which in this case is only row 1. So the output would be:
A B C
Now, I'm wondering if there is a modification to this command that will allow me to find matches between the current row and the row that is 2 rows before it, or 3 rows before it, or even 4 rows before it.
So using the same sample input file, if I was trying to make matches for "2 rows before", on field 2, it would now only output
B B B
which is row 2, because it is the only instance of the 2nd field ("B") matching with the second field in the row that is 2 rows ahead (i.e. row 4).
I'm not terribly familiar with arrays. I'm guessing the run time will suffer but is the original command modifiable in this way ?
You could use this awk:
awk 'a[FNR%n,m]==$m {print a[FNR%n]}{a[FNR%n]=$0; a[FNR%n,m]=$m}' n=2 m=3 file.txt
The above will print the nth line, before the current line if field m in both lines match.
The above will keep the memory nicely in check: if you don't care too much about memory consumption, you can do this:
awk '(FNR-n,$m) in a {print a[FNR-n,$m]}{a[FNR,$m]=$0}' n=2 m=3 file.txt
You may use this awk solution:
cat prev.awk
FNR > p && n = split(row[FNR-p], cols) && $2 == cols[2] {
print row[FNR-p]
}
{
row[FNR] = $0
}
Then use it for current-2 row matching:
awk -v p=2 -f prev.awk file
B B B
and current-1 row matching:
awk -v p=1 -f prev.awk file
A B C

Bash - Extract a column from a tsv file whose header matches a given pattern

I've got a tab-delimited file called dataTypeA.txt. It looks something like this:
Probe_ID GSM24652 GSM24653 GSM24654 GSM24655 GSM24656 GSM24657
1007_s_at 1149.82818866431 1156.14191288693 743.515922643437 1219.55564561635 1291.68030259557 1110.83793199643
1053_at 253.507372571459 150.907554200493 181.107054946649 99.0610660103702 147.953428467212 178.841519788697
117_at 157.176825094869 147.807257232552 162.11169957066 248.732378039521 176.808414979907 112.885784025819
121_at 1629.87514240262 1458.34809770171 1397.36209234134 1601.83045996129 1777.53949459116 1256.89054921471
1255_g_at 91.9622298972477 29.644137111864 61.3949774595639 41.2554576367652 78.4403716513328 66.5624213750532
1294_at 313.633291641829 305.907304474766 218.567756319376 335.301256439494 337.349552407502 316.760658896597
1316_at 195.799277107983 163.176402437481 111.887056644528 194.008323756222 211.992656497053 135.013920706472
1320_at 34.5168433158599 19.7928225262233 21.7147425051394 25.3213322300348 22.4410631949167 29.6960283168278
1405_i_at 74.938724593443 24.1084307838881 24.8088845994911 113.28326338746 74.6406975005947 70.016519414531
1431_at 88.5010900723741 21.0652011409692 84.8954961447585 110.017339630928 84.1264201735067 49.8556999547353
1438_at 26.0276274326623 45.5977459152141 31.8633816890024 38.568939176828 43.7048363737468 28.5759163094148
1487_at 1936.80799770498 2049.19167519573 1902.85054762899 2079.84030768241 2088.91036902825 1879.84684705068
1494_f_at 358.11266607978 271.309665853292 340.738488775022 477.953251687206 388.441738062896 329.43505750512
1598_g_at 2908.90515715761 4319.04621682741 2405.62061966298 3450.85255814957 2573.97860992156 2791.38660060659
160020_at 416.089910909237 327.353902186303 385.030831004533 385.199279534446 256.512900212781 217.754025190117
1729_at 43.1079499314469 114.654670657195 133.191500889286 86.4106614983387 122.099426341898 218.536976034472
177_at 75.9653827137444 27.4348937420347 16.5837374743166 50.6758325717831 58.7568500760629 18.8061888366161
1773_at 31.1717741953018 158.225161489953 161.976679771553 139.173486349393 218.572194156366 103.916119454
179_at 1613.72113870554 1563.35465407698 1725.1817757679 1694.82209331327 1535.8108561345 1650.09670894426
Let's say I have a variable col="GSM24655". I want to extract the column from dataTypeA.txt that corresponds to this column name.
Additionally, I'd like to put this in a function, where I can just give it a file (i.e. dataTypeA.txt), and a column (i.e. GSM24655), and it'll return that column.
I'm not very proficient in Bash, so I've been having some trouble with this. I'd appreciate the help.
Below script using awk can be used to achieve the objective.
col="GSM24655";
awk -v column_val="$col" '{ if (NR==1) {val=-1; for(i=1;i<=NF;i++) { if ($i == column_val) {val=i;}}} if(val != -1) print $val} ' dataTypeA.txt
Working: Initially, value of col is passed to awk script using -v column_val="$col" . Then the column number is find out. (when NR==1, i.e the first row, it iterates through all the fields (for(i=1;i<=NF;i++), awk variable NF contains the number of columns) and then compare the value of column_val (if ($i == column_val)), when a match is found the corresponding column number is found and stored ( val=i )). After that, from next row onwards, the values in that column is printed (print $val).
If you copy the below code into a file called say find_column.sh, you can call sh find_column.sh GSM24655 dataTypeA.txt to display the column having value of first parameter (GSM24655) in the file named second parameter (dataTypeA.txt). $1 and $2 are positional parameters. The lines column=$1 and file=$2 will assign the input values to the variables.
column=$1;
file=$2;
awk -v column_val="$column" '{ if (NR==1) {val=-1; for(i=1;i<=NF;i++) { if ($i == column_val) {val=i;}}} if(val != -1) print $val} ' $file
I would use the following, it is quick and easy.
In your script, you get the name of the file, let's say $1, and word, $2.
Then, in my for each I am using the whole header, but you can just add a head -1 $1, and in the IF, the $2, this is going to output column name.
c=0;
for each in `echo "Probe_ID GSM24652 GSM24653 GSM24654 GSM24655 GSM24656 GSM24657"`;do if [[ $each == "Probe_ID" ]];then
echo $c;
col=$c;
else c=$(( c + 1 ));
fi;
done
Right after this, you just do a cat $1| cut -d$'\t' -f$col

GAWK - Multiple BEGIN and END sections

I'm trying to process a bunch of files extracting data using gawk.
File area fixed width space formatted file
I'm trying to extract data from two different lines matched by two different regular expressions but return the data from both of these lines on the ONE print statement.
I can achieve this with the following in a.awk file and use gawk -f to run it. the first BEGIN section setup up input file format (FIELDWIDTHs) and the second BEGIN I am trying to use a loop per file to output based on extracted data. The first END complete the inner BEGIN and the second to match the outer BEGIN.
However I can only apply this to one file at a time because if I apply to a bunch of files (as in gawk -f regex.awk km*.txt , I only get the last file's output.
Can I get a one line of output per file input without having to resort to a script file looping over the input files and running the awk script each time.
Thanks
BEGIN{
OFS=","; FIELDWIDTHS ="2 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12";
printf("Date, Turnover, SalesA, SalesB, SalesC, SalesD, Other Data\n");
}
BEGIN{ Sales = 0;
SalesA = 0;
SalesB = 0;
SalesC = 0;
SalesD = 0;
JointSales = 0;
Turnover = 0;
OtherData = 0;}
/^03/ || /^06/ {
if ($1 == "03") {
Sales = $15/100;
SalesA = $17/100;
SalesB = $26/100;
SalesC = $20/100;
SalesD = $22/100;
JointSales = SalesA - SalesB;
Turnover = JointSales + SalesB + SalesC + SalesD; }
else if ( $1 == "06") {
OtherData = substr($0,183,12)/100; }
# printf("%s, %10.2f, %10.2f, %10.2f, %10.2f, %10.2f, %10.2f\n", getDate(FILENAME), Sales, JointSales, SalesB, SalesC, SalesD, OtherData )
}
END{printf("%s, %10.2f, %10.2f, %10.2f, %10.2f, %10.2f, %10.2f\n", getDate(FILENAME), Sales, JointSales, SalesB, SalesC, SalesD, OtherData ) }
END {}
function getDate(str)
{ date = substr(str,3,6);
year = substr(date,1,2);
month= substr(date,3,2);
day=substr(date,5,2);
odate=(day"/"month"/"year);
return odate
}
If you are using gawk, you're in luck. In addition to BEGIN and END blocks, gawk implements BEGINFILE and ENDFILE blocks, which are executed just as you want: before and after processing each file. See the handy gawk programming guide.
Like all awk implementations, Gnu awk allows you to have multiple BEGIN and END blocks. All BEGIN blocks are run in order (first to last) before the first file is read, and all END blocks are run in the same first-to-last order after the last file is done. Since the same order is used for both types of special block, they don't "nest".
awk only allows one begin and end action set per run (though they can be spread across multiple blocks, they're all combined into one action set) and a run includes all files that you process.
If you want to do something between each file as well, the can use the ARGIND variable which holds the index of the current argument (zero-based). You just need to maintain the last argument index (initially zero) and, if the actual argument index is different, execute your special actions and update the last index.
With empty files (for which no code would be run), the current argument index may be more than one higher than the last so you may need to loop, incrementing the last index until it reaches the current one.
For example, let's print the lines of each file but with special markers for before, within and after. With the file a.in:
xyzzy
plugh
and a b.in file containing nothing, you can use the following script demo.awk:
function middleCheck() {
while (lastArgInd != ARGIND) {
print "MIDDLE after "lastArgInd":"ARGV[lastArgInd]
lastArgInd++
}
}
BEGIN { print "BEGIN"
lastArgInd = 1
}
{ middleCheck()
print " "$0
}
END { middleCheck()
print "END"
}
to effect an action between each file:
pax> vi demo.awk ; awk -f demo.awk b.in a.in a.in b.in a.in b.in b.in
BEGIN
MIDDLE after 1:b.in
xyzzy
plugh
MIDDLE after 2:a.in
xyzzy
plugh
MIDDLE after 3:a.in
MIDDLE after 4:b.in
xyzzy
plugh
MIDDLE after 5:a.in
MIDDLE after 6:b.in
END
You just have to make that action match what you need, your current "inner" end followed by your current "inner" begin.

AWK - Search for a pattern-add it as a variable-search for next line that isn't a variable & print it + variable

I have a given file:
application_1.pp
application_2.pp
#application_2_version => '1.0.0.1-r1',
application_2_version => '1.0.0.2-r3',
application_3.pp
#application_3_version => '2.0.0.1-r4',
application_3_version => '2.0.0.2-r7',
application_4.pp
application_5.pp
#application_5_version => '3.0.0.1-r8',
application_5_version => '3.0.0.2-r9',
I would like to be able to read this file and search for the string
".pp"
When that string is found, it adds that line into a variable and stores it.
It then reads the next line of the file. If it encounters a line preceded by a # it ignores it and moves onto the next line.
If it comes across a line that does not contain ".pp" and doesn't start with # it should print out that line next to a the last stored variable in a new file.
The output would look like this:
application_1.pp
application_2.pp application_2_version => '1.0.0.2-r3',
application_3.pp application_3_version => '2.0.0.2-r7',
application_4.pp
application_5.pp application_5_version => '3.0.0.2-r9',
I would like to achieve this with awk. If somebody knows how to do this and it is a simple solution i would be happy if they could share it with me. If it is more complex, it would be helpful to know what in awk I need to understand in order to know how to do this (arrays, variables, etc). Can it even be achieved with awk or is another tool necessary?
Thanks,
I'd say
awk '/\.pp/ { if(NR != 1) print line; line = $0; next } NF != 0 && substr($1, 1, 1) != "#" { line = line $0 } END { print line }' filename
This works as follows:
/\.pp/ { # if a line contains ".pp"
if(NR != 1) { # unless we just started
print line # print the last assembled line
}
line = $0 # and remember this new one
next # and we're done here.
}
NF != 0 && substr($1, 1, 1) != "#" { # otherwise, unless the line is empty
# or a comment
line = line $0 # append it to the line we're building
}
END { # in the end,
print line # print the last line.
}
You can use sed:
#n
/\.pp/{
h
:loop
n
/[^#]application.*version/{
H
g
s/\n[[:space:]]*/\t/
p
b
}
/\.pp/{
x
p
}
b loop
}
If you save this as s.sed and run
sed -f s.sed file
You will get this output
application_1.pp
application_2.pp application_2_version => '1.0.0.2-r3',
application_3.pp application_3_version => '2.0.0.2-r7',
application_4.pp
application_5.pp application_5_version => '3.0.0.2-r9',
Explanation
The #n supresses normal output.
Once we match the /\.pp/, we store that line into the hold space with h, and start the loop.
We go to the next line with n
If it matches /[^#]application.*version/, meaning it doesn't start with a #, then we append the line to the hold space with H, then copy the hold space to the pattern space with g, and substitute the newline and any subsequent whitespace for a tab. Finally we print with p, and skip to the end of the script with b
If it matches /\.pp/, then we swap the pattern and hold spaces with x, and print with p.

Error in writing output file through AWK scripting

I have a AWK script to write specific values matching with specific pattern to a .csv file.
The code is as follows:
BEGIN{print "Query Start,Query End, Target Start, Target End,Score, E,P,GC"}
/^\>g/ { Query=$0 }
/Query =/{
split($0,a," ")
query_start=a[3]
query_end=a[5]
query_end=gsub(/,/,"",query_end)
target_start=a[8]
target_end=a[10]
}
/Score =/{
split($0,a," ")
score=a[3]
score=gsub(/,/,"",score)
e=a[6]
e=gsub(/,/,"",e)
p=a[9]
p=gsub(/,/,"",p)
gc=a[12]
printf("%s,%s,%s,%s,%s,%s,%s,%s\n",query_start, query_end,target_start,target_end,score,e,p,gc)
}
The input file is as follows:
>gi|ABCDEF|
Plus strand results:
Query = 100 - 231, Target = 100 - 172
Score = 20.92, E = 0.01984, P = 4.309e-08, GC = 51
But I received the output in a .csv file as provided below:
100 0 100 172 0 0 0 51
The program failed to copy the values of:
Query end
Score
E
P
(Note: all the failed values are present before comma (,))
Any help to obtain the right output will be great.
Best regards,
Amit
As #Jidder mentioned, you don't need to call split() and as #jaypal mentioned you're using gsub() incorrectly, but also you don't need to call gsub() at all if you just include , in your FS.
Try this:
BEGIN {
FS = "[[:space:],]+"
OFS = ","
print "Query Start","Query End","Target Start","Target End","Score","E","P","GC"
}
/^\>g/ { Query=$0 }
/Query =/ {
query_start=$4
query_end=$6
target_start=$9
target_end=$11
}
/Score =/ {
score=$4
e=$7
p=$10
gc=$13
print query_start,query_end,target_start,target_end,score,e,p,gc
}
That work? Note the field numbers are bumped out by 1 because when you don't use the default FS awk no longer skips leading white space so there's an empty field before the white space in your input.
Obviously, you are not using your Query variable so the line that populates it is redundant.