Getting the last column of a grep match for each line - regex

Let's say I have
this is a test string
this is a shest string
this est is another example of sest string
I want the number of the character in string of the last "t" IN THE WORDS [tsh]EST, how do I get it? (In bash)
EDIT2: I can get the wanted substring with [tsh]*est if I'm not wrong.
I cannot rely on the first match (awk where=match(regex,$0) ) since it gives the first character position but the size of the match is not always the same.
EDIT: Expected output ->
last t of [tsh]*est at char number: 14
last t of [tsh]*est at char number: 15
last t of [tsh]*est at char number: 35
Hope I was clear, I think I edited the question too many times sorry !

What you got wrong
where=match(regex,$0)
the syntax of match is wrong. its string followd by the regex. That is match($0, regex)
Correction
$ awk '{print match($0, "t[^t]*$")}' input
17
18
38
EDIT
Get number of the character in string of the last "t" IN THE WORDS [tsh]EST,
$ awk '{match($0, "(t|sh|s)est"); print RSTART+RLENGTH-1}' input
14
15
35
OR
a much simpler version
$ awk 'start=match($0, "(t|sh|s)est")-1{$0=start+RLENGTH}1' input
14
15
35
Thanks Jidder for the suggestion
EDIT
To use the regex same as OP has provided
$ awk '{for(i=NF; match($i, "(t|sh|s)*est") == 0 && i > 0; i--); print index($0,$i)+RLENGTH-1;}' input
14
15
35

You can use this awk using same regex as provided by OP:
awk -v re='[tsh]*est' '{
i=0;
s=$0;
while (p=match(s, re)) {
p+=RLENGTH;
i+=p-1;
s=substr(s, p)
}
print i;
}' file
14
15
35

Try:
awk '{for (i=NF;i>=0;i--) { if(index ($i, "t") != 0) {print i; break}}}' myfile.txt

This will print the column with the last word that contains t
awk '{s=0;for (i=1;i<=NF;i++) if ($i~/t/) s=i;print s}' file
5
5
8
awk '{s=w=0;for (i=1;i<=NF;i++) if ($i~/t/) {s=i;w=$i};print "last t found in word="w,"column="s}'
last t found in word=string column=5
last t found in word=string column=5
last t found in word=string column=8

Related

bash: extraction of substrings by pattern, empty fields and multiple occurrences

I would like to extract the Pfam_A information from each line of a file:
item_1 ID=HJNANFJJ_180142;inference=ab initio prediction:Prodigal_v2.6.3;locus_tag=HJNANFJJ_180142;partial=01;product=unannotated protein;KEGG=K03531
item_4 ID=HJNANFJJ_87662;inference=ab initio prediction:Prodigal_v2.6.3;locus_tag=HJNANFJJ_87662;partial=10;product=unannotated protein;KEGG=K15725;Pfam_A=OEP;Resfams=adeC-adeK-oprM
item_8 ID=HJNANFJJ_328505;inference=ab initio prediction:Prodigal_v2.6.3;locus_tag=HJNANFJJ_328505;partial=11;product=unannotated protein;KEGG=K03578;Pfam_A=OB_NTP_bind
item_2 ID=HJNANFJJ_512995;inference=ab initio prediction:Prodigal_v2.6.3;locus_tag=HJNANFJJ_512995;partial=11;product=unannotated protein;KEGG=K00674;Pfam_A=Hexapep;Pfam_A=Hexapep_2;metacyc=TETHYDPICSUCC-RXN
item_0 ID=HJNANFJJ_188729;inference=ab initio prediction:Prodigal_v2.6.3;locus_tag=HJNANFJJ_188729;partial=11;product=unannotated protein
In some lines this information is missing at all, in some there can be multiple occurrences.
Finally, I want to get a table like this, so that instead of empty fields there is NaN and multiple occurrences are put tab separated into different fields:
item_1 NaN
item_4 OEP
item_8 OB_NTP_bind
item_2 Hexapep Hexapep_2
item_0 NaN
You may use this awk:
awk -v OFS='\t' 'NF > 1 {
s = ""
n = split($NF, a, /;/)
for (i=1; i<=n; i++)
if (split(a[i], b, /=/) == 2 && b[1] == "Pfam_A")
s = s OFS b[2]
print $1 (s ? s : OFS "NaN")
}' file
item_1 NaN
item_4 OEP
item_8 OB_NTP_bind
item_2 Hexapep Hexapep_2
item_0 NaN
A quick and dirty way would be:
awk '{ s=$0;t="";
while (match(s,"Pfam_A=[^;]*")) {
t = t (t?OFS:"") substr(s,RSTART+7,RLENGTH-7);
s = substr(s,RSTART+RLENGTH)
}
}{print $1, (t?t:"NaN")}' file
With the presumptions that in each input line, there are no other ; characters except for the ; characters that separate the data fields, and no tab characters unless they delimit the first column, a simple sed command could do the job:
sed -E 's/\s+/;/; s/;Pfam_A=/;\t/g; s/;[^\t]*//g; /\t/!s/$/\tNaN/' file

Using gawk to extract rows with string in a column

I was trying to extract rows from a tab separated file, if it contained a certain word in the 4th column. For example, if input file test.txt is:
chr 8 1234 abc ; xyz
chr 8 1255 abc
chr 8 987 xyz
chr 8 5467 jxyzm
The following code correctly outputs only the 1st and 3rd line:
gawk -F"\t" ' { if($4 ~ /\<xyz\>/) print $0 } ' test.txt >> test.out
However, when I try to run this in a loop, in a bash script, my output file is blank. the code I am using is:
while read id
do
OFILE=${ODIR}/${id}.txt
gawk -v id="$id" -F"\t" ' { if($4 ~ /\<id\>/) print $0 } ' ${IFILE} >> ${OFILE}
done < ${GFILE}
The file ${GFILE} has one word per line, e.g.:
xyz
fg45
tre2y
What am I doing wrong?
thanks!
Edited to:
Add fourth row in input file
Added -v id="$id" to command...script still doesn't work!
You can very well use awk to read search patterns from one file and find matches in other like this:
awk -F '\t' '
NR == FNR {
words[$1]
next
}
{
for (w in words)
if (index($4, w)) {
print > w ".txt"
break
}
}' "$GFILE" "$IFILE"
Then check output:
cat xyz.txt
chr 8 1234 abc ; xyz
chr 8 987 xyz
If you really-really want to fix your shell script then here it is:
while read id; do
awk -F '\t' -v id="$id" '$4 ~ id' "$IFILE" > "$id.txt"
done < "$GFILE"

How can I group unknown (but repeated) words to create an index?

I have to create a shellscript that indexes a book (text file) by taking any words that are encapsulated in angled brackets (<>) and making an index file out of that. I have two questions that hopefully you can help me with!
The first is how to identify the words in the text that are encapsulated within angled brackets.
I found a similar question that was asked but required words inside of square brackets and tried to manipulate their code but am getting an error.
grep -on \\<.*> index.txt
The original code was the same but with square brackets instead of the angled brackets and now I am receiving an error saying:
line 5: .*: ambiguous redirect
This has been answered
I also now need to take my index and reformat it like so, from:
1:big
3:big
9:big
2:but
4:sun
6:sun
7:sun
8:sun
Into:
big: 1 3 9
but: 2
sun: 4 6 7 8
I know that I can flip the columns with an awk command like:
awk -F':' 'BEGIN{OFS=":";} {print $2,$1;}' index.txt
But am not sure how to group the same words into a single line.
Thanks!
Could you please try following(if you are not worried about sorting order, in case you need to sort it then append sort to following code).
awk '
BEGIN{
FS=":"
}
{
name[$2]=($2 in name?name[$2] OFS:"")$1
}
END{
for(key in name){
print key": "name[key]
}
}
' Input_file
Explanation: Adding detailed explanation for above code.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here.
FS=":" ##Setting field separator as : here.
}
{
name[$2]=($2 in name?name[$2] OFS:"")$1 ##Creating array named name with index of $2 and value of $1 which is keep appending to its same index value.
}
END{ ##Starting END block of this code here.
for(key in name){ ##Traversing through name array here.
print key": "name[key] ##Printing key colon and array name value with index key
}
}
' Input_file ##Mentioning Input_file name here.
If you want to extract multiple occurrences of substrings in between angle brackets with GNU grep, you may consider a PCRE regex based solution like
grep -oPn '<\K[^<>]+(?=>)' index.txt
The PCRE engine is enabled with the -P option and the pattern matches:
< - an open angle bracket
\K - a match reset operator that discards all text matched so far
[^<>]+ - 1 or more (due to the + quantifier) occurrences of any char but < and > (see the [^<>] bracket expression)
(?=>) - a positive lookahead that requires (but does not consume) a > char immediately to the right of the current location.
Something like this might be what you need, it outputs the paragraph number, line number within the paragraph, and character position within the line for every occurrence of each target word:
$ cat book.txt
Wee, <sleeket>, cowran, tim’rous beastie,
O, what a panic’s in <thy> breastie!
Thou need na start <awa> sae hasty,
Wi’ bickerin brattle!
I wad be laith to rin an’ chase <thee>
Wi’ murd’ring pattle!
I’m <truly> sorry Man’s dominion
Has broken Nature’s social union,
An’ justifies that ill opinion,
Which makes <thee> startle,
At me, <thy> poor, earth-born companion,
An’ fellow-mortal!
.
$ cat tst.awk
BEGIN { RS=""; FS="\n"; OFS="\t" }
{
for (lineNr=1; lineNr<=NF; lineNr++) {
line = $lineNr
idx = 1
while ( match( substr(line,idx), /<[^<>]+>/ ) ) {
word = substr(line,idx+RSTART,RLENGTH-2)
locs[word] = (word in locs ? locs[word] OFS : "") NR ":" lineNr ":" idx + RSTART
idx += (RSTART + RLENGTH)
}
}
}
END {
for (word in locs) {
print word, locs[word]
}
}
.
$ awk -f tst.awk book.txt | sort
awa 1:3:21
sleeket 1:1:7
thee 1:5:34 2:4:24
thy 1:2:23 2:5:9
truly 2:1:6
Sample input courtesy of Rabbie Burns
GNU datamash is a handy tool for working on groups of columnar data (Plus some sed to massage its output into the right format):
$ grep -oPn '<\K[^<>]+(?=>)' index.txt | datamash -st: -g2 collapse 1 | sed 's/:/: /; s/,/ /g'
big: 1 3 9
but: 2
sun: 4 6 7 8
To transform
index.txt
1:big
3:big
9:big
2:but
4:sun
6:sun
7:sun
8:sun
into:
big: 1 3 9
but: 2
sun: 4 6 7 8
you can try this AWK program:
awk -F: '{ if (entries[$2]) {entries[$2] = entries[$2] " " $1} else {entries[$2] = $2 ": " $1} }
END { for (entry in entries) print entries[entry] }' index.txt | sort
Shorter version of the same suggested by RavinderSingh13:
awk -F: '{
{ entries[$2] = ($2 in entries ? entries[$2] " " $1 : $2 ": " $1 }
END { for (entry in entries) print entries[entry] }' index.txt | sort

awk with joined field

I am trying to extract data from one file, based on another.
The substring from file1 serves as an index to find matches in file2.
All works when the string to be searched inf file2 is beetween spaces or isolated, but when is joined to other fields awk cannot find it. is there a way to have awk match any part of the strings in file2 ?
awk -vv1="$Var1" -vv2="$var2" '
NR==FNR {
if ($4==v1 && $5==v2) {
s=substr($0,4,8)
echo $s
a[s]++
}
next
}
!($1 in a) {
print
}' /tmp/file1 /tmp/file2
example that works:
file1:
1 554545352014-01-21 2014-01-21T16:18:01 FS 14001 1 1.10
1 554545362014-01-21 2014-01-21T16:18:08 FS 14002 1 5.50
file2:
55454535 11 17 102 850Sande Fiambre 1.000
55454536 11 17 17 238Pesc. Dourada 1.000
example that does not work:
file2:
5545453501/21/20142 1716:18 1 1 116:18
5545453601/21/20142 1716:18 1 1 216:18
the string to be searched, for instance : 55454535 finds a match in the working example, but it doesn't in the bottom one.
You probably want to replace this:
!($1 in a) {
print
}
with this (or similar - your requirements are unclear):
{
found = 0
for (s in a) {
if ($1 ~ "^"s) {
found = 1
}
}
if (!found) {
print
}
}
Use a regex comparison ~ instead of ==
ex. if ($4 ~ v1 && $5 ~ v2)
Prepend v1/v2 with ^ if you want to the word to only begin with string and $ if you want to word to only end with it

Separate string of digits into 3 columns using awk/sed

I have a string of digits in rows as below:
6390212345678912011012112121003574820069121409100000065471234567810
6390219876543212011012112221203526930428968109100000065478765432196
That I need to split into 6 columns as below:
639021234567891,201101211212100,3574820069121409,1000000,654712345678,10
639021987654321,201101211222120,3526930428968109,1000000,654787654321,96
Conditions:
Field 1 = 15 Char
Field 2 = 15 Char
Field 3 = 15 or 16 Char
Field 4 = 7 Char
Field 5 = 12 Char
Field 6 = 2 Char
Final Output:
639021234567891,3574820069121409,654712345678
639021987654321,3526930428968109,654787654321
It's not clear how detect whether field 3 should have 15 or 16 chars. But as draft for the first 3 fields you could use something like that:
echo 63902910069758520110121121210035748200670169758510 |
awk '{ printf("%s,%s,%s",substr($1,1,15),substr($1,16,15),substr($1,30,15)); }'
Or with sed:
echo $NUM | sed -r 's/^([0-9]{16})([0-9]{15})([0-9]{15,16}) ...$/\1,\2,\3, .../'
This will use 15 or 16 for the length of field 3 based the length of the whole string.
If you're using gawk:
gawk -v f3w=16 'BEGIN {OFS=","; FIELDWIDTHS="15 15 " f3w " 7 12 2"} {print $1, $3, $5}'
Do you know ahead of time what the width of Field 3 should be? Do you need it to be programatically determined? How? Based on the total length of the line? Does it change line-by-line?
Edit:
If you don't have gawk, then this is a similar approach:
awk -v f3w=16 'BEGIN {OFS=","; FIELDWIDTHS="15 15 " f3w " 7 12 2"; n=split(FIELDWIDTHS,fw," ")} { p=1; r=$0; for (i=1;i<=n;i++) { $i=substr(r,p,fw[i]); p += fw[i]}; print $1,$3,$5}'