use grep to extract multiple values from one line - regex

Question

file:
timestamp1 KKIE ABC=123 [5454] GHI=547 JKL=877 MNO=878
timestamp2 GGHI ABC=544 [ 24548] GHI=883 JKL=587 MNO=874
timestamp3 GGGIO ABC=877 [3487] GHI=77422 JKL=877 MNO=877
timestamp4 GGDI ABC=269 [ 1896] GHI=887 JKL=877 MNO=123
note: You sometimes have a space between '[' and the next digit).
when JKL=877, I want timestampx, ABC and GHI
solution 1:
timestamp1 ABC=123 GHI=547
timestamp3 ABC=877 GHI=77422
timestamp4 ABC=269 GHI=887
solution 2 (the best one):
TIMESTAMP ABC GHI
timestamp1 123 547
timestamp3 877 77422
timestamp4 269 887
I know how to have these values individually but not all of them in once.
A. solution 1:
grep JKL=877 file | awk '{print $1}'
grep JKL=877 file | grep -o '.ABC=[0-9]\{3\}'
grep JKL=877 file | grep -o '.GHI=[0-9]\{3,5\}'
without the '[' issue, I would do:
grep JKL=877 | awk '{print $1,$3,$5}' file
B. for solution 2:
grep JKL=877 file | grep -o '.ABC=[0-9]\{3\}' | tr 'ABC=' ' ' | awk '{print $1}'
(I use awk to remove the space created by tr function)
grep JKL=877 file | grep -o '.GHI=[0-9]\{3,5\}' | tr 'ABC=' ' ' | awk '{print $1}'
without the '[' issue, I would do:
printf "TIMESTAMP ABC GHI\n";
awk '{print $1,$3,$5}' file | tr 'ABC=' ' ' | tr 'GHI=' ' '
C. Now to have them all in once, I was thinking of a loop and puting matches in a variable (see https://unix.stackexchange.com/questions/37313/how-do-i-grep-for-multiple-patterns):
MATCH=".ABC=[0-9]\{3\} .GHI=[0-9]\{3,5\}" but something is wrong with my syntax; furthermore, it does not include timestampx.
printf "TIMESTAMP ABC GHI\n"
grep JKL=877 file | while read line
do
?
done
Thanx for your help.

Answer 1

Try using sed
printf "TIMESTAMP\tABC\tGHI\n"
sed -nr '/JKL=877/s/^(\w+).*ABC=([0-9]+).*GHI=([0-9]+).*/\1\t\2\t\3/p' file
Output:
TIMESTAMP ABC GHI
timestamp1 123 547
timestamp3 877 77422
timestamp4 269 887

Answer 2

With these kinds of problems it's usually best to first build an array that maps the names to the values for the name=value type of fields. That way you can simply use the fields values by addressing the array with their names however you like:
$ cat file
timestamp1 KKIE ABC=123 [5454] GHI=547 JKL=877 MNO=878
timestamp2 GGHI ABC=544 [ 24548] GHI=883 JKL=587 MNO=874
timestamp3 GGGIO ABC=877 [3487] GHI=77422 JKL=877 MNO=877
timestamp4 GGDI ABC=269 [ 1896] GHI=887 JKL=877 MNO=123
$
$ cat tst.awk
{
for (i=1;i<=NF;i++) {
split($i,tmp,/=/)
val[tmp[1]] = tmp[2]
fld[tmp[1]] = $i
}
if (val["JKL"] == 877) {
print $1, fld["ABC"], fld["GHI"]
}
}
$
$ awk -f tst.awk file
timestamp1 ABC=123 GHI=547
timestamp3 ABC=877 GHI=77422
timestamp4 ABC=269 GHI=887

Answer 3

#!/bin/bash
cat input.txt
echo ""
echo "############"
echo "TIMESTAMP ABC GHI"
sed -ne 's/\(timestamp[0-9]\).*ABC=\([0-9]*\).*GHI=\([0-9]*\).*JKL=877.*$/\1 \2 \3/gp' input.txt
output is
timestamp1 KKIE ABC=123 [5454] GHI=547 JKL=877 MNO=878
timestamp2 GGHI ABC=544 [ 24548] GHI=883 JKL=587 MNO=874
timestamp3 GGGIO ABC=877 [3487] GHI=77422 JKL=877 MNO=877
timestamp4 GGDI ABC=269 [ 1896] GHI=887 JKL=877 MNO=123
############
TIMESTAMP ABC GHI
timestamp1 123 547
timestamp3 877 77422
timestamp4 269 887
if you are not using things between [ and ] then just ignore them

Answer 4

Here's an awk version:
awk -F'=| +' -v OFS=$'\t' 'BEGIN {
print "TIMESTAMP", "ABC", "GHI"
}{
sub(/\[[^]]+\]/, "");
if ($8==877) print $1, $4, $6
}' input-file

Answer 5

With perl :
$ perl -lne '
print "$1 $2 $3"
if m/^(timestamp\d+).*?(ABC=\d+).*?(GHI=\d+)\s+JKL=877/i
' file
Output
timestamp1 ABC=123 GHI=547
timestamp3 ABC=877 GHI=77422
timestamp4 ABC=269 GHI=887

Answer 6

For the solution 1, you could try something like :
[ ~]$ awk 'BEGIN {str=""}{str=str"\n"; for (i=1;i<=NF;i++){if($i ~ "^(timestamp\(ABC|GHI)=)"){str=str""$i" "}}} END {print str}' file.txt|sed "1d;s/\ $//g"
timestamp1 ABC=123 GHI=547
timestamp2 ABC=544 GHI=883
timestamp3 ABC=877 GHI=77422
timestamp4 ABC=269 GHI=887
If you need to catch all values which match the pattern "[A-Z]+=[0-9]+" :
[ ~]$ awk 'BEGIN {str=""} {str=str"\n"; for (i=1;i<=NF;i++){if($i ~ "^(timestamp|[A-Z]+=[0-9]+)"){str=str""$i" "}}} END {print str}' file.txt|sed "1d;s/\ $//g"
timestamp1 ABC=123 GHI=547 JKL=877 MNO=878
timestamp2 ABC=544 GHI=883 JKL=587 MNO=874
timestamp3 ABC=877 GHI=77422 JKL=877 MNO=877
timestamp4 ABC=269 GHI=887 JKL=877 MNO=123
For the solution 2 :
[ ~]$ head=$(head -n1 file.txt|egrep -o "[A-Z]+=[0-9]+"|awk -F "=" 'BEGIN{s=""}{s=s""$1" "} END {print "TIMESTAMP "s}'|sed "s/\ $//g")
[ ~]$ content=$(i=1; while read; do echo $REPLY|egrep -o "[A-Z]+=[0-9]+"|awk -F "=" 'BEGIN{s=""} {s=s""$2" "} END {print "timestamp'$i' "s}'|sed "s/\ $//g"; ((i++)); done < file.txt)
[ ~]$ echo -e "$head\n$content"
TIMESTAMP ABC GHI JKL MNO
timestamp1 123 547 877 878
timestamp2 544 883 587 874
timestamp3 877 77422 877 877
timestamp4 269 887 877 123

Answer 7

If the number of matches on a line is constant, you can get away with a grep-only solution with a little help from paste:
grep JKL=877 file |
grep -o -e '^timestamp[0-9]' -e '\bABC=[0-9]\{3\}' -e '\bGHI=[0-9]\{3,5\}' |
grep -o '[^=]*$' |
paste - - -
Output:
timestamp1 123 547
timestamp3 877 77422
timestamp4 269 887
To include the desired header do something like this:
(
printf "TIMESTAMP\tABC\tGHI\n"
grep JKL=877 file |
grep -o -e '^timestamp[0-9]' -e '\bABC=[0-9]\{3\}' -e '\bGHI=[0-9]\{3,5\}' |
grep -o '[^=]*$' |
paste - - -
)
Output:
TIMESTAMP ABC GHI
timestamp1 123 547
timestamp3 877 77422
timestamp4 269 887

Answer 8

If you can make some assumptions about the order of the input and the number of fields, e.g. no white space at the end lines, you can use the simple field referencing you attempted in "solution 2", e.g.:
awk '/JKL=877/ { print $1, $4, $(NF==11 ? 7 : 8) }' FS='=| +' file
Output:
timestamp1 123 547
timestamp3 877 77422
timestamp4 269 887