Using regex in Bash with mapfile - regex

Question

Edit 2:
Minimal input file: input/input.txt
#-----------
snapshot=83
#-----------
time=30142088
mem_heap_B=20224
mem_heap_extra_B=8
mem_stacks_B=240480
heap_tree=empty
#-----------
snapshot=84
#-----------
time=30408368
mem_heap_B=20224
mem_heap_extra_B=8
mem_stacks_B=240552
heap_tree=empty
#-----------
snapshot=85
#-----------
time=30674648
mem_heap_B=20224
mem_heap_extra_B=8
mem_stacks_B=240464
heap_tree=empty
#-----------
snapshot=86
#-----------
Actual output:
input.txt/*
time, heap, stack
input/input.txt
time, heap, stack
30674648, 20224, 240464
input/input.txt
time, heap, stack
input/input.txt
time, heap, stack
input/input.txt
time, heap, stack
30674648, 20224, 240464
Expected output:
input.txt
time, heap, stack,
30142088, 20224, 240480
30408368, 20224, 240552
30674648, 20224, 240464
Edit: Originally, the problem may have been due to Bash's regex's lack of multiline capability. However, after stripping newlines from the text, the problem remains, with the exception that the output now has between one to five lines instead of zero.
I'm trying to write a Bash script to parse a text file into a desirable CSV file with the needed information.
As part of the script, I iterate through n files. Each of the files contains m matches for a given regex, and each match contains three capture groups.
I want to format the three capture groups into a CSV row, then concatenate all the rows of all the matches of all the files and write them to a *.csv file.
I'm quite comfortable using Regex in high level languages such as Kotlin or C#, however I have no experience with Regex in Bash. I used this answer as a starting point, however it doesn't seem to be working for me (mapfile -t matches < <( format_row "$text" "$regex" ) doesn't do anything.
Here's the full code with the relevant portion noted:
#!/bin/bash
# RELEVANT CODE BELOW
regex="time=([0-9]+)\nmem_heap_B=([0-9]+)\n.*\nmem_stacks_B=([0-9]+)"
format_row() {
local s=$1 regex=$2
while [[ $s =~ $regex ]]
do
time="${BASH_REMATCH[1]}"
heap="${BASH_REMATCH[2]}"
stack="${BASH_REMATCH[3]}"
echo "${time}, ${heap}, ${stack}"
echo ""
s=${s#*"${BASH_REMATCH[3]}"}
done
}
for file in $1/*
do
echo "Parsing ${file}..."
echo $file >> $2
echo "time, heap, stack" >> $2
text=$(<${file})
mapfile -t matches < <( format_row "$text" "$regex" )
printf "%s\n" "${matches[#]}" >> $2
echo "" >> $2
done
echo ""
echo "Done"
Thanks!

Answer 1

There are two problems here:
Although bash's =~ operator can match newlines, it does not understand the \n escape sequence. You have to use actual newlines in your regex. This can also be achieved by C-style strings $'\n'.
The regex quantifier * is greedy. When matching ...
[[ "a=1,b=1 a=2,b=2 a=3,b=3" =~ a=(.).*b=(.) ]]
... you end up with BATCH_REMATCH=(1 3) instead of (1 1).
In other regex dialects like PCRE you could use the non-greedy quantifier *?. However, in bash we have to use a workaround and have to replace .* with something that cannot match more than wanted, for instance
[[ "a=1,b=1 a=2,b=2 a=3,b=3" =~ a=(.)[^=]*b=(.) ]]
In your case we have to make sure that the next mem_stacks is not matched
As you didn't post any example input and expected output, I can only guess. However, I assume the following regex could work for you:
regex=$'time=([0-9]+)
mem_heap_B=([0-9]+)
([^\n]*\n){TODO set number of lines allowed here}
mem_stacks_B=([0-9]+)'
Note that now you have to use BASH_REMATCH[4] instead of [3].
At the marked location you have to insert the numbers of lines allowed between mem_heap and mem_stacks. The number can be constant (e.g. {5}) or a range (e.g. {1,10}). In case of ranges you have to make sure that the maximum bound is not so high that you could accidentally skip the next mem_stacks and match another mem_stacks instead. Thus, in case of ranges it might be more appropriate to use two matches. Something like
regex1='time=([0-9]+)
mem_heap_B=([0-9]+)'
regex2='mem_stacks_B=([0-9]+)'
while
[[ "$s" =~ $regex1 ]] &&
time="${BASH_REMATCH[1]}" &&
heap="${BASH_REMATCH[2]}" &&
[[ "$s" =~ $regex2 ]] &&
stack="${BASH_REMATCH[1]}"
do
echo "$time, $heap, $stack"
s="${s#*$stack}"
done >> "$2"
By the way:
https://www.shellcheck.net/ helps you to make your script more robust.
First and foremost: quote your variables.
You can use do cmd1; cmd2 done >> file instead of do cmd1 >> file; cmd2 >> file; done.
mapfile -t matches < <(format_row "$text" "$regex")
printf "%s\n" "${matches[#]}" >> "$2"
could be written as just
format_row "$text" "$regex" >> "$2"