Regex in a bash scipt - regex

I've got the following text file which contains:
12.3-456, test
test test test
If the line contains xx.x-xxx, then I want to print the line out. (X's are numbers)
I think I have the correct regex and have tested it here:
http://regexr.com/3clu3
I have then used this in a bash script but the line containing the text is not printed out.
What have I messed up?
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
if [[ $line =~ /\d\d.\d-\d\d\d,/g ]]; then
echo $line
fi
done < input.txt

You need to use [0-9] instead of a \d in Bash regex. No regex delimiters are necessary, and the global flag is not necessary either. Also, you can contract it a bit using limiting quantifiers (like {3} that will match 3 occurrences of the pattern next to it). Besides, a dot matches any character in regex, so you need to escape it if you want to match a literal dot symbol.
Use
regex="[0-9]{2}\.[0-9]-[0-9]{3},"
if [[ $line =~ $regex ]]
...

This works:
#!/bin/bash
#regex="/\d\d.\d-\d\d\d,/g"
regex="[0-9\.\-]+\, [A-Za-z]+"
while IFS='' read -r line || [[ -n "$line" ]]; do
echo $line
if [[ $line =~ $regex ]]; then
echo "match"
fi
done
regex is [any of 0-9, '.', '-'] followed by ',' followed by alphachars. This could be refined in a number of ways - e.g. explicit places before/ after '-'.
Testing indicates:
$ ./sqltrace2.sh < input.txt
12.3-456, test
match
123.3-456, test
match
12.3-456,
test test test
test test test

Related

Bash regex =~ doesn’t support multiline mode?

using =~ operator to match output of a command and grab group from it. Code is as follows:
Comamndout=$(cmd) Match=‘^hello world’ If $Comamndout =~ $Match; then
echo something fi
Commandout is in pattern
Something
Hello world
But if statement is failing.
Is bash regex support multiline search with everyline start with ^ and end with $.
No, the =~ operator doesn't perform a multiline search. A newline must be matched literally:
string=$(cmd)
regexp='(^|'$'\n'')hello world'
if [[ $string =~ $regexp ]]; then
echo matches
fi
=~ would treat multiple lines as one line.
if [[ $(echo -e "abc\nd") =~ ^a.*d$ ]]; then
echo "find a string '$(echo -e "abc\nd")' that starts with a and ends with d"
fi
Output:
find a string 'abc
d' that starts with a and ends with d
P.S.
When processing multiple lines, it is common to use grep or read with either re-direct or pipeline.
For a grep and pipeline example:
# to find a line start with either a or e
echo -e "abc\nd\ne" | grep -E "^[ae]"
Output:
abc
e
For a read and redirect example:
while read line; do
if [[ $line =~ ^a} ]] ; then
echo "find a line '${line}' start with a"
fi
done <<< $(echo -e "abc\nd\ne")
Output:
find a line 'abc' start with a
P.S.
-e of echo means translate following \n into new line. -E of grep means using the extended regular expression to match.

compare regex with grep output - bash script

I'm trying to find the word "PASS_MAX_DAYS" in a file using grep
grep "^PASS_MAX_DAYS" /etc/login.defs
then I save it in a variable and compare it to a regular expression that has the value 90 or less.
regex = "PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)"
grep output is: PASS_MAX_DAYS 120
so my function should print a fail, however it matches:
function audit_Control () {
if [[ $cmd =~ $regex ]]; then
echo match
else
echo fail
fi
}
cmd=`grep "^PASS_MAX_DAYS" /etc/login.defs`
regex="PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)"
audit_Control "$cmd" "$regex"
The problem is that the bash test [[ using the regex match operator =~ does not support the common escapes such as \s for whitespace or \W for non-word-characters.
It does support posix predefined character classes, so you can use [[:space:]] in place of \s
Your regex would then be:
regex="PASS_MAX_DAYS[[:space:]]*([0-9]|[1-8][0-9]|90)"
You may want to add anchors ^ and $ to ensure a whole-line match, then the regex is
regex="^PASS_MAX_DAYS[[:space:]]*([0-9]|[1-8][0-9]|90)$"
Without the end-of-line anchor you could match lines that have trailing numbers after the match, so PASS_MAX_DAYS 9077 would match PASS_MAX_DAYS 90 and the trailing "77" would not prevent the match.
This answer also has some very useful information about bash's [[ ]] construction with the =~ operator.
I believe you have a problem with your regex, please try that version:
PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)$
[lucas#lucasmachine ~]$ cat test.sh
#!/bin/bash
function audit_Control () {
if [[ $cmd =~ $regex ]]; then
echo match
else
echo fail
fi
}
regex="PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)$"
audit_Control "$cmd" "$regex"
[lucas#lucasmachine ~]$ export cmd="PASS_MAX_DAYS 123"
[lucas#lucasmachine ~]$ ./test.sh
fail
[lucas#lucasmachine ~]$ export cmd="PASS_MAX_DAYS 1"
[lucas#lucasmachine ~]$ ./test.sh
match
I can explain the problem was the other regex was not checking the end of line, so, your were matching "PASS_MAX_DAYS 1"23 so 23 were not being "counted" to your regex. Your regex was really matching part of the text.. Now with the end of line it should match exactly 1 digit find a end of line, or [1-8][0-9] end of line or 90 end of line.

Using regular expressions in a ksh Script

I have a file (file.txt) that contains some text like:
000000000+000+0+00
000000001+000+0+00
000000002+000+0+00
and I am trying to check each line to make sure that it follows the format:
character*9, "+", character*3, "+", etc
so far I have:
#!/bin/ksh
file=file.txt
line_number=1
for line in $(cat $file)
do
if [[ "$line" != "[[.]]{9}+[[.]]{3}+[[.]]{1}+[[.]]{2} ]" ]]
then
echo "Invalid number ($line) check line $line_number"
exit 1
fi
let "line_number++"
done
however this does not evaluate correctly, no matter what I put in the lines the program terminates.
When you want line numbers of the mismatches, you can use grep -vn. Be careful with writing a correct regular expression, and you will have
grep -Evn "^.{9}[+].{3}[+].[+].{2}$" file.txt
This is not in the layout that you want, so change the layout with sed:
grep -Evn "^.{9}[+].{3}[+].[+].{2}$" file.txt |
sed -r 's/([^:]*):(.*)/Invalid number (\2) check line number \1./'
EDIT:
I changed .{1} into ..
The sed is also over the top. When you need spme explanation, you can start with echo "Linenr:Invalid line"
I'm having funny results putting the regex in the condition directly:
$ line='000000000+000+0+00'
$ [[ $line =~ ^.{9}\+.{3}\+.\+..$ ]] && echo ok
ksh: syntax error: `~(E)^.{9}\+.{3}\+.\+..$ ]] && echo ok
' unexpected
But if I save the regex in a variable:
$ re="^.{9}\+.{3}\+.\+..$"
$ [[ $line =~ $re ]] && echo ok
ok
So you can do
#!/bin/ksh
file=file.txt
line_number=1
re="^.{9}\+.{3}\+.\+..$"
while IFS= read -r line; do
if [[ ! $line =~ $re ]]; then
echo "Invalid number ($line) check line $line_number"
exit 1
fi
let "line_number++"
done < "$file"
You can also use a plain glob pattern:
if [[ $line != ?????????+???+?+?? ]]; then echo error; fi
ksh glob patterns have some regex-like syntax. If there's an optional space in there, you can handle that with the ?(sub-pattern) syntax
pattern="?????????+???+?( )?+??"
line1="000000000+000+0+00"
line2="000000000+000+ 0+00"
[[ $line1 == $pattern ]] && echo match || echo no match # => match
[[ $line2 == $pattern ]] && echo match || echo no match # => match
Read the "File Name Generation" section of the ksh man page.
Your regex looks bad - using sites like https://regex101.com/ is very helpful. From your description, I suspect it should look more like one of these;
^.{9}\+.{3}\+.{1}\+.{2}$
^[^\+]{9}\+[^\+]{3}\+[^\+]{1}\+[^\+]{2}$
^[0-9]{9}\+[0-9]{3}\+[0-9]{1}\+[0-9]{2}$
From the ksh manpage section on [[ - you would probably want to be using =~.
string =~ ere
True if string matches the pattern ~(E)ere where ere is an extended regular expression.
Note: As far as I know, ksh regex doesn't follow the normal syntax
You may have better luck with using grep:
# X="000000000+000+0+00"
# grep -qE "^[^\+]{9}\+[^\+]{3}\+[^\+]{1}\+[^\+]{2}$" <<<"${X}" && echo true
true
Or:
if grep -qE "^[^\+]{9}\+[^\+]{3}\+[^\+]{1}\+[^\+]{2}$" <<<"${line}"
then
exit 1
fi
You may also prefer to use a construct like below for handling files:
while read line; do
echo "${line}";
done < "${file}"

Regex not matching name in filepath

I have a folder with ipa files. I need to identify them by having a appstore or enterprise in the filename.
mles:drive-ios-swift mles$ ls build
com.project.drive-appstore.ipa
com.project.test.swift.dev-enterprise.ipa
com.project.drive_v2.6.0._20170728_1156.ipa
I've tried:
#!/bin/bash -veE
fileNameRegex="**appstore**"
for appFile in build-test/*{.ipa,.apk}; do
if [[ $appFile =~ $fileNameRegex ]]; then
echo "$appFile Matches"
else
echo "$appFile Does not match"
fi
done
However nothing matches:
mles:drive-ios-swift mles$ ./test.sh
build-test/com.project.drive-appstore.ipa Does not match
build-test/com.project.drive_v2.6.0._20170728_1156.ipa Does not match
build-test/com.project.test.swift.dev-enterprise.ipa Does not match
build-test/*.apk Does not match
How would the correct script look like to match build-test/com.project.drive-appstore.ipa?
You are confusing between the glob string match with a regex match. For a greedy glob match like * you can just use the test operator with ==,
#!/usr/bin/env bash
fileNameGlob='*appstore*'
# ^^^^^^^^^^^^ Single quote the regex string
for appFile in build-test/*{.ipa,.apk}; do
# To skip non-existent files
[[ -e $appFile ]] || continue
if [[ $appFile == *${fileNameGlob}* ]]; then
echo "$appFile Matches"
else
echo "$appFile Does not match"
fi
done
produces a result
build-test/com.project.drive_v2.6.0._20170728_1156.ipa Does not match
build-test/com.project.drive-appstore.ipa Matches
build-test/com.project.test.swift.dev-enterprise.ipa Does not match
(or) with a regex use greedy match .* as
fileNameRegex='.*appstore.*'
if [[ $appFile =~ ${fileNameRegex} ]]; then
# rest of the code
That said to match your original requirement to match enterprise or appstore string in file name use extended glob matches in bash
Using glob:
shopt -s nullglob
shopt -s extglob
fileExtGlob='*+(enterprise|appstore)*'
if [[ $appFile == ${fileExtGlob} ]]; then
# rest of the code
and with regex,
fileNameRegex2='enterprise|appstore'
if [[ $appFile =~ ${fileNameRegex2} ]]; then
# rest of the code
You can use the following regex to match appstore and enterprise in a filename:
for i in build-test/*; do if [[ $i =~ appstore|enterprise ]]; then echo $i; fi; done

How do you match ^[\s]* in bash?

I'm trying to write a bash script that reads in a file skips commented lines.
I have:
#!/bin/bash
### read file
IFS=$'\r\n'
while read line; do
match_pattern="^[:space:]*#"
if [[ "$line" =~ $match_pattern ]];
then
echo "#####"
continue
fi
#semicolons and commas are removed everywhere...
array+=($line)
done <list.txt
And this skips lines that begin with a "#", but not lines that begin with spaces and then a pound. ie: "^\s+#"
I get the same results using [:blank:].
How should this regular expression be written?
You are missing brackets in your pattern:
match_pattern="^[[:space:]]*#"
does what you want.
This works for me:
while read line; do
match_pattern="^\s*#"
if [[ "$line" =~ $match_pattern ]]; then
echo "#####"
fi
done
Input
One
#Two
#Three
# Four
# Five
####Six
Output
One
#Two
#####
#Three
#####
# Four
#####
# Five
#####
####Six
#####
Doesn't start with a hash
Doesn't start with infinite space followed by a hash
(?!^#|^\s+#)^.*$
Yields this result from the code in your question:
IFS=$'\r\n'
while read line; do
match_pattern="^[:space:]*#"
if [[ "$line" =~ $match_pattern ]];
then
echo "#####"
continue
fi
array+=($line)
done <list.txt
It will match lines which look like this though:
while read line; do #while loop
[:space:] is a bracket expression that will match any of the characters :, a, c, e, p, s.
[[:space:]] is a bracket expression containing a character class: it will match a whitespace character.
$ s=" # x"
$ [[ $s =~ ^[:blank:]*# ]] && echo match || echo no match
no match
$ [[ $s =~ ^[[:blank:]]*# ]] && echo match || echo no match
match
bash's extended patterns can handle this as well
$ shopt -s extglob
$ [[ $s == *([[:blank:]])#* ]] && echo match || echo no match
match