Difference between grep -E regex and Bash regex in conditional expression - regex

For the same regex applied to the same string, why does grep -E match, but the Bash =~ operator in [[ ]] does not?
$ D=Dw4EWRwer
$ echo $D|grep -qE '^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_-\ ]{1,22}$' || echo wrong pattern
$ [[ "${D}" =~ ^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_-\ ]{1,22}$ ]] || echo wrong pattern
wrong pattern
Update: I confirm this worked:
[[ "${D}" =~ ^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]\ _-]{1,22}$ ]] || echo wrong pattern

The problem (for both versions of the code) is on this character class:
[[:alnum:]_-\ ]
In the grep version, because the regex is enclosed in single quotes, the backslash doesn't escape anything and the character range received by grep is exactly how it is represented above.
In the bash version, the backslash (\) escapes the space that follows it and the actual character class used by [[ ]] to test is [[:alnum:]_- ].
Because in ASCII table the underscore (_) comes after both space () and backslash (\), neither of these character classes is correct.
For the bash version you can use:
[[ "${D}" =~ ^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_-\ ]{1,22}$ ]]; echo $?
to verify its outcome. If the regex is incorrect, the exit code is 2.
If you want to put a dash (-) into a character class you have to put it either as the first character in the class (just after [ or [^ if it is a negating class) or as the last character in the class (right before the closing]`).
The grep version of the code should be (there is no need to escape anything inside a string enclosed in single quotes):
$ echo $D | grep -qE '^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_ -]{1,22}$' || echo wrong pattern
The bash version of your code should be:
[[ "${D}" =~ ^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_\ -]{1,22}$ ]] || echo wrong pattern

Based on your comment, you want the bracket expression to contain alphanumeric characters, spaces, underscores and dashes, so the dash is not supposed to indicate a range. To add a hyphen to a bracket expression, it has to be the first or last character in it. Additionally, you don't have to escape things in bracket expressions, so you can drop the backslash. Your grep regex includes a literal \ in the bracket expression:
$ grep -q '[\]' <<< '\' && echo "Match"
Match
In the Bash regex, the space has to be escaped because the string is first read by the shell, but see below how to avoid that.
First, fixing your regex:
^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_ -]{1,22}$
The backslash is gone, and the hyphen is moved to the end. Using this with grep works fine:
$ D=Dw4EWRwer
$ grep -E '^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_ -]{1,22}$' <<< "$D"
Dw4EWRwer
To use the regex within [[ ]] directly, the space has to be escaped:
$ [[ $D =~ ^[A-Z][A-Za-z0-9]{1,2}[[:alnum:]_\ -]{1,22}$ ]] && echo "Match"
Match
I would make the following changes:
Use character classes where possible: [A-Z] is [[:upper:]], [A-Za-z0-9] is [[:alnum:]]
Store the regex in a variable for usage in [[ ]]; this has two advantages: no escaping characters special to the shell, and compatibility with older Bash versions, as the quoting requirements changed between 3.1 and 3.2 (see the Patterns article in the BashGuide).
The regex would then become this for grep:
$ grep -E '^[[:upper:]][[:alnum:]][[:alnum:]_ -]{1,22}$' <<< "$D"
Dw4EWRwer
and this in Bash:
$ re='^[[:upper:]][[:alnum:]][[:alnum:]_ -]{1,22}$'
$ [[ $D =~ $re ]] && echo "Match"
Match

Related

compare regex with grep output - bash script

I'm trying to find the word "PASS_MAX_DAYS" in a file using grep
grep "^PASS_MAX_DAYS" /etc/login.defs
then I save it in a variable and compare it to a regular expression that has the value 90 or less.
regex = "PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)"
grep output is: PASS_MAX_DAYS 120
so my function should print a fail, however it matches:
function audit_Control () {
if [[ $cmd =~ $regex ]]; then
echo match
else
echo fail
fi
}
cmd=`grep "^PASS_MAX_DAYS" /etc/login.defs`
regex="PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)"
audit_Control "$cmd" "$regex"
The problem is that the bash test [[ using the regex match operator =~ does not support the common escapes such as \s for whitespace or \W for non-word-characters.
It does support posix predefined character classes, so you can use [[:space:]] in place of \s
Your regex would then be:
regex="PASS_MAX_DAYS[[:space:]]*([0-9]|[1-8][0-9]|90)"
You may want to add anchors ^ and $ to ensure a whole-line match, then the regex is
regex="^PASS_MAX_DAYS[[:space:]]*([0-9]|[1-8][0-9]|90)$"
Without the end-of-line anchor you could match lines that have trailing numbers after the match, so PASS_MAX_DAYS 9077 would match PASS_MAX_DAYS 90 and the trailing "77" would not prevent the match.
This answer also has some very useful information about bash's [[ ]] construction with the =~ operator.
I believe you have a problem with your regex, please try that version:
PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)$
[lucas#lucasmachine ~]$ cat test.sh
#!/bin/bash
function audit_Control () {
if [[ $cmd =~ $regex ]]; then
echo match
else
echo fail
fi
}
regex="PASS_MAX_DAYS\s*([0-9]|[1-8][0-9]|90)$"
audit_Control "$cmd" "$regex"
[lucas#lucasmachine ~]$ export cmd="PASS_MAX_DAYS 123"
[lucas#lucasmachine ~]$ ./test.sh
fail
[lucas#lucasmachine ~]$ export cmd="PASS_MAX_DAYS 1"
[lucas#lucasmachine ~]$ ./test.sh
match
I can explain the problem was the other regex was not checking the end of line, so, your were matching "PASS_MAX_DAYS 1"23 so 23 were not being "counted" to your regex. Your regex was really matching part of the text.. Now with the end of line it should match exactly 1 digit find a end of line, or [1-8][0-9] end of line or 90 end of line.

Regex stored in a shell variable doesn't work between double brackets

The below is a small part of a bigger script I'm working on, but the below is giving me a lot of pain which causes a part of the bigger script to not function properly. The intention is to check if the variable has a string value matching red hat or Red Hat. If it is, then change the variable name to redhat. But it doesn't quite match the regex I've used.
getos="red hat"
rh_reg="[rR]ed[:space:].*[Hh]at"
if [ "$getos" =~ "$rh_reg" ]; then
getos="redhat"
fi
echo $getos
Any help will be greatly appreciated.
There are a multiple things to fix here
bash supports regex pattern matching within its [[ extended test operator and not within its POSIX standard [ test operator
Never quote our regex match string. bash 3.2 introduced a compatibility option compat31 (under New Features in Bash 1.l) which reverts bash regular expression quoting behavior back to 3.1 which supported quoting of the regex string.
Fix the regex to use [[:space:]] instead of just [:space:]
So just do
getos="red hat"
rh_reg="[rR]ed[[:space:]]*[Hh]at"
if [[ "$getos" =~ $rh_reg ]]; then
getos="redhat"
fi;
echo "$getos"
or enable the compat31 option from the extended shell option
shopt -s compat31
getos="red hat"
rh_reg="[rR]ed[[:space:]]*[Hh]at"
if [[ "$getos" =~ "$rh_reg" ]]; then
getos="redhat"
fi
echo "$getos"
shopt -u compat31
But instead of messing with those shell options just use the extended test operator [[ with an unquoted regex string variable.
There are two issues:
First, replace:
rh_reg="[rR]ed[:space:].*[Hh]at"
With:
rh_reg="[rR]ed[[:space:]]*[Hh]at"
A character class like [:space:] only works when it is in square brackets. Also, it appears that you wanted to match zero or more spaces and that is [[:space:]]* not [[:space:]].*. The latter would match a space followed by zero or more of anything at all.
Second, replace:
[ "$getos" =~ "$rh_reg" ]
With:
[[ "$getos" =~ $rh_reg ]]
Regex matches requires bash's extended test: [[...]]. The POSIX standard test, [...], does not have the feature. Also, in bash, regular expressions only work if they are unquoted.
Examples:
$ rh_reg='[rR]ed[[:space:]]*[Hh]at'
$ getos="red Hat"; [[ "$getos" =~ $rh_reg ]] && getos="redhat"; echo $getos
redhat
$ getos="RedHat"; [[ "$getos" =~ $rh_reg ]] && getos="redhat"; echo $getos
redhat

Regular expression in bash s character

I have very strange issue with s character.
This works:
[[ "import scala" =~ ^import\s*.+cala$ ]] && echo "yes"
but this doesn't work:
[[ "import scala" =~ ^import\s*scala$ ]] && echo "yes"
I tried to escape s and but it didn't works.
How to solve this issue?
\s doesn't work with bash regex. Use [[:blank:]] instead to match a space or tab character:
[[ "import scala" =~ ^import[[:blank:]].*scala$ ]] && echo "yes"
yes
PS: However [[:space:]] is equivalent of \s that also matches \n
Also note that you must use .* instead of .+ before scala to match 0 or more characters instead of 1+ because space has already been matched using [[:blank:]]
\s will lose its meaning in shell (escaped as 's'), try to use a variable to store regex expression as suggested in bash manual:
ex='^import\s+scala$'; [[ "import scala" =~ $ex ]] && echo "yes"
This works on my machine.

Regex in a bash scipt

I've got the following text file which contains:
12.3-456, test
test test test
If the line contains xx.x-xxx, then I want to print the line out. (X's are numbers)
I think I have the correct regex and have tested it here:
http://regexr.com/3clu3
I have then used this in a bash script but the line containing the text is not printed out.
What have I messed up?
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
if [[ $line =~ /\d\d.\d-\d\d\d,/g ]]; then
echo $line
fi
done < input.txt
You need to use [0-9] instead of a \d in Bash regex. No regex delimiters are necessary, and the global flag is not necessary either. Also, you can contract it a bit using limiting quantifiers (like {3} that will match 3 occurrences of the pattern next to it). Besides, a dot matches any character in regex, so you need to escape it if you want to match a literal dot symbol.
Use
regex="[0-9]{2}\.[0-9]-[0-9]{3},"
if [[ $line =~ $regex ]]
...
This works:
#!/bin/bash
#regex="/\d\d.\d-\d\d\d,/g"
regex="[0-9\.\-]+\, [A-Za-z]+"
while IFS='' read -r line || [[ -n "$line" ]]; do
echo $line
if [[ $line =~ $regex ]]; then
echo "match"
fi
done
regex is [any of 0-9, '.', '-'] followed by ',' followed by alphachars. This could be refined in a number of ways - e.g. explicit places before/ after '-'.
Testing indicates:
$ ./sqltrace2.sh < input.txt
12.3-456, test
match
123.3-456, test
match
12.3-456,
test test test
test test test

Bash regex: replace string with any number of characters

I'm trying to remove colouring codes from a string; e.g. from: \033[36;1mDISK\033[0m to: DISK
my regex looks like this: \033.*?m so match '\033' followed by any number of chars, terminated by 'm'
when I search for the pattern, it finds a match; [[ "$var" =~ $regex ]] evaluates to true
however when I try to replace matches, nothing happens and the same string is returned.
Here's my complete script:
regex="\033.*?m"
var="\033[36;1mDISK\033[0m"
if [[ "$var" =~ $regex ]]
then
echo "matches"
echo ${var//$regex}
else
echo "doesn't match!"
fi
The problem appears to be with the match any number of any character part of the regex. I can successfully replace DISK but if I change that to D.*K or D.*?K it fails.
Note in all above cases the pattern claims to match the string but fails when replacing. Not too sure where to go with this now, any help appreciated.
Thanks
The following should do it:
$ var="\033[36;1mDISK\033[0m"
$ newvar=$(printf ${var} | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g")
$ echo ${newvar}
returns:
DISK
Now verify!
$ echo $var | od
0000000 030134 031463 031533 035466 066461 044504 045523 030134
0000020 031463 030133 005155
0000026
$ echo $newvar | od
0000000 044504 045523 000012
0000005
To use the parameter expansion substitution operator, you need to use an extended glob.
shopt -s extglob
newvar=${var//\\033\[*([0-9;])m}
To break it down:
\\033\[ - match the encoded escape character and [.
*([0-9;]) - match zero or more digits or semicolons. You could use +([0-9;]) to (more correctly?) match one or more digits or semicolons
m - the trailing m.