Bash script checking command line parameters - regex

I am still on the learning path of bash, shell, Linux, regex etc. Today I share this bash shell script that I have programmed and that I want to use as a "module" in future scripts. It thoroughly tests the validity of a set of command line parameters. I would like to ask the experienced adepts for advice and comments on how to better archive things in terms of syntax, different approaches or alternative commands. There were a lot that I tried differently, but I couldn't figure it out. Especially I don't like the 'case' structures. I would rather define a set of option letters like 'cmds=(h o g a m)' and then loop through that with 'for c in "${cmds[#]}"; do'. But that leads to the problem that I would have to use dynamic variable names, and I couldn't figure it out. Another problem is, that I am able to assign a boolean 'true' but I can't negate it with something like 'a=!a'. Etc. Any suggestions very welcome!
#!/bin/bash
# Usage: -p <path> -g <group> -o <owner> -m <mask> -h (help)
# Extensive option and parameter check:
expecting_option=true # alternatingly expecting option switch and parameter on command line
for i do # loop $i trough all parameters
# display help:
if [ $i = "-h" ]; then
echo "Usage:"
echo "-p <path> (default .)"
echo "-o <owner>"
echo "-g <group>"
echo "-m <permission mask> (000 - 777)"
exit
fi;
if [ "$expecting_option" = true ]; then # next arg supposed to be an option
if [[ "$i" =~ ^(.)(.*?)(.*?)$ ]]; then # retrieve up to 3 single characters
# Does it begin with '-' ?
if [ ${BASH_REMATCH[1]} != "-" ]; then
echo "ERROR: Option to begin with '-' expected at '"$i"'" >&2
exit
fi
# only one letter length for options
if [ -n "${BASH_REMATCH[3]}" ]; then
echo "ERROR: Invalid option '"$i"'. Use -h for help" >&2
exit
fi
switch=${BASH_REMATCH[2]} # save the current option switch
# has this option already been set?
# is option valid?
case $switch in
o) if [ $o ]; then
echo 'ERROR: duplicate option: -o' >&2
exit
fi;;
g) if [ $g ]; then
echo 'ERROR: duplicate option: -g' >&2
exit
fi;;
m) if [ $m ]; then
echo 'ERROR: duplicate option: -m' >&2
exit
fi;;
p) if [ $p ]; then
echo 'ERROR: duplicate option: -p' >&2
exit
fi;;
*) echo "ERROR: Invalid option '"$i"'. Use -h for help" >&2
exit;;
esac
fi
# next arg supposed to be the parameter
expecting_option=!true # it's not true, so it works. But is it 'false'?
else # this is supposed to be a parameter for the previous option switch
if [[ "$i" =~ ^\- ]]; then # starts with '-' ?
echo "ERROR: Parameter for "$switch" missing." >&2
exit
fi
case $switch in
o) # check if designated owner exists (=0):
if ! [ $(id -u "$i" > /dev/null 2>&1; echo $?) -eq 0 ]; then
echo "ERROR: user '"$i"' does not exist." >&2
exit
fi
o="$i";;
g) # check if designated group exists:
if [ -z $(getent group "$i") ]; then
echo "ERROR: group '"$i"' does not exist." >&2
exit
fi
g="$i";;
m) if ! [[ $i =~ ^[0-7][0-7][0-7]$ ]]; then
echo "ERROR: Invalid right mask '"$i"'" >&2
exit
fi
m="$i";;
p) # check if path exists
if ! [ -d "${i}" ]; then
echo "ERROR: Directory '"$i"' not found." >&2
exit
fi
p="$i";;
esac
expecting_option=true
fi
done
# last arg must be a parameter:
if [ "$expecting_option" != true ]; then
echo "ERROR: Parameter for "$switch" missing." >&2
exit
fi
# at least o, g or m must be specified:
if ! [ $g ] && ! [ $o ] && ! [ $m ] ; then
# this didn't work: if ! [ [ $g ] || [ $o ] || [ $m ] ] ; then
echo "Nothing to do. Specify at least owner, group or mask. Use -h for help."
exit
fi
# defaults: path = . owner = no change group = no change mask = no change
# set defaults:
[[ -z $p ]] && p="."
# All necessary options are given and checked:
# p defaults to . otherwise valid path
# if o is given, than the user exists
# if g is given, than the group exists
# if m is given, than the mask is valid
# at least one of o,g or m are given
# no option dupes
# no missing parameters
# ok, now let's do something:
# set group:owner + mask of whole directory tree:
if [ $g ] || [ $o ] ; then
[[ -n $g ]] && g=":"$g # use chown's column only if group change required, with or without owner
sudo find $p -type f -exec chown $o$g {} + &&
sudo find $p -type d -exec chown $o$g {} +
fi
if [ $m ]; then
sudo find $p -type f -exec chmod $m {} + &&
sudo find $p -type d -exec chmod $m {} +
fi

With 'getopts', as suggested by #Shawn in a comment, the parsing will become:
# Default values for options
g=
o=
P=.
m=
while getopts g:o:p:m: opt ; do
case "$opt" in
g) g=$OPTARG ;;
o) o=$OPTARG ;;
p) p=$OPTARG ;;
m) m=$OPTARG ;;
# Abort on illgal option
*) exit 2 ;;
esac
done
shift $((OPTIND-1))
# Rest of code goes here

Related

How to structure a compound conditional with several tests (at least 1 of which is regex)

I searched for this but haven't found an answer to this particular situation. I'm familiar with file tests in shells and with using the [[ ]] syntax to perform regex matching.
Is there a way to combine these two operations in a compound conditional that doesn't require multiple nested ifs?
So far I've tried the following (...and other much crazier variations):
if [ -e ~/.profile -a $0 =~ bash ]; then echo yes ; fi
if [ -e ~/.profile -a ( $0 =~ bash ) ]; then echo yes ; fi
if [ -e ~/.profile -a [ $0 =~ bash ] ]; then echo yes ; fi
if [[ -e ~/.profile -a $0 =~ bash ]]; then echo yes ; fi
if [[ ( -e ~/.profile ) -a ( $0 =~ bash ) ]]; then echo yes ; fi
if [ -e ~/.profile -a $0 =~ bash ]; then echo yes; fi
if [ -e ~/.profile -a $( [ $0 =~ bash ] ) ]; then echo yes; fi
if [ -e ~/.profile -a [[ $0 =~ bash ]] ]; then echo yes; fi
if [ -e ~/.profile -a $([[ $0 =~ bash ]]) ]; then echo yes; fi
-a is treated as an AND when using single brackets, eg:
$ [ 3 -gt 1 -a 2 -lt 3 ] && echo 'true'
true
For double brackets you want to use &&, eg:
$ [[ 3 -gt 1 && 2 -lt 3 ]] && echo 'true'
true
Alternatively you can && to separate tests regardless of whether you're using single or double brackets, eg:
$ [ 3 -gt 1 ] && [ 2 -lt 3 ] && echo 'true'
true
$ [[ 3 -gt 1 ]] && [[ 2 -lt 3 ]] && echo 'true'
true
$ [ 3 -gt 1 ] && [[ 2 -lt 3 ]] && echo 'true'
true
NOTE: same rules apply for -o vs || (aka OR)
Apparently, when you want to represent a LOGICAL AND between these two statements, you must use && instead of -a (which the shell interprets as "does this file exist" file test in double brackets). Also, for the regex to work, the statement must be within [[ ]]. What was unknown to me at the time is that even though -a changes its meaning in double brackets, the -e -w -r and other file tests don't change their functionality (e.g. it's the same for single or double brackets).
if [[ -w ~/.bash_profile && $0 =~ bash ]]; then ( echo 1 ; echo 2 ) >> .bash_profile
elif [[ -w ~/.profile && <someothercondition> ]]; then
( echo 3
echo 4
echo 5
) >> .profile
fi

incrementing list variable and then loop on it

in a sh script, I am trying to make a list of filename in a folder, and then loop on it to check if two consecutive filename respond well to "expression criteria".
in a folder I have:
file1.nii
file1_mask.nii
file2.nii
file2_mask.nii
etc ...
undefined number of files. but if filex.nii exists, it must have filex_mask.nii
in a .txt file that the user modify.
it contains:
file1.nii tab some parameter \n
file2.nii tab some parameter \n
etc ...
the script take long hours after to run, and for example, the mask files are used only after few hours.
so I want at the beginning of the .sh to check if filenames are well spelled and if any files in the .txt is present in the folder.
and in case not, stop the .sh and warn the user. not wait hours before noticing the problem.
For now I tried:
test=""
for entry in "${search_dir}"/*
do
echo "$entry"
test="${test} $entry"
done
I have then a string variable with space between filenames, but it has the folder name as well.
./search_dir/file1.nii ./search_dir/file1_mask.nii
I wanted file1.nii file1_mask.nii etc ...
and now I read my .txt file and check if the filename specified in it are in my test variable.
while read -r line
do
set -- $line
stack=$1
check=False
check2=False
for i in $test; do
echo "$stack.nii"
echo "$i"
if "${stack}.nii" == "$i";
then
check=True
fi
if "${stack}_mask.nii"=="$i";
then
check2=True
fi
done
done < "$txt_file"
but it is not working.
"$stack_mask.nii"=="$i"
doesn't seems to be the good way to compare strings
it generates the error:
"file1.nii" not found
Here is my solution for now, based on glenn answer:
errs=0
while read -r line; do
set -- $line
prefix="${1}.nii"
prefix2="${1}.nii.gz"
if [ -e ${PATH}/$prefix2 ]; then
echo "File found: ${PATH}/$prefix2" >&2
elif [ -e ${PATH}/$prefix ]; then
echo "File found: ${PATH}/$prefix" >&2
else
echo "File not found: ${PATH}/$prefix" >&2
errs=$((errs + 1))
fi
prefixmask="${1}_brain_mask.nii"
prefixmask2="${1}_brain_maskefsd.nii.gz"
if [ -e ${PATH}/$prefixmask ]; then
echo "Mask file found for ${PATH}/$prefixmask" >&2
elif [ -e ${PATH}/$prefixmask2 ]; then
echo "Mask file found for ${PATH}/$prefixmask2" >&2
else
echo "Mask file not found: ${PATH}/$prefixmask" >&2
errs=$((errs + 1))
fi
done < "$INPUT"
echo $errs
if [ $errs > 0 ]; then
echo "Errors found"
exit 3
fi
then only problem now is that it always exit, even if errs is equal to 0 and I don't know why ...
I would do this:
errs=0
for f in "$search_dir"/*.mii; do
[[ $f == *_mask.mii ]] && continue # skip the mask files
prefix=${f%.mii} # strip off the extension
if [[ ! -f "${prefix}_mask.mii" ]]; then
echo "Error: $f has no mask file" >&2
((errs++))
fi
done
if [[ $errs -gt 0 ]]; then
echo "Aborting due to errors" >&2
exit 2
fi
That should be pretty efficient, since it just loops through the files once.
Now that we see the input file:
errs=0
while read -r mii_file other_stuff; do
prefix="${mii_file%.mii}"
if [[ ! -f ./"$mii_file" ]]; then # adjust your relative path accordingly
echo "File not found: $mii_file" >&2
((errs++))
elif [[ ! -f ./"${prefix}_mask.mii" ]]; then
echo "Mask file missing for $mii_file" >&2
((errs++))
fi
done < "$txt_file"
if (( errs > 0 )); then
echo "Errors found"
exit 2
fi

Frustraiting How are there too many arguments in this if statement

if [ $? -ne 0 ]; then
STATUS=$(cat $LOG.$i)
if [ $STATUS != "$i-DOWN!" ]; then
echo "$(date): ping failed, $i host is down!" |
mail -s "$(date) $i host is down!" $EMAIL
fi
echo "$(date) $i-DOWN!" > $LOG.$i
else
STATUS=$(cat $LOG.$i)
**------->>>**if [ $STATUS != "$i-UP!" ]; then
echo "$(date): ping OK, $i host is up!" |
mail -s "$(date) $i host is up!" $EMAIL
fi
echo "$(date) $i-UP!" > $LOG.$i
fi
I keep getting the error "Too many arguments" for that specific line with teh arrow pointing to it.. but I have the exact same line above it and do not get the error...
This is frustrating anyone have any idea why?
My guess is that $STATUS contains spaces, try to wrap the expression in quotes:
if [ "$STATUS" != "$i-UP!" ]
The line above was not evaluated so it can't ever trigger an error. Note that shell scripts are not compiled and every line is evaluated only if it's reached at runtime.

Splitting all txt files in a folder into smaller files based on a regular expression using bash

I have a folder containing large text files. Each file is a collection of 1000 files separated by [[ file name ]]. I want to split the files and make 1000 files out of them and put them in a new folder. Is there a way in bash to do it? Any other fast method will also do.
for f in $(find . -name '*.txt')
do mkdir $f
mv
cd $f
awk '/[[.*]]/{g++} { print $0 > g".txt"}' $f
cd ..
done
You are trying to create a folder with the same name of the already existing file.
for f in $(find . -name '*.txt')
do mkdir $f
Here, "find" will list the files in the current path, and for each of these files you will try to create a directory with exactly the same name. One way of doing it would be first creating a temporary folder:
for f in $(find . -name '*.txt')
do mkdir temporary # create a temporary folder
mv $f temporary # move the file into the folder
mv temporary $f # rename the temporary folder to the name of the file
cd $f # enter the folder and go on....
awk '/[[.*]]/{g++} { print $0 > g".txt"}' $f
cd ..
done
Note that all your folders will have the ".txt" extension. If you don't want that, you can cut it out before creating the folder; that way, you won't need the temporary folder, because the folder you're trying to create has a different name from the .txt file.
Example:
for f in $(find . -name '*.txt' | rev | cut -b 5- | rev)
Although not awk and written and written by a drunk person, not guaranteed to work.
import re
import sys
def main():
pattern = re.compile(r'\[\[(.+)]]')
with open (sys.argv[1]) as f:
for line in f:
m = re.search(pattern, line)
if m:
try:
with open(fname, 'w+') as g:
g.writelines(lines)
except NameError:
pass
fname = m.group(1)
lines = []
else:
lines.append(line)
with open(fname, 'w+') as g:
g.writelines(lines)
if __name__ == '__main__':
main()
Write a bash script. Here, I've done it for you.
Notice the structure and features of this script:
explain what it does in a usage() function, which is used for the -h option.
provide a set of standard options: -h, -n, -v.
use getopts to do option processing
do lots of error checking on the arguments
be careful about filename parsing (notice that blanks surrounding the file names are ignored.
hide details within functions. Notice the 'talk', 'qtalk', 'nvtalk' functions? Those are from a bash library I've built to make this kind of scripting easy to do.
explain what is going on to the user if in $verbose mode.
provide the user the ability to see what would be done without actually doing it (the -n option, for $norun mode).
never run commands directly. but use the run function, which pays attention to the $norun, $verbose, and $quiet variables.
I'm not just fishing for you, but teaching you how to fish.
Good luck with your next bash script.
Alan S.
#!/bin/bash
# split-collections IN-FOLDER OUT-FOLDER
PROG="${0##*/}"
usage() {
cat 1>&2 <<EOF
usage: $PROG [OPTIONS] IN-FOLDER OUT-FOLDER
This script splits a collection of files within IN-FOLDER into
separate, named files into the given OUT-FOLDER. The created file
names are obtained from formatted text headers within the input
files.
The format of each input file is a set of HEADER and BODY pairs,
where each HEADER is a text line formatted as:
[[input-filename1]]
text line 1
text line 2
...
[[input-filename2]]
text line 1
text line 2
...
Normal processing will show the filenames being read, and file
names being created. Use the -v (verbose) option to show the
number of text lines being written to each created file. Use
-v twice to show the actual lines of text being written.
Use the -n option to show what would be done, without actually
doing it.
Options
-h Show this help
-n Dry run -- do NOT create any files or make any changes
-o Overwrite existing output files.
-v Be verbose
EOF
exit
}
talk() { echo 1>&2 "$#" ; }
chat() { [[ -n "$norun$verbose" ]] && talk "$#" ; }
nvtalk() { [[ -n "$verbose" ]] || talk "$#" ; }
qtalk() { [[ -n "$quiet" ]] || talk "$#" ; }
nrtalk() { talk "${norun:+(norun) }$#" ; }
error() {
local code=2
case "$1" in [0-9]*) code=$1 ; shift ;; esac
echo 1>&2 "$#"
exit $code
}
talkf() { printf 1>&2 "$#" ; }
chatf() { [[ -n "$norun$verbose" ]] && talkf "$#" ; }
nvtalkf() { [[ -n "$verbose" ]] || talkf "$#" ; }
qtalkf() { [[ -n "$quiet" ]] || talkf "$#" ; }
nrtalkf() { talkf "${norun:+(norun) }$#" ; }
errorf() {
local code=2
case "$1" in [0-9]*) code=$1 ; shift ;; esac
printf 1>&2 "$#"
exit $code
}
# run COMMAND ARGS ...
qrun() {
( quiet=1 run "$#" )
}
run() {
if [[ -n "$norun" ]]; then
if [[ -z "$quiet" ]]; then
nrtalk "$#"
fi
else
if [[ -n "$verbose" ]]; then
talk ">> $#"
fi
if ! eval "$#" ; then
local code=$?
return $code
fi
fi
return 0
}
show_line() {
talkf "%s:%d: %s\n" "$in_file" "$lines_in" "$line"
}
# given an input filename, read it and create
# the output files as indicated by the contents
# of the text in the file
split_collection() {
in_file="$1"
out_file=
lines_in=0
lines_out=0
skipping=
while read line ; do
: $(( lines_in++ ))
[[ $verbose_count > 1 ]] && show_line
# if a line with the format of "[[foo]]" occurs,
# close the current output file, and open a new
# output file called "foo"
if [[ "$line" =~ ^\[\[[[:blank:]]*([^ ]+.*[^ ]|[^ ])[[:blank:]]*\]\][[:blank:]]*$ ]] ; then
new_file="${BASH_REMATCH[1]}"
# close out the current file, if any
if [[ "$out_file" ]]; then
nrtalkf "%d lines written to %s\n" $lines_out "$out_file"
fi
# check the filename for bogosities
case "$new_file" in
*..*|*/*)
[[ $verbose_count < 2 ]] && show_line
error "Badly formatted filename"
;;
esac
out_file="$out_folder/$new_file"
if [[ -e "$out_file" ]]; then
if [[ -n "$overwrite" ]]; then
nrtalk "Overwriting existing '$out_file'"
qrun "cat /dev/null >'$out_file'"
else
error "$out_file already exists."
fi
else
nrtalk "Creating new output file: '$out_file' ..."
qrun "touch '$out_file'"
fi
lines_out=0
elif [[ -z "$out_file" ]]; then
# apparently, there are text lines before the filename
# header; ignore them (out loud)
if [[ ! "$skipping" ]]; then
talk "Text preceding first filename ignored.."
skipping=1
fi
else # next line of input for the file
qrun "echo \"$line\" >>'$out_file'"
: $(( lines_out++ ))
fi
done
}
norun=
verbose=
verbose_count=0
overwrite=
quiet=
while getopts 'hnoqv' opt ; do
case "$opt" in
h) usage ;;
n) norun=1 ;;
o) overwrite=1 ;;
q) quiet=1 ;;
v) verbose=1 ; : $(( verbose_count++ )) ;;
esac
done
shift $(( OPTIND - 1 ))
in_folder="${1:?Missing IN-FOLDER; see $PROG -h for details}"
out_folder="${2:?Missing OUT-FOLDER; see $PROG -h for details}"
# validate the input and output folders
#
# It might be reasonable to create the output folder for the
# user, but that's left as an exercise for the user.
in_folder="${in_folder%/}" # remove trailing slash, if any
out_folder="${out_folder%/}"
[[ -e "$in_folder" ]] || error "$in_folder does not exist"
[[ -d "$in_folder" ]] || error "$in_folder is not a directory."
[[ -e "$out_folder" ]] || error "$out_folder does not exist."
[[ -d "$out_folder" ]] || error "$out_folder is not a directory."
for collection in $in_folder/* ; do
talk "Reading $collection .."
split_collection "$collection" <$collection
done
exit

shell scripting and regular expression

#!bin/bash
echo enter your password :
read password
passlength=$(echo ${#password})
if [ $passlength -le 8 ];
then
echo you entered correct password
else
echo entered password is incorrect
fi
if [[$password == [a-z]*[0-9][a-z]*]];
then
echo match found
else
echo match not found
fi
I am not getting what's wrong with this code. If I enter any string as a password, let's say hello123, it gives me an error:
hello123 : command not found
What is wrong with my script?
You can do the following to make it work cross-platforms with any the bourne shell (/bin/sh) based shell, no bash specific primitives -
echo "$password" | grep -q "[a-z]*[0-9][a-z]*"
if [ $? -eq 0 ] ;then
echo "match found"
else
echo "match not found"
fi
Also feel free to use quotes around the variable names. It will save you hours and hours worth of useless debugging. :)
Technically it should give you an error like [[hello123 : command not found.
The issue is that [[$password is not expanded how you think it is. Bash will first resolve the $password variable to what you entered (i.e. hello123). This will yield the string [[hello123 which bash will then try to invoke (and fail, as there is nothing with that name).
Simply add a space () after [[ and bash will recognise [[ as the command to run (although it is a builtin).
if [[ "$password" == [a-z]*[0-9][a-z]* ]]
then
...
The corrected script is below. The errors were:
#!/bin/bash, not #!bin/bash
To read password length, just do passlength=${#password}, not
passlength=$(echo ${#password})
Always put a space after [ or [[
#!/bin/bash
echo "enter your password :"
read password
passlength=${#password}
if [[ $passlength -le 8 ]]
then
echo "you entered correct password"
else
echo "entered password is incorrect"
fi
if [[ $password == [a-z]*[0-9][a-z]* ]]
then
echo "match found"
else
echo "match not found"
fi
In the bash [[ construct, the == operator will match glob-style patterns, and =~ will match regular expressions. See the documentation.
#!/bin/bash
read -s -p "Enter Password: " password
password_length=${#password}
if [ $password_length -lt 8 -o $password_length -gt 20 ] ;then
echo -e "Invalid password - should be between 8 and 20 characters in length.";
echo ;
else
# Check for invalid characters
case $password in
*[^a-zA-Z0-9]* )
echo -e "Password contains invalid characters.";
echo ;
;;
* )
echo "Password accepted.";
echo ;
break;
;;
esac
fi
More tuned example..
Try to replace line
if [[$password == [a-z]*[0-9][a-z]*]];
with following
if echo "$password" | grep -qs '[a-z]*[0-9][a-z]*'
HTH