Pass a regular expression as a parameter to a shell script - regex

How to pass a regular expression as a parameter to a shell script?
I need to write a shell script, which will take parameters and give them to unix commands. And I'd like to use regular expressions there. Is this possible at all?
Or reformulation - how to write the script equivalent to "cp" command, using only this command?
I'm trying to make a file "mycp"
#!/bin/bash -fx
cp $2 $1
and call it by
mycp myDir "*sh"
and want it to do the same as "cp *sh myDir".
But resulting bash interpretation is:
+ cp '*sh' myDir
cp: *sh: No such file or directory

Revised question
I'm trying to make a file "mycp"
#!/bin/bash -fx
cp $2 $1
and call it by
mycp myDir "*sh"
In that case, you still need eval, but you'd write:
#!/bin/bash -x
eval cp "$2" "$1"
You're running into problems because you have specified the -f option. man bash says (in part):
After word splitting, unless the -f option has been set, bash scans each word for the characters *, ?, and [.
Remove the f from the 'shebang' (first) line of the script.
Original question
Given that you want mycp "*sh*" aa to do shell expansion on the argument, you'll probably end up using eval in your script:
eval cp "$#"
However, the use of eval is dangerous; it can lead to unexpected side-effects. The use of "$#" is important; it preserves the number of arguments and spaces in them. Unfortunately, using eval then undoes that, but we can't have everything — or not easily.
For quite a long time (say 1987* to 1999), I used this script as a cover for cp:
: "#(#)$Id: cp.sh,v 1.3 1997/06/02 21:45:00 johnl Exp $"
#
# Alternative copy command
case $# in
0) /bin/cp ;;
1) /bin/cp $1 . ;;
2) /bin/cp "$#" ;;
*) if [ -d `la "$#"` ]
then /bin/cp "$#"
else /bin/cp "$#" .
fi;;
esac
It uses a very simple C program called la (for 'last argument') to get the last argument and checks whether the given last argument is a directory. More than 99% of the time, if I typed (by accident) 'cp /some/where/sh' rather than cp /some/where/*sh* ., the second was what I meant, and the script fixed things. I haven't used it for quite some time; it may have been in the last millennium, but was probably sometime earlier in this one that I gave up using it.
* Although the version string says '1997', the code is identical to the 1987 version. Version 1.1 and 1.2 were under SCCS and used different SCCS ID strings; the conversion to RCS made them identical. Version 1.3 reinstated the #(#) identifier string used by the SCCS what command to the RCS version handling. So, the script is ancient — 1987, really.

Try this: eval cp $1 $2
Bash reference manual: eval

If all you want to do is specify the directory first, look at the -t option to GNU cp:
alias mycp='cp -t'
mycp mydir *.sh
Otherwise:
mycp() {
local dir=$1
shift
cp "$#" "$dir"
}
mycp mydir *.sh
Either way, let the shell expand the wildcards and avoid the use of eval
(note, *.sh is not a regular expression, it's a shell "pattern", often referred to as a "glob pattern")

Related

/bin/sh: 1: Syntax error: "(" unexpected after running Makefile [duplicate]

I often find Bash syntax very helpful, e.g. process substitution like in diff <(sort file1) <(sort file2).
Is it possible to use such Bash commands in a Makefile? I'm thinking of something like this:
file-differences:
diff <(sort file1) <(sort file2) > $#
In my GNU Make 3.80 this will give an error since it uses the shell instead of bash to execute the commands.
From the GNU Make documentation,
5.3.2 Choosing the Shell
------------------------
The program used as the shell is taken from the variable `SHELL'. If
this variable is not set in your makefile, the program `/bin/sh' is
used as the shell.
So put SHELL := /bin/bash at the top of your makefile, and you should be good to go.
BTW: You can also do this for one target, at least for GNU Make. Each target can have its own variable assignments, like this:
all: a b
a:
#echo "a is $$0"
b: SHELL:=/bin/bash # HERE: this is setting the shell for b only
b:
#echo "b is $$0"
That'll print:
a is /bin/sh
b is /bin/bash
See "Target-specific Variable Values" in the documentation for more details. That line can go anywhere in the Makefile, it doesn't have to be immediately before the target.
You can call bash directly, use the -c flag:
bash -c "diff <(sort file1) <(sort file2) > $#"
Of course, you may not be able to redirect to the variable $#, but when I tried to do this, I got -bash: $#: ambiguous redirect as an error message, so you may want to look into that before you get too into this (though I'm using bash 3.2.something, so maybe yours works differently).
One way that also works is putting it this way in the first line of the your target:
your-target: $(eval SHELL:=/bin/bash)
#echo "here shell is $$0"
If portability is important you may not want to depend on a specific shell in your Makefile. Not all environments have bash available.
You can call bash directly within your Makefile instead of using the default shell:
bash -c "ls -al"
instead of:
ls -al
There is a way to do this without explicitly setting your SHELL variable to point to bash. This can be useful if you have many makefiles since SHELL isn't inherited by subsequent makefiles or taken from the environment. You also need to be sure that anyone who compiles your code configures their system this way.
If you run sudo dpkg-reconfigure dash and answer 'no' to the prompt, your system will not use dash as the default shell. It will then point to bash (at least in Ubuntu). Note that using dash as your system shell is a bit more efficient though.
It's not a direct answer to the question, makeit is limited Makefile replacement with bash syntax and it can be useful in some cases (I'm the author)
rules can be defined as bash-functions
auto-completion feature
Basic idea is to have while loop in the end of the script:
while [ $# != 0 ]; do
if [ "$(type -t $1)" == 'function' ]; then
$1
else
exit 1
fi
shift
done
https://asciinema.org/a/435159

How to pass a command which contains special characters through SSH?

I would like to run the following command from Jenkins:
ssh -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname "supervisorctl start company-$app ; awk -v app=$app '$0 ~ "program:company-"app {p=NR} p && NR==p+6 && /^autostart/ {$0="autostart=true" ; p=0} 1' /etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf”
This is one of the last steps of a job which creates a CloudFormation stack.
Running the command from the target server's terminal works properly.
In this step, I'd like to ssh to each one of the servers (members of ASG's within the new stack) and search and replace a specific line as shown above in the /etc/supervisord.conf, basically setting one specific service to autostart.
When I run the command I get the following error:
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
I've tried escaping the double quotes but got the same error, any idea what I'm doing wrong?
You are running in to this issue due to the way the shell handles nested quotes. This is a use case for a HERE DOCUMENT or heredoc - A HERE DOCUMENT allows you to write multi-line commands passed through bash without worrying about quotes. The structure is as follows:
$ ssh -t user#server.com <<'END'
command |\
command2 |\
END
<--- Oh yeah, the -t is important to the ssh command as it lets the shell know to behave as if being used interactively, and will avoid warnings and unexpected results.
In your specific case, you should try something like:
$ ssh -t -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname <<'END'
supervisorctl start company-$app |\
awk -v app=$app '$0 ~ \"program:company-\"app {p=NR} p && NR==p+6 \
&& /^autostart/ {$0="autostart=true" ; p=0} 1' \
/etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf
END
Just a note, since I can't be sure about your desired output of the command you are running, be advised to keep track of your own " and ' marks, and to escape them accordingly in your awk command as you would at an interactive terminal. I notice the "'s around program:company and I am confused a bit by them If they are a part of the pattern in the string being searched they will need to be escaped accordingly. P.S.

UNIX: error when I try to use array

I have this code and when I try it in FreeBSD it shows me a lot of errors... how can I fix it? I check directories, if it match with variable IGN. NAME_d should be an array.
max_d=$(find "${DIR}" -type d | wc -l)
for i in `seq 1 $max_d`
do
check_d=$(find "${DIR}" -type d | head -n "${i}" | tail -n -1 | tr '\/' '\n' | egrep -n "${IGN}")
if [ ! -z "$check_d" ]; then
NAME_d+=$i"d "
fi
done
directory_d=${NAME_d[*]}
sedCmds_d=${directory_d// /;}
Arrays are a bashism not supported by the Almquist shell, the default bourne style shell on FreeBSD (i.e. /bin/sh). An advantage of the shell is that most scripts run about 3 times faster.
If you want to use bashisms, use bash to execute your script. E.g. call it bash dirstat.sh or change the shebang.
This is the correct one for FreeBSD.
#!/usr/local/bin/bash
This is the portable version but requires PATH to be set:
#!/usr/bin/env bash
You also might have to install bash first: pkg add bash
Not sure if it will solve it, but if you're using bash, you should initiate NAME_d as an array
NAME_d=()
and then adding to the array you should also use parens, e.g.
NAME_d+=("${i}d ")

Grep across the file system has no output in a shell script

I'm trying to create a pre-commit hook in Git that will check for any debugging code and prompt the user to fix it. I have a regex that I'm grepping for (ignore the fact that it won't exclude occurrences in multiline comments!):
grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{module,inc,install,php,js} -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /path/to/code/
This works fine when I run it normally in the console, but when I try it in an executable .sh script it does nothing. None of the following has worked for me:
#!/bin/sh
grep ...
MYVAR =`grep ...` # Note the backticks!
echo $MYVAR
MYVAR =$(grep ...)
echo $MYVAR
MYVAR ="`grep ...`"
echo $MYVAR
I tried doing it with Python and os.system() but that did nothing either. It seems to just have no STDOUT. There's possibly something obvious I'm missing but I'm at a loose end.
Any help would be much appreciated! Thanks.
Edit:
This is the exact script, even though it's at the earliest possible stage due to not being able to actually do the first bit. I've hidden the exact folder names because it's probably best to not share my company's code base on SO ;)
#!/bin/bash
echo "Test!"
ONE=`grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{module,inc,install,php,js} -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /company/projects/company/www/sites/all/modules/custom/`
TWO=$(grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{coffee} -P '^\s*(?!\#)\s*(dpm\(|dsm\(|console.log)' /company/projects/company/www/sites/all/modules/custom/)
echo $ONE
echo "$TWO"
... and running bash -x pre-commit returns:
ubuntu#ip-12-34-56-78:/company/projects/company/scripts$ bash -x pre-commit
+ echo 'Test!'
Test!
++ grep -IiRn --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=public --exclude-dir=lib --exclude-dir=contrib '--include=*.module' '--include=*.inc' '--include=*.install' '--include=*.php' '--include=*.js' -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /company/projects/company/www/sites/all/modules/custom/
+ ONE='/company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: dsm('\''test'\'');
/company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: console.log(e.stack);
/company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: console.log('\''fetch success'\'');
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: console.log('\''wtf?'\'');
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: console.log(resp);
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: console.log(ui.placeholder);
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: console.log($( ".required" ));
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: console.log(resp);
/company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: console.log(data);'
++ grep -IiRn --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=public --exclude-dir=lib --exclude-dir=contrib '--include=*.{coffee}' -P '^\s*(?!\#)\s*(dpm\(|dsm\(|console.log)' /company/projects/company/www/sites/all/modules/custom/
+ TWO=
+ echo /company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: 'dsm('\''test'\'');' /company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: 'console.log(e.stack);' /company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: 'console.log('\''fetch' 'success'\'');' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: 'console.log('\''wtf?'\'');' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: 'console.log(resp);' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: 'console.log(ui.placeholder);' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: 'console.log($(' '".required"' '));' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: 'console.log(resp);' /company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: 'console.log(data);'
/company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: dsm('test'); /company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: console.log(e.stack); /company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: console.log('fetch success'); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: console.log('wtf?'); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: console.log(resp); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: console.log(ui.placeholder); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: console.log($( ".required" )); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: console.log(resp); /company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: console.log(data);
+ echo ''
... but running it without the -x flag STILL doesn't work.
Edit two:
In case anyone is wondering, my env is as follows...
ubuntu#ip-12-34-56-78:~$ uname -a
Linux ip-12-34-56-78 3.2.0-31-virtual #50-Ubuntu SMP Fri Sep 7 16:36:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
ubuntu#ip-12-34-56-78:~$ whereis sh && whereis bash
sh: /bin/sh /bin/sh.distrib /usr/share/man/man1/sh.1.gz
bash: /bin/bash /etc/bash.bashrc /usr/share/man/man1/bash.1.gz
I can't say for sure until you post the actual script you're running, but in your current code snippet have
#!/bin/sh
Depending on your OS, this may be a link to /bin/bash, for example, or it may be the actual Bourne shell, which does not support brace expansion (e.g. {a, b, c}). Even if /bin/sh does point to /bin/bash on your machine, you should only use portable constructs if your shebang is #!/bin/sh (i.e. say what you mean). If you want to use brace expansion in your script, change the shebang to #!/bin/bash.
If you put
set -x
at the top of your script, it will print detailed information that can help with debugging. You can also do this by invoking the shell directly instead of modifying your script, for example
sh -x /path/to/script
or
bash -x /path/to/script
EDIT: On Ubuntu, /bin/sh is dash, the Debian Almquist shell. Like the Bourne shell, dash is fairly restrictive, and does not support brace expansion. See this page for a discussion of portability issues and dash.

pattern matching while using ls command in bash script

In a sh script, I am trying to loop over all files that match the following pattern
abc.123 basically abc. followed by only numbers, number following . can be of any length.
Using
$ shopt -s extglob
$ ls abc.+([0-9])
does the job but on terminal and not through the script. How can I get only files that match the pattern?
if I understood you right, the pattern could be translated into regex:
^abc\.[0-9]+$
so you could
keep using ls and grep the output. for example:
ls *.*|xargs -n1|grep -E '^abc\.[0-9]+$'
or use find
find has an option -regex
If you're using sh and not bash, and presumably you also want to be POSIX compliant, you can use:
for f in ./*
do
echo "$f" | grep -Eq '^\./abc.[0-9]+$' && continue
echo "Something with $f here"
done
It will work fine with filenames with spaces, quotes and such, but may match some filenames with line feeds in them that it shouldn't.
If you tagged your question bash because you're using bash, then just use extglob like you described.