How to pass a command which contains special characters through SSH? - regex

I would like to run the following command from Jenkins:
ssh -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname "supervisorctl start company-$app ; awk -v app=$app '$0 ~ "program:company-"app {p=NR} p && NR==p+6 && /^autostart/ {$0="autostart=true" ; p=0} 1' /etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf”
This is one of the last steps of a job which creates a CloudFormation stack.
Running the command from the target server's terminal works properly.
In this step, I'd like to ssh to each one of the servers (members of ASG's within the new stack) and search and replace a specific line as shown above in the /etc/supervisord.conf, basically setting one specific service to autostart.
When I run the command I get the following error:
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
I've tried escaping the double quotes but got the same error, any idea what I'm doing wrong?

You are running in to this issue due to the way the shell handles nested quotes. This is a use case for a HERE DOCUMENT or heredoc - A HERE DOCUMENT allows you to write multi-line commands passed through bash without worrying about quotes. The structure is as follows:
$ ssh -t user#server.com <<'END'
command |\
command2 |\
END
<--- Oh yeah, the -t is important to the ssh command as it lets the shell know to behave as if being used interactively, and will avoid warnings and unexpected results.
In your specific case, you should try something like:
$ ssh -t -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname <<'END'
supervisorctl start company-$app |\
awk -v app=$app '$0 ~ \"program:company-\"app {p=NR} p && NR==p+6 \
&& /^autostart/ {$0="autostart=true" ; p=0} 1' \
/etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf
END
Just a note, since I can't be sure about your desired output of the command you are running, be advised to keep track of your own " and ' marks, and to escape them accordingly in your awk command as you would at an interactive terminal. I notice the "'s around program:company and I am confused a bit by them If they are a part of the pattern in the string being searched they will need to be escaped accordingly. P.S.

Related

Bash ! causes command to be recalled from history

I'm adding a rule to iptables however since i'm using a port exclusion with ! it seems to be egreping the command from history.
iptables -t raw -A OUTPUT -p tcp --destination-ports !80,!443
What i get is:
-bash: !80: event not found
before that it was egrepping a command earlier onto this command
iptables -t raw -A OUTPUT -p tcp -m multiport --destination-ports !443
output was
iptables -t raw -A OUTPUT -p tcp -m multiport --destination-ports egrep "47443|47080" netstat_*
iptables v1.4.21: invalid port/service `egrep' specified
Try `iptables -h' or 'iptables --help' for more information.
any ideas how i can stop it escaping my command?
The exclamation mark is part of history expansion in bash.
You need to escape it either by wrapping around with single quotes or with backslashes.
So you can escape the exclamation marks as
--destination-ports '!80,!443'
--destination-ports \!80,\!443
But I suspect whether it is a correct way of a port exclusion.
You can try instead:
iptables -t raw -A OUTPUT -p tcp ! --destination-ports 80,443
and also check it from this post
Just quote it. This will prevent bash from treating ! as the history expansion character.
$ ... --destination-ports '!80,!443'
Incidentally, if you run history at the bash prompt, you will get a listing of previous commands along with their numbers, and that will tell you what !<number> (unquoted) would be interpreted as. For example,
$ history
...
25157 echo
25158 history
$ echo !25158
echo history
history
Here !25158 is equivalent to history. For more information read the section on "History" in the bash man page.
You mean you want the exclamation escaped, I believe.
$ anycmd '!80' 'siNGLe quo** contain $crazy stuff'

perl: replace string over ssh from a bash script

I need to replace a string in a file on a remote server with this:
ssh username#${TARGETSERVER} -i /path/to/ssh-key perl -p -i -e "s#\$user = \'${SDBUSER}\'\;#\$user = \'${TDBUSER}\'\;#g" ${TARGETDIR}/configuration.php
That would replace the db user on a remote server in a joomla installation, btw.
If I execute that interactively on the remote server with all the variables filled, it IS working.
If I put the above line in a bash script, I get this message and no replacement takes place:
"Substitution pattern not terminated at -e line 1."
Any clue how I can make this work? I've already tried several escapes like \\' and so forth, but I didn't succeed.
Thanks in advance.
BINGO - Got it working:
ssh -T username#${TARGETSERVER} -i /path/to/ssh-key <<EOI
perl -p -i -e "s#\$user = \'${SDBUSER}\'\;#\$user = \'${TDBUSER}\'\;#g" ${TARGETDIR}/configuration.php
exit
EOI
From there, I could add several perl commands so I wouldn't have to have the payload of sshing in each time.

Grep across the file system has no output in a shell script

I'm trying to create a pre-commit hook in Git that will check for any debugging code and prompt the user to fix it. I have a regex that I'm grepping for (ignore the fact that it won't exclude occurrences in multiline comments!):
grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{module,inc,install,php,js} -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /path/to/code/
This works fine when I run it normally in the console, but when I try it in an executable .sh script it does nothing. None of the following has worked for me:
#!/bin/sh
grep ...
MYVAR =`grep ...` # Note the backticks!
echo $MYVAR
MYVAR =$(grep ...)
echo $MYVAR
MYVAR ="`grep ...`"
echo $MYVAR
I tried doing it with Python and os.system() but that did nothing either. It seems to just have no STDOUT. There's possibly something obvious I'm missing but I'm at a loose end.
Any help would be much appreciated! Thanks.
Edit:
This is the exact script, even though it's at the earliest possible stage due to not being able to actually do the first bit. I've hidden the exact folder names because it's probably best to not share my company's code base on SO ;)
#!/bin/bash
echo "Test!"
ONE=`grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{module,inc,install,php,js} -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /company/projects/company/www/sites/all/modules/custom/`
TWO=$(grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{coffee} -P '^\s*(?!\#)\s*(dpm\(|dsm\(|console.log)' /company/projects/company/www/sites/all/modules/custom/)
echo $ONE
echo "$TWO"
... and running bash -x pre-commit returns:
ubuntu#ip-12-34-56-78:/company/projects/company/scripts$ bash -x pre-commit
+ echo 'Test!'
Test!
++ grep -IiRn --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=public --exclude-dir=lib --exclude-dir=contrib '--include=*.module' '--include=*.inc' '--include=*.install' '--include=*.php' '--include=*.js' -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /company/projects/company/www/sites/all/modules/custom/
+ ONE='/company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: dsm('\''test'\'');
/company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: console.log(e.stack);
/company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: console.log('\''fetch success'\'');
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: console.log('\''wtf?'\'');
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: console.log(resp);
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: console.log(ui.placeholder);
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: console.log($( ".required" ));
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: console.log(resp);
/company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: console.log(data);'
++ grep -IiRn --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=public --exclude-dir=lib --exclude-dir=contrib '--include=*.{coffee}' -P '^\s*(?!\#)\s*(dpm\(|dsm\(|console.log)' /company/projects/company/www/sites/all/modules/custom/
+ TWO=
+ echo /company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: 'dsm('\''test'\'');' /company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: 'console.log(e.stack);' /company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: 'console.log('\''fetch' 'success'\'');' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: 'console.log('\''wtf?'\'');' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: 'console.log(resp);' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: 'console.log(ui.placeholder);' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: 'console.log($(' '".required"' '));' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: 'console.log(resp);' /company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: 'console.log(data);'
/company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: dsm('test'); /company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: console.log(e.stack); /company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: console.log('fetch success'); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: console.log('wtf?'); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: console.log(resp); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: console.log(ui.placeholder); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: console.log($( ".required" )); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: console.log(resp); /company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: console.log(data);
+ echo ''
... but running it without the -x flag STILL doesn't work.
Edit two:
In case anyone is wondering, my env is as follows...
ubuntu#ip-12-34-56-78:~$ uname -a
Linux ip-12-34-56-78 3.2.0-31-virtual #50-Ubuntu SMP Fri Sep 7 16:36:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
ubuntu#ip-12-34-56-78:~$ whereis sh && whereis bash
sh: /bin/sh /bin/sh.distrib /usr/share/man/man1/sh.1.gz
bash: /bin/bash /etc/bash.bashrc /usr/share/man/man1/bash.1.gz
I can't say for sure until you post the actual script you're running, but in your current code snippet have
#!/bin/sh
Depending on your OS, this may be a link to /bin/bash, for example, or it may be the actual Bourne shell, which does not support brace expansion (e.g. {a, b, c}). Even if /bin/sh does point to /bin/bash on your machine, you should only use portable constructs if your shebang is #!/bin/sh (i.e. say what you mean). If you want to use brace expansion in your script, change the shebang to #!/bin/bash.
If you put
set -x
at the top of your script, it will print detailed information that can help with debugging. You can also do this by invoking the shell directly instead of modifying your script, for example
sh -x /path/to/script
or
bash -x /path/to/script
EDIT: On Ubuntu, /bin/sh is dash, the Debian Almquist shell. Like the Bourne shell, dash is fairly restrictive, and does not support brace expansion. See this page for a discussion of portability issues and dash.

Pass a regular expression as a parameter to a shell script

How to pass a regular expression as a parameter to a shell script?
I need to write a shell script, which will take parameters and give them to unix commands. And I'd like to use regular expressions there. Is this possible at all?
Or reformulation - how to write the script equivalent to "cp" command, using only this command?
I'm trying to make a file "mycp"
#!/bin/bash -fx
cp $2 $1
and call it by
mycp myDir "*sh"
and want it to do the same as "cp *sh myDir".
But resulting bash interpretation is:
+ cp '*sh' myDir
cp: *sh: No such file or directory
Revised question
I'm trying to make a file "mycp"
#!/bin/bash -fx
cp $2 $1
and call it by
mycp myDir "*sh"
In that case, you still need eval, but you'd write:
#!/bin/bash -x
eval cp "$2" "$1"
You're running into problems because you have specified the -f option. man bash says (in part):
After word splitting, unless the -f option has been set, bash scans each word for the characters *, ?, and [.
Remove the f from the 'shebang' (first) line of the script.
Original question
Given that you want mycp "*sh*" aa to do shell expansion on the argument, you'll probably end up using eval in your script:
eval cp "$#"
However, the use of eval is dangerous; it can lead to unexpected side-effects. The use of "$#" is important; it preserves the number of arguments and spaces in them. Unfortunately, using eval then undoes that, but we can't have everything — or not easily.
For quite a long time (say 1987* to 1999), I used this script as a cover for cp:
: "#(#)$Id: cp.sh,v 1.3 1997/06/02 21:45:00 johnl Exp $"
#
# Alternative copy command
case $# in
0) /bin/cp ;;
1) /bin/cp $1 . ;;
2) /bin/cp "$#" ;;
*) if [ -d `la "$#"` ]
then /bin/cp "$#"
else /bin/cp "$#" .
fi;;
esac
It uses a very simple C program called la (for 'last argument') to get the last argument and checks whether the given last argument is a directory. More than 99% of the time, if I typed (by accident) 'cp /some/where/sh' rather than cp /some/where/*sh* ., the second was what I meant, and the script fixed things. I haven't used it for quite some time; it may have been in the last millennium, but was probably sometime earlier in this one that I gave up using it.
* Although the version string says '1997', the code is identical to the 1987 version. Version 1.1 and 1.2 were under SCCS and used different SCCS ID strings; the conversion to RCS made them identical. Version 1.3 reinstated the #(#) identifier string used by the SCCS what command to the RCS version handling. So, the script is ancient — 1987, really.
Try this: eval cp $1 $2
Bash reference manual: eval
If all you want to do is specify the directory first, look at the -t option to GNU cp:
alias mycp='cp -t'
mycp mydir *.sh
Otherwise:
mycp() {
local dir=$1
shift
cp "$#" "$dir"
}
mycp mydir *.sh
Either way, let the shell expand the wildcards and avoid the use of eval
(note, *.sh is not a regular expression, it's a shell "pattern", often referred to as a "glob pattern")

Use GDB to debug a C++ program called from a shell script

I have a extremely complicated shell script, within which it calls a C++ program I want to debug via GDB. It is extremely hard to separate this c++ program from the shell since it has a lot of branches and a lot of environmental variables setting.
Is there a way to invoke GDB on this shell script? Looks like gdb requires me to call on a C++ program directly.
In addition to options mentioned by #diverscuba23, you could do the following:
gdb --args bash <script>
(assuming it's a bash script. Else adapt accordingly)
There are two options that you can do:
Invoke GDB directly within the shell script. This would imply that you don't have standard in and standard out redirected.
Run the shell script and then attach the debugger to the already running C++ process like so: gdb progname 1234 where 1234 is the process ID of the running C++ process.
If you need to do things before the program starts running then option 1 would be the better choice, otherwise option 2 is the cleaner way.
Modify the c++ application to print its pid and sleep 30 seconds (perhaps based on environment or an argument). Attach to the running instance with gdb.
I would probably modify the script to always call gdb (and revert this later) or add an option to call gdb. This will almost always be the easiest solution.
The next easiest would be to temporarily move your executable and replace it with a shell script that runs gdb on the moved program. For example, in the directory containing your program:
$ mv program _program
$ (echo "#!/bin/sh"; echo "exec gdb $PWD/_program") > program
$ chmod +x program
Could you just temporarily add gdb to your script?
Although the answers given are valid, sometimes you don't have permissions to change the script to execute gdb or to modify the program to add additional output to attach through pid.
Luckily, there is yet another way through the power of bash
Use ps, grep and awk to pick-out the pid for you after its been executed. You can do this by either wrapping the other script with your own or by just executing a command yourself.
That command might look something like this:
process.sh
#!/usr/bin/env bash
#setup for this example
#this will execute vim (with cmdline options) as a child to bash
#we will attempt to attach to this process
vim ~/.vimrc
To get gdb to attach, we'd just need to execute the following:
gdb --pid $(ps -ef | grep -ve grep | grep vim | awk '{print $2}')
I use ps -ef here to list the processes and their arguments. Sometimes, you'll have multiple instances of a program running and need to further grep down to the one you want
the grep -ve grep is there because the f option to ps will include the next grep in its list. If you don't need the command arguments for additional filtering, don't include the -f option for ps and ignore this piece
grep vim is where we're finding our desired process. If you needed more filtering, you could just do something like grep -E "vim.*vimrc" and filter down to exactly the process that you're trying to attach to
awk '{print $2}' simply outputs just the process' pid to stdout. Use $1 if you're using ps -e instead of ps -ef
My normal setup is to run such script that starts my process in 1 tmux pane and having typed something similar to the above in a bottom pane. That way if I need to adjust the filtering (for whatever reason), I can do it pretty quickly.
Usually though, it will be the same for a specific instance and I want to just attach automatically after its been started. I'll do the following instead:
runGdb.py
#!/usr/bin/env bash
./process.sh &
PID=$(ps -ef | grep -ve grep | grep -E "vim.*vimrc" | awk '{print $2}')
#or
#PID=$(ps -e | grep vim | awk '{print $1}')
gdb --pid $PID
This assumes that the original process can be safely run in the background.