I'm adding a rule to iptables however since i'm using a port exclusion with ! it seems to be egreping the command from history.
iptables -t raw -A OUTPUT -p tcp --destination-ports !80,!443
What i get is:
-bash: !80: event not found
before that it was egrepping a command earlier onto this command
iptables -t raw -A OUTPUT -p tcp -m multiport --destination-ports !443
output was
iptables -t raw -A OUTPUT -p tcp -m multiport --destination-ports egrep "47443|47080" netstat_*
iptables v1.4.21: invalid port/service `egrep' specified
Try `iptables -h' or 'iptables --help' for more information.
any ideas how i can stop it escaping my command?
The exclamation mark is part of history expansion in bash.
You need to escape it either by wrapping around with single quotes or with backslashes.
So you can escape the exclamation marks as
--destination-ports '!80,!443'
--destination-ports \!80,\!443
But I suspect whether it is a correct way of a port exclusion.
You can try instead:
iptables -t raw -A OUTPUT -p tcp ! --destination-ports 80,443
and also check it from this post
Just quote it. This will prevent bash from treating ! as the history expansion character.
$ ... --destination-ports '!80,!443'
Incidentally, if you run history at the bash prompt, you will get a listing of previous commands along with their numbers, and that will tell you what !<number> (unquoted) would be interpreted as. For example,
$ history
...
25157 echo
25158 history
$ echo !25158
echo history
history
Here !25158 is equivalent to history. For more information read the section on "History" in the bash man page.
You mean you want the exclamation escaped, I believe.
$ anycmd '!80' 'siNGLe quo** contain $crazy stuff'
Related
I would like to run the following command from Jenkins:
ssh -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname "supervisorctl start company-$app ; awk -v app=$app '$0 ~ "program:company-"app {p=NR} p && NR==p+6 && /^autostart/ {$0="autostart=true" ; p=0} 1' /etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf”
This is one of the last steps of a job which creates a CloudFormation stack.
Running the command from the target server's terminal works properly.
In this step, I'd like to ssh to each one of the servers (members of ASG's within the new stack) and search and replace a specific line as shown above in the /etc/supervisord.conf, basically setting one specific service to autostart.
When I run the command I get the following error:
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
I've tried escaping the double quotes but got the same error, any idea what I'm doing wrong?
You are running in to this issue due to the way the shell handles nested quotes. This is a use case for a HERE DOCUMENT or heredoc - A HERE DOCUMENT allows you to write multi-line commands passed through bash without worrying about quotes. The structure is as follows:
$ ssh -t user#server.com <<'END'
command |\
command2 |\
END
<--- Oh yeah, the -t is important to the ssh command as it lets the shell know to behave as if being used interactively, and will avoid warnings and unexpected results.
In your specific case, you should try something like:
$ ssh -t -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname <<'END'
supervisorctl start company-$app |\
awk -v app=$app '$0 ~ \"program:company-\"app {p=NR} p && NR==p+6 \
&& /^autostart/ {$0="autostart=true" ; p=0} 1' \
/etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf
END
Just a note, since I can't be sure about your desired output of the command you are running, be advised to keep track of your own " and ' marks, and to escape them accordingly in your awk command as you would at an interactive terminal. I notice the "'s around program:company and I am confused a bit by them If they are a part of the pattern in the string being searched they will need to be escaped accordingly. P.S.
I'm writing a daemon that will log in to other machines to confirm that a service is running and also start, stop, or kill it. Because of this, the unix commands get a little long and obfuscated.
The basic shape of the commands that are forming are like:
bash -c 'ssh -p 22 user#host.domain.com pgrep -fl "APP.*APP_id=12345"'
Where APP is the name of the remote executable and APP_id is a parameter passed to the application when started.
The executable running on the remote side will be started with something like:
/path/to/APP configs/config.xml -v APP_id=12345 APP_port=2345 APP_priority=7
The exit status of this command is used to determine if the remote service is running or was successfully started or killed.
The problem I'm having is that when testing on my local machine, ssh connects to the local machine to make things easier, but pgrep called this way will also identify the ssh command that the server is running to do the check.
For example, pgrep may return:
26308 ./APP configs/config.xml APP_id=128bb8da-9a0b-474b-a0de-528c9edfc0a5 APP_nodeType=all APP_exportPort=6500 APP_clientPriority=11
27915 ssh -p 22 user#localhost pgrep -fl APP.*APP_id=128bb8da-9a0b-474b-a0de-528c9edfc0a5
So the logical next step was to change the pgrep pattern to exclude 'ssh', but this seems impossible because pgrep does not seem to be compiled with a PCRE version that allows lookaheads, for example:
bash -c -'ssh -p 22 user#localhost preg -fl "\(?!ssh\).*APP.*APP_id=12345"
This will throw a regex error, so as a workaround I was using grep:
bash -c 'ssh -p 22 user#host.domain.com pgrep -fl "APP.*APP_id=12345" \\| grep -v ssh'
This works well for querying with pgrep even though it's a workaround. However, the next step using pkill doesn't work because there's no opportunity for grep to be effective:
bash -c 'ssh -p 22 user#host.domain.com pkill -f "APP.*APP_id=12345"'
Doesn't work well because pkill also kills the ssh connection which causes the exit status to be bad. So, I'm back to modifying my pgrep/pkill pattern and not having much luck.
This environment can be simulated with something simple on a local machine that can ssh to itself without a password (in this case, APP would be 'watch'):
watch echo APP_id=12345
Here is the question simply put: How do I match 'APP' but not 'ssh user#host APP' in pgrep?
It's kind of a workaround, but does the job:
bash -c 'ssh -p 22 user#host.domain.com pgrep -fl "^[^\s]*APP.*APP_id=12345"'
...which only matches commands that have no space before the application name. This isn't entirely complete, because it's possible that the path to the executable may contain a directory with spaces, but without lookaround syntax I haven't thought of another way to make this work.
really old q but!
export VAR="agent.py"; pkill -f .*my$VAR;
I need to replace a string in a file on a remote server with this:
ssh username#${TARGETSERVER} -i /path/to/ssh-key perl -p -i -e "s#\$user = \'${SDBUSER}\'\;#\$user = \'${TDBUSER}\'\;#g" ${TARGETDIR}/configuration.php
That would replace the db user on a remote server in a joomla installation, btw.
If I execute that interactively on the remote server with all the variables filled, it IS working.
If I put the above line in a bash script, I get this message and no replacement takes place:
"Substitution pattern not terminated at -e line 1."
Any clue how I can make this work? I've already tried several escapes like \\' and so forth, but I didn't succeed.
Thanks in advance.
BINGO - Got it working:
ssh -T username#${TARGETSERVER} -i /path/to/ssh-key <<EOI
perl -p -i -e "s#\$user = \'${SDBUSER}\'\;#\$user = \'${TDBUSER}\'\;#g" ${TARGETDIR}/configuration.php
exit
EOI
From there, I could add several perl commands so I wouldn't have to have the payload of sshing in each time.
I have a file that I want to grep out lines with strings like "Nov 30" or "Nov 30" (basically I don't want to specify the number of spaces in the middle.
In the terminal this would be fine, I'd just do:
grep 'Nov * 30' file
However, it'd be nice to keep it general for my purposes, so in fact I'd like to do something like:
grep "$(date +%b) * $(date +%d)" file
That works fine BUT I actually want to do this through ssh so it'd be like
ssh -t host "grep "$(date +%b) * $(date +%d)" file"
At this point I run into problems. Instead of grepping only for Nov 30 type date strings, it returns all sorts of different November dates. I feel like the problem has something to do with double quotation usage (perhaps the first set of double quotes for the -t argument is affecting the second lot, but I don't see how to get around that) , and I see from this answer that "bash will evaluate and substitute variables inside double-quotes on the local machine but will do it on the target machine inside single quotes". So I tried replacing the above with
ssh -t host "grep '$(date +%b) * $(date +%d)' file"
But now the grep returns no results at all! I assume this is because I'm grepping for literal '$(date +%b)...' and not the substituted 'Nov..', but then I don't understand why the first attempt with double quotes didn't work.
Welcome any help
Escape your quotes:
ssh -t host "grep \"$(date +%b) * $(date +%d)\" file"
Alternately, single-quote the command line you wish to execute on the remote machine. (In this case the date commands will execute on the remote end.)
ssh -t host 'grep "$(date +%b) * $(date +%d)" file'
In this version the date command will be executed locally:
ssh -t host "grep '$(date +%b) * $(date +%d)' file"
In this version the date command will be executed on the remote host:
ssh -t host 'grep "$(date +%b) * $(date +%d)" file'
This can make a difference when your local PC and server are in different time zone. Right now for example, it's Dec 1 in France, but Nov 30 on my server in the US.
In the 1st version the $() within the double quotes are evaluated before sending to the server. So the command sent to the server is grep 'Dec * 1' file in my timezone.
In the 2nd version the $() within single quotes are NOT evaluated before sending to the server. So the command sent to the server is grep "$(date +%b) * $(date +%d)" file, and so the server will evaluate the $() within double quotes.
I'm trying to create a pre-commit hook in Git that will check for any debugging code and prompt the user to fix it. I have a regex that I'm grepping for (ignore the fact that it won't exclude occurrences in multiline comments!):
grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{module,inc,install,php,js} -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /path/to/code/
This works fine when I run it normally in the console, but when I try it in an executable .sh script it does nothing. None of the following has worked for me:
#!/bin/sh
grep ...
MYVAR =`grep ...` # Note the backticks!
echo $MYVAR
MYVAR =$(grep ...)
echo $MYVAR
MYVAR ="`grep ...`"
echo $MYVAR
I tried doing it with Python and os.system() but that did nothing either. It seems to just have no STDOUT. There's possibly something obvious I'm missing but I'm at a loose end.
Any help would be much appreciated! Thanks.
Edit:
This is the exact script, even though it's at the earliest possible stage due to not being able to actually do the first bit. I've hidden the exact folder names because it's probably best to not share my company's code base on SO ;)
#!/bin/bash
echo "Test!"
ONE=`grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{module,inc,install,php,js} -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /company/projects/company/www/sites/all/modules/custom/`
TWO=$(grep -IiRn --exclude-dir={node_modules,vendor,public,lib,contrib} --include=\*.{coffee} -P '^\s*(?!\#)\s*(dpm\(|dsm\(|console.log)' /company/projects/company/www/sites/all/modules/custom/)
echo $ONE
echo "$TWO"
... and running bash -x pre-commit returns:
ubuntu#ip-12-34-56-78:/company/projects/company/scripts$ bash -x pre-commit
+ echo 'Test!'
Test!
++ grep -IiRn --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=public --exclude-dir=lib --exclude-dir=contrib '--include=*.module' '--include=*.inc' '--include=*.install' '--include=*.php' '--include=*.js' -P '^\s*(?!\/\/)\s*(dpm\(|dsm\(|console.log\()' /company/projects/company/www/sites/all/modules/custom/
+ ONE='/company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: dsm('\''test'\'');
/company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: console.log(e.stack);
/company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: console.log('\''fetch success'\'');
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: console.log('\''wtf?'\'');
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: console.log(resp);
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: console.log(ui.placeholder);
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: console.log($( ".required" ));
/company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: console.log(resp);
/company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: console.log(data);'
++ grep -IiRn --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=public --exclude-dir=lib --exclude-dir=contrib '--include=*.{coffee}' -P '^\s*(?!\#)\s*(dpm\(|dsm\(|console.log)' /company/projects/company/www/sites/all/modules/custom/
+ TWO=
+ echo /company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: 'dsm('\''test'\'');' /company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: 'console.log(e.stack);' /company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: 'console.log('\''fetch' 'success'\'');' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: 'console.log('\''wtf?'\'');' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: 'console.log(resp);' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: 'console.log(ui.placeholder);' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: 'console.log($(' '".required"' '));' /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: 'console.log(resp);' /company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: 'console.log(data);'
/company/projects/company/www/sites/all/modules/custom/some_module/some_module.report.inc:594: dsm('test'); /company/projects/company/www/sites/all/modules/custom/goals_app/goals_app.module:170: console.log(e.stack); /company/projects/company/www/sites/all/modules/custom/company_usage_reports/js/script.js:300: console.log('fetch success'); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_change_workgroup.js:19: console.log('wtf?'); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:33: console.log(resp); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder_table.js:39: console.log(ui.placeholder); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_goal_form.js:4: console.log($( ".required" )); /company/projects/company/www/sites/all/modules/custom/another_module/js/another_module_reorder.js:40: console.log(resp); /company/projects/company/www/sites/all/modules/custom/company_goals/js/views/goal-list.js:87: console.log(data);
+ echo ''
... but running it without the -x flag STILL doesn't work.
Edit two:
In case anyone is wondering, my env is as follows...
ubuntu#ip-12-34-56-78:~$ uname -a
Linux ip-12-34-56-78 3.2.0-31-virtual #50-Ubuntu SMP Fri Sep 7 16:36:36 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
ubuntu#ip-12-34-56-78:~$ whereis sh && whereis bash
sh: /bin/sh /bin/sh.distrib /usr/share/man/man1/sh.1.gz
bash: /bin/bash /etc/bash.bashrc /usr/share/man/man1/bash.1.gz
I can't say for sure until you post the actual script you're running, but in your current code snippet have
#!/bin/sh
Depending on your OS, this may be a link to /bin/bash, for example, or it may be the actual Bourne shell, which does not support brace expansion (e.g. {a, b, c}). Even if /bin/sh does point to /bin/bash on your machine, you should only use portable constructs if your shebang is #!/bin/sh (i.e. say what you mean). If you want to use brace expansion in your script, change the shebang to #!/bin/bash.
If you put
set -x
at the top of your script, it will print detailed information that can help with debugging. You can also do this by invoking the shell directly instead of modifying your script, for example
sh -x /path/to/script
or
bash -x /path/to/script
EDIT: On Ubuntu, /bin/sh is dash, the Debian Almquist shell. Like the Bourne shell, dash is fairly restrictive, and does not support brace expansion. See this page for a discussion of portability issues and dash.