Let's say I have a C program, and I run it from bash:
$ ./a.out 123 *
The program would output all the command line arguments,
but it will show these instead:
Argument 1: 123
Argument 2: a.out
What can I do in my program to fix this?
The shell is replacing the asterisk with the name of each file in the directory.
To pass a literal asterisk, you should be able to escape it:
$ ./a.out 123 \*
Another option is to use set -f to turn off expansion. Compare:
echo *
v.s.
set -f
echo *
You can turn it back on with set +f:
set -f
echo *
set +f
echo *
You can quote it in the shell
./a.out 123 '*'
There is nothing you can do in your program, because the * expansion is done by the shell (in contrast to Windows, where it's done by the program).
Another alternative is to start your script with #!/bin/bash -f as the first line, which will allow you to accept literal strings as arguments (including the asterisk), and thus that will allow you to run ./a.out 123 * with the desired input, but note that bash -f disables expansions completely, and that can have adverse effects in your script depending on what you do.
This doesn't have anything to do with your program.
The * is a wildcard in Bash, it means "all files in the current directory". If you want to pass an asterisk as an argument to your program, you do it the same way you do it with every other program: you escape it with a backslash or quote it.
Related
I am having trouble understanding what this command does and more importantly the significance of both the $s in the command. I started by making a file foo and filled it with words like SHELL, WELL, etc. I then tried the command, however it did not print anything, so I thought that maybe something was wrong with my terminal. See my terminal below:
ifsmy12#cloudshell:~/3300/tests/t2 (cs3300-301722)$ cat foo
SHELL
HELL
WELL
SHELL
SWELL
DWELL
FAREWELL
ifsmy12#cloudshell:~/3300/tests/t2 (cs3300-301722)$ grep "$SHELL$" foo
ifsmy12#cloudshell:~/3300/tests/t2 (cs3300-301722)$
I then decided to play around with the command by first trying it without the first $ and then doing the same thing except without the second $. My results are below:
ifsmy12#cloudshell:~/3300/tests/t2 (cs3300-301722)$ grep "$SHELL" foo
ifsmy12#cloudshell:~/3300/tests/t2 (cs3300-301722)$ grep "SHELL$" foo
SHELL
SHELL
I got some results but I am still confused as to what each $ does and why the command does not work with both in place. Can someone explain?
$SHELL expands to the path of your currently executing shell. $ without text after it does not expand and stays unaltered.
If executed in the Bash shell, your final command would look like:
grep '/bin/bash$' foo
$ as part of a regular-expression – which is what a grep pattern is – means "end of line".
You can verify it by prepending your command with echo, or enabling command tracing in your shell: set -x
All that said, using $ unescaped and expecting it to not expand is bad practice (in other words: waiting for surprising stuff to happen). Better to be explicit where you want shell expansion to be performed and where not:
grep "$SHELL"'$' foo
grep "$SHELL\$" foo
This seems like a relatively basic question, but I can't find it anywhere after an hour of searching. Many (there are a lot!) of the similar questions do not seem to hit the point.
I am writing a script ("vims") to use vim in a sed-like mode (so I can call normal vim commands on a stream input without actually opening vim), so I need to pass each argument to vim with a "-c" flag prepended to it. There are also many characters that need to be escaped (I need to pass regex expressions), so some of the usual methods on SO do not work.
Basically, when I write:
cat myfile.txt | vims ':%g/foo/exe "norm yyPImyfile: \<esc>\$dF,"' ':3p'
which are two command-line vim arguments to run on stdout,
I need these two single-quoted arguments to be passed exactly the way they are to my function vims(), which then tags each of them with a -c flag, so they are interpreted as commands in vim.
Here's what I've tried so far:
vims() {
vim - -nes -u NONE -c '$1' -c ':q!' | tail -n +2
}
This seems to work perfectly for a single command. No characters get escaped, and the "-c" flag is there.
Then, using the oft-duplicated question-answer, the "$#" trick, I tried:
vims() {
vim - -nes -u NONE $(for arg in "$#"; do echo -n " -c $arg "; done) -c ':q!' | tail -n +2
}
This seems to break the spaces within each string I pass it, so does not work. I also tried a few variations of the printf command, as suggested in other questions, but this has weird interactions with the vim command sequences. I've tried many other different backslash-quote-combinations in a perpetual edit-test loop, but have always found a quirk in my method.
What is the command sequence I am missing?
Add all the arguments to an array one at a time, then pass the entire array to vim with proper quoting to ensure whitespace is correctly preserved.
vims() {
local args=()
while (($# > 0)); do
args+=(-c "$1")
shift
done
vim - -nes -u NONE "${args[#]}" -c ':q!' | tail -n +2
}
As a rule of thumb, if you find yourself trying to escape things, add backslashes, use printf, etc., you are likely going down the wrong path. Careful use of quoting and arrays will cover most scenarios.
I think I have uncovered an error in grep. If I run this grep statement against a db log on the command line it runs fine.
grep "Query Executed in [[:digit:]]\{5\}.\?" db.log
I get this result:
Query Executed in 19699.188 ms;"select distinct * from /xyztable.....
when I run it in a script
LONG_QUERY=`grep "Query Executed in [[:digit:]]\{5\}.\?" db.log`
the asterisk in the result is replaced with a list of all files in the current directory.
echo $LONG_QUERY
Result:
Query Executed in 19699.188 ms; "select distinct <list of files in
current directory> from /xyztable.....
Has anyone seen this behavior?
This is not an error in grep. This is an error in your understanding of how scripts are interpreted.
If I write in a script:
echo *
I will get a list of filenames, because an unquoted, unescaped, asterisk is interpreted by the shell (not grep, but /bin/bash or /bin/sh or whatever shell you use) as a request to substitute filenames matching the pattern '*', which is to say all of them.
If I write in a script:
echo "*"
I will get a single '*', because it was in a quoted string.
If I write:
STAR="*"
echo $STAR
I will get filenames again, because I quoted the star while assigning it to a variable, but then when I substituted the variable into the command it became unquoted.
If I write:
STAR="*"
echo "$STAR"
I will get a single star, because double-quotes allow variable interpolation.
You are using backquotes - that is, ` characters - around a command. That captures the output of the command into a variable.
I would suggest that if you are going to be echoing the results of the command, and little else, you should just redirect the results into a file. (After all, what are you going to do when your LONG_QUERY contains 10,000 lines of output because your log file got really full?)
Barring that, at the very least do echo "$LONG_QUERY" (in double quotes).
The following command pipes the output of echo to findstr and tries to match a regular expression with it. I use it to check if the echoed line only consists of (one or more) digits:
echo 123 | findstr /r /c:"^[0-9][0-9]*$"
The expected output of findstr is 123, which means that the expression could be matched with this string. The output is correct when I execute the command with powershell.exe.
Executing the command in cmd.exe however does not give a match. It only outputs an empty line and sets %ERRORLEVEL% to 1, which means that no match was found.
What causes the different behavior? Is there a way to make this command run correctly on cmd as well?
My OS is Windows 7 Professional, 64 Bit.
In Powershell the command echoes the string 123 to the pipeline and that matches your regular expression.
In cmd, your command echos 123<space> to the pipeline. The trailing space isn't allowed for in your regular expression so you don't get a match.
Try:
echo 123| findstr /r /c:"^[0-9][0-9]*$"
and it will work just fine. Or just switch entirely to Powershell and stop having to worry about the vagaries of cmd.exe.
Edit:
Yes, cmd and powershell handle parameters very differently.
With cmd all programs are passed a simple text command line. The processing that cmd performs is pretty minimal: it will terminate the command at | or &, removes i/o redirection and will substitute in any variables. Also of course it identifies the command and executes it. Any argument processing is done by the command itself, so a command can choose whether spaces separate arguments or what " characters mean. Mostly commands have a fairly common interpretation of these things but they can just do their own thing with the string they were given. echo does it's own thing.
Powershell on the other hand has a complex syntax for arguments. All of the argument parsing is done by Powershell. The parsed arguments are then passed to Powershell functions or cmdlets as a sequence of .Net objects: that means you aren't limited to just passing simple strings around. If the command turns out not to be a powershell command and runs externally it will attempt to convert the objects into a string and puts quotes round any arguments that have a space. Sometimes the conversion can be a bit confusing, but it does mean that something like this:
echo (1+1)
will echo 2 in Powershell where cmd would just echo the input string.
It is worth always remembering that with Powershell you are working with objects, so for example:
PS C:\> echo Today is (get-date)
Today
is
17 April 2014 20:03:15
PS C:\> echo "Today is $(get-date)"
Today is 04/17/2014 20:03:20
In the first case echo gets 3 objects, two strings and a date. It outputs each object on a separate line (and a blank line when the type changes). In the second case it gets a single object which is a string (and unlike the cmd echo it never sees the quote marks).
I'm writing a program, foo, in C++. It's typically invoked on the command line like this:
foo *.txt
My main() receives the arguments in the normal way. On many systems, argv[1] is literally *.txt, and I have to call system routines to do the wildcard expansion. On Unix systems, however, the shell expands the wildcard before invoking my program, and all of the matching filenames will be in argv.
Suppose I wanted to add a switch to foo that causes it to recurse into subdirectories.
foo -a *.txt
would process all text files in the current directory and all of its subdirectories.
I don't see how this is done, since, by the time my program gets a chance to see the -a, then shell has already done the expansion and the user's *.txt input is lost. Yet there are common Unix programs that work this way. How do they do it?
In Unix land, how can I control the wildcard expansion?
(Recursing through subdirectories is just one example. Ideally, I'm trying to understand the general solution to controlling the wildcard expansion.)
You program has no influence over the shell's command line expansion. Which program will be called is determined after all the expansion is done, so it's already too late to change anything about the expansion programmatically.
The user calling your program, on the other hand, has the possibility to create whatever command line he likes. Shells allow you to easily prevent wildcard expansion, usually by putting the argument in single quotes:
program -a '*.txt'
If your program is called like that it will receive two parameters -a and *.txt.
On Unix, you should just leave it to the user to manually prevent wildcard expansion if it is not desired.
As the other answers said, the shell does the wildcard expansion - and you stop it from doing so by enclosing arguments in quotes.
Note that options -R and -r are usually used to indicate recursive - see cp, ls, etc for examples.
Assuming you organize things appropriately so that wildcards are passed to your program as wildcards and you want to do recursion, then POSIX provides routines to help:
nftw - file tree walk (recursive access).
fnmatch, glob, wordexp - to do filename matching and expansion
There is also ftw, which is very similar to nftw but it is marked 'obsolescent' so new code should not use it.
Adrian asked:
But I can say ls -R *.txt without single quotes and get a recursive listing. How does that work?
To adapt the question to a convenient location on my computer, let's review:
$ ls -F | grep '^m'
makefile
mapmain.pl
minimac.group
minimac.passwd
minimac_13.terminal
mkmax.sql.bz2
mte/
$ ls -R1 m*
makefile
mapmain.pl
minimac.group
minimac.passwd
minimac_13.terminal
mkmax.sql.bz2
mte:
multithread.ec
multithread.ec.original
multithread2.ec
$
So, I have a sub-directory 'mte' that contains three files. And I have six files with names that start 'm'.
When I type 'ls -R1 m*', the shell notes the metacharacter '*' and uses its equivalent of glob() or wordexp() to expand that into the list of names:
makefile
mapmain.pl
minimac.group
minimac.passwd
minimac_13.terminal
mkmax.sql.bz2
mte
Then the shell arranges to run '/bin/ls' with 9 arguments (program name, option -R1, plus 7 file names and terminating null pointer).
The ls command notes the options (recursive and single-column output), and gets to work.
The first 6 names (as it happens) are simple files, so there is nothing recursive to do.
The last name is a directory, so ls prints its name and its contents, invoking its equivalent of nftw() to do the job.
At this point, it is done.
This uncontrived example doesn't show what happens when there are multiple directories, and so the description above over-simplifies the processing.
Specifically, ls processes the non-directory names first, and then processes the directory names in alphabetic order (by default), and does a depth-first scan of each directory.
foo -a '*.txt'
Part of the shell's job (on Unix) is to expand command line wildcard arguments. You prevent this with quotes.
Also, on Unix systems, the "find" command does what you want:
find . -name '*.txt'
will list all files recursively from the current directory down.
Thus, you could do
foo `find . -name '*.txt'`
I wanted to point out another way to turn off wildcard expansion. You can tell your shell to stop expanding wildcards with the the noglob option.
With bash use set -o noglob:
> touch a b c
> echo *
a b c
> set -o noglob
> echo *
*
And with csh, use set noglob:
> echo *
a b c
> set noglob
> echo *
*