I want to use ncap2 to make a large number of variables a function of time in a netcdf file. I therefore wanted to build up the command argument in a loop and store in a bash variable command and then apply it as a variable, which I often do with cdo like this
cdo $command in.nc out.nc
I build up my command like this:
varlist="hfls hfss prlr prw rlds rlus rlut rsds rsdt rsus rsut clt evspsbl"
command=""
for var in $varlist ; do
command+=" -s '${var}[time,lat,lon]=${var}'"
done
echo $command
when I then attempt to do this :
ncap2 $command in.nc out.nc
I get a segmentation fault and core dump. But if I cut and paste the result of the echo command into the ncap2 statement,
ncap2 -s 'hfls[time,lat,lon]=hfls' -s 'hfss[time,lat,lon]=hfss' -s 'prlr[time,lat,lon]=prlr' -s 'prw[time,lat,lon]=prw' -s 'rlds[time,lat,lon]=rlds' -s 'rlus[time,lat,lon]=rlus' -s 'rlut[time,lat,lon]=rlut' -s 'rsds[time,lat,lon]=rsds' -s 'rsdt[time,lat,lon]=rsdt' -s 'rsus[time,lat,lon]=rsus' -s 'rsut[time,lat,lon]=rsut' -s 'clt[time,lat,lon]=clt' -s 'evspsbl[time,lat,lon]=evspsbl' $dir/$file /scratch/b/$USER/${file%???}_corrected.nc
It works fine... I'm sure it is something to do with the way I'm handling the strings, but my experiments trying to change around quotes etc, didn't seem to work.
Yes, it seems to be a shell-quoting issue.
Rather than try to create a simpler analogue for testing, let me start by just asking, does it work if you use this instead?
eval ncap2 $command in.nc out.nc
Related
I often find Bash syntax very helpful, e.g. process substitution like in diff <(sort file1) <(sort file2).
Is it possible to use such Bash commands in a Makefile? I'm thinking of something like this:
file-differences:
diff <(sort file1) <(sort file2) > $#
In my GNU Make 3.80 this will give an error since it uses the shell instead of bash to execute the commands.
From the GNU Make documentation,
5.3.2 Choosing the Shell
------------------------
The program used as the shell is taken from the variable `SHELL'. If
this variable is not set in your makefile, the program `/bin/sh' is
used as the shell.
So put SHELL := /bin/bash at the top of your makefile, and you should be good to go.
BTW: You can also do this for one target, at least for GNU Make. Each target can have its own variable assignments, like this:
all: a b
a:
#echo "a is $$0"
b: SHELL:=/bin/bash # HERE: this is setting the shell for b only
b:
#echo "b is $$0"
That'll print:
a is /bin/sh
b is /bin/bash
See "Target-specific Variable Values" in the documentation for more details. That line can go anywhere in the Makefile, it doesn't have to be immediately before the target.
You can call bash directly, use the -c flag:
bash -c "diff <(sort file1) <(sort file2) > $#"
Of course, you may not be able to redirect to the variable $#, but when I tried to do this, I got -bash: $#: ambiguous redirect as an error message, so you may want to look into that before you get too into this (though I'm using bash 3.2.something, so maybe yours works differently).
One way that also works is putting it this way in the first line of the your target:
your-target: $(eval SHELL:=/bin/bash)
#echo "here shell is $$0"
If portability is important you may not want to depend on a specific shell in your Makefile. Not all environments have bash available.
You can call bash directly within your Makefile instead of using the default shell:
bash -c "ls -al"
instead of:
ls -al
There is a way to do this without explicitly setting your SHELL variable to point to bash. This can be useful if you have many makefiles since SHELL isn't inherited by subsequent makefiles or taken from the environment. You also need to be sure that anyone who compiles your code configures their system this way.
If you run sudo dpkg-reconfigure dash and answer 'no' to the prompt, your system will not use dash as the default shell. It will then point to bash (at least in Ubuntu). Note that using dash as your system shell is a bit more efficient though.
It's not a direct answer to the question, makeit is limited Makefile replacement with bash syntax and it can be useful in some cases (I'm the author)
rules can be defined as bash-functions
auto-completion feature
Basic idea is to have while loop in the end of the script:
while [ $# != 0 ]; do
if [ "$(type -t $1)" == 'function' ]; then
$1
else
exit 1
fi
shift
done
https://asciinema.org/a/435159
I intend to automate compile and run process in C++, I wrote the following code as compile-run.sh
#! /bin/bash
clang++ $1.cpp -o $1.out && ./$1.out
I put this compile-run.sh in /usr/local/bin for global usage,
and when I type the command compile-run.sh XXX.cpp, it intend to compile and run the specified cpp file. But the problem now is I have to manually removed ".cpp" in the command.
Is there any way to trim the last X number of character and assign to a variable in general?
Is there any way to trim the .cpp and apply trimmed $1 in the code?
Is there better way to automate compile and run process?
well, an ugly way could be by using something like:
#! /bin/bash
filename=$1
temp="${filename%%.cpp}"
clang++ $temp.cpp -o $temp.out && ./$temp.out
another way, if you want to trim the last 4 characters whatever the last part is:
#! /bin/bash
filename=$1
temp="${filename::-4}"
clang++ $temp.cpp -o $temp.out && ./$temp.out
but for substrings you could also use cut: ie. https://stackabuse.com/substrings-in-bash/
I would like to run the following command from Jenkins:
ssh -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname "supervisorctl start company-$app ; awk -v app=$app '$0 ~ "program:company-"app {p=NR} p && NR==p+6 && /^autostart/ {$0="autostart=true" ; p=0} 1' /etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf”
This is one of the last steps of a job which creates a CloudFormation stack.
Running the command from the target server's terminal works properly.
In this step, I'd like to ssh to each one of the servers (members of ASG's within the new stack) and search and replace a specific line as shown above in the /etc/supervisord.conf, basically setting one specific service to autostart.
When I run the command I get the following error:
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
I've tried escaping the double quotes but got the same error, any idea what I'm doing wrong?
You are running in to this issue due to the way the shell handles nested quotes. This is a use case for a HERE DOCUMENT or heredoc - A HERE DOCUMENT allows you to write multi-line commands passed through bash without worrying about quotes. The structure is as follows:
$ ssh -t user#server.com <<'END'
command |\
command2 |\
END
<--- Oh yeah, the -t is important to the ssh command as it lets the shell know to behave as if being used interactively, and will avoid warnings and unexpected results.
In your specific case, you should try something like:
$ ssh -t -i ~/.ssh/company.pem -o StrictHostKeyChecking=no user#$hostname <<'END'
supervisorctl start company-$app |\
awk -v app=$app '$0 ~ \"program:company-\"app {p=NR} p && NR==p+6 \
&& /^autostart/ {$0="autostart=true" ; p=0} 1' \
/etc/supervisord.conf > $$.tmp && sudo mv $$.tmp /etc/supervisord.conf
END
Just a note, since I can't be sure about your desired output of the command you are running, be advised to keep track of your own " and ' marks, and to escape them accordingly in your awk command as you would at an interactive terminal. I notice the "'s around program:company and I am confused a bit by them If they are a part of the pattern in the string being searched they will need to be escaped accordingly. P.S.
I can split one large mp3 file into several files based on silence using the mp3split command / program below
mp3splt -f -t 4.0 -a -d split audio_file.mp3
and I get
split/audio_file_000m_00s_005m_00s.mp3
but how can I get
split/000m_00s_005m_00s_audio_file.mp3
or increment by one in the front
split/000_audio_file_000m_00s_005m_00s.mp3
split/001_audio_file_005m_00s_010m_00s.mp3
I looked at the syntax http://wiki.librivox.org/index.php/How_To_Split_With_Mp3Splt but couldn't figure out what needs to change in my syntax.
I'm using ubuntu 16.04 64bit linux
You need to set the -o (output format) option.
Try something like:
mp3splt -o #N3_#f -f -t 4.0 -a -d split audio_file.mp3
Giving you:
001_audio_file.mp3,
002_audio_file.mp3,
003_audio_file.mp3…
The man page is a little messy, but it's all there.
I used
mp3splt -o #N3_#mm_#ss_#f -f -t 4.0 -a -d split audio_file.mp3
which gives me
/split/001_000m_00s_audio_file.mp3
/split/002_004m_00s_audio_file.mp3
When I directly run a command in my Linux terminal, say "ls", the output is with color. However, when I run a C++ program which calls system("ls"), the output does not have color.
Is there way to get the latter way to also display colored output?
Thanks!
The answer for why there's no color lies here.
system() executes a command specified in command by calling /bin/sh -c
command, and returns after the command has been completed.
sh -c ignores aliases. Perhaps somewhere you have an alias where ls means ls --color=auto.
So for example, if I do sh -c 'ls', I will get no color.
Proof:
wow ♪[01:04 AM][vnbraun#chernobyl ~]$ which ls
alias ls='ls --color=auto'
/bin/ls
wow ♪[01:08 AM][vnbraun#chernobyl ~]$ sh -c 'which ls'
/bin/ls
Therefore, you can try doing system("ls --color=auto");.
You could run
system("/bin/ls --color=auto");
But I don't think you really should run ls from your C++ program. Perhaps you want to use -some combination of- readdir(3), stat(2), nftw(3), glob(3), wordexp(3) etc etc....
I don't think that forking a shell which then runs /bin/ls is useful from a C++ program. There are simpler ways to achieve your goal (which I cannot guess).
You probably should read Advanced Linux Programming
Try invoking ls --color=auto or ls --color=always to display ls with colors.
This is likely due to a bash configuration file somewhere in your system aliasing "ls" to "ls --color".
Using "ls --color" in your program should work.