I am trying to write a bash script in which the user can pass a regex as a parameter. So for example you run the program this way.
./clean_txt.sh s/.*\(my_choices.*gz\)/\1p
In the program, I am using that parameter this way.
ls /home/user/ | sed -n '$1' > cleaned_file.txt
echo "sed -n '$1'"
In my echo, I see the regular expression passed when when program was initiated. But my cleaned_file.txt is empty. The only way this works is if I hardcode the a regular expression into the program itself but that defeats the purpose of what I am trying to do.
Any idea on how I can pass that parameter into the sed command?
The problem is that your variable is not being expanded. You need to wrap it in double quotes (which is what you're doing in the echo already):
ls /home/user/ | sed -n "$1" > cleaned_file.txt
Note that ls is not needed:
files=( /home/user/* )
sed -n "$1" <<<"${files[#]}" > cleaned_file.txt
Would do the same thing. This uses a glob to create an array containing all the filenames, which is used as input to sed.
Related
I have a configuration file with placeholders like this (stored in /tmp/var for this example)
ldap_bind_dn='${bind_dn}'
Now I'd like replace ${bind_dn} with the the environment variable of the same name (which is set inside Docker).
export $bind_dn=CN=my-user,CN=Users,DC=example,DC=com
The expected result after processing aboves test file would be
ldap_bind_dn='CN=my-user,CN=Users,DC=example,DC=com'
I tried sed but it doesn't replace it to the value of the env variable
$ sed "s#\${\(.*\)}#$\1#" /tmp/var
bind_dn='$bind_dn'
Why sed replace with $bind_dn instead of the value? I'd expect that the variable is processed because I haven't escaped the $ sign.
The expression itself works, only the substitution doesn't:
$ sed "s#\${\(.*\)}#test123#" /tmp/var
bind_dn='test123'
The replacement is also done correctly when the target variable is hardcoded
$ sed "s#\${\(.*\)}#$bind_dn#" /tmp/var
bind_dn='CN=my-user,CN=Users,DC=example,DC=com'
But since we have a bunch of configuration variables, I'd like to automatically replace all env variables in the format ${NAME} automatically.
Not the most elegant solution, but this one works in bash:
sed 's#\${\(.*\)}#`{echo,"$\1"}`#' /tmp/var | xargs -n1 -I{} echo echo "{}" | bash -s
It is a little bit tricky because you need bash execution for the variable replacement, that's why I'm piping it to bash -s
While the env variable could be parsed by executing eval with mapfile, this seems not suiteable for me because its a ini configuration file. The sections marked with brackets like [general] would throw errors. And I also have concerns to just execute the WHOLE line, which allows executing any command.
This is fixed by the following awk:
awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < /tmp/var > /tmp/var
Why sed replace with $bind_dn instead of the value?
Because sed is not supposed to do shell parameter expansion in its pattern space. That's the job of the shell mainly.
Using GNU sed:
~> cat /tmp/var
ldap_bind_dn='${bind_dn}'
~> export bind_dn=CN=my-user,CN=Users,DC=example,DC=com
~> sed -E 's/^\w+=.*/echo "&"/e' /tmp/var
ldap_bind_dn='CN=my-user,CN=Users,DC=example,DC=com'
The e command (a GNU extension) in
sed -E 's/^\w+=.*/echo "&"/e'
executes the command that is found in pattern space and replaces the pattern space with the output. For this example, pattern space is ldap_bind_dn='${bind_dn}', and is replaced with the output of echo "ldap_bind_dn='${bind_dn}'" (& references the whole matched portion of the pattern space in echo "&"). Since the argument of echo is in double quotes, it is subject to parameter expansion when it is executed by the shell.
Caveat: Make sure that the file (/tmp/var in this example) comes from a trusted source. Otherwise it may contain lines like foo='$(some_nasty_command)', which is executed when the sed command above runs.
I need to insert a command (as string) to an existing file after a certain match. The existing string is a long make command and I only need to modify it by inserting another string at specific location. I tried using sed but it either adds a new line before/after the matching string or replaces it. I'd like to know if at least it is possible to accomplish what I want with sed or should I be using something else? Could you please provide me with some hints?
Example:
The file contains two make commands and I am only interested in the second one without bbnote.
oe_runmake_call() {
bbnote make -j 8 CROSS_COMPILE=arm-poky-linux-gnueabi- CC="arm-poky-linux-gnueabi-gcc" "$#"
make -j 8 CROSS_COMPILE=arm-poky-linux-gnueabi- CC="my_command_here arm-poky-linux-gnueabi-gcc" --sysroot=/some/path "$#"
}
Thanks in advance!
Here's the code:
http://hastebin.com/tigatoquje.go
You could do something like this using Sed:
sed -r 's:(^\s+make.+ CC=\"):\1your_command_here :g' file.log >outfile.log
or with sed in-place edit:
sed -ir 's:(^\s+make.+ CC=\"):\1your_command_here :g' file.log
Without sed regex option:
sed 's:\(^\s\+make.\+ CC=\"\):\1your_command_here :g' file.log > outfile.log
Outputs:
oe_runmake_call() {
bbnote make -j 8 CROSS_COMPILE=arm-poky-linux-gnueabi- CC="arm-poky-linux-gnueabi-gcc" "$#"
make -j 8 CROSS_COMPILE=arm-poky-linux-gnueabi- CC="your_command_here arm-poky-linux-gnueabi-gcc" --sysroot=/some/path "$#"
}
How:
sed -r 's:(^\s+make.+ CC=\"):\1your_command_here :g'
-r = regex option
^make(CC=\") = starts with make and set a capture group on CC="
\1your_command_here = \1 reference capture group then add command text
You could use perl.
Replace YOUR_COMMAND with what you want added. This assumes your file is in file.txt:
perl -i.bak -pl -e '/^make/ and s/(CC=".*")/$1 YOUR_COMMAND /' file.txt
I am trying to write a shell script to rename the result of 7-zipped folders. The resulting filenames contains backslash \ in the filename.
I wrote a simple :
#! /bin/sh
for n in * do
OldName=$n
NewName=`echo "$n" | tr -s '\' "#" | tr -s " " "_"`
echo $NewName
mv "$OldName" "$NewName"
done
The problem I have is that \01 is interpreted by echo, and my files are :
FLD\01.02.2015 thefile.pdf
Thus, echo "FLD\01.02.2015 thefile.pdf" returns FLD?.2015.
I have tried various replacement solutions, s/\/#/g, sed, tr.
I tried to use printf instead
I have search all over the net without finding a valid solution.
Nothing works. I need a solution that would work on Unix and Mac OS X.
the only "working" solution would be to
ls > liste.txt
sed -e 's/\\/,/g' liste.txt
and then parse liste.txt, escape the backslash, generate a rename.sh and execute it. But this seems really dirty to me.
Does anyone has a suggestion ?
If you care about literal data, don't use echo. Its results are not well defined.
You can instead use printf, also POSIX, which doesn't mangle data.
NewName=`printf "%s" "$n" | tr -s '\' "#" | tr -s " " "_"`
Instead of using echo to rename the filename you can do this (in Bash):
NewName="${OldName//\\/}"
Example:
OldName="FLD\01.02.2015 thefile.pdf"
NewName="${OldName//\\/}"
echo "$NewName"
It will print out:
FLD01.02.2015 thefile.pdf
Note: You will need to change the shebang to #!/bin/bash to use bash parameter expansion.
Im trying to iterate over each file and folder inside a directory and extract part of the file name into a variable, but I can't make sed work correctly. I either get all of the file name or none of it.
This version of the script should capture the entire file name:
#!/bin/bash
for f in *
do
substring=`echo $f | sed -E -n 's/(.*)/\1/'`
echo "sub: $substring"
done
But instead I get nothing:
sub:
sub:
sub:
sub:
...
This version should give me just the first character in the filename:
#!/bin/bash
for f in *
do
substring=`echo $f | sed -E 's/^([a-zA-Z])/\1/'`
echo "sub: $substring"
done
But instead I get the whole file name:
sub: Adlm
sub: Applications
sub: Applications (Parallels)
sub: Desktop
...
I've tried numerous iterations of it and what it basically boils down to is that if I use -n I get nothing and if I don't I get the whole file name.
Can someone show me how to get just the first character?
Or, my overall goal is to be able to extract a substring and store it into a variable, if anybody has a better approach to it, that would be appreciated as well.
Thanks in advance.
If you want to modify a shell parameter, you probably want to use a parameter expansion.
for f in *; do
# This version should expand to the whole parameter
echo "$f"
# This version should expand to the first character in the filename
echo "${f::1}"
done
Parameter expansions are not as powerful as sed, but they are built in to the shell (no launching a separate process or subshell necessary) and there are expansions for:
Substrings (as above)
Replacing and substituting characters
Altering the case of strings (bash 4+)
and more.
This version of the script should capture the entire file name:
sed -E -n 's/(.*)/\1/'
But instead I get nothing.
You used -n so naturally it won't yield anything. Perhaps you should remove -n or add p:
sed -E -n 's/(.*)/\1/p'
This version should give me just the first character in the filename:
sed -E 's/^([a-zA-Z])/\1/'
But instead I get the whole file name,
You didn't replace anything there. Perhaps what you wanted was
sed -E 's/^([a-zA-Z]).*/\1/'
Also I suggest quoting your arguments well:
substring=`echo "$f" | sed ...'`
Finally the simpler method is to use substring expansion if you're using Bash as suggested by kojiro.
You forget to add .* after the capturing group in sed,
$ for i in *; do substring=`echo $i | sed -E 's/^(.).*$/\1/'`; echo "sub: $substring"; done
It's better to use . instead of [a-zA-Z] because it may fail if the first character starts with any special character.
I prefer awk to sed. It seems to be easier for me to understand.
#!/bin/bash
#set -x
for f in *
do
substring=`echo $f | awk '{print substr($1,1,1)}'`
echo "sub: $substring"
done
I just want to match some text in a Bash script. I've tried using sed but I can't seem to make it just output the match instead of replacing it with something.
echo -E "TestT100String" | sed 's/[0-9]+/dontReplace/g'
Which will output TestTdontReplaceString.
Which isn't what I want, I want it to output 100.
Ideally, it would put all the matches in an array.
edit:
Text input is coming in as a string:
newName()
{
#Get input from function
newNameTXT="$1"
if [[ $newNameTXT ]]; then
#Use code that im working on now, using the $newNameTXT string.
fi
}
You could do this purely in bash using the double square bracket [[ ]] test operator, which stores results in an array called BASH_REMATCH:
[[ "TestT100String" =~ ([0-9]+) ]] && echo "${BASH_REMATCH[1]}"
echo "TestT100String" | sed 's/[^0-9]*\([0-9]\+\).*/\1/'
echo "TestT100String" | grep -o '[0-9]\+'
The method you use to put the results in an array depends somewhat on how the actual data is being retrieved. There's not enough information in your question to be able to guide you well. However, here is one method:
index=0
while read -r line
do
array[index++]=$(echo "$line" | grep -o '[0-9]\+')
done < filename
Here's another way:
array=($(grep -o '[0-9]\+' filename))
Pure Bash. Use parameter substitution (no external processes and pipes):
string="TestT100String"
echo ${string//[^[:digit:]]/}
Removes all non-digits.
I Know this is an old topic but I came her along same searches and found another great possibility apply a regex on a String/Variable using grep:
# Simple
$(echo "TestT100String" | grep -Po "[0-9]{3}")
# More complex using lookaround
$(echo "TestT100String" | grep -Po "(?i)TestT\K[0-9]{3}(?=String)")
With using lookaround capabilities search expressions can be extended for better matching. Where (?i) indicates the Pattern before the searched Pattern (lookahead),
\K indicates the actual search pattern and (?=) contains the pattern after the search (lookbehind).
https://www.regular-expressions.info/lookaround.html
The given example matches the same as the PCRE regex TestT([0-9]{3})String
Use grep. Sed is an editor. If you only want to match a regexp, grep is more than sufficient.
using awk
linux$ echo -E "TestT100String" | awk '{gsub(/[^0-9]/,"")}1'
100
I don't know why nobody ever uses expr: it's portable and easy.
newName()
{
#Get input from function
newNameTXT="$1"
if num=`expr "$newNameTXT" : '[^0-9]*\([0-9]\+\)'`; then
echo "contains $num"
fi
}
Well , the Sed with the s/"pattern1"/"pattern2"/g just replaces globally all the pattern1s to pattern 2.
Besides that, sed while by default print the entire line by default .
I suggest piping the instruction to a cut command and trying to extract the numbers u want :
If u are lookin only to use sed then use TRE:
sed -n 's/.*\(0-9\)\(0-9\)\(0-9\).*/\1,\2,\3/g'.
I dint try and execute the above command so just make sure the syntax is right.
Hope this helped.
using just the bash shell
declare -a array
i=0
while read -r line
do
case "$line" in
*TestT*String* )
while true
do
line=${line#*TestT}
array[$i]=${line%%String*}
line=${line#*String*}
i=$((i+1))
case "$line" in
*TestT*String* ) continue;;
*) break;;
esac
done
esac
done <"file"
echo ${array[#]}