So I have the following string in my config.fish, and init.vim:
Fish: eval sh ~/.config/fish/colors/base16-monokai.dark.sh
Vim: colorscheme base16-monokai
Vim: let g:airline_theme='base16_monokai'
And I have the following shell script:
#!/bin/sh
theme=$1
background=$2
if [ -z '$theme' ]; then
echo "Please provide a theme name."
else
if [ -z '$background' ]; then
$background = 'dark'
fi
base16-builder -s $theme -t vim -b $background > ~/.config/nvim/colors/base16-$theme.vim &&
base16-builder -s $theme -t shell -b $background > ~/.config/fish/colors/base16-$theme.$background.sh &&
base16-builder -s $theme -t vim-airline -b $background > ~/.vim/plugged/vim-airline-themes/autoload/airline/themes/base16_$theme.vim
sed -i -e 's/foo/eval sh ~/.config/fish/colors/base16-$theme.$background.sh/g' ~/Developer/dotfiles/config.fish
sed -i -e 's/foo/colorscheme base16-$theme/g' ~/Developer/dotfiles/init.vim
sed -i -e 's/foo/let g:airline_theme='base16_$theme'/g' ~/Developer/dotfiles/init.vim
fi
Basically the idea is the script will generate whichever theme is passed through using this builder.
I have tried referring this documentation but I am not very skilled at regex so if anybody could give me a hand I would appreciate it.
What I need to happen is once the script is generated sed will look for the above strings and replace theme with the newly generated theme ones.
Try this :
sed -i "s|\(eval sh ~/\.config/fish/colors/base16-\)\([^.]*\)\.\([^.]*\)\\(.*\)|\1$theme.$background\4|
" ~/Developer/dotfiles/config.fish
sed -i "s/\(base16\)\([-_]\)\([a-zA-Z]*\)/\1\2$theme/g" ~/Developer/dotfiles/init.vim
Assuming in the second sed command that the theme is an alphanumeric string. If not, you can complete the character range : [a-zA-Z] with additional characters (eg [a-zA-Z0-9]).
You can replace something in sed using this syntax: sed "s#regex#replacement#g". Because you have /s and 's in your strings, it's easiest not to need to escape them.
There are some characters that need to be escaped to make the regexes. . and $ need to be escaped with a \. The $ in the replacement string needs to be escaped too.
If you want to capture a certain part from match, it's easiest to use char classes. For example, eval sh ~/\.config/fish/colors/base16-([^.]+)\.dark\.sh would be the regex to use if you want your replacement to be airline_theme='$1_base16_\$theme'. In that case, the $1 in the replacement is the thing captured in the regex.
[^.]+ will capture everything up to the next .
I hope this helps you to better understand regexes! This should be detailed enough to show you how to write your own.
You need to use double quotes for parameter expansion not single quotes.
You need to escape the single quotes: 'hello'\''world'
I will make one line for you and leave it as an exercise to fix the other lines
sed -i -e 's~\(let g:airline_theme='\''\)[^'\'']*\('\'\)'~base16_'"$theme"~' ~/Developer/dotfiles/init.vim
The first character after the s in the sed expression string is used as the pattern separator, so by putting / first you have specified / as the separator.
Additionally using the single quote tells the shell not to expand any variables, you are going to want to use double quotes instead.
try something like
sed -i -e "s#foo#eval sh ~/.config/fish/colors/base16-$theme.$background.sh#g" ~/Developer/dotfiles/config.fish
as you've now commented that you needed to find the previous theme string instead of foo
sed -i -e "s#eval sh \~/\.config/fish/colors/base16-.*?\..*?\.sh#eval sh ~/.config/fish/colors/base16-$theme.$background.sh#g" ~/Developer/dotfiles/config.fish
i want to replace this
#!/usr/bin/env bash
with this
#!/bin/bash
i have tried two approaches
Approach 1
original_str="#!/usr/bin/env bash"
replace_str="#!/bin/bash"
sed s~${original_str}~${replace_str}~ filename
Approach 2
line=`grep -n "/usr/bin" filename`
awk NR==${line} {sub("#!/usr/bin/env bash"," #!/bin/bash")}
But both of them are not working.
You cannot use ! inside a double quotes in BASH otherwise history expansion will take place.
You can just do:
original_str='/usr/bin/env bash'
replace_str='/bin/bash'
sed "s~$original_str~$replace_str~" file
#!/bin/bash
Using escape characters :
$ cat z.sh
#!/usr/bin/env bash
$ sed -i "s/\/usr\/bin\/env bash/\/bin\/bash/g" z.sh
$ cat z.sh
#!/bin/bash
Try this out in the terminal:
echo "#!/usr/bin/env bash" | sed 's:#!/usr/bin/env bash:#!/bin/bash:g'
In this cases I use : because sed gets confused between the different slashes and it isn't able to tell anymore with one separates and wich one is part of the text.
Plus it looks really clean.
The cool thing is that you can use every symbol you want as a separator.
For example a semicolon ; or the pipe symbol | .
By using the escape character \ I think that the code would look too messy and wouldn't be very readable, considering the fact that you have to put it before every forward slash in the command.
The command above will just print out the replaced line, but if you want to modify the file, than you need to specify the input and output file, like this:
sed 's:#!/usr/bin/env bash:#!/bin/bash:g' <inputfile >outputfile-new
Remember to put that -new if the inputfile and the output file have the same name, because without it your original one will be cleared completely: this happend me in the past, and it's not the best thing at all. For example:
<test.txt >test-new.txt
I'm trying to create a function within a bash script that queries a log file. Within the query function, I have something that resembles the following:
if [ -n "$(cat some-log-file.log | grep \"$1\")" ]; then
echo "There is a match."
else
echo "No lines matched the search term."
fi
If I send something I know will be in the log file as $1, like "I don't", I get the output:
$ ./y.sh query "I don't"
grep: don't": No such file or directory
No lines matched the search term.
If I try to single quote the $() expression, it sends the literal string and always evaluates true. I'm guessing it has something to do with the way grep interprets backslashes, but I can't figure it out. Maybe I'm overseeing something simple, but I've been at this for hours looking on forums and plugging in all kinds of strange combinations of quotes and escape characters. Any help or advice is appreciated.
It's actually really easy, if you realize that $() is allowed to have unescaped quotes:
if [ -n "$(cat some-log-file.log | grep "$1")" ]; then
echo "There is a match."
else
echo "No lines matched the search term."
fi
You can actually even skip that step, though, because grep gives an appropriate exit code:
if grep -q "$1" some-log-file.log; then
echo "There is a match."
else
echo "No lines matched the search term."
fi
In short, this happens for the same reason that "$1" works: Shell parameter expansion and command substitution happen before word splitting and quote removal. See more about how bash parses commands in the Shell Expansions section of the bash manual.
Im using a shell script to get a file using wget and search it for a pattern. My shell script is as follows:
#Execute commands one by one
while read line
do
STARTTIME=$(($(date +%s%N)/1000000))
line2=$(wget -q --post-data "$line" -O PHPFiles/test.php http://localhost:1234/XSS/XSS2/test.php)
ENDTIME=$(($(date +%s%N)/1000000))
GAP=$(($ENDTIME-$STARTTIME))
DIFF=$(($DIFF+($ENDTIME-$STARTTIME)))
echo "Time Taken "$GAP
finalSearchLine1="${line/&name2=/ }"
finalSearchLine2="${finalSearchLine1/name=/}"
echo "$finalSearchLine2"
if grep -q -F "$finalSeachLine2" -a PHPFiles/test.php;
then
echo found
success=$((success+1))
else
echo not found
failure=$((failure+1))
fi
rm PHPFiles/test.php
done < $1
echo "***************"
echo "Success "$success
echo "Failure "$failure
echo "Total Time "$DIFF
echo "Average Time "$((DIFF/(success+failure)))
However, I'm having trouble with the grep command. Sometimes, the data $finalSearchLine2 contains quotes such as:
<script >alert("XSS"); </script>
This seem to cause trouble with the grep command. For the if statement, I always seem to get the result as found even when there is no matching pattern in the $finalSearchLine2 variable. I dont know if its possible to use escape strings within the variable for grep. Can anyone suggest a possible solution for this?
Grep needs double quotes to be escaped like this \"
So as a first solution you could try:
temp_variable=$(sed 's/"/\\"/g' <<< $temp)
if grep -q -F "$temp_variable" -a /PHPFiles/test.php;
So you first escape the double quotes with sed and you store the result in temp_variable. Then you use temp_variable in grep.
What is the correct way to parse a string using regular expressions in a linux shell script? I wrote the following script to print my SO rep on the console using curl and sed (not solely because I'm rep-crazy - I'm trying to learn some shell scripting and regex before switching to linux).
json=$(curl -s http://stackoverflow.com/users/flair/165297.json)
echo $json | sed 's/.*"reputation":"\([0-9,]\{1,\}\)".*/\1/' | sed s/,//
But somehow I feel that sed is not the proper tool to use here. I heard that grep is all about regex and explored it a bit. But apparently it prints the whole line whenever a match is found - I am trying to extract a number from a single line of text. Here is a downsized version of the string that I'm working on (returned by curl).
{"displayName":"Amarghosh","reputation":"2,737","badgeHtml":"\u003cspan title=\"1 silver badge\"\u003e\u003cspan class=\"badge2\"\u003e●\u003c/span\u003e\u003cspan class=\"badgecount\"\u003e1\u003c/span\u003e\u003c/span\u003e"}
I guess my questions are:
What is the correct way to parse a string using regular expressions in a linux shell script?
Is sed the right thing to use here?
Could this be done using grep?
Is there any other command that's more easier/appropriate?
The grep command will select the desired line(s) from many but it will not directly manipulate the line. For that, you use sed in a pipeline:
someCommand | grep 'Amarghosh' | sed -e 's/foo/bar/g'
Alternatively, awk (or perl if available) can be used. It's a far more powerful text processing tool than sed in my opinion.
someCommand | awk '/Amarghosh/ { do something }'
For simple text manipulations, just stick with the grep/sed combo. When you need more complicated processing, move on up to awk or perl.
My first thought is to just use:
echo '{"displayName":"Amarghosh","reputation":"2,737","badgeHtml"'
| sed -e 's/.*tion":"//' -e 's/".*//' -e 's/,//g'
which keeps the number of sed processes to one (you can give multiple commands with -e).
You may be interested in using Perl for such tasks. As a demonstration, here is a Perl script which prints the number you want:
#!/usr/local/bin/perl
use warnings;
use strict;
use LWP::Simple;
use JSON;
my $url = "http://stackoverflow.com/users/flair/165297.json";
my $flair = get ($url);
my $parsed = from_json ($flair);
print "$parsed->{reputation}\n";
This script requires you to install the JSON module, which you can do with just the command cpan JSON.
For working with JSON in shell script, use jsawk which like awk, but for JSON.
json=$(curl -s http://stackoverflow.com/users/flair/165297.json)
echo $json | jsawk 'return this.reputation' # 2,747
My proposition:
$ echo $json | sed 's/,//g;s/^.*reputation...\([0-9]*\).*$/\1/'
I put two commands in sed argument:
s/,//g is used to remove all commas, in particular the ones that are present in the reputation value.
s/^.*reputation...\([0-9]*\).*$/\1/ locates the reputation value in the line and replaces the whole line by that value.
In this particular case, I find that sed provides the most compact command without loss of readability.
Other tools for manipulating strings (not only regex) include:
grep, awk, perl mentioned in most of other answers
tr for replacing characters
cut, paste for handling multicolumn inputs
bash itself with its rich $(...) syntax for accessing variables
tail, head for keeping last or first lines of a file
sed is appropriate, but you'll spawn a new process for every sed you use (which may be too heavyweight in more complex scenarios). grep is not really appropriate. It's a search tool that uses regexps to find lines of interest.
Perl is one appropriate solution here, being a shell scripting language with powerful regexp features. It'll do most everything you need without spawning out to separate processes (unlike normal Unix shell scripting) and has a huge library of additional functions.
You can do it with grep. There is -o switch in grep witch extract only matching string not whole line.
$ echo $json | grep -o '"reputation":"[0-9,]\+"' | grep -o '[0-9,]\+'
2,747
1) What is the correct way to parse a string using regular expressions in a linux shell script?
Tools that include regular expression capabilities include sed, grep, awk, Perl, Python, to mention a few. Even newer version of Bash have regex capabilities. All you need to do is look up the docs on how to use them.
2) Is sed the right thing to use here?
It can be, but not necessary.
3) Could this be done using grep?
Yes it can. you will just construct similar regex as you would if you use sed, or others. Note that grep just does what it does, and if you want to modify any files, it will not do it for you.
4) Is there any other command that's easier/more appropriate?
Of course. regex can be powerful, but its not necessarily the best tool to use everytime. It also depends on what you mean by "easier/appropriate".
The other method to use with minimal fuss on regex is using the fields/delimiter approach. you look for patterns that can be "splitted". for eg, in your case(i have downloaded the 165297.json file instead of using curl..(but its the same)
awk 'BEGIN{
FS="reputation" # split on the word "reputation"
}
{
m=split($2,a,"\",\"") # field 2 will contain the value you want plus the rest
# Then split on ":" and save to array "a"
gsub(/[:\",]/,"",a[1]) # now, get rid of the redundant characters
print a[1]
}' 165297.json
output:
$ ./shell.sh
2747
sed is a perfectly valid command for your task, but it may not be the only one.
grep may be useful too, but as you say it prints the whole line. It's most useful for filtering the lines of a multi-line file, and discarding the lines you don't want.
Efficient shell scripts can use a combination of commands (not just the two you mentioned), exploiting the talents of each.
Blindly:
echo $json | awk -F\" '{print $8}'
Similar (the field separator can be a regex):
awk -F'{"|":"|","|"}' '{print $5}'
Smarter (look for the key and print its value):
awk -F'{"|":"|","|"}' '{for(i=2; i<=NF; i+=2) if ($i == "reputation") print $(i+1)}'
You can use a proper library (as others noted):
E:\Home> perl -MLWP::Simple -MJSON -e "print from_json(get 'http://stackoverflow.com/users/flair/165297.json')->{reputation}"
or
$ perl -MLWP::Simple -MJSON -e 'print from_json(get "http://stackoverflow.com/users/flair/165297.json")->{reputation}, "\n"'
depending on OS/shell combination.
Simple RegEx via Shell
Disregarding the specific code in question, there may be times when you want to do a quick regex replace-all from stdin to stdout using shell, in a simple way, using a string syntax similar to JavaScript.
Below are some examples for anyone looking for a way to do this. Perl is a better bet on Mac since it lacks some sed options. If you want to get stdin as a variable you can use MY_VAR=$(cat);.
echo 'text' | perl -pe 's/search/replace/g'; # using perl
echo 'text' | sed -e 's/search/replace/g'; # using sed
And here's an example of a custom, reusable regex function. Arguments are source string (or -- for stdin), search, replace, and options.
regex() {
case "$#" in
( '0' ) exit 1 ;; ( '1' ) echo "$1"; exit 0 ;;
( '2' ) REP='' ;; ( '3' ) REP="$3"; OPT='' ;;
( * ) REP="$3"; OPT="$4" ;;
esac
TXT="$1"; SRCH="$2";
if [ "$1" = "--" ]; then [ ! -t 0 ] && read -r TXT; fi
echo "$TXT" | perl -pe 's/'"$SRCH"'/'"$REP"'/'"$OPT";
}
echo 'text' | regex -- search replace g;