I am working on building a .sed file to start scripting the setup of multiple apache servers. I am trying to get sed to match the default webmaster email addresses in the .conf file which works great with this egrep. However when I use sed to try and so a substitute search and replace i get no errors back but it also does not do any substituting. I test this by running the same egrep command again.
egrep -o '\b[A-Za-z0-9._%-]+#[A-Za-z0-9.-]+(\.[A-Za-z]{2,4})?\b' /home/test/httpd.conf
returns
admin#your-domain.com
root#localhost
webmaster#dummy-host.example.com
The sed command I'm trying to use is
sed -i '/ServerAdmin/ s/\b[A-Za-z0-9._%-]+#[A-Za-z0-9.-]+(\.[A-Za-z]{2,4})?\b/MY_ADMIN_ADDRESS#gmail.com/g' /home/test/httpd.conf
After running I try and verify the results by running the egrep again and it returns the same 3 email address indicating nothing was replaced.
Don't assume that any two tools use the same regular expression syntax. If you're going to be doing replacements with sed, use sed to test - not egrep. It's easy to use sed as if it were a grep command: sed -ne '/pattern/p'.
sed must be told that it needs to use extended regular expressions using the -r option then making the sed command as follows.
sed -ir '/ServerAdmin/ s/\b[A-Za-z0-9._%-]+#[A-Za-z0-9.-]+(\.[A-Za-z]{2,4})?\b/MY_ADMIN_ADDRESS#gmail.com/g' /home/test/httpd.conf
Much thanks to Kent for pointing out that the address it was missing wasnt following a ServerName
Related
I'm trying to run a sed command on a Windows machine from the command prompt (CMD.exe) but I am struggling to understand the regular expression and how to escape the string properly when running on Windows. Ideally, I want to develop a solution that works across UNIX and Windows.
sed is not available on Windows so I have installed it via the gnuWin32 project which works well.
The unix format for the command is:
sed -i '' -e 's/\\/_next/\\.\\/next/g' out/**.html
Through a process of trial an error I have managed to get this far:
sed -i \'\' -e \'s/\\/_next/\\.\\/next/g\' out/**.html
but I get an error:
sed: -e expression #1, char 27: unterminated address regex
So there's definitely something wrong with my regex, probably the escaping of various parts?
Any ideas how I might go about fixing this?
Update:
I'm getting the code from here where unfortunately only Linux and OSX are covered.
You need to use
sed -i "s/\\/_next/\\.\\/next/g" out/**.html
The Windows GNU sed does not require the '' empty argument after -i option, they can be safely removed.
Also, the sed command in Windows console should be used in double quotes.
For integration tests, I have output that contains full file paths. I want to have my test script replace the user-specific start of the file path (e.g. /Users/uli/) with a generic word (USER_DIR) so that I can compare the files.
The problem, of course, are the slashes in the path. I tried the solutions given here and here, but they don't work for me:
#!/bin/bash
old_path="/Users/uli/"
new_path="USERDIR"
sed -i "s#$old_path#$new_path#g" /Users/uli/Desktop/replacetarget.txt
I get the error
sed: 1: "/Users/uli/Desktop/repl ...": invalid command code u
This is the version of sed that comes with macOS 10.14.6 (it has no --version option and is installed in /usr/bin/, so no idea what exact version).
Update:
I also tried
#!/bin/bash
old_path="/Users/uli/"
old_path=${old_path//\//\\\/}
new_path="USERDIR"
regex="s/$old_path/$new_path/g"
echo $old_path
echo $regex
sed -i $regex /Users/uli/Desktop/replacetarget.txt
But I get the same error. What am I doing wrong?
BSD sed requires an argument following -i (the empty string '' indicates no backup, similar to argumentless -i in GNU sed). As a result, your script is being treated as the backup-file extention, and your input file as the script.
old_path="/Users/uli/"
new_path="USERDIR"
sed -i '' "s#$old_path#$new_path#g" /Users/uli/Desktop/replacetarget.txt
However, sed is a stream editor, based on the file editor ed, so using -i is an indication you are using the wrong tool to begin with. Just use ed.
old_path="/Users/uli/"
new_path="USERDIR"
printf 's#%s#%s#g\nwq\n' "$old_path" "$new_path" | ed /Users/uli/Desktop/replacetarget.txt
Obligatory warning: neither editor is parameterized as such; you are simpling generating the script dynamically, which means it's your responsibility to ensure that the resulting script is valid. (For example, if either parameter contains a ;, it had better be escaped to prevent (s)ed from seeing it as a command separator.)
I'm trying to use SED to extract text from a log file. I can do a search-and-replace without too much trouble:
sed 's/foo/bar/' mylog.txt
However, I want to make the search case-insensitive. From what I've googled, it looks like appending i to the end of the command should work:
sed 's/foo/bar/i' mylog.txt
However, this gives me an error message:
sed: 1: "s/foo/bar/i": bad flag in substitute command: 'i'
What's going wrong here, and how do I fix it?
Update: Starting with macOS Big Sur (11.0), sed now does support the I flag for case-insensitive matching, so the command in the question should now work (BSD sed doesn't reporting its version, but you can go by the date at the bottom of the man page, which should be March 27, 2017 or more recent); a simple example:
# BSD sed on macOS Big Sur and above (and GNU sed, the default on Linux)
$ sed 's/ö/#/I' <<<'FÖO'
F#O # `I` matched the uppercase Ö correctly against its lowercase counterpart
Note: I (uppercase) is the documented form of the flag, but i works as well.
Similarly, starting with macOS Big Sur (11.0) awk now is locale-aware (awk --version should report 20200816 or more recent):
# BSD awk on macOS Big Sur and above (and GNU awk, the default on Linux)
$ awk 'tolower($0)' <<<'FÖO'
föo # non-ASCII character Ö was properly lowercased
The following applies to macOS up to Catalina (10.15):
To be clear: On macOS, sed - which is the BSD implementation - does NOT support case-insensitive matching - hard to believe, but true. The formerly accepted answer, which itself shows a GNU sed command, gained that status because of the perl-based solution mentioned in the comments.
To make that Perl solution work with foreign characters as well, via UTF-8, use something like:
perl -C -Mutf8 -pe 's/öœ/oo/i' <<< "FÖŒ" # -> "Foo"
-C turns on UTF-8 support for streams and files, assuming the current locale is UTF-8-based.
-Mutf8 tells Perl to interpret the source code as UTF-8 (in this case, the string passed to -pe) - this is the shorter equivalent of the more verbose -e 'use utf8;'.Thanks, Mark Reed
(Note that using awk is not an option either, as awk on macOS (i.e., BWK awk and BSD awk) appears to be completely unaware of locales altogether - its tolower() and toupper() functions ignore foreign characters (and sub() / gsub() don't have case-insensitivity flags to begin with).)
A note on the relationship of sed and awk to the POSIX standard:
BSD sed and awk limit their functionality mostly to what the POSIX sed and
POSIX awk specs mandate, whereas their GNU counterparts implement many more extensions.
Editor's note: This solution doesn't work on macOS (out of the box), because it only applies to GNU sed, whereas macOS comes with BSD sed.
Capitalize the 'I'.
sed 's/foo/bar/I' file
Another work-around for sed on Mac OS X is to install gsedfrom MacPorts or HomeBrew and then create the alias sed='gsed'.
If you are doing pattern matching first, e.g.,
/pattern/s/xx/yy/g
then you want to put the I after the pattern:
/pattern/Is/xx/yy/g
Example:
echo Fred | sed '/fred/Is//willma/g'
returns willma; without the I, it returns the string untouched (Fred).
The sed FAQ addresses the closely related case-insensitive search. It points out that a) many versions of sed support a flag for it and b) it's awkward to do in sed, you should rather use awk or Perl.
But to do it in POSIX sed, they suggest three options (adapted for substitution here):
Convert to uppercase and store original line in hold space; this won't work for substitutions, though, as the original content will be restored before printing, so it's only good for insert or adding lines based on a case-insensitive match.
Maybe the possibilities are limited to FOO, Foo and foo. These can be covered by
s/FOO/bar/;s/[Ff]oo/bar/
To search for all possible matches, one can use bracket expressions for each character:
s/[Ff][Oo][Oo]/bar/
The Mac version of sed seems a bit limited. One way to work around this is to use a linux container (via Docker) which has a useable version of sed:
cat your_file.txt | docker run -i busybox /bin/sed -r 's/[0-9]{4}/****/Ig'
Use following to replace all occurrences:
sed 's/foo/bar/gI' mylog.txt
I had a similar need, and came up with this:
this command to simply find all the files:
grep -i -l -r foo ./*
this one to exclude this_shell.sh (in case you put the command in a script called this_shell.sh), tee the output to the console to see what happened, and then use sed on each file name found to replace the text foo with bar:
grep -i -l -r --exclude "this_shell.sh" foo ./* | tee /dev/fd/2 | while read -r x; do sed -b -i 's/foo/bar/gi' "$x"; done
I chose this method, as I didn't like having all the timestamps changed for files not modified. feeding the grep result allows only the files with target text to be looked at (thus likely may improve performance / speed as well)
be sure to backup your files & test before using. May not work in some environments for files with embedded spaces. (?)
Following should be fine:
sed -i 's/foo/bar/gi' mylog.txt
I am trying to search string and replace string in a file. I used the below code:
sed -e 's/{"AP_SESSION_ID"\1\"787"}/{"AP_SESSION_ID"\1\"800"}/g' FILE|tee FILE
but it is not working and the output is like this:
sed: number in \[0-9] invalid
My environment is CYGWIN.
sample file is:
DP_SESSION_ID is a sting for values
DP_SESSION_ID is aplicat
"DP_S42SETTACC_TYPE"\1\"02"
"DP_SAP_CLIENT"\1\"460"
"DP_SAP_COMM_CONNECTION"\1\"JAVA_COMM_TOOL_ANALYZER"
"DP_SAP_CONNECTION"\1\"JAVA_TOOL_ANALYZER"
"DP_SAP_TOOLBI_CONNECTION"\1\"JAVA_TOOLBI_ANALYZER"
"DP_SESSION_ID"\1\"808"
I want search this "DP_SESSION_ID"\1\" sting and replace corresponding number like 808 in file prenatally(windows env), and i wand sing line command in windows bat command or perl command i don't want scrip or program
even i have installed cygwin tool in my server so unix also ok but single line command
server: windows 2008,cygwin x
using tool : datastage server jobs
perl -pi -e 's{" "DP_SESSION_ID"\1\"808 '"}{' "DP_SESSION_ID"\1\"900 '"'"}g' " file name
this code is not working
Please give good solution
You need to "escape" the backslashes by using two in a row:
sed -e 's/{"AP_SESSION_ID"\\1\\"787"}/{"AP_SESSION_ID"\\1\\"800"}/g' FILE|tee FILE
Otherwise the \1 is treated as a backreference, and you have no subgroups (parenthesized expressions) to reference.
Apart from back-slashes, IMO you also need to escape the quotes.
sed -e 's/{\"AP_SESSION_ID\"\\1\\\"787\"}/{\"AP_SESSION_ID\"\\1\\\"800\"}/g' FILE|tee FILE
I have sed running with the following argument fine if I copy and paste this into an open shell:
cat test.txt | sed '/[,0-9]\{0,\}[0-9]\{1,\}[acd][0-9]\{1,\}[,0-9]\{0,\}/{N
s/[,0-9]\{0,\}[0-9]\{1,\}[acd][0-9]\{1,\}[,0-9]\{0,\}\n\-\-\-//}'
The problem is that when I try to move this into a KornShell (ksh) script, the ksh throws errors because of what I think is that new line character. Can anyone give me a hand with this? FYI: the regular expression is supposed to be a multiple line replacement.
Thank you!
This: \{0,\} can be replaced by this: *
This: \{1,\} can be replaced by this: \+
It's not necessary to escape hyphens.
The newline can be replaced by -e (or by a semicolon)
The cat can be replaced by using the filename as an argument to sed
The result:
sed -e '/[,0-9]*[0-9]\+[acd][0-9]\+[,0-9]*/{N' -e 's/[,0-9]*[0-9]\+[acd][0-9]\+[,0-9]*\n---//}' test.txt
or
sed '/[,0-9]*[0-9]\+[acd][0-9]\+[,0-9]*/{N;s/[,0-9]*[0-9]\+[acd][0-9]\+[,0-9]*\n---//}' test.txt
(untested)
can you try to put your regex in a file and call sed with the option -f ?
cat test.txt | sed -f file.sed
Can you try to replace the new line character with `echo -e \\r`
The Korn Shell - unlike the C Shell - has no problem with newlines in strings. The newline is very unlikely to be your problem, therefore. The same comments apply to Bourne and POSIX shells, and to Bash. I've copied your example and run it on Linux under both Bash and Korn shell without any problem.
If you use C Shell for your work, are you sure you're running 'ksh ./script' and not './script'?
Otherwise, there is some other problem - an unbalanced quote somewhere, perhaps.
Check out the '-v' and '-n' options as well as the '-x' option to the Korn Shell. That may tell you more about where the problem is.