Is there a way of telling a regular expression (specifically sed) to prefer using an optional component when the input also matches without using that component?
I'm trying to extract a number from a string that may optionally be preceded by prefix. It works in the following cases:
echo dummy/123456/dummy | sed "s:.*/\(prefix\)\?\([0-9]\{3,\}\)/.*:\2:"
123456
echo dummy/prefix123456/dummy | sed "s:.*/\(prefix\)\?\([0-9]\{3,\}\)/.*:\2:"
123456
but if the string contains both a prefixed number and a "bare" number, it choses the bare number:
echo dummy/prefix123456/987654/dummy | sed "s:.*/\(prefix\)\?\([0-9]\{3,\}\)/.*:\2:"
987654
Is there a way of forcing sed to prefer the match including the prefix (123456)? All search results I've found talk of greedy/lazy options, which – as far as I can tell – don't apply here.
Clarifications
The dummy portions in the examples above may contain slashes.
The bit I'm interested in is either the first slash-delimited run of three or more digits (.../123456/...) or the first slash-delimited run of 3+ digits with a prefix (.../prefix123456/...), whichever occurs first.
You may try this sed command:
sed '
/.*\/prefix\([0-9]\{3,\}\)\/.*/{
s//\1/
b
}
s/.*\/\([0-9]\{3,\}\)\/.*/\1/
' file
which will print out
123456
123456
123456
123456
where the content of file is
dummy/123456/dummy
dummy/prefix123456/dummy
dummy/prefix123456/987654/dummy
dummy/987654/prefix123456/dummy
With GNU awk you could try following code. Written and tested with shown samples only.
awk 'match($0,/\/(prefix){0,1}([0-9]+)/,arr){print arr[2]}' Input_file
Explanation: Simple explanation would be, using GNU awk's match function. In it using regex (prefix){0,1}([0-9]+) which is having 2 capturing groups and its matched values are getting stored into array named arr and if condition is fine then printing 2nd element of that array.
sed BRE or ERE doesn't have a way to use lazy quantifier in starting .*?.
However, based on your use-cases, you may use this sed:
sed -E 's~[^/]*/(prefix){0,1}([0-9]{3,})/.*~\2~' file
123456
123456
123456
where input is:
cat file
dummy/123456/dummy
dummy/prefix123456/dummy
dummy/prefix123456/987654/dummy
Here we are using negated character class (bracket expression) [^/]* instead of .* to allow pattern to match 0 or more of any char that is not a /.
If you can consider perl then .*? with a negative lookahead will work for you:
perl -pe 's~^.*?/(?:prefix)?(\d{3,})(?!.*prefix\d{3}).*~$1~' file
RegEx Demo
I need to modify an ntp configuration file by adding some options to the line containing Ip addresses.
I have been trying it for so long using sed command, but no able to modify the line unless i don't know the IP addresses.
Let say, i have few lines as,
server 172.0.0.1
server 10.0.0.1
I need to add iburst option after the ip address.
I have tried command like.. sed -e 's/(\d{1,3}\.\d{1.3}\.\d{1,3}\.\d{1,3})/ \1 iburst/g' ntp_file
or sed -e 's/^server +\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/server \1\.\2\.\3\ iburst/g' ntp_file
but its not modifying the line. Any kind of suggestions would be really appriciated.
The regex you have used as POSIX BRE cannot match the expected strings due to \d shorthand class that sed does not support, the misused dot inside a range quantifier and incorrect escaping of grouping and range quantifier delimiters.
You may use
sed -E -i 's/[0-9]{1,3}(\.[0-9]{1,3}){3}/ & iburst/g' ntp_file
The POSIX ERE (enabled with the -E option) expression means to match
[0-9]{1,3} - one to three digits
(\.[0-9]{1,3}){3} - three occurrences of a dot and one to three digits
The replacement pattern is & iburst where & stands for the whole match.
The g flag replaces all occurrences.
I'd like to run a grep command that searches a text file and should match email address with a certain tld.
Example, if the text file contains the following lines
tom#google.com
mark#google.com
tom.comber#google.cz
And I'm searching for the .com tld emails:
It should match tom#google.com and mark#google.com but not tom.comber#google.cz
I'm currently using the follow grep command, which matches pretty much every string that contains a .com. I want it to match specifically the tld of the domain
grep -rnwi "/Users/Me/Desktop/Folder/" -e ".com"
EDIT
grep -rnwi '#.+\.com$' "/Users/Me/Desktop/Folder/" matches nothing. but grep -rnwi "/Users/Me/Desktop/Folder/" -e "hotmail.com" matches plenty. I don't want just hotmail.com but all .com emails
EDIT2, this seem to match nothing either. is it because I'm searching in multiple text files in a folder?
grep -rnwi '#.\+\.com$' "/Users/Me/Desktop/Folder/"
EDIT3: wasn't totally clear. There are characters after the .tld extension so I had to leave off the trailing $. That works.
Do:
grep '#.\+\.com$' file.txt
#.\+ matches a # followed by one or more characters
\.com$ matches literal .com at the end
to do the same for other TLDs, replace com at the end with that TLD.
I am wantin to match IP address that are from 10.0-29.x.x, 10.31-39.x.x, and 10.41-253.x.x.
Of the lines below, I want to capture the 3rd line and below.
network 10.40.5.0 0.0.0.255
network 10.255.5.0 0.0.0.255
network 10.23.3.0 0.0.0.255
netowrk 10.273.255.0 0.255.255
So the way it will work, is if there is a match, it will set a flag that the configuration is invalid. I may have 10 invalid lines, or just 1. It doesn't matter.
Regex are not designed to do math.
However, you can try something like [3-4]{1} if you want a 3 or a 4.
For bigger processing you might have to match it first with a general IP regex, then process it with any language.
The core of your problem is a regex that matches these number ranges: 0-29, 31-39, 41-253
An extended regex that matches this is:
^network 10\.([0-9]|1[0-9]|2[0-9]|3[1-9]|4[1-9]|[5-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-3])\.[0-9]+\.[0-9]+
The regex is divided in these steps:
0-9, 10-19, 20-29, 31-39, 41-49, 50-99, 100-199, 200-249, 250-253
A shell script that would work is:
if {
cat input_file |
egrep -q '^network 10.([0-9]|1[0-9]|2[0-9]|3[1-9]|4[1-9]|[5-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-3]).[0-9]+.[0-9]+ '
}
then
echo action if matched
else
echo action if not matched
fi
I have a file that looks something like this:
# cat $file
...
ip access-list extended DOG-IN
permit icmp 10.10.10.1 0.0.0.7 any
permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
deny ip any any log
ip access-list extended CAT-IN
permit icmp 10.13.10.0 0.0.0.255 any
permit ip 10.14.10.0 0.0.0.255 host 10.15.10.10
permit tcp 10.16.10.0 0.0.0.255 host 10.17.10.10 eq smtp
...
I want to be able to search by name (using a script) to get 'section' output for independent access-lists. I want the output to look like this:
# grep -i dog $file | sed <options??>
ip access-list extended DOG-IN
permit icmp 10.10.10.1 0.0.0.7 any
permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
deny ip any any log
...with no further output of inapplicable non-indented lines.
I have tried the following:
grep -A 10 DOG $file | sed -n '/^[[:space:]]\{1\}/p'
...Which only gives me the 10 lines after DOG which begin with a single space (including lines not applicable to the searched access-list).
sed -n '/DOG/,/^[[:space:]]\{1\}/p' $file
...Which gives me the line containing DOG, and the next line beginning with a single space. (Need all the applicable lines of the access-list...)
I want the line containing DOG, and all lines after DOG which begin with a single space, until the next un-indented line. There are too many variables in the content to depend on any patterns other than the leading space (there is not always a deny on the end, etc...).
Using GNU sed (Linux):
name='dog' # case-INsensitive name of section to extract
sed -n "/$name/I,/^[^[:space:]]/ { /$name/I {p;d}; /^[^[:space:]]/q; p }" file
To make matching case-sensitive, remove the I after the occurrences of /I above.
-n suppresses default output so that output must explicitly be requested inside the script with functions such as p.
Note the use of double quotes ("...") around the sed script, so as to allow references to the shell variable $name: The double quotes ensure that the shell variable references are expanded BEFORE the script is handed to sed (sed itself has no access to shell variables).
Caveat: This technique is tricky, because (a) you must use shell escaping to escape shell metacharacters you want to pass through to sed, such as $ as \$, and (b) the shell-variable value must not contain sed metacharacters that could break the sed script; for generic escaping of shell-variable values for use in sed scripts, see this answer of mine, or use my awk-based answer.
/$name/I,/^[^[:space:]]/ uses a range to match the line of interest (/$name/I; the trailing I is GNU sed's case-insensitivity matching option) through the start of the next section (/^[^[:space:]]/ - i.e., the next line that does NOT start with whitespace); since sed ranges are always inclusive, the challenge is to selectively remove the last line of the range, IF it is the start of the next section - note that this will NOT be the case if the section of interest is the LAST one in the file.
Note that the commands inside { ... } are only executed for each line in the range.
/$name/I {p;d}; unconditionally prints the 1st line of the range: d deletes the line (which has already been printed) and starts the next cycle (proceeds to the next input line).
/^[^[:space:]]/q matches the last line in the range, IF it is the next section's first line, and quits processing altogether (q), without printing the line.
p is then only reached for section-interior lines and prints them.
Note:
The assumption is that header lines can be identified by NOT starting with a whitespace char., and that any other lines are non-header lines - if more sophisticated matching is required, see my awk-based answer.
This solution has the slight disadvantage that the range regexes must be duplicated, although you could mitigate that with shell variables.
FreeBSD/macOS sed can almost do the same, except that it lacks the case-insensitivity option, I.
name='DOG' # case-SENSITIVE name of section to extract
sed -n -e "/$name/,/^[^[:space:]]/ { /$name/ {p;d;}; /^[^[:space:]]/q; p; }" file
Note that FreeBSD/OSX sed generally has stricter syntax requirements, such as the ; after a command even when followed by }.
If you do need case-insensitivity, see my awk-based answer.
awk -vfound=0 '
/DOG/{
found = !found;
print;
next
}
/^[[:space:]]/{
if (found) {
print;
next
}
}
{ found = !found }
'
You can substitute any ERE in place of /DOG/, such as /(DOG)|(CAT)/, and the rest of the script will do the work. You can condense it if you like of course.
Note that just because a line begins with a space, that doesn't mean there is only one space. /^[[:space:]]{1}/ will match the leading space, even in a string like
nonspace
meaning it is equivalent to /^[[:space:]]/. If your format is so rigid that there must always only be a single space, use /^[[:space:]][^[:space:]]/ instead. Lines like the one with "nonspace" above will not be matched.
I added a second answer as mklement0 pointed a flaw on my logic.
This is yet a very simple way to do that in Perl:
perl -ne ' /^\w+/ && {$p=0}; /DOG/ && {$p=1}; $p && {print}'
EXAMPLES:
cat /tmp/file | perl -ne ' /^\w+/ && {$p=0}; /DOG/ && {$p=1}; $p && {print}'
ip access-list extended DOG-IN
permit icmp 10.10.10.1 0.0.0.7 any
permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
deny ip any any log
cat /tmp/file | perl -ne ' /^\w+/ && {$p=0}; /CAT/ && {$p=1}; $p && {print}'
ip access-list extended CAT-IN
permit icmp 10.13.10.0 0.0.0.255 any
permit ip 10.14.10.0 0.0.0.255 host 10.15.10.10
permit tcp 10.16.10.0 0.0.0.255 host 10.17.10.10 eq smtp
EXPLANATION:
If the line starts with [a-z0-9_] set $p false
If the line contains PATTERN in this case DOG sets $p true
if $p true prints
#mklement0 squeezed my already-inscrutable sed down to this:
sed '/^ip/!{H;$!d};x; /DOG/I!d'
which swaps accumulated multiline groups into the pattern buffer for processing -- the main logic (/DOG/I!d here) operates on whole groups.
The /^ip/! identifies continuation lines by the absence of a first-line marker and accumulates them, so the x only runs when an entire group has been accumulated.
Some corner cases don't apply here:
The first x swaps in a phantom empty group at the start. If that doesn't get dropped during ordinary processing, adding a 1d fixes that.
The last x also swaps out the last line of the file. That's usually just last line of the last group, already accumulated by the H, but if some command might produce one-line groups you need to supply a fake one at the end (with e.g. echo "header phantom" | sed '/^header/!{H;$!d};x' realdata.txt -, or { showgroups; echo header phantom; } | sed '/^header/!{H;$!d};x'.
A shorter, POSIX-compliant awk solution, which is a generalized and optimized translation of #Tiago's excellent Perl-based answer.
One advantage of these answers over the sed solutions is that they use literal substring matching rather than regular expressions, which allows passing in arbitrary search strings, without needing to worry about escaping. That said, if you did want regex matching, use the ~ operator rather than the index() function; e.g., index($0, name) would become $0 ~ name. You then have to make sure that the value passed for name either contains no accidental regex metacharacters meant to be treated as literals or is an intentionally crafted regex.
name='DOG' # Case-sensitive name to search for.
awk -v name="$name" '/^[^[:space:]]/ {if (p) exit; if (index($0,name)) {p=1}} p' file
Option -v name="$name" defines awk variable name based on the value of shell variable $name (awk has no direct access to shell variables).
Variable p is used as a flag to indicate whether the current line should be printed, i.e., whether it is part of the section of interest; as long as p is not initialized, it is treated as 0 (false) in a Boolean context.
Pattern /^[^[:space:]]/ matches only header lines (lines that start with a non-whitespace character), and the associated action ({...}) is only processed for them:
if (p) exit exits processing altogether, if p is already set, because that implies that the next section has been reached. Exiting right away has the benefit of not having to process the remainder of the file.
if (index($0, name)) looks for the name of interest as a literal substring in the header line at hand, and, if found (in which case index() returns the 1-based position at which the substring was found, which is interpreted astruein a Boolean context), sets flagpto1({p=1}`).
p simply prints the current line, if p is 1, and does nothing otherwise. That is, once the section header of interest has been found, it and subsequent lines are printed (up until the next section or the end of the input file).
Note that this is an example of a pattern-only command: only a pattern (condition) is specified, without an associated action ({...}), in which case the default action is to print the current line, if the pattern evaluates to true. (That technique is used in the common shorthand 1 to simply unconditionally print the current record.)
If case-INsensitivity is needed:
name='dog' # Case-INsensitive name to search for.
awk -v name="$name" \
'/^[^[:space:]]/ {if(p) exit; if(index(tolower($0),tolower(name))) {p=1}} p' file
Caveat: The BSD-based awk that comes with macOS (still applies as of 10.12.1) is not UTF-8-aware.: the case-insensitive matching won't work with non-ASCII letters such as ü.
GNU awk alternative, using the special IGNORECASE variable:
awk -v name="$name" -v IGNORECASE=1 \
'/^[^[:space:]]/ {if(p) exit; if(index($0,name)) {p=1}} p' file
Another POSIX-compliant awk solution:
name='dog' # Case-insensitive name of section to extract.
awk -v name="$name" '
index(tolower($0),tolower(name)) {inBlock=1; print; next} # 1st section line found.
inBlock && !/^[[:space:]]/ {exit} # Exit at start of next section.
inBlock # Print 2nd, 3rd, ... section line.
' file
Note:
next skips the remaining pattern-action pairs and proceeds to the next line.
/^[[:space:]]/ matches lines that start with at least one whitespace char. As #Chrono Kitsune explains in his answer, if you wanted to match lines that start with exactly one whitespace char., use /^[[:space:]][^[:space:]]/. Also note that, despite its name, character class [:space:] matches ANY form of whitespace, not just spaces - see man isspace.
There's no need to initialize flag variable inBlock, as it defaults to 0 in numeric/Boolean contexts.
If you have GNU awk, you can more easily achieve case-insensitive matching by setting the IGNORECASE variable to a nonzero value (-v IGNORECASE=1) and simply using index($0, name) inside the program.
A GNU awk solution, IF, you can assume that all section header lines start with 'ip' (so as to break the input into sections that way, rather than looking for leading whitespace):
awk -v RS='(^|\n)ip' -F'\n' -v name="$name" -v IGNORECASE=1 '
index($1, name) { sub(/\n$/, ""); print "ip" $0; exit }
' file
-v RS='(^|\n)ip' breaks the input into records by lines that fall between line-starting instances of string 'ip'.
-F'\n' then breaks each record into fields ($1, ...) by lines.
index($1, name) looks for the name on the current record's first line - case-INsensitively, thanks to -v IGNORECASE=1.
sub(/\n$/, "") removes any trailing \n, which can stem from the section of interest being the last in the input file.
print "ip" $0 prints the matching record, comprising the entire section of interest - since, however the record doesn't include the separator, 'ip', it is prepended.
The simplest way I can think of is: sed '/DOG/, /^ip/ !d' | sed '$d'
cat file | sed '/DOG/, /^ip/ !d' | sed '$d'
ip access-list extended DOG-IN
permit icmp 10.10.10.1 0.0.0.7 any
permit tcp 10.11.10.1 0.0.0.7 eq www 443 10.12.10.0 0.0.0.63
deny ip any any log
Explanation:
first sed command prints from the line containing DOG to the next line starting with ip
second sed command deletes the last line(which is the line starting with ip)