Related
My RAKU Code:
sub comments {
if ($DEBUG) { say "<filtering comments>\n"; }
my #filteredtitles = ();
# This loops through each track
for #tracks -> $title {
##########################
# LAB 1 TASK 2 #
##########################
## Add regex substitutions to remove superflous comments and all that follows them
## Assign to $_ with smartmatcher (~~)
##########################
$_ = $title;
if ($_) ~~ s:g:mrx/ .*<?[\(^.*]> / {
# Repeat for the other symbols
########################## End Task 2
# Add the edited $title to the new array of titles
#filteredtitles.push: $_;
}
}
# Updates #tracks
return #filteredtitles;
}
Result when compiling:
Error Compiling! Placeholder variable '#_' may not be used here because the surrounding block doesn't take a signature.
Is there something obvious that I am missing? Any help is appreciated.
So, in contrast with #raiph's answer, here's what I have:
my #tracks = <Foo Ba(r B^az>.map: { S:g / <[\(^]> // };
Just that. Nothing else. Let's dissect it, from the inside out:
This part: / <[\(^]> / is a regular expression that will match one character, as long as it is an open parenthesis (represented by the \() or a caret (^). When they go inside the angle brackets/square brackets combo, it means that is an Enumerated character class.
Then, the: S introduces the non-destructive substitution, i.e., a quoting construct that will make regex-based substitutions over the topic variable $_ but will not modify it, just return its value with the modifications requested. In the code above, S:g brings the adverb :g or :global (see the global adverb in the adverbs section of the documentation) to play, meaning (in the case of the substitution) "please make as many as possible of this substitution" and the final / marks the end of the substitution text, and as it is adjacent to the second /, that means that
S:g / <[\(^]> //
means "please return the contents of $_, but modified in such a way that all its characters matching the regex <[\(^]> are deleted (substituted for the empty string)"
At this point, I should emphasize that regular expressions in Raku are really powerful, and that reading the entire page (and probably the best practices and gotchas page too) is a good idea.
Next, the: .map method, documented here, will be applied to any Iterable (List, Array and all their alikes) and will return a sequence based on each element of the Iterable, altered by a Code passed to it. So, something like:
#x.map({ S:g / foo /bar/ })
essencially means "please return a Sequence of every item on #x, modified by substituting any appearance of the substring foo for bar" (nothing will be altered on #x). A nice place to start to learn about sequences and iterables would be here.
Finally, my one-liner
my #tracks = <Foo Ba(r B^az>.map: { S:g / <[\(^]> // };
can be translated as:
I have a List with three string elements
Foo
Ba(r
B^az
(This would be a placeholder for your "list of titles"). Take that list and generate a second one, that contains every element on it, but with all instances of the chars "open parenthesis" and "caret" removed.
Ah, and store the result in the variable #tracks (that has my scope)
Here's what I ended up with:
my #tracks = <Foo Ba(r B^az>;
sub comments {
my #filteredtitles;
for #tracks -> $_ is copy {
s:g / <[\(^]> //;
#filteredtitles.push: $_;
}
return #filteredtitles;
}
The is copy ensures the variable set up by the for loop is mutable.
The s:g/...//; is all that's needed to strip the unwanted characters.
One thing no one can help you with is the error you reported. I currently think you just got confused.
Here's an example of code that generates that error:
do { #_ }
But there is no way the code you've shared could generate that error because it requires that there is an #_ variable in your code, and there isn't one.
One way I can help in relation to future problems you may report on StackOverflow is to encourage you to read and apply the guidance in Minimal Reproducible Example.
While your code did not generate the error you reported, it will perhaps help you if you know about some of the other compile time and run time errors there were in the code you shared.
Compile-time errors:
You wrote s:g:mrx. That's invalid: Adverb mrx not allowed on substitution.
You missed out the third slash of the s///. That causes mayhem (see below).
There were several run-time errors, once I got past the compile-time errors. I'll discuss just one, the regex:
.*<?[...]> will match any sub-string with a final character that's one of the ones listed in the [...], and will then capture that sub-string except without the final character. In the context of an s:g/...// substitution this will strip ordinary characters (captured by the .*) but leave the special characters.
This makes no sense.
So I dropped the .*, and also the ? from the special character pattern, changing it from <?[...]> (which just tries to match against the character, but does not capture it if it succeeds) to just <[...]> (which also tries to match against the character, but, if it succeeds, does capture it as well).
A final comment is about an error you made that may well have seriously confused you.
In a nutshell, the s/// construct must have three slashes.
In your question you had code of the form s/.../ (or s:g/.../ etc), without the final slash. If you try to compile such code the parser gets utterly confused because it will think you're just writing a long replacement string.
For example, if you wrote this code:
if s/foo/ { say 'foo' }
if m/bar/ { say 'bar' }
it'd be as if you'd written:
if s/foo/ { say 'foo' }\nif m/...
which in turn would mean you'd get the compile-time error:
Missing block
------> if m/⏏bar/ { ... }
expecting any of:
block or pointy block
...
because Raku(do) would have interpreted the part between the second and third /s as the replacement double quoted string of what it interpreted as an s/.../.../ construct, leading it to barf when it encountered bar.
So, to recap, the s/// construct requires three slashes, not two.
(I'm ignoring syntactic variants of the construct such as, say, s [...] = '...'.)
I'm new to ruby so please excuse my ignorance :)
I just learned about eval and I read about its dark sides.
what I've read so far:
When is eval in Ruby justified?
Is 'eval' supposed to be nasty?
Ruby Eval and the Execution of Ruby Code
so what I have to do is to read a file in which there are some text such as /e/ 3 which will replace each e with a 3 after evaluation.
so here what i did so far:(working but..)
def evaluate_lines
result="elt"
IO.foreach("test.txt") do |reg|
reg=reg.chomp.delete(' ')
puts reg
result=result.gsub(eval(reg[0..2]),"#{reg[3..reg.length]}" )
p result
end
end
contents of the test.txt file
/e/ 3
/l/ 1
/t/ 7
/$/ !
/$/ !!
this only works because I know the length of the lines in the file.
so assuming my file has the following /a-z/ 3 my program would be not able to do what is expected from it.
Note
I tried using Regexp.new reg and this resulted with the following /\/e\/3/ which isn't very helpful in this case.
simple example to the `Regexp
str="/e/3"
result="elt"
result=result.gsub(Regexp.new str)
p result #outputs: #<Enumerator: "elt":gsub(/\/e\/3/)>
i already tried stripping off the slashes but even though this wont deliver the desired result thus the gsub() takes two parameters, such as this gsub(/e/, "3").
for the usage of the Regexp, I have already read Convert a string to regular expression ruby
While you can write something to parse that file, it rapidly gets complicated because you have to parse regular expressions. Consider /\/foo\\/.
There are a number of incomplete solutions. You can split on whitespace, but this will fail on /foo bar/.
re, replace = line.split(/\s+/, 2)
You can use a regex. Here's a first stab.
match = "/3/ 4".match(%r{^/(.*)/\s+(.+)})
This fails on escaped /, we need something more complex.
match = '/3\// 4'.match(%r{\A / ((?:[^/]|\\/)*) / \s+ (.+)}x)
I'm going to guess it was not your teacher's intent to have you parsing regexes. For the purposes of the assignment, splitting on whitespace is probably fine. You should clarify with your teacher.
This is a poor data format. It is non-standard, difficult to parse, and has limitations on the replacement. Even a tab-delimited file would be better.
There's little reason to use a non-standard format these days. The simplest thing is to use a standard data format for the file. YAML or JSON are the most obvious choices. For such simple data, I'd suggest JSON.
[
{ "re": "e", "replace": "3" },
{ "re": "l", "replace": "1" }
]
Parsing the file is trivial, use the built-in JSON library.
require 'json'
specs = JSON.load("test.json")
And then you can use them as a list of hashes.
specs.each do |spec|
# No eval necessary.
re = Regexp.new(spec["re"])
# `gsub!` replaces in place
result.gsub!(re, spec["replace"])
end
The data file is extensible. For example, if later you want to add regex options.
[
{ "re": "e", "replace": "3" },
{ "re": "l", "replace": "1", "options": ['IGNORECASE'] }
]
While the teacher may have specified a poor format, pushing back on bad requirements is good practice for being a developer.
Here's a really simple example that uses vi notation like s/.../.../ and s/.../.../g:
def rsub(text, spec)
_, mode, repl, with, flags = spec.match(%r[\A(.)\/((?:[^/]|\\/)*)/((?:[^/]|\\/)*)/(\w*)\z]).to_a
case (mode)
when 's'
if (flags.include?('g'))
text.gsub(Regexp.new(repl), with)
else
text.sub(Regexp.new(repl), with)
end
end
end
Note the matcher looks for non-slash characters ([^/]) or a literal-slash combination (\\/) and splits out the two parts accordingly.
Where you can get results like this:
rsub('sandwich', 's/and/or/')
# => "sorwich"
rsub('and/or', 's/\//,/')
# => "and,or"
rsub('stack overflow', 's/o/O/')
# => "stack Overflow"
rsub('stack overflow', 's/o/O/g')
# => "stack OverflOw"
The principle here is you can use a very simple regular expression to parse out your input regular expression and feed that cleaned up data into Regexp.new. There is absolutely no need for eval here, and if anything that severely limits what you can do.
With a little work you could alter that regular expression to parse what's in your existing file and make it do what you want.
I have text files containing lines like:
2/17/2018 400000098627 =2,000.0 $2.0994 $4,387.75
3/7/2018 1)0000006043 2,000.0 $2.0731 $4,332.78
3/26/2018 4 )0000034242 2,000.0 $2.1729 $4,541.36
4/17/2018 2)0000008516 2,000.0 $2.219 $4,637.71
I am matching them with /^\s*(\S+)\s+(?:[0-9|\)| ]+)+\s+([0-9|.|,]+)\s+\$/ But I also have some files with lines in a completely different format, which I match with a different regex. When I open a file I determine which format and assign $pat = '<regex-string>'; in a switch/case block:
$pat = '/^\s*(\S+)\s+(?:[0-9|\)| ]+)+\s+([0-9|.|,]+)\s+\$/'
But the ? character that introduces the non-capturing group I use to match repeats after the date and before the first currency amount causes the Perl interpreter to fail to compile the script, reporting on abort:
syntax error at ./report-dates-amounts line 28, near "}continue "
If I delete the ? character, or replace ? with \? escaped character, or first assign $q = '?' then replace ? with $q inside a " string assignment (ie. $pat = "/^\s*(\S+)\s+($q:[0-9|\)| ]+)+\s+([0-9|.|,]+)\s+\$/"; ) the script compiles and runs. If I assign the regex string outside the switch/case block that also works OK. Perl v5.26.1 .
My code also doesn't have any }continue in it, which as reported in the compilation failure is probably some kind of transformation of the switch/case code by Switch.pm into something native the compiler chokes on. Is this some kind of bug in Switch.pm? It fails even when I use given/when in exactly the same way.
#!/usr/local/bin/perl
use Switch;
# Edited for demo
switch($format)
{
# Format A eg:
# 2/17/2018 400000098627 =2,000.0 $2.0994 $4,387.75
# 3/7/2018 1)0000006043 2,000.0 $2.0731 $4,332.78
# 3/26/2018 4 )0000034242 2,000.0 $2.1729 $4,541.36
# 4/17/2018 2)0000008516 2,000.0 $2.219 $4,637.71
#
case /^(?:april|snow)$/i
{ # This is where the ? character breaks compilation:
$pat = '^\s*(\S+)\s+(?:[0-9|\)| ]+)+\s+\D?(\S+)\s+\$';
# WORKS:
# $pat = '^\s*(\S+)\s+(' .$q. ':[0-9|\)| ]+)+\s+\D' .$q. '(\S+)\s+\$';
}
# Format B
case /^(?:umberto|petro)$/i
{
$pat = '^(\S+)\s+.*Think 1\s+(\S+)\s+';
}
}
Don't use Switch. As mentionned by #choroba in the comments, Switch uses a source filter, which leads to mysterious and hard to debug errors, as you constated.
The module's documentation itself says:
In general, use given/when instead. It were introduced in perl 5.10.0. Perl 5.10.0 was released in 2007.
However, given/when is not necessarily a good option as it is experimental and likely to change in the future (it seems that this feature was almost removed from Perl v5.28; so you definitely don't want to start using it now if you can avoid it). A good alternative is to use for:
for ($format) {
if (/^(?:april|snow)$/i) {
...
}
elsif (/^(?:umberto|petro)$/i) {
...
}
}
It might look weird a first, but once you get used to it, it's actually reasonable in my opinion. Or, of course, you can use none of this options and just do:
sub pattern_from_format {
my $format = shift;
if ($format =~ /^(?:april|snow)$/i) {
return qr/^\s*(\S+)\s+(?:[0-9|\)| ]+)+\s+\D?(\S+)\s+\$/;
}
elsif ($format =~ /^(?:umberto|petro)$/i) {
return qr/^(\S+)\s+.*Think 1\s+(\S+)\s+/;
}
# Some error handling here maybe
}
If, for some reason, you still want to use Switch: use m/.../ instead of /.../.
I have no idea why this bug is happening, however, the documentation says:
Also, the presence of regexes specified with raw ?...? delimiters may cause mysterious errors. The workaround is to use m?...? instead.
Which I misread at first, and therefore tried to use m/../ instead of /../, which fixed the issue.
Another option instead of an if/elsif chain would be to loop over a hash which maps your regular expressions to the values which should be assigned to $pat:
#!/usr/local/bin/perl
my %switch = (
'^(?:april|snow)$' => '^\s*(\S+)\s+(?:[0-9|\)| ]+)+\s+\D?(\S+)\s+\$',
'^(?:umberto|petro)$' => '^(\S+)\s+.*Think 1\s+(\S+)\s+',
);
for my $re (keys %switch) {
if ($format =~ /$re/i) {
$pat = $switch{$re};
last;
}
}
For a more general case (i.e., if you're doing more than just assigning a string to a scalar) you could use the same general technique, but use coderefs as the values of your hash, thus allowing it to execute an arbitrary sub based on the match.
This approach can cover a pretty wide range of the functionality usually associated with switch/case constructs, but note that, because the conditions are pulled from the keys of a hash, they'll be evaluated in a random order. If you have data which could match more than one condition, you'll need to take extra precautions to handle that, such as having a parallel array with the conditions in the proper order or using Tie::IxHash instead of a regular hash.
I have files with sequences of conversations where speakers are tagged. The format of my files is:
<SPEAKER>John</SPEAKER>
I am John
<SPEAKER>Lisa</SPEAKER>
And I am Lisa
I am now looking to identify the first sequence in each document in which John speaks and Lisa speaks right afterwards (and I then want to then retain the entire part of the document that follows this sequence, including the sequence).
I built this regex:
^.*?(<SPEAKER>John<\/SPEAKER>.*?<SPEAKER>Lisa<\/SPEAKER>.*)
but it of course also captures the case where there is a sequence of speakers is John-Michael-Lisa, i.e. where there is someone speaking between John and Lisa.
How can I get the right match?
Here is a regex you can use to match what you describe:
(<SPEAKER>John<\/SPEAKER>(?:(?!<SPEAKER>).)*<SPEAKER>Lisa<\/SPEAKER>.*)
And a small demo showing that it works: https://regex101.com/r/iW8vS5/1
However, as both kchinger and owler mentioned, regex probably isn't the best way to do this. A regex solution would likely be significantly slower than a small snippet of code for any long document.
This isn't a purely regex solution, maybe someone else can do that, but instead I wrote a small loop to check each line. If it finds what you want, it will keep the rest of the document. You would need to feed it the correct sequence if it wasn't a full document. A regex to do what you want might be kind of slow since it will be relatively complicated, but you'd have to benchmark against a pure regex solution (if someone comes up with one) if speed is important.
edit to note: ?!Lisa is a negative lookahead if you haven't seen it. Some combined negative lookaheads might be what you need to use to do it in one regex, but good luck reading it later.
open(my $input,"input2.txt")||die "can't open the file";
my $output = "";
my $wanted = 0;
while(<$input>)
{
$wanted = 1 if(/<SPEAKER>John<\/SPEAKER>/);
$wanted = 2 if(/<SPEAKER>Lisa<\/SPEAKER>/ && $wanted == 1);
if(/<SPEAKER>(?!Lisa)/ && /<SPEAKER>(?!John)/ && $wanted == 1)
{
$wanted = 0;
last;
}
$output = $output . $_ if($wanted);
}
print "$output" if $wanted;
--Edit-- The current answers have some useful ideas but I want something more complete that I can 100% understand and reuse; that's why I set a bounty. Also ideas that work everywhere are better for me than not standard syntax like \K
This question is about how I can match a pattern except some situations s1 s2 s3. I give a specific example to show my meaning but prefer a general answer I can 100% understand so I can reuse it in other situations.
Example
I want to match five digits using \b\d{5}\b but not in three situations s1 s2 s3:
s1: Not on a line that ends with a period like this sentence.
s2: Not anywhere inside parens.
s3: Not inside a block that starts with if( and ends with //endif
I know how to solve any one of s1 s2 s3 with a lookahead and lookbehind, especially in C# lookbehind or \K in PHP.
For instance
s1 (?m)(?!\d+.*?\.$)\d+
s3 with C# lookbehind (?<!if\(\D*(?=\d+.*?//endif))\b\d+\b
s3 with PHP \K (?:(?:if\(.*?//endif)\D*)*\K\d+
But the mix of conditions together makes my head explode. Even more bad news is that I may need to add other conditions s4 s5 at another time.
The good news is, I don't care if I process the files using most common languages like PHP, C#, Python or my neighbor's washing machine. :) I'm pretty much a beginner in Python & Java but interested to learn if it has a solution.
So I came here to see if someone think of a flexible recipe.
Hints are okay: you don't need to give me full code. :)
Thank you.
Hans, I'll take the bait and flesh out my earlier answer. You said you want "something more complete" so I hope you won't mind the long answer—just trying to please. Let's start with some background.
First off, this is an excellent question. There are often questions about matching certain patterns except in certain contexts (for instance, within a code block or inside parentheses). These questions often give rise to fairly awkward solutions. So your question about multiple contexts is a special challenge.
Surprise
Surprisingly, there is at least one efficient solution that is general, easy to implement and a pleasure to maintain. It works with all regex flavors that allow you to inspect capture groups in your code. And it happens to answer a number of common questions that may at first sound different from yours: "match everything except Donuts", "replace all but...", "match all words except those on my mom's black list", "ignore tags", "match temperature unless italicized"...
Sadly, the technique is not well known: I estimate that in twenty SO questions that could use it, only one has one answer that mentions it—which means maybe one in fifty or sixty answers. See my exchange with Kobi in the comments. The technique is described in some depth in this article which calls it (optimistically) the "best regex trick ever". Without going into as much detail, I'll try to give you a firm grasp of how the technique works. For more detail and code samples in various languages I encourage you to consult that resource.
A Better-Known Variation
There is a variation using syntax specific to Perl and PHP that accomplishes the same. You'll see it on SO in the hands of regex masters such as CasimiretHippolyte and HamZa. I'll tell you more about this below, but my focus here is on the general solution that works with all regex flavors (as long as you can inspect capture groups in your code).
Thanks for all the background, zx81... But what's the recipe?
Key Fact
The method returns the match in Group 1 capture. It does not care at
all about the overall match.
In fact, the trick is to match the various contexts we don't want (chaining these contexts using the | OR / alternation) so as to "neutralize them". After matching all the unwanted contexts, the final part of the alternation matches what we do want and captures it to Group 1.
The general recipe is
Not_this_context|Not_this_either|StayAway|(WhatYouWant)
This will match Not_this_context, but in a sense that match goes into a garbage bin, because we won't look at the overall matches: we only look at Group 1 captures.
In your case, with your digits and your three contexts to ignore, we can do:
s1|s2|s3|(\b\d+\b)
Note that because we actually match s1, s2 and s3 instead of trying to avoid them with lookarounds, the individual expressions for s1, s2 and s3 can remain clear as day. (They are the subexpressions on each side of a | )
The whole expression can be written like this:
(?m)^.*\.$|\([^\)]*\)|if\(.*?//endif|(\b\d+\b)
See this demo (but focus on the capture groups in the lower right pane.)
If you mentally try to split this regex at each | delimiter, it is actually only a series of four very simple expressions.
For flavors that support free-spacing, this reads particularly well.
(?mx)
### s1: Match line that ends with a period ###
^.*\.$
| ### OR s2: Match anything between parentheses ###
\([^\)]*\)
| ### OR s3: Match any if(...//endif block ###
if\(.*?//endif
| ### OR capture digits to Group 1 ###
(\b\d+\b)
This is exceptionally easy to read and maintain.
Extending the regex
When you want to ignore more situations s4 and s5, you add them in more alternations on the left:
s4|s5|s1|s2|s3|(\b\d+\b)
How does this work?
The contexts you don't want are added to a list of alternations on the left: they will match, but these overall matches are never examined, so matching them is a way to put them in a "garbage bin".
The content you do want, however, is captured to Group 1. You then have to check programmatically that Group 1 is set and not empty. This is a trivial programming task (and we'll later talk about how it's done), especially considering that it leaves you with a simple regex that you can understand at a glance and revise or extend as required.
I'm not always a fan of visualizations, but this one does a good job of showing how simple the method is. Each "line" corresponds to a potential match, but only the bottom line is captured into Group 1.
Debuggex Demo
Perl/PCRE Variation
In contrast to the general solution above, there exists a variation for Perl and PCRE that is often seen on SO, at least in the hands of regex Gods such as #CasimiretHippolyte and #HamZa. It is:
(?:s1|s2|s3)(*SKIP)(*F)|whatYouWant
In your case:
(?m)(?:^.*\.$|\([^()]*\)|if\(.*?//endif)(*SKIP)(*F)|\b\d+\b
This variation is a bit easier to use because the content matched in contexts s1, s2 and s3 is simply skipped, so you don't need to inspect Group 1 captures (notice the parentheses are gone). The matches only contain whatYouWant
Note that (*F), (*FAIL) and (?!) are all the same thing. If you wanted to be more obscure, you could use (*SKIP)(?!)
demo for this version
Applications
Here are some common problems that this technique can often easily solve. You'll notice that the word choice can make some of these problems sound different while in fact they are virtually identical.
How can I match foo except anywhere in a tag like <a stuff...>...</a>?
How can I match foo except in an <i> tag or a javascript snippet (more conditions)?
How can I match all words that are not on this black list?
How can I ignore anything inside a SUB... END SUB block?
How can I match everything except... s1 s2 s3?
How to Program the Group 1 Captures
You didn't as for code, but, for completion... The code to inspect Group 1 will obviously depend on your language of choice. At any rate it shouldn't add more than a couple of lines to the code you would use to inspect matches.
If in doubt, I recommend you look at the code samples section of the article mentioned earlier, which presents code for quite a few languages.
Alternatives
Depending on the complexity of the question, and on the regex engine used, there are several alternatives. Here are the two that can apply to most situations, including multiple conditions. In my view, neither is nearly as attractive as the s1|s2|s3|(whatYouWant) recipe, if only because clarity always wins out.
1. Replace then Match.
A good solution that sounds hacky but works well in many environments is to work in two steps. A first regex neutralizes the context you want to ignore by replacing potentially conflicting strings. If you only want to match, then you can replace with an empty string, then run your match in the second step. If you want to replace, you can first replace the strings to be ignored with something distinctive, for instance surrounding your digits with a fixed-width chain of ###. After this replacement, you are free to replace what you really wanted, then you'll have to revert your distinctive ### strings.
2. Lookarounds.
Your original post showed that you understand how to exclude a single condition using lookarounds. You said that C# is great for this, and you are right, but it is not the only option. The .NET regex flavors found in C#, VB.NET and Visual C++ for example, as well as the still-experimental regex module to replace re in Python, are the only two engines I know that support infinite-width lookbehind. With these tools, one condition in one lookbehind can take care of looking not only behind but also at the match and beyond the match, avoiding the need to coordinate with a lookahead. More conditions? More lookarounds.
Recycling the regex you had for s3 in C#, the whole pattern would look like this.
(?!.*\.)(?<!\([^()]*(?=\d+[^)]*\)))(?<!if\(\D*(?=\d+.*?//endif))\b\d+\b
But by now you know I'm not recommending this, right?
Deletions
#HamZa and #Jerry have suggested I mention an additional trick for cases when you seek to just delete WhatYouWant. You remember that the recipe to match WhatYouWant (capturing it into Group 1) was s1|s2|s3|(WhatYouWant), right? To delete all instance of WhatYouWant, you change the regex to
(s1|s2|s3)|WhatYouWant
For the replacement string, you use $1. What happens here is that for each instance of s1|s2|s3 that is matched, the replacement $1 replaces that instance with itself (referenced by $1). On the other hand, when WhatYouWant is matched, it is replaced by an empty group and nothing else — and therefore deleted. See this demo, thank you #HamZa and #Jerry for suggesting this wonderful addition.
Replacements
This brings us to replacements, on which I'll touch briefly.
When replacing with nothing, see the "Deletions" trick above.
When replacing, if using Perl or PCRE, use the (*SKIP)(*F) variation mentioned above to match exactly what you want, and do a straight replacement.
In other flavors, within the replacement function call, inspect the match using a callback or lambda, and replace if Group 1 is set. If you need help with this, the article already referenced will give you code in various languages.
Have fun!
No, wait, there's more!
Ah, nah, I'll save that for my memoirs in twenty volumes, to be released next Spring.
Do three different matches and handle the combination of the three situations using in-program conditional logic. You don't need to handle everything in one giant regex.
EDIT: let me expand a bit because the question just became more interesting :-)
The general idea you are trying to capture here is to match against a certain regex pattern, but not when there are certain other (could be any number) patterns present in the test string. Fortunately, you can take advantage of your programming language: keep the regexes simple and just use a compound conditional. A best practice would be to capture this idea in a reusable component, so let's create a class and a method that implement it:
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;
public class MatcherWithExceptions {
private string m_searchStr;
private Regex m_searchRegex;
private IEnumerable<Regex> m_exceptionRegexes;
public string SearchString {
get { return m_searchStr; }
set {
m_searchStr = value;
m_searchRegex = new Regex(value);
}
}
public string[] ExceptionStrings {
set { m_exceptionRegexes = from es in value select new Regex(es); }
}
public bool IsMatch(string testStr) {
return (
m_searchRegex.IsMatch(testStr)
&& !m_exceptionRegexes.Any(er => er.IsMatch(testStr))
);
}
}
public class App {
public static void Main() {
var mwe = new MatcherWithExceptions();
// Set up the matcher object.
mwe.SearchString = #"\b\d{5}\b";
mwe.ExceptionStrings = new string[] {
#"\.$"
, #"\(.*" + mwe.SearchString + #".*\)"
, #"if\(.*" + mwe.SearchString + #".*//endif"
};
var testStrs = new string[] {
"1." // False
, "11111." // False
, "(11111)" // False
, "if(11111//endif" // False
, "if(11111" // True
, "11111" // True
};
// Perform the tests.
foreach (var ts in testStrs) {
System.Console.WriteLine(mwe.IsMatch(ts));
}
}
}
So above, we set up the search string (the five digits), multiple exception strings (your s1, s2 and s3), and then try to match against several test strings. The printed results should be as shown in the comments next to each test string.
Your requirement that it's not inside parens in impossible to satify for all cases.
Namely, if you can somehow find a ( to the left and ) to the right, it doesn't always mean you are inside parens. Eg.
(....) + 55555 + (.....) - not inside parens yet there are ( and ) to left and right
Now you might think yourself clever and look for ( to the left only if you don't encounter ) before and vice versa to the right. This won't work for this case:
((.....) + 55555 + (.....)) - inside parens even though there are closing ) and ( to left and to right.
It is impossible to find out if you are inside parens using regex, as regex can't count how many parens have been opened and how many closed.
Consider this easier task: using regex, find out if all (possibly nested) parens in a string are closed, that is for every ( you need to find ). You will find out that it's impossible to solve and if you can't solve that with regex then you can't figure out if a word is inside parens for all cases, since you can't figure out at a some position in string if all preceeding ( have a corresponding ).
Hans if you don't mind I used your neighbor's washing machine called perl :)
Edited:
Below a pseudo code:
loop through input
if line contains 'if(' set skip=true
if skip= true do nothing
else
if line match '\b\d{5}\b' set s0=true
if line does not match s1 condition set s1=true
if line does not match s2 condition set s2=true
if s0,s1,s2 are true print line
if line contains '//endif' set skip=false
Given the file input.txt:
tiago#dell:~$ cat input.txt
this is a text
it should match 12345
if(
it should not match 12345
//endif
it should match 12345
it should not match 12345.
it should not match ( blabla 12345 blablabla )
it should not match ( 12345 )
it should match 12345
And the script validator.pl:
tiago#dell:~$ cat validator.pl
#! /usr/bin/perl
use warnings;
use strict;
use Data::Dumper;
sub validate_s0 {
my $line = $_[0];
if ( $line =~ \d{5/ ){
return "true";
}
return "false";
}
sub validate_s1 {
my $line = $_[0];
if ( $line =~ /\.$/ ){
return "false";
}
return "true";
}
sub validate_s2 {
my $line = $_[0];
if ( $line =~ /.*?\(.*\d{5.*?\).*/ ){
return "false";
}
return "true";
}
my $skip = "false";
while (<>){
my $line = $_;
if( $line =~ /if\(/ ){
$skip = "true";
}
if ( $skip eq "false" ) {
my $s0_status = validate_s0 "$line";
my $s1_status = validate_s1 "$line";
my $s2_status = validate_s2 "$line";
if ( $s0_status eq "true"){
if ( $s1_status eq "true"){
if ( $s2_status eq "true"){
print "$line";
}
}
}
}
if ( $line =~ /\/\/endif/) {
$skip="false";
}
}
Execution:
tiago#dell:~$ cat input.txt | perl validator.pl
it should match 12345
it should match 12345
it should match 12345
Not sure if this would help you or not, but I am providing a solution considering the following assumptions -
You need an elegant solution to check all the conditions
Conditions can change in future and anytime.
One condition should not depend on others.
However I considered also the following -
The file given has minimal errors in it. If it doe then my code might need some modifications to cope with that.
I used Stack to keep track of if( blocks.
Ok here is the solution -
I used C# and with it MEF (Microsoft Extensibility Framework) to implement the configurable parsers. The idea is, use a single parser to parse and a list of configurable validator classes to validate the line and return true or false based on the validation. Then you can add or remove any validator anytime or add new ones if you like. So far I have already implemented for S1, S2 and S3 you mentioned, check classes at point 3. You have to add classes for s4, s5 if you need in future.
First, Create the Interfaces -
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace FileParserDemo.Contracts
{
public interface IParser
{
String[] GetMatchedLines(String filename);
}
public interface IPatternMatcher
{
Boolean IsMatched(String line, Stack<string> stack);
}
}
Then comes the file reader and checker -
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using FileParserDemo.Contracts;
using System.ComponentModel.Composition.Hosting;
using System.ComponentModel.Composition;
using System.IO;
using System.Collections;
namespace FileParserDemo.Parsers
{
public class Parser : IParser
{
[ImportMany]
IEnumerable<Lazy<IPatternMatcher>> parsers;
private CompositionContainer _container;
public void ComposeParts()
{
var catalog = new AggregateCatalog();
catalog.Catalogs.Add(new AssemblyCatalog(typeof(IParser).Assembly));
_container = new CompositionContainer(catalog);
try
{
this._container.ComposeParts(this);
}
catch
{
}
}
public String[] GetMatchedLines(String filename)
{
var matched = new List<String>();
var stack = new Stack<string>();
using (StreamReader sr = File.OpenText(filename))
{
String line = "";
while (!sr.EndOfStream)
{
line = sr.ReadLine();
var m = true;
foreach(var matcher in this.parsers){
m = m && matcher.Value.IsMatched(line, stack);
}
if (m)
{
matched.Add(line);
}
}
}
return matched.ToArray();
}
}
}
Then comes the implementation of individual checkers, the class names are self explanatory, so I don't think they need more descriptions.
using FileParserDemo.Contracts;
using System;
using System.Collections.Generic;
using System.ComponentModel.Composition;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
namespace FileParserDemo.PatternMatchers
{
[Export(typeof(IPatternMatcher))]
public class MatchAllNumbers : IPatternMatcher
{
public Boolean IsMatched(String line, Stack<string> stack)
{
var regex = new Regex("\\d+");
return regex.IsMatch(line);
}
}
[Export(typeof(IPatternMatcher))]
public class RemoveIfBlock : IPatternMatcher
{
public Boolean IsMatched(String line, Stack<string> stack)
{
var regex = new Regex("if\\(");
if (regex.IsMatch(line))
{
foreach (var m in regex.Matches(line))
{
//push the if
stack.Push(m.ToString());
}
//ignore current line, and will validate on next line with stack
return true;
}
regex = new Regex("//endif");
if (regex.IsMatch(line))
{
foreach (var m in regex.Matches(line))
{
stack.Pop();
}
}
return stack.Count == 0; //if stack has an item then ignoring this block
}
}
[Export(typeof(IPatternMatcher))]
public class RemoveWithEndPeriod : IPatternMatcher
{
public Boolean IsMatched(String line, Stack<string> stack)
{
var regex = new Regex("(?m)(?!\\d+.*?\\.$)\\d+");
return regex.IsMatch(line);
}
}
[Export(typeof(IPatternMatcher))]
public class RemoveWithInParenthesis : IPatternMatcher
{
public Boolean IsMatched(String line, Stack<string> stack)
{
var regex = new Regex("\\(.*\\d+.*\\)");
return !regex.IsMatch(line);
}
}
}
The program -
using FileParserDemo.Contracts;
using FileParserDemo.Parsers;
using System;
using System.Collections.Generic;
using System.ComponentModel.Composition;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace FileParserDemo
{
class Program
{
static void Main(string[] args)
{
var parser = new Parser();
parser.ComposeParts();
var matches = parser.GetMatchedLines(Path.GetFullPath("test.txt"));
foreach (var s in matches)
{
Console.WriteLine(s);
}
Console.ReadLine();
}
}
}
For testing I took #Tiago's sample file as Test.txt which had the following lines -
this is a text
it should match 12345
if(
it should not match 12345
//endif
it should match 12345
it should not match 12345.
it should not match ( blabla 12345 blablabla )
it should not match ( 12345 )
it should match 12345
Gives the output -
it should match 12345
it should match 12345
it should match 12345
Don't know if this would help you or not, I do had a fun time playing with it.... :)
The best part with it is that, for adding a new condition all you have to do is provide an implementation of IPatternMatcher, it will automatically get called and thus will validate.
Same as #zx81's (*SKIP)(*F) but with using a negative lookahead assertion.
(?m)(?:if\(.*?\/\/endif|\([^()]*\))(*SKIP)(*F)|\b\d+\b(?!.*\.$)
DEMO
In python, i would do easily like this,
import re
string = """cat 123 sat.
I like 000 not (456) though 111 is fine
222 if( //endif if(cat==789 stuff //endif 333"""
for line in string.split('\n'): # Split the input according to the `\n` character and then iterate over the parts.
if not line.endswith('.'): # Don't consider the part which ends with a dot.
for i in re.split(r'\([^()]*\)|if\(.*?//endif', line): # Again split the part by brackets or if condition which endswith `//endif` and then iterate over the inner parts.
for j in re.findall(r'\b\d+\b', i): # Then find all the numbers which are present inside the inner parts and then loop through the fetched numbers.
print(j) # Prints the number one ny one.
Output:
000
111
222
333