Non-Greedy Regular Expression Matching in Flex - regex

Question

I have just started with Flex and can't seem to figure out how to match the following Expression :
"Dog".*"Cat"
------------------
Input :
Dog Ca Cat Cc Cat
------------------
Output:
Dog Ca Cat Cc Cat
But I want a non-greedy matching, with the following output :
Output:
Dog Ca Cat
How can this be acheived on Flex ?
EDIT
Tried the following :
%%
Dog.*Cat/.*Cat printf("Matched : ||%s||", yytext);
dog.*cat printf("Matched : ||%s||", yytext);
dOg[^c]*cAt printf("Matched : ||%s||", yytext);
DOG.*?CAT printf("Matched : ||%s||", yytext);
%%
Input :
Dog Ca Cat Cc Cat
dog Ca cat Cc cat
dOg Ca cAt Cc cAt
DOG CA CAT CC CAT
Output :
Matched : ||Dog Ca Cat Cc Cat||
Matched : ||dog Ca cat Cc cat||
Matched : ||dOg Ca cAt|| Cc cAt
Matched : ||DOG CA CAT CC CAT||
Also receiving a warning :
lex4.l:2: warning, dangerous trailing context
Flex Version :
flex 2.5.35 Apple(flex-31)

Answer 1

This is quite a common issue with using the lex/flex tools that stumps beginners (and sometime non-beginners). There are two solutions to the problem that require two different advanced features of the tools. A phrase like dog ... cat is much the same problem as matching comments in various programming languages, such as the C comment form /* ... */ or even 'comment' ... 'tnemmoc'. These have exactly the same characteristics as your example. Consider the following C code:
/* This is a comment */ "This is a String */"
A greedy lexical match of that would match the wrong comment terminator (and is a good test of a student lexer BTW!).
There are suggested solutions on several university compiler courses. The one that explains it well is here (at Manchester). Which cites a couple of good books which also cover the problems:
J.Levine, T.Mason & D.Brown: Lex and Yacc (2nd ed.)
M.E.Lesk & E.Schmidt: Lex - A Lexical Analyzer Generator
The two techniques described are to use Start Conditions to explicity specify the state machine, or manual input to read characters directly.
For your cat ... dog problem they can be programmed in the following ways:
Start Conditions
In this solution we need several states. The keyword dog causes causes it to enter the DOG state which continues until a letter c is encountered. This then enters the LETTERC state which must be followed by a letter a, if not the DOG state continues; a letter a causes the CAT state to be entered which must be followed by a letter t which causes the entire phrase to be matched and returns to the INITIAL state. The yymore causes the entire dog ... cat text to be retained for use.
%x DOG LETTERC CAT
d [dD]
o [oO]
g [gG]
c [cC]
a [aA]
t [tT]
ws [ \t\r\n]+
%%
<INITIAL>{d}{o}{g} {
BEGIN(DOG);
printf("DOG\n");
yymore();
}
<DOG>[^cC]*{c} {
printf("C: %s\n",yytext);
yymore();
BEGIN(LETTERC);
}
<LETTERC>{a} {
printf("A: %s\n",yytext);
yymore();
BEGIN(CAT);
}
<LETTERC>[^aA] {
BEGIN(DOG);
yymore();
}
<CAT>{t} {
printf("CAT: %s\n",yytext);
BEGIN(INITIAL);
}
<CAT>[^tT] {
BEGIN(DOG);
yymore();
}
<INITIAL>{ws} /* skip */ ;
Manual Input
The Manual input method just matches the start phrase dog and the enters C code which swallows up input characters until the desired cat sequence is encountered. (I did not bother with both upper and lower case letters). The problem with this solution is that it is hard to retain the input text value in yytext for later use in the parser. It discards it, which would be OK if the construct is a comment, but no so useful otherwise.
d [dD]
o [oO]
g [gG]
ws [ \t\r\n]+
%%
{d}{o}{g} {
register int c;
for ( ; ; )
{
/* Not dealt with upper case .. left as an exercise */
while ( (c = input()) != 'c' &&
c != EOF )
; /* eat up text of dog */
if ( c == 'c' )
{
if ( ( c = input()) == 'a' )
if ( (c = input()) == 't' )
break; /* found the end */
}
if ( c == EOF )
{
REJECT;
break;
}
}
/* because we have used input() yytext always contains "dog" */
printf("cat: %s\n", yytext);
}
{ws} /* skip */ ;
(Both these solutions have been tested)

Answer 2

A good question. Here is a pure regex solution, without using the non-greedy .*? syntax:
Dog([^C]|C+(aC+)*([^Ca]|a[^Ct]))*C+(aC+)*at

Answer 3

Here's a minimal C++ flex lexer for this problem. The key for nongreedy matching is start conditions as mentioned in the flex manual and elsewhere.
A start condition is just another state for the lexer. When nongreedy matching is needed there's some pattern that needs to terminate the matching on its first occurrence
In general regardless of state if you're looking for a target string or pattern you just need to make sure there are no other more general patterns that could match a longer stretch of input containing the target pattern
Start conditions help when the target pattern is conditional and needs to be enabled after some earlier match. You turn on the start condition to enable matching the target pattern and turn it off by resetting the state to 0 or INITIAL - or switching to another state for even more conditional matching
States are switched with BEGIN - there's also a state stack for use through yy_push_state and yy_pop_state
There are many examples of start conditions in the flex manual
Here are the flex rules that show nongreedy matching with flex start conditions - the lexer matches the first occurrence of dog on a line till the first occurrence of cat - matching is case insensitive
The complete file is posted at the end - for people unfamiliar with flex please note many lines begin with a space - this is not accidental and required by flex
%%
/* flex rules section */
string match;
dog {
// found a dog, change state to HAVE_DOG to start looking for a cat
BEGIN(HAVE_DOG);
// save the found dog
match = yytext;
}
/* save and keep going till cat is found */
<HAVE_DOG>. match += yytext;
<HAVE_DOG>cat {
// save the found cat
match += yytext;
// output the matched dog and cat
cout << match << "\n";
// ignore rest of line
BEGIN(SKIP_LINE);
}
/* no cat on this line, reset state */
<HAVE_DOG>\n BEGIN(0);
/* rules to ignore rest of the line then reset state */
<SKIP_LINE>{
.*
\n BEGIN(0);
}
/* nothing to do yet */
.|\n
Here's some test input
$ cat dogcat.in.txt
Dog Ca Cat Cc Cat
dog Ca cat Cc cat
dOg Ca cAt Cc cAt
DOG CA CAT CC CAT
cat dog dog cat cat
dog kitten cat dog cat
dig cat dog can dog cut
dig dug dog cow cat cat
doc dogcat catdog
dog dog dog
cat cat cat
Build with
flex -o dogcat.flex.cpp dogcat.flex.l && g++ -o dogcat dogcat.flex.cpp
Run with
$ ./dogcat < dogcat.in.txt
Dog Ca Cat
dog Ca cat
dOg Ca cAt
DOG CA CAT
dog dog cat
dog kitten cat
dog cow cat
dogcat
The complete flex file
/* dogcat.flex.l */
/*
Build with:
flex -o dogcat.flex.cpp dogcat.flex.l && g++ -o dogcat dogcat.flex.cpp
*/
/*
A minimal C++ flex lexer that shows nongreedy matching with flex
start conditions
matches the first occurrence of dog on a line till the first
occurrence of cat
matching is case insensitive
*/
/* C++ lexer using yyFlexLexer in FlexLexer.h */
%option c++
/* case-insensitive patterns */
%option case-insensitive
/* generate main function for executable */
%option main
/* all input must be matched, no echo by default */
%option nodefault
/* debug output with lexer.set_debug(1) */
%option debug
/* start condition means dog was matched */
%x HAVE_DOG
/* start condition means to ignore remaining line */
%x SKIP_LINE
%{
#include <string>
#include <iostream>
// C++ flex lexer class
// needed because header itself has no guard
#ifndef yyFlexLexerOnce
# include <FlexLexer.h>
#endif
using namespace std;
namespace {
// the C++ lexer class from flex
yyFlexLexer lexer;
// main generated by flex still calls free yylex function even for C++ lexer
int yylex() {
return lexer.yylex();
}
}
%}
%%
/* flex rules section */
string match;
dog {
// found a dog, change state to HAVE_DOG to start looking for a cat
BEGIN(HAVE_DOG);
// save the found dog
match = yytext;
}
/* save and keep going till cat is found */
<HAVE_DOG>. match += yytext;
<HAVE_DOG>cat {
// save the found cat
match += yytext;
// output the matched dog and cat
cout << match << "\n";
// ignore rest of line
BEGIN(SKIP_LINE);
}
/* no cat on this line, reset state */
<HAVE_DOG>\n BEGIN(0);
/* rules to ignore rest of the line then reset state */
<SKIP_LINE>{
.*
\n BEGIN(0);
}
/* nothing to do yet */
.|\n