Lex only detects symbols when there is whitespace between them - regex

Question

I want Lex, when given an input of "foo+1", to first return the identifier "foo", then the character '+', and then the integer 1. This works if I lex "foo + 1", but for some reason with the grammar I have, it doesn't work if I omit the spaces, and it skips over the '+', just returning "foo" and then 1. I can't figure out why. Is there anything here that seems problematic?
%{
#include "expression.h"
#include "picoScanner.h"
static int block_comment_num = 0;
static char to_char(char *str);
int yylineno = 0;
%}
%option nodefault yyclass="FlexScanner" noyywrap c++
%x LINE_COMMENT
%x BLOCK_COMMENT
%%
Any { return pico::BisonParser::token::ANY; }
Int { return pico::BisonParser::token::INT; }
Float { return pico::BisonParser::token::FLOAT; }
Char { return pico::BisonParser::token::CHAR; }
List { return pico::BisonParser::token::LIST; }
Array { return pico::BisonParser::token::ARRAY; }
Table { return pico::BisonParser::token::TABLE; }
alg { return pico::BisonParser::token::ALG; }
if { return pico::BisonParser::token::IF; }
then { return pico::BisonParser::token::THEN; }
else { return pico::BisonParser::token::ELSE; }
is { return pico::BisonParser::token::IS; }
or { return pico::BisonParser::token::OR; }
and { return pico::BisonParser::token::AND; }
not { return pico::BisonParser::token::NOT; }
when { return pico::BisonParser::token::WHEN; }
[A-Z][a-zA-Z0-9_]* { yylval->strval = new std::string(yytext);
return pico::BisonParser::token::TYPENAME; }
[a-z_][a-zA-Z0-9_]* { printf("saw '%s'\n", yytext); yylval->strval = new std::string(yytext);
return pico::BisonParser::token::ID; }
"==" { return pico::BisonParser::token::EQ; }
"<=" { return pico::BisonParser::token::LEQ; }
">=" { return pico::BisonParser::token::GEQ; }
"!=" { return pico::BisonParser::token::NEQ; }
"->" { return pico::BisonParser::token::RETURN; }
[\+\-\*/%] { return yytext[0]; }
[-+]?[0-9]+ { yylval->ival = atoi(yytext);
return pico::BisonParser::token::INT_LITERAL; }
([0-9]+|([0-9]*\.[0-9]+)([eE][-+]?[0-9]+)?) { yylval->fval = atof(yytext);
return pico::BisonParser::token::FLOAT_LITERAL; }
\"(\\.|[^\\"])*\" { yylval->strval = new std::string(strndup(yytext+1, strlen(yytext) - 2));
return pico::BisonParser::token::STRING_LITERAL; }
\'(\\.|[^\\'])*\' { yylval->cval = to_char(yytext+1);
return pico::BisonParser::token::CHAR_LITERAL; }
[ \t\r]+ { /* ignore */ }
\n { yylineno++; }
. { printf("~~~~~~~~~~munched %s\n", yytext); return yytext[0]; }
%%
static char to_char(char *str) {
if (strlen(str) <= 1) {
fprintf(stderr, "Error: empty character constant (line %d)\n", yylineno);
exit(1);
} else if (str[0] != '\\') {
return str[0];
} else {
if (strlen(str) == 1) {
fprintf(stderr, "Error: somehow we got a single slash character\n");
exit(1);
}
switch (str[1]) {
case 'n': return '\n';
case 'r': return '\r';
case 't': return '\t';
case 'a': return '\a';
case 'b': return '\b';
case 'f': return '\f';
case 'v': return '\v';
case '\'': return '\'';
case '"': return '"';
case '\\': return '\\';
case '?': return '\?';
case 'x':
fprintf(stderr, "Error: unicode not yet supported (line %d)\n", yylineno);
exit(1);
default:
fprintf(stderr, "Error: unrecognized escape sequence '\\%c' (line %d)\n",
str[1], yylineno);
exit(1);
}
}
}

Answer 1

I am not familair with lex, but I'm pretty sure the following causes the error:
[-+]?[0-9]+ { yylval->ival = atoi(yytext);
return pico::BisonParser::token::INT_LITERAL; }
foo is parsed as an identifier, but then "+0" is parsed as an int literal (due to the atoi conversion, the sign is discarded).
It is probably a good idea to only consider unsigned numeric literals at a lexer level, and handle signs at the level of the parser (treating the + and - tokens differently depending on their context).
Not only does this resolve the ambiguity, but it also enables you to "correctly" (in the sense that these are legal in C, C++, Java etc.) parse integer literals such as - 5 instead of -5.
Moreover: are the escaping backslashs in the arithmetic operator rule really necessary? Afaik, the only characters with special meaning inside a character class are -, ^, and ] (but I might be wrong).

Answer 2

Looks to me like it's matching foo+1 as foo and +1 (an INT_LITERAL). See related thread: Is it possible to set priorities for rules to avoid the "longest-earliest" matching pattern?
You could add an explicit rule to match + as a token, otherwise it sounds like Lex is going to take the longest match it can (+1 is longer than +).