So I have a grammar that includes the empty string. The grammar is something like this:
S->ε
S->expression ;; S
I'm getting the error "No more states to discard" when I run my parser so I believe I'm not representing the empty string correctly. So how would I go about representing it, specifically in the lexer .mll file?
I know I need to make a rule for it so I think I have that down. This is what I think it should look like for the parser .mly file, excluding the stuff for expression.
s:
| EMPTY_STRING { [] }
| expression SEMICOLON s { $1::$3 }
You're thinking of epsilon as a token, but it's not a token. It's a 0-length sequence of tokens. Since there are no tokens there, it's not something your scanner needs to know about. Just the parser needs to know about it.
Here's a grammar something like what I think you want:
%token X
%token SEMICOLON
%token EOF
%start main
%type <char list> main
%%
main :
s EOF { $1 }
s :
| epsilon { $1 }
| X SEMICOLON s { 'x' :: $3 }
epsilon :
{ [] }
Note that epsilon is a non-terminal (not a token). Its definition is an empty sequence of symbols.
Related
I am trying to write Grammar for java specification
for example:-
COMPILATION_UNIT: PACKAGE_DEC? IMPORT_DECS? TYPE_DECS?
but it doesn't work
I have the following error:
invalid character: `?'
for each question mark I use in my file.y
I know that Bison has special characters and it should handle it
Please help
Bison does not allow a ? meaning that the prior token is optional, you have to write out the grammar with the optional elements:
package_decl_opt: %empty
| SOME_TOKEN
;
package: package)_dec_opt TOKEN_PACKAGE TOKEN_IDENTIFIER
;
would allow both of the following:
SOME_TOKEN TOKEN_PACKAGE TOKEN_IDENTIFIER
TOKEN_PACKAGE TOKEN_IDENTIFIER
As you have seen, bison does not implement the ? regular expression optionality operator. Nor does it implement + or * repetition operators. That's because the right-hand sides of productions in contex-free grammars are not regular expressions.
Yacc/bison context-free grammars do allow the | alternation operator, but as an abbreviation:
a : b | c
Is exactly the same as writing
a : b
a : c
and semantic actions only apply to the alternative in which they are specified, so that
a : b | c { /* C action; */ }
Is equivalent to:
a : b { /* Implicit default action*/ }
a : c { /* C action; */ }
It is tempting to create X_opt non-terminals to capture the semantics of X?:
X_opt: X | %empty { $$ = default_value; }
In many simple cases that will work fine, but there are also many grammars in which that introduces an unnecessary shift-reduce conflict. Consider, for example:
label: IDENT ':'
label_opt: label | %empty
statement: label_opt expr
Since expr can start with an identifier, there is no way to know if an IDENT token starts a label or if it starts an expr following an empty label_opt. But LR(1) requires that the empty label_opt be reduced before the IDENT is consumed. So the above grammar is LR(2) and cannot be correctly parsed by an LR(1) parser.
That problem does not occur without the use of the label_opt shortcut:
label: IDENT ':'
statement: label expr
| expr
Since the parser now does not have decide between label and expr before the ':' is encountered.
I have been tasked with a project that involves me taking a Grammar (in BNF form) and creating a lexical scanner (using lex) and a parser (using bison). I've never worked with any of these programs and I think a good reference would be to see how these items are created from a grammar. I am looking for a grammar and it's associated .l and .ypp files, preferably in C++. I've been able to find sample files or sample grammars, but not both of them. I've spent some time searching and I could not find anything. I figure I'd post here in hopes that someone has something for me, but I will continue searching in the meantime.
I am currently reading Tom Niemann's
http://epaperpress.com/lexandyacc/download/LexAndYaccTutorial.pdf which seems to be pretty well written and understandable.
Thanks
Edit: I am still searching, I am starting to think that what I am looking for does not exist. Google usually never fails me!
Edit 2: Maybe if I provide some of the grammar, you folks could show me what the appropriate .l and .ypp files would look like. This is just a snippet of the grammar, I just need a little 'taste' of how this works and I think I can take it from there.
Grammar:
Program ::= Compound
Statements ::= Compound | Assignment | ...
Assignment ::= Var ASSIGN Expression
Expression ::= Var | Operator Expression Expression | Number
Compound := START Statements END
Number ::= NUMBER
Descriptions:
Assignment is the equal sign ":="
Var is an identifier that begins with a lower case letter and is followed by lower case letters or digits
START is the "start" keyword
END is the "end keyword
Operator is "+", "-", "*", "/"
Number is decimal digits which could potentially be negative (minus sign in front)
Most of this is fairly straightforward. One part, however, is decidedly problematic. You've defined a number to (potentially) include a leading -, and that's a problem.
The problem is pretty simple. Given an input like 321-123, it's essentially impossible for the lexer (which won't normally keep track of current state) to guess at whether that's supposed to be two tokens (321 and -123 or three 321, -, 123). In this case, the - is almost certainly intended to be separate from the 123, but if the input were 321 + -123 you'd apparently want -123 as a single token instead.
To deal with that, you probably want to change your grammar so the leading - isn't part of the number. Instead, you always want to treat the - as an operator, and the number itself is composed solely of the digits. Then it's up to the parser to sort out expressions where the - is unary vs. binary.
Taking that into account, the lexer file would look something like this:
%{
#include "y.tab.h"
%}
%option noyywrap case-insensitive
%%
:= { return ASSIGN; }
start { return START; }
end { return END; }
[+/*] { return OPERATOR; }
- { return MINUS; }
[0-9]+ { return NUMBER; }
[a-z][a-z0-9]* { return VAR; }
[ \r\n] { ; }
%%
void yyerror(char const *s) { fputs(s, stderr); }
The matching yacc file would look something like this:
%token ASSIGN START END OPERATOR MINUS NUMBER VAR
%left '-' '+' '*' '/'
%%
program : compound
statement : compound
| assignment
;
assignment : VAR ASSIGN expression
;
statements :
| statements statement
;
expression : VAR
| expression OPERATOR expression
| expression MINUS expression
| value
;
value: NUMBER
| MINUS NUMBER
;
compound : START statements END
%%
int main() {
yyparse();
return 0;
}
Note: I've tested these only extremely minimally--enough to verify input I believe is grammatical, such as: start a:=1 b:=2 end and start a:=1+3*3 b:=a+4 c:=b*3 end is accepted (no error message printed out) and input I believe is un-grammatical, such as: 9:=13 and a=13 do both print out syntax error messages. Since this doesn't attempt to do any more with the expressions than recognize those which are or are not grammatical, that's about the best we can do though.
OK, so I'm experimenting with token aliases and facing some issues.
Let's take this part of my (ultra-simplified) Bison grammar as an example :
/****************************************
Definitions
****************************************/
%union
{
char* str;
}
/****************************************
Tokens & Types
****************************************/
%token <str> ID "identifier"
%token <str> NUMBER_DEC "number"
%token <str> NUMBER_HEX "number"
%token <str> NUMBER_BIN "number"
%token <str> NUMBER_FLOAT "number"
%type <str> identifier number
%type <str> assignment_st
%type <str> statements statement
%type <str> program
/****************************************
Directives
****************************************/
%glr-parser
%locations
%start program
%define parse.error verbose
%%
/****************************************
Grammar Rules
****************************************/
identifier : ID
;
number : NUMBER_DEC
| NUMBER_HEX
| NUMBER_BIN
| NUMBER_FLOAT
;
assignment_st : identifier '=' number ';' { printf("assignment : %s = %s\n",$identifier,$number); }
;
statement : assignment_st
;
statements : statement
| statements statement
;
program : statements
;
%%
Now, if I try a = 2;, this is obviously ok with the grammar.
If I try a = b; this is an error as it expects a number. In this case, the parser reports :
syntax error, unexpected identifier, expecting number or NUMBER_HEX or NUMBER_BIN or NUMBER_FLOAT
(Well, the "number" alias is a duplicate since it's used in 4 tokens).
However, I'd be looking for something more like unexpected identifier, expected number.
How would you go about it?
Also, is there any chance I could incorporate the error line as well in the message?
P.S. I have been looking into the latest Bison documentation for hours, but I feel as if I'll end up building a... rocket instead of fixing the error messages... lol
How would you go about it?
I'd use a single NUMBER token. I don't see any reason to make the parser care which type of numeric literal it's looking at.
Of course, it's possible that your full grammar does actually involve places where only certain formats of numeric literals are allowed, although my general inclination about that sort of thing is "yuk". The most likely possibility is that there is some rule in which an integer constant is ok, but a floating point constant is not. In that case, you're not going to get good error messages for that particular production unless you provide a different alias for floating point numbers than for integers. On the whole, though, I'm sticking with "yuk".
If you have a single numeric token type, with alias "number", then the errors should work out fine.
I am running into a problem when compiling the following rule for my parser:
%%
expr:
| expr ASN expr { Asn ($1, $2) }
This is an assignment rule that takes an integer, then the assignment (equal sign) and an expression, as defined in my AST:
type expr =
Asn of int * expr
Of course, the compiler is complaining because I am defining "expr ASN expr", and the first argument should be an integer, not an expression. However, I have not been able to figure out the syntax to specify this.
If somebody could lead me in the right direction, I would really appreciate it.
Thanks!
You don't supply enough details to give a good answer. What do you mean by an integer? I'll assume you mean an integer literal.
Assuming your lexical definitions have a token named INT that represents an integer literal, you might want something like this.
expr:
| INT ASN expr { Asn ($1, $2) }
Probably what you want is the assignment as:
type expr = Asn of var * int
and then define expr in the parser as:
expr:
| VAR ASN INT { Asn ($1, $2) }
in the lexer you should have defined VAR as string and INT as integer literal too, just as examples:
| [a-zA-z]+ { VAR($1) }
| [0-9]+ as i { INT(int_of_string i) }
I am writing a compiler with Yacc and having trouble figuring out how to write productions to match a function. In my language, functions are defined like this:
function foo(a, b, c);
I created lex patterns to match the word function to FUNC, and any C style name to NAME.
Ideally, I would want something like this:
FUNC NAME OBRACKET NAME (COMMA NAME)* CBRACKET
Which would allow some unknown number of pairs of COMMA NAME in between NAME and CBRACKET.
Additionally, how would I know how many it found?
You might try something like this:
funcdecl: FUNC NAME OBRACKET arglist CBRACKET SEMI
;
arglist: nonemptyarglist
|
;
nonemptyarglist: nonemptyarglist COMMA NAME
| NAME
;
I'd suggest using the grammar to build a syntax tree for your language and then doing whatever you need to the syntax tree after parsing has finished. Bison and yacc have features that make this rather simple; look up %union and %type in the info page.
After a little experimentation, I found this to work quite well:
int argCount;
int args[128];
arglist: nonemptyarglist
|
;
nonemptyarglist: nonemptyarglist COMMA singleArgList
| singleArgList
{
};
singleArgList:
REGISTER
{
args[argCount++] = $1;
};