I am trying to write Grammar for java specification
for example:-
COMPILATION_UNIT: PACKAGE_DEC? IMPORT_DECS? TYPE_DECS?
but it doesn't work
I have the following error:
invalid character: `?'
for each question mark I use in my file.y
I know that Bison has special characters and it should handle it
Please help
Bison does not allow a ? meaning that the prior token is optional, you have to write out the grammar with the optional elements:
package_decl_opt: %empty
| SOME_TOKEN
;
package: package)_dec_opt TOKEN_PACKAGE TOKEN_IDENTIFIER
;
would allow both of the following:
SOME_TOKEN TOKEN_PACKAGE TOKEN_IDENTIFIER
TOKEN_PACKAGE TOKEN_IDENTIFIER
As you have seen, bison does not implement the ? regular expression optionality operator. Nor does it implement + or * repetition operators. That's because the right-hand sides of productions in contex-free grammars are not regular expressions.
Yacc/bison context-free grammars do allow the | alternation operator, but as an abbreviation:
a : b | c
Is exactly the same as writing
a : b
a : c
and semantic actions only apply to the alternative in which they are specified, so that
a : b | c { /* C action; */ }
Is equivalent to:
a : b { /* Implicit default action*/ }
a : c { /* C action; */ }
It is tempting to create X_opt non-terminals to capture the semantics of X?:
X_opt: X | %empty { $$ = default_value; }
In many simple cases that will work fine, but there are also many grammars in which that introduces an unnecessary shift-reduce conflict. Consider, for example:
label: IDENT ':'
label_opt: label | %empty
statement: label_opt expr
Since expr can start with an identifier, there is no way to know if an IDENT token starts a label or if it starts an expr following an empty label_opt. But LR(1) requires that the empty label_opt be reduced before the IDENT is consumed. So the above grammar is LR(2) and cannot be correctly parsed by an LR(1) parser.
That problem does not occur without the use of the label_opt shortcut:
label: IDENT ':'
statement: label expr
| expr
Since the parser now does not have decide between label and expr before the ':' is encountered.
So I have a grammar that includes the empty string. The grammar is something like this:
S->ε
S->expression ;; S
I'm getting the error "No more states to discard" when I run my parser so I believe I'm not representing the empty string correctly. So how would I go about representing it, specifically in the lexer .mll file?
I know I need to make a rule for it so I think I have that down. This is what I think it should look like for the parser .mly file, excluding the stuff for expression.
s:
| EMPTY_STRING { [] }
| expression SEMICOLON s { $1::$3 }
You're thinking of epsilon as a token, but it's not a token. It's a 0-length sequence of tokens. Since there are no tokens there, it's not something your scanner needs to know about. Just the parser needs to know about it.
Here's a grammar something like what I think you want:
%token X
%token SEMICOLON
%token EOF
%start main
%type <char list> main
%%
main :
s EOF { $1 }
s :
| epsilon { $1 }
| X SEMICOLON s { 'x' :: $3 }
epsilon :
{ [] }
Note that epsilon is a non-terminal (not a token). Its definition is an empty sequence of symbols.
I can't, for the life of me, figure out what's wrong with my regex's.
What I'd like to tokenize are two (2) types of strings, both of which to be contained on a single line. One string can be anything (other than a new line), and the other, any alpha-numeric (ASCII) character and literal '_', '/' '-', and '.'.
The snippet of flex code is:
nl \n|\r\n|\r|\f|\n\r
...
%%
...
\"[^\"]+{nl} { frx_parser_error("Label is missing trailing double quote."); }
\"[a-zA-Z0-9_\.\/\-]+\" {
if (yyleng > 1024) frx_parser_error("File name too long.");
yytext[yyleng - 1] = '\0';
frx_parser_lval.str = strdup(yytext+1);
fprintf(stderr,"TOSP_FILENAME: %s\n", frx_parser_lval.str);
return (TOSP_FILENAME);
}
\"[^{nl}]+\" {
yytext[yyleng - 1] = '\0';
frx_parser_lval.str = strdup(yytext+1);
fprintf(stderr,"TOSP_IDENTIFIER:\n%s\n", frx_parser_lval.str);
return (TOSP_IDENTIFIER);
}
And when I run the parser, the fprintf's spit this out:
TOSP_FILENAME: ModStar-Picture-Analysis.txt
TOSP_FILENAME: ModStar-Rubric.log.txt
TOSP_IDENTIFIER:
picture-A"
Progress (26,255) camera 'C' root("picture-C-
Syntax (line 34): syntax error
For whatever reason, the quote after picture-A is being ... missed. Why? I checked the ASCII values for the eight locations the quote character appears and they're all 0x22 (where the double quutoes appear that is).
If I add some characters to the end of the "picture-A" it can work sometimes; adding ".par", ".pbr" doesn't work as expected, but ".pnr" does.
I've even added a specific non-regexy token:
\"picture-A\" { frx_parser_lval.str = strdup("picture-A"); return TOSP_FILENAME; }
to the lex file and it gets skipped.
I'm using flex 2.5.39, no flex libraries, one option (%option prefix=frx_parser_) in the lex file and the flex command line is:
flex -t script-lexer.l > script-lexer.c
What gives?
EDIT I need to test this on the actual system, but unit tests show this tokenizer to be much more robust (based on rici's answer):
nl \n|\r\n|\r|\f|\n\r
...
%%
...
["][^"]+{nl} { printf("Missing trailing quote.\n%s\n",yytext); }
["][[:alnum:]_./-]+["] { printf("File name:\n%s\n",yytext); }
["][^"]+["] { printf("String:\n%s\n",yytext); }
EDIT The rule ["].+["] swallows consecutive multiple strings as one big string. It was changed to ["][^"]+["]
The problem is your pattern:
\"[^{nl}]+\"
You're attempting to expand a definition inside a character class, but that is not possible; inside a character class, { is always just a {, not a flex operator. See the flex manual:
Note that inside of a character class, all regular expression operators lose their special meaning except escape (‘\’) and the character class operators, ‘-’, ‘]]’, and, at the beginning of the class, ‘^’.
A definition is not a macro. Rather, a definition defines a new regular expression operator.
As a consequence of the above, you can write [^\"] as simply [^"] and \"[a-zA-Z0-9_\.\/\-]+\" as \"[a-zA-Z0-9_./-]+\" (The - needs to be either at the end or at the beginning.) Personally, I'd write the second pattern as:
["][[:alnum:]_./-]+["]
But everyone has their own style.
I am writing a compiler with Yacc and having trouble figuring out how to write productions to match a function. In my language, functions are defined like this:
function foo(a, b, c);
I created lex patterns to match the word function to FUNC, and any C style name to NAME.
Ideally, I would want something like this:
FUNC NAME OBRACKET NAME (COMMA NAME)* CBRACKET
Which would allow some unknown number of pairs of COMMA NAME in between NAME and CBRACKET.
Additionally, how would I know how many it found?
You might try something like this:
funcdecl: FUNC NAME OBRACKET arglist CBRACKET SEMI
;
arglist: nonemptyarglist
|
;
nonemptyarglist: nonemptyarglist COMMA NAME
| NAME
;
I'd suggest using the grammar to build a syntax tree for your language and then doing whatever you need to the syntax tree after parsing has finished. Bison and yacc have features that make this rather simple; look up %union and %type in the info page.
After a little experimentation, I found this to work quite well:
int argCount;
int args[128];
arglist: nonemptyarglist
|
;
nonemptyarglist: nonemptyarglist COMMA singleArgList
| singleArgList
{
};
singleArgList:
REGISTER
{
args[argCount++] = $1;
};
Roughly speaking in C++ there are:
operators (+, -, *, [], new, ...)
identifiers (names of classes, variables, functions,...)
const literals (10, 2.5, "100", ...)
some keywords (int, class, typename, mutable, ...)
brackets ({, }, <, >)
preprocessor (#, ## ...).
But what is the semicolon?
The semicolon is a punctuator, see 2.13 §1
The lexical representation of C++ programs includes a number of preprocessing tokens which are used in
the syntax of the preprocessor or are converted into tokens for operators and punctuators
It is part of the syntax and therein element of several statements. In EBNF:
<do-statement>
::= 'do' <statement> 'while' '(' <expression> ')' ';'
<goto-statement>
::= 'goto' <label> ';'
<for-statement>
::= 'for' '(' <for-initialization> ';' <for-control> ';' <for-iteration> ')' <statement>
<expression-statement>
::= <expression> ';'
<return-statement>
::= 'return' <expression> ';'
This list is not complete. Please see my comment.
The semicolon is a terminal, a token that terminates something. What exactly it terminates depends on the context.
Semicolon denotes sequential composition. It is also used to delineate declarations.
Semicolon is a statement terminator.
The semicolon isn't given a specific name in the C++ standard. It's simply a character that's used in certain grammar productions (and it just happens to be at the end of them quite often, so it 'terminates' those grammatical constructs). For example, a semicolon character is at the end of the following parts of the C++ grammar (not necessarily a complete list):
an expression-statement
a do/while iteration-statement
the various jump-statements
the simple-declaration
Note that in an expression-statement, the expression is optional. That's why a 'run' of semicolons, ;;;;, is valid in many (but not all) places where a single one is.
';'s are often used to delimit one bit of C++ source code, indicating it's intentionally separate from the following code. To see how it's useful, let's imagine we didn't use it:
For example:
#include <iostream>
int f() { std::cout << "f()\n"; }
int g() { std::cout << "g()\n"; }
int main(int argc)
{
std::cout << "message"
"\0\1\0\1\1"[argc] ? f() : g(); // final ';' needed to make this compile
// but imagine it's not there in this new
// semicolon-less C++ variant....
}
This (horrible) bit of code, called with no arguments such that argc is 1, prints:
ef()\n
Why not "messagef()\n"? That's what might be expected given first std::cout << "message", then "\0\1\0\1\1"[1] being '\1' - true in a boolean sense - suggests a call to f() printing f()\n?
Because... (drumroll please)... in C++ adjacent string literals are concatenated, so the program's parsed like this:
std::cout << "message\0\1\0\1\1"[argc] ? f() : g();
What this does is:
find the [argc/1] (second) character in "message\0\1\0\1\1", which is the first 'e'
send that 'e' to std::cout (printing it)
the ternary operator '?' triggers casting of std::cout to bool which produces true (because the printing presumably worked), so f() is called...!
Given this string literal concatenation is incredibly useful for specifying long strings
(and even shorter multi-line strings in a readable format), we certainly wouldn't want to assume that such strings shouldn't be concatenated. Consequently, if the semicolon's gone then the compiler must assume the concatenation is intended, even though visually the layout of the code above implies otherwise.
That's a convoluted example of how C++ code with and with-out ';'s changes meaning. I'm sure if I or other readers think on it for a few minutes we could come up with other - and simpler - examples.
Anyway, the ';' is necessary to inform the compiler that statement termination/separation is intended.
The semicolon lets the compiler know that it's reached the end of a command AFAIK.
The semicolon (;) is a command in C++. It tells the compiler that you're at the end of a command.
If I recall correctly, Kernighan and Ritchie called it punctuation.
Technically, it's just a token (or terminal, in compiler-speak), which
can occur in specific places in the grammar, with a specific semantics
in the language. The distinction between operators and other punctuation
is somewhat artificial, but useful in the context of C or C++, since
some tokens (,, = and :) can be either operators or punctuation,
depending on context, e.g.:
f( a, b ); // comma is punctuation
f( (a, b) ); // comma is operator
a = b; // = is assignment operator
int a = b; // = is punctuation
x = c ? a : b; // colon is operator
label: // colon is punctuation
In the case of the first two, the distinction is important, since a user
defined overload will only affect the operator, not punctuation.
It represents the end of a C++ statement.
For example,
int i=0;
i++;
In the above code there are two statements. The first is for declaring the variable and the second one is for incrementing the value of variable by one.