In C++, a famous parsing ambiguity happens with code like
x<T> a;
Is it if T is a type, it is what it looks like (a declaration of a variable a of type x<T>, otherwise it is (x < T) > a (<> are comparison operators, not angle brackets).
In fact, we could make a change to make this become unambiguous: we can make < and > nonassociative. So x < T > a, without brackets, would not be a valid sentence anyway even if x, T and a were all variable names.
How could one resolve this conflict in Menhir? At first glance it seems we just can't. Even with the aforementioned modification, we need to lookahead an indeterminate number of tokens before we see another closing >, and conclude that it was a template instantiation, or otherwise, to conclude that it was an expression. Is there any way in Menhir to implement such an arbitrary lookahead?
Different languages (including the ones listed in your title) actually have very different rules for templates/generics (like what type of arguments there can be, where templates/generics can appear, when they are allowed to have an explicit argument list and what the syntax for template/type arguments on generic methods is), which strongly affect the options you have for parsing. In no language that I know is it true that the meaning of x<T> a; depends on whether T is a type.
So let's go through the languages C++, Java, Rust and C#:
In all four of those languages both types and functions/methods can be templates/generic. So we'll not only have to worry about an ambiguity with variable declarations, but also function/method calls: is f<T>(x) a function/method call with an explicit template/type argument or is it two relational operators with the last operand parenthesized? In all four languages template/generic functions/methods can be called without template/type when those can be inferred, but that inference isn't always possible, so just disallowing explicit template/type arguments for function/method calls is not an option.
Even if a language does not allow relational operators to be chained, we could get an ambiguity in expressions like this: f(a<b, c, d>(e)). Is this calling f with the three arguments a<b, c and d>e or with the single argument a<b, c, d>(e) calling a function/method named a with the type/template arguments b,c,d?
Now beyond this common foundation, most everything else is different between these languages:
Rust
In Rust the syntax for a variable declaration is let variableName: type = expr;, so x<T> a; couldn't possibly be a variable declaration because that doesn't match the syntax at all. In addition it's also not a valid expression statement (anymore) because comparison operators can't be chained (anymore).
So there's no ambiguity here or even a parsing difficulty. But what about function calls? For function calls, Rust avoided the ambiguity by simply choosing a different syntax to provide type arguments: instead of f<T>(x) the syntax is f::<T>(x). Since type arguments for function calls are optional when they can be inferred, this ugliness is thankfully not necessary very often.
So in summary: let a: x<T> = ...; is a variable declaration, f(a<b, c, d>(e)); calls f with three arguments and f(a::<b, c, d>(e)); calls a with three type arguments. Parsing is easy because all of these are sufficiently different to be distinguished with just one token of lookahead.
Java
In Java x<T> a; is in fact a valid variable declaration, but it is not a valid expression statement. The reason for that is that Java's grammar has a dedicated non-terminal for expressions that can appear as an expression statement and applications of relational operators (or any other non-assignment operators) are not matched by that non-terminal. Assignments are, but the left side of assignment expressions is similarly restricted. In fact, an identifier can only be the start of an expression statement if the next token is either a =, ., [ or (. So an identifier followed by a < can only be the start of a variable declaration, meaning we only need one token of lookahead to parse this.
Note that when accessing static members of a generic class, you can and must refer to the class without type arguments (i.e. FooClass.bar(); instead of FooClass<T>.bar()), so even in that case the class name would be followed by a ., not a <.
But what about generic method calls? Something like y = f<T>(x); could still run into the ambiguity because relational operators are of course allowed on the right side of =. Here Java chooses a similar solution as Rust by simply changing the syntax for generic method calls. Instead of object.f<T>(x) the syntax is object.<T>f(x) where the object. part is non-optional even if the object is this. So to call a generic method with an explicit type argument on the current object, you'd have to write this.<T>f(x);, but like in Rust the type argument can often be inferred, allowing you to just write f(x);.
So in summary x<T> a; is a variable declaration and there can't be expression statements that start with relational operations; in general expressions this.<T>f(x) is a generic method call and f<T>(x); is a comparison (well, a type error, actually). Again, parsing is easy.
C#
C# has the same restrictions on expression statements as Java does, so variable declarations aren't a problem, but unlike the previous two languages, it does allow f<T>(x) as the syntax for function calls. In order to avoid ambiguities, relational operators need to be parenthesized when used in a way that could also be valid call of a generic function. So the expression f<T>(x) is a method call and you'd need to add parentheses f<(T>(x)) or (f<T)>(x) to make it a comparison (though actually those would be type errors because you can't compare booleans with < or >, but the parser doesn't care about that) and similarly f(a<b, c, d>(e)) calls a generic method named a with the type arguments b,c,d whereas f((a<b), c, (d<e)) would involve two comparisons (and you can in fact leave out one of the two pairs of parentheses).
This leads to a nicer syntax for method calls with explicit type arguments than in the previous two languages, but parsing becomes kind of tricky. Considering that in the above example f(a<b, c, d>(e)) we can actually place an arbitrary number of arguments before d>(e) and a<b is a perfectly valid comparison if not followed by d>(e), we actually need an arbitrary amount of lookahead, backtracking or non-determinism to parse this.
So in summary x<T> a; is a variable declaration, there is no expression statement that starts with a comparison, f<T>(x) is a method call expression and (f<T)>(x) or f<(T>(x)) would be (ill-typed) comparisons. It is impossible to parse C# with menhir.
C++
In C++ a < b; is a valid (albeit useless) expression statement, the syntax for template function calls with explicit template arguments is f<T>(x) and a<b>c can be a perfectly valid (even well-typed) comparison. So statements like a<b>c; and expressions like a<b>(c) are actually ambiguous without additional information. Further, template arguments in C++ don't have to be types. That is, Foo<42> x; or even Foo<c> x; where c is defined as const int x = 42;, for example, could be perfectly valid instantiations of the Foo template if Foo is defined to take an integer as a template argument. So that's a bummer.
To resolve this ambiguity, the C++ grammar refers to the rule template-name instead of identifier in places where the name of a template is expected. So if we treated these as distinct entities, there'd be no ambiguity here. But of course template-name is defined simply as template-name: identifier in the grammar, so that seems pretty useless, ... except that the standard also says that template-name should only be matched when the given identifier names a template in the current scope. Similarly it says that identifiers should only be interpreted as variable names when they don't refer to a template (or type name).
Note that, unlike the previous three languages, C++ requires all types and templates to be declared before they can be used. So when we see the statement a<b>c;, we know that it can only be a template instantiation if we've previously parsed a declaration for a template named a and it is currently in scope.
So, if we keep track of scopes while parsing, we can simply use if-statements to check whether the name a refers to a previously parsed template or not in a hand-written parser. In parser generators that allow semantic predicates, we can do the same thing. Doing this does not even require any lookahead or backtracking.
But what about parser generators like yacc or menhir that don't support semantic predicates? For these we can use something known as the lexer hack, meaning we make the lexer generate different tokens for type names, template names and ordinary identifiers. Then we have a nicely unambiguous grammar that we can feed our parser generator. Of course the trick is getting the lexer to actually do that. In order to accomplish that, we need to keep track of which templates and types are currently in scope using a symbol table and then access that symbol table from the lexer. We'll also need to tell the lexer when we're reading the name of a definition, like the x in int x;, because then we want to generate a regular identifier even if a template named x is currently in scope (the definition int x; would shadow the template until the variable goes out of scope).
This same approach is used to resolve the casting ambiguity (is (T)(x) a cast of x to type T or a function call of a function named T?) in C and C++.
So in summary, foo<T> a; and foo<T>(x) are template instantiations if and only if foo is a template. Parsing's a bitch, but possible without arbitrary lookahead or backtracking and even using menhir when applying the lexer hack.
AFAIK C++'s template syntax is a well-known example of real-world non-LR grammar. Strictly speaking, it is not LR(k) for any finite k... So C++ parsers are usually hand-written with hacks (like clang) or generated by a GLR grammar (LR with branching). So in theory it is impossible to implement a complete C++ parser in Menhir, which is LR.
However even the same syntax for generics can be different. If generic types and expressions involving comparison operators never appear under the same context, the grammar may still be LR compatible. For example, consider the rust syntax for variable declaration (for this part only):
let x : Vec<T> = ...
The : token indicates that a type, rather than an expression follows, so in this case the grammar can be LR, or even LL (not verified).
So the final answer is, it depends. But for the C++ case it should be impossible to implement the syntax in Menhir.
Related
In c++17 template <auto> allows to declare templates with arbitrary type parameters. Partially inspired by this question, it would be useful to have an extension of template <auto> that captures both type and nontype template parameters, and also allows for a variadic version of it.
Are there plans for such an extension in the next c++20 release? Is there some fundamental problem in having a syntax like template<auto... X>, with X any type or nontype template parameter?
Are there plans for such an extension in the next c++20 release?
No.
Is there some fundamental problem in having a syntax like template<auto... X>, with X any type or nontype template parameter?
It would be a totally new concept in the language - having a name refer to either a type or a value in the same place. So it'd come with all sorts of additional questions - and probably additional language features to check if X is a type or not.
The syntax likely cannot be template <auto... X> struct Y { }; since that syntax already has meaning and means a bunch of values and Y<int>{} is ill-formed.
There are definitely places where such a thing would be useful though. A proposal would just have to address these issues.
The big issue with trying to do something like that is grammar. Template parameters state up-front whether they are templates, types, or values, and the most important reason for this is grammatical.
C++ is a context-sensitive grammar. That means that you cannot know, just from a sequence of tokens, what a particular sequence of tokens means. For example, IDENTIFIER LEFT_PAREN RIGHT_PAREN SEMICOLON. What does that mean?
It could mean to call a function named by IDENTIFIER with no parameters. It could mean to default initialize a prvalue of a class named by IDENTIFIER. These are rather different things; you might conceptually see them as similar, but C++'s grammar does not.
Templates are not macros; they're not doing token pasting. There is some understanding that a piece of code in a template is supposed to mean a specific thing. And you can only do that if you at least know what kind of thing a template parameter is.
In order to retain this ability, these "omni template parameters" cannot be utilized until you actually know what they mean. So in order to create such a feature in C++, you would need to:
Create a new syntax to declare omni template parameters (auto isn't going to fly, as it already has a specific meaning).
Provide a syntax for determining what kind of thing an omni template parameter is.
Require the user to invoke that syntax before they can use such parameter names in most ways. This would typically be via some form of specialized if constexpr block, but pattern matching proposals represent an interesting alternative/additional way to handle them (since they can be expressions as well as statements). And expansion statements represent a possible way to access all of the omni parameters in a parameter pack.
I can't see how it would be useful that a template argument could be dynamically either a type or a value? The code statements that use types are very different to those that which use constant values introduced through the template argument.
The only way would be a big "if constexpr" which would make it pointless in my view.
Ok, having looked more closely at the referenced question, I guess there is room there for generically pass-through wrapping the various explicit base template implementations that use different parameter orderings. I still fail to see a huge benefit. The compiler errors when it goes wrong are going to be unfathomable, if nothing else!
I remember being told that overloading and templates were going to rid the world of the unfathomable error messages generated from macros. I have yet to see it!
I'm going through the full tutorial at cplusplus.com, coding and compiling each example manually. Regularly, I stumble upon something that leaves me perplexed.
I am currently learning this section: http://www.cplusplus.com/doc/tutorial/structures/ . There are some subtleties that could easily be overlooked by only reading the tutorial. The advantage of typing everything by hand is that such details do stand out.
In the above page, there are two sample programs. One has this line:
stringstream(mystr) >> yours.year;
The other one has this line:
(stringstream) mystr >> pmovie->year;
What I don't understand is the difference (if any) between type (myVar) = x; and (type) myVar = x;.
I am not doing the whole tutorial in sequential order. I checked but didn't find this addressed anywhere, though I may have missed it.
Is there a difference?
Is there a preferred way to do it one way rather than the other?
There is no difference between type(x) and (type)x. These two are completely equivalent. Most people prefer type(x) for classes and (type)x for non-class types, but that's purely up to one's own choice. Both call constructors for classes with one argument x.
The preferred way for classes is type(x), because this allows passing more than one argument to the constructor, as in type(x, y). Trying to apply the other form, (type)x, y will not work: It casts x, and then applies the comma operator and evalutes y in isolation. Parentheses like (type)(x, y) do not help: This will evaluate x and y in isolation using the comma operator and then cast y to type.
For non-class types, such a cast is often too powerful. C++ has static_cast<type>(x) for roughly doing the reverse of an implicit conversion (such as casting base classes to derived classes and casting void* to another pointer), which often is what fits in. See When should static_cast, dynamic_cast and reinterpret_cast be used?.
stringstream is not a function, though. Doing function(x) will call it the function, but doing (function)x is illegal, beause there are two expressions next to each other, with no operator in between.
For those who don't believe this answer, and downvote it on gut feeling, please consult the Standard at 5.2.3/1
A simple-type-specifier (7.1.5) followed by a parenthesized expression-list constructs a value of the specified type given the expression list. If the expression list is a single expression, the type conversion expression is equivalent (in definedness, and if defined in meaning) to the corresponding cast expression (5.4).
The page you cite is not what I would consider a authority on C++ in general.
Anyway,
(stringstream) mystr >> pmovie->year;
casts a std::string to a std::stringstream object. This is a C-style cast. Rather dangerous if you don't know what you are doing. This would create a stringstream object and the value is extracted to pmovie->year next.
stringstream(mystr) >> yours.year;
Creates an anonymous std::stringstream object and initializes it with mystr and then the value is extracted to pmovie->year. The object vanishes at the end of its lexical scope which in this case would be the ; at the end of the line.
Not much of a difference (as others have noted so far) among the two w.r.t class objects.
On the other hand, with identifiers (of functions/macros) this gets tricky: function (myVar) = x; works irrespective of whether function is an actual function or a macro. However, (function) (myVar) = x; only works for real functions.
Some standard library identifiers are allowed to have both forms (most notably the tolower and friends) and therefore if you want to invoke the function always then you should go for the former.
In C++, the symbols '<' and '>' are used for comparisons as well as for signifying a template argument. Thus, the code snippet
[...] Foo < Bar > [...]
might be interpreted as any of the following two ways:
An object of type Foo with template argument Bar
Compare Foo to Bar, then compare the result to whatever comes next
How does the parser for a C++ compiler efficiently decide between those two possibilities?
If Foo is known to be a template name (e.g. a template <...> Foo ... declaration is in scope, or the compiler sees a template Foo sequence), then Foo < Bar cannot be a comparison. It must be a beginning of a template instantiation (or whatever Foo < Bar > is called this week).
If Foo is not a template name, then Foo < Bar is a comparison.
In most cases it is known what Foo is, because identifiers generally have to be declared before use, so there's no problem to decide one way or the other. There's one exception though: parsing template code. If Foo<Bar> is inside a template, and the meaning of Foo depends on a template parameter, then it is not known whether Foo is a template or not. The language standard directs to treat it as a non-template unless preceded by the keyword template.
The parser might implement this by feeding context back to the lexer. The lexer recognizes Foo as different types of tokens, depending on the context provided by the parser.
The important point to remember is that C++ grammar is not context-free. I.e., when the parser sees Foo < Bar (in most cases) knows that Foo refers to a template definition (by looking it up in the symbol table), and thus < cannot be a comparison.
There are difficult cases, when you literally have to guide the parser. For example, suppose that are writing a class template with a template member function, which you want to specialize explicitly. You might have to use syntax like:
a->template foo<int>();
(in some cases; see Calling template function within template class for details)
Also, comparisons inside non-type template arguments must be surrounded by parentheses, i.e.:
foo<(A > B)>
not
foo<A > B>
Non-static data member initializers bring more fun: http://open-std.org/JTC1/SC22/WG21/docs/cwg_active.html#325
C and C++ parsers are "context sensitive", in other words, for a given token or lexeme, it is not guaranteed to be distinct and have only one meaning - it depends on the context within which the token is used.
So, the parser part of the compiler will know (by understanding "where in the source it is") that it is parsing some kind of type or some kind of comparison (This is NOT simple to know, which is why reading the source of competent C or C++ compiler is not entirely straight forward - there are lots of conditions and function calls checking "is this one of these, if so do this, else do something else").
The keyword template helps the compiler understand what is going on, but in most cases, the compiler simply knows because < doesn't make sense in the other aspect - and if it doesn't make sense in EITHER form, then it's an error, so then it's just a matter of trying to figure out what the programmer might have wanted - and this is one of the reasons that sometimes, a simple mistake such as a missing } or template can lead the entire parsing astray and result in hundreds or thousands of errors [although sane compilers stop after a reasonable number to not fill the entire universe with error messages]
Most of the answers here confuse determining the meaning of the symbol (what I call "name resolution") with parsing (defined narrowly as "can read the syntax of the program").
You can do these tasks separately..
What this means is that you can build a completely context-free parser for C++ (as my company, Semantic Designs does), and leave the issues of deciding what the meaning of the symbol is to a explicitly seperate following task.
Now, that task is driven by the possible syntax interpretations of the source code. In our parsers, these are captured as ambiguities in the parse.
What name resolution does is collect information about the declarations of names, and use that information to determine which of the ambiguous parses doesn't make sense, and simply drop those. What remains is a single valid parse, with a single valid interpretation.
The machinery to accomplish name resolution in practice is a big mess. But that's the C++ committee's fault, not the parser or name resolver. The ambiguity removal with our tool is actually done automatically, making that part actually pretty nice but if you don't look inside our tools you would not appreciate that, but we do because it means a small engineering team was able to build it.
See an example of resolution of template-vs-less than on C++s most vexing parse done by our parser.
I've been lately using templates and macros, but i have to say i have barely found information about these important types. This is my superficial understanding:
typed/expr is something that must exists previously, but you can use .immediate. to overcome them.
untyped/stmt is something that doesn't to be defined previously/one or more statements.
This is a very vague notion of the types. I'd like to have a better explanation of them, including which types should be used as return.
The goal of these different parameter types is to give you several increasing levels of precision in specifying what the compiler should accept as a parameter to the macro.
Let's imagine a hypothetical macro that can solve mathematical equations. It will be used like this:
solve(x + 10 = 25) # figures out that the correct value for x is 15
Here, the macro just cares about the structure of the supplied AST tree. It doesn't require that the same tree is a valid expression in the current scope (i.e. that x is defined and so on). The macro just takes advantage of the Nim parser that already can decode most of the mathematical equations to turn them into easier to handle AST trees. That's what untyped parameters are for. They don't get semantically checked and you get the raw AST.
On the next step in the precision ladder are the typed parameters. They allow us to write a generic kind of macro that will accept any expression, as long as it has a proper meaning in the current scope (i.e. its type can be determined). Besides catching errors earlier, this also has the advantage that we can now work with the type of the expression within the macro body (using the macros.getType proc).
We can get even more precise by requiring an expression of a specific type (either a concrete type or a type class/concept). The macro will now be able to participate in overload resolution like a regular proc. It's important to understand that the macro will still receive an AST tree, as it will accept both expressions that can be evaluated at compile-time and expressions that can only be evaluated at run-time.
Finally, we can require that the macro receives a value of specific type that is supplied at compile-time. The macro can work with this value to parametrise the code generation. This is realm of the static parameters. Within the body of the macro, they are no longer AST trees, but rather ordinary well typed values.
So far, we've only talked about expressions, but Nim's macros also accept and produce blocks and this is the second axis, which we can control. expr generally means a single expression, while stmt denotes a list of expressions (historically, its name comes from StatementList, which existed as a separate concept before expressions and statements were unified in Nim).
The distinction is most easily illustrated with the return types of templates. Consider the newException template from the system module:
template newException*(exceptn: typedesc, message: string): expr =
## creates an exception object of type ``exceptn`` and sets its ``msg`` field
## to `message`. Returns the new exception object.
var
e: ref exceptn
new(e)
e.msg = message
e
Even thought it takes several steps to construct an exception, by specifying expr as the return type of the template, we tell the compiler that only that last expression will be considered as the return value of the template. The rest of the statements will be inlined, but cleverly hidden from the calling code.
As another example, let's define a special assignment operator that can emulate the semantics of C/C++, allowing assignments within if statements:
template `:=` (a: untyped, b: typed): bool =
var a = b
a != nil
if f := open("foo"):
...
Specifying a concrete type has the same semantics as using expr. If we had used the default stmt return type instead, the compiler wouldn't have allowed us to pass a "list of expressions", because the if statement obviously expects a single expression.
.immediate. is a legacy from a long-gone past, when templates and macros didn't participate in overload resolution. When we first made them aware of the type system, plenty of code needed the current untyped parameters, but it was too hard to refactor the compiler to introduce them from the start and instead we added the .immediate. pragma as a way to force the backward-compatible behaviour for the whole macro/template.
With typed/untyped, you have a more granular control over the individual parameters of the macro and the .immediate. pragma will be gradually phased out and deprecated.
I know how to declare aliases for simple types, like class types, primitive types and, say, pointers to functions returning the value of that types. Actually:
typedef int T; //T := int
typedef int* T; // T := int*
typedef int (*T)() //T := int (*)(). OK, but it's a bit unclear to me.
//Seems a little bit confused
typedef int (*T[])() // T := array of int(*)(). Totally confused. What the hell is going on?
I can't understand how the compiler should parse such typedef declarations. Maybe someone can explain on the simple example that I cited? I know, that c++ introduced alias decalrtion as follows:
using T = int*;
It could be more readable, but now I'm interested in only typedef decalration.
The grammar of a typedef is exactly the same as that of a variable
declaration; the only difference is that the name being declared becomes
an alias for the type, rather than an object, reference or function.
Note that typedef is part of the decl-specifier-seq of the
declaration; a full declaration consists of three parts: an
attribute-specifier-seq (new to C++11), a decl-specifier-seq, and an
init-declarator-list, in that order. All parts may in principle be
empty, but only for certain types of declarations; in the case of a
typedef, for example, only the attribute-specifier-seq may be empty.
To understand a declaration, you have to first break it down into the
three parts: the attribute-specifier-seq is easy: it will always be
within [[...]] and you won't see it too often, since it is very new,
and only for special uses. We'll ignore it for now. The
decl-specifier-seq is a sequence of keywords or symbols which name a
type (although there are special cases after some keywords, like
struct or enum); just collect all of the symbols until you encounter
something which isn't a keyword or a type. typedef included. Order
here isn't important, so:
int typedef const CI;
would be perfectly legal, although certainly not typical. If the
keyword typedef is present, the declaration is a typedef (which
means that some other keywords, like extern or static, aren't
allowed). The decl-specifier gives the final type in an English
expression of the type.
Everything that follows is part of the init-declarator-seq which is a
comma separated list of init-declarator. A typedef requires at
least one init-declarator, and in fact, doesn't allow the init part,
so it is just a declarator (but there can in fact be several, although
Microsoft is the only ones I know that go in for this bit of
obfuscation). Each declarator is basically an expression, with
the operators on the right (() and []) having precedence over the
operators on the left (* and &), and parentheses being used to
modify the precedence. So if you have something like (&ra)[10], ra
is a reference to an array[10] of... what:w
ever type is specified by the
decl-specified. Or where precedence is not given by parentheses:
*ra[10] is an array[10] of pointers to...
typedefs follow the same rule as variable declarations, so I will first cover these. The intended principle is: if you type the declaration as an expression, you will get the type. So let's analyse this variable:
int (*a[])();
Now we proceed step by step:
Typing (*a[42])() gives an int. Substitute x1 for (*a[42]). x1() is of type int, so clearly x1 (which is (*a[42])) is a function taking no parameters and returning an int.
Therefore, a[42] must be a pointer to a "function taking no parameters and returning an int."
Therefore, a must be an array of "pointers to a function taking no parameters and returning an int."
With typedefs, the only difference is that instead of the variable a, we're talking about its type. So typedef int (*T[])(); means:
T is the type a variable a would have if it were declared as int (*a[])();
So in your case, T is "an array of pointers to function-taking-no-parameter-and-returning-int."
I can't understand how the compiler should parse such typedef declarations.
Since this seemed to be a main topic of concern I decided to add an important note regarding the higher-level overview of how this is possible.
Since there is now context in the language you need to have information available about both the syntax and the semantics of the language when parsing.
In your example the original solution used the lex hack, which is the actual name for the method that C++ compiler designers implemented to solve the problem of the language no longer being context-free due to the typedef token. The basic idea of this 'hack' is to have an additional backchannel from the semantic analyzer to the lexer to provide the needed context.
There are also other ways to solve the problem of parsing context-sensitive grammars such has lexerless parsing.