As a corollary to a question I asked previously, I am curious why the type of the type of a name ('category' of that name) in a template is set in the first phase of the 2 phase lookup, when the category itself can also depend on the template parameter. What is the real gain of this behavior?
A little clarification - I think I have a fair understanding of how the 2 phase look-up works; what I'm trying to understand is why a category of a token is definitively determined in phase 1, which differs from when dependent types are determined (in phase 2). My argument is that there is a very real gain in simplifying a difficult syntax, to make code easier to write and to read, so I am curious what the compelling reason to restrict category evaluation to phase 1 is. Is it simply for better template validation/error messages before template instantiation, or a marginal increase in speed? Or is there some fundamental attribute of templates that makes phase 2 category evaluation unfeasible?
The question could be two fold: why do we want two phase lookup in the first place, and given that we have two phase lookup, why are the interpretation of the tokens fixed during the first phase. The first is the harder question to answer, as it is a design decision in the language and as such it has its advantages and disadvantages and depending on where you stand the ones or the others will have more weight.
The second part, which is what you are interested in, is actually much simpler. Why, in a C++ language with two phase lookup are the token meaning fixed during the first phase and cannot be left to be interpreted in the second phase. The reason is that C++ has a contextual grammar, and the interpretation of the tokens is highly dependent on the context. Without fixating the meaning of the tokens during the first phase you won't even know what names need to be looked up in the first place.
Consider a slightly modified version of your original code, where the literal 5 is substituted by a constant expression, and assuming that you did not need to provide the template or typename keywords that bit you the last time:
const int b = 5;
template<typename T>
struct Derived : public Base<T> {
void Foo() {
Base<T>::Bar<false>(b); // [1]
std::cout << b; // [2]
}
};
What are the possible meanings of [1] (ignoring the fact that in C++ this is determined by adding typename and template)?
Bar is a static template function that takes a single bool as template argument and an integer as argument. b is a non-dependent name that refers to the constant 5. *
Bar is a nested template type that takes a single bool as template argument. b is an instance of that type defined inside the function Derived<T>::Foo and not used.
Bar is a static member variable of a type X for which there is a comparison operator< that takes a bool and yields as result an object of type U that can be compared with operator> with an integer.
Now the question is how do we proceed resolving the names before the template arguments are substituted in (i.e. during the first phase). If we are in case 1. or 3. then b needs to be looked up and the result can be substituted in the expression. In the first case yielding your original code: Base<T>::template Bar<false>(5), in the latter case yielding operator>( operator<( Base<T>::Bar,false ), 5 ). In the third case (2.) the code after the first phase would be exactly the same as the original code: Base<T>::Bar<false> b; (removing the extra ()).
The meaning of the second line [2] is then dependent on how we interpreted the first one [1]. In the 2. case it represents a call to operator<<( std::cout, Base<T>::Bar<false> & ), while in the other two cases it represents operator<<( std::cout, 5 ). Again the implications extend beyond what type is the second argument, as in the 2. case name b within Derived<T>::Foo is dependent, and thus it cannot be resolved during the first phase but rather postponed to the second phase (where it will also affect lookup by adding the namespaces of Base and the instantiating type T to the Argument Dependent Lookup).
As the example shows, the interpretation of the tokens impact the meaning of the names, and that in turn affects what the rest of the code means, what names are dependent or not and thus what else needs to be looked up or not during the first phase. At the same time, the compiler does perform checks during the first pass, and if the tokens could be reinterpreted during the second pass, then the checks and the results of the lookup during the first pass would be rendered useless (imagine that during the first pass b had been substituted with 5 only to find out that we are in case 2. during the second phase!), and everything would have to be checked during the second phase.
The existence of two phase lookup depends on the tokens being interpreted and it's meaning selected during the first phase. The alternative is a single pass lookup as VS does.
* I am simplifying the cases here, in the Visual Studio compiler, that does not implement two-phase lookup, b could also be a member of Base<T> for the currently instantiating type T (i.e. it can be a dependent name)
Much of the advantages of C++ is that it's a strictly checked language. You express the intent of your program as clearly as possible, and the compiler tells you if that intent is violated.
I can't imagine that you would ever write Base<T>::Bar<false>(b); (from Dribeas's example) and not have a particular interpretation that you want. By telling the interpretation to the compiler (Base<T>::typename Bar<false>(b);), it can generate a meaningful error if someone provides a type that has a static member Bar or a nested template type instead of a member function template.
Other languages are designed to stress terseness over static analysis; for example many dynamic languages have a great number of "do what I mean" rules. Which causes fun when the compiler turns non-sensible code into something unpredictable, with no errors. (Case in point: Perl. I love it for text manipulation, but goodness DWIM is annoying. Almost everything is a runtime error, there's barely any static checking to speak of)
Related
Why does accumulate in C++ have two templates defined when the job can be done with just one template (the one with the binaryOperation and default value to sum)?
I am referring to the accumulate declaration from http://www.cplusplus.com/reference/numeric/accumulate/
Because that's how the standard has been specified.
It is often a matter of taste whether to use an overload or a default argument. In this case, overload was chosen (by committee, by Alexander Stepanov, or by whoever happened to be responsible for the choice).
Default values are more limited than overloads. For example, you can have a function pointer T (*)(InputIterator, InputIterator, T) pointing to the first overload, which would not be possible if there was only one function (template) with 4 arguments. This flexibility can be used as an argument for using overloads rather than default arguments when possible.
It's true you would get mostly the same behavior from a single template like
template <class InputIt, class T, class BinaryOperation = std::plus<>>
accumulate(InputIt first, InputIt last, T init, BinaryOperation op = {});
But note that in earlier versions of C++, this would be difficult or impossible:
Prior to C++11, a function template could not have default template arguments.
Prior to C++14, std::plus<> (which is the same as std::plus<void>) was not valid: the class template could only be instantiated with one specific argument type.
The accumulate template is even older than the first C++ Standard of 1998: it goes back to the SGI STL library. At that time, compiler support for templates was rather inconsistent, so it was advisable to keep templates as simple as possible.
So the original two declarations were kept. As noted in bobah's answer, combining them into one declaration could break existing code, since for example code might be using a function pointer to an instantiation of the three-argument version (and function pointers cannot represent a default function argument, whether the function is from a template or not).
Sometimes the Standard library will add additional overloads to an existing function, but usually only for a specific purpose that would improve the interface, and when possible without breaking old code. There hasn't been any such reason for std::accumulate.
(But note member functions in the standard library can change more often than non-member functions like std::accumulate. The Standard gives implementations permission to declare member functions with different overloads, default arguments, etc. than specified as long as the effects are as described. This means it's generally a bad idea to take pointers to member functions to standard library class members, or otherwise assume very specific declarations, in the first place.)
The motivtion for the 2 functions is the same reason that we have both a copy and a transform function, to give the coder the flexability to apply a function on a per element basis. But perhaps some real world code would be helpful in understanding where this would be used. I've used both these snipits professionally in coding:
The 1st instance of accumulate can be used to sum the elements of a range. For example, given const int input[] = { 13, 42 } I can do this to get the sum of all elements in input:
accumulate(cbegin(input), cend(input), 0) /* Returns 55 */
I personally most commonly use the 2nd instance to generate strings (because it's the closest thing c++ has to a join) but it can also be used when special preprocessing is needed before the element is added. For example:
accumulate(next(cbegin(input)), cend(input), to_string(front(input)), [](const auto& current_sum, const auto i){ return current_sum + ", " + to_string(i); }) /* Returns "13, 42"s */
It's worth noting P0616R0 when considering my use of the 2nd function. This proposal has been accepted into c++20 and will move rather than copy the first parameter to accumulate's functor, which, "Can lead to massive improvements (particularly, it
means accumulating strings is linear rather than quadratic)."
I try to create a c++ flex/bison parser. I used this tutorial as a starting point and did not change any bison/flex configurations. I am stuck now to the point of trying to unit test the lexer.
I have a function in my unit tests that directly calls yylex, and checks the result of it:
private: static void checkIntToken(MyScanner &scanner, Compiler *comp, unsigned long expected, unsigned char size, char isUnsigned, unsigned int line, const std::string &label) {
yy::MyParser::location_type loc;
yy::MyParser::semantic_type semantic; // <---- is seems like the destructor of this variable causes the crash
int type = scanner.yylex(&semantic, &loc, comp);
Assert::equals(yy::MyParser::token::INT, type, label + "__1");
MyIntToken* token = semantic.as<MyIntToken*>();
Assert::equals(expected, token->value, label + "__2");
Assert::equals(size, token->size, label + "__3");
Assert::equals(isUnsigned, token->isUnsigned, label + "__4");
Assert::equals(line, loc.begin.line, label + "__5");
//execution comes to this point, and then, program crashes
}
The error message is:
program: ../src/__autoGenerated__/MyParser.tab.hh:190: yy::variant<32>::~variant() [S = 32]: Assertion `!yytypeid_' failed.
I have tried to follow the logic in the auto-generated bison files, and make some sense out of it. But I did not succeed on that and ultimately gave up. I searched then for any advice on the web about this error message but did not find any.
The location indicated by the error has the following code:
~variant (){
YYASSERT (!yytypeid_);
}
EDIT: The problem disappears only if I remove the
%define parse.assert
option from the bison file. But I am not sure if this is a good idea...
What is the proper way to obtain the value of the token generated by flex, for unit testing purposes?
Note: I've tried to explain bison variant types to the best of my knowledge. I hope it is accurate but I haven't used them aside from some toy experiments. It would be an error to assume that this explanation in any way implies an endorsement of the interface.
The so-called "variant" type provided by bison's C++ interface is not a general-purpose variant type. That was a deliberate decision based on the fact that the parser is always able to figure out the semantic type associated with a semantic value on the parser stack. (This fact also allows a C union to be used safely within the parser.) Recording type information within the "variant" would therefore be redundant. So they don't. In that sense, it is not really a discriminated union, despite what one might expect of a type named "variant".
(The bison variant type is a template with an integer (non-type) template argument. That argument is the size in bytes of the largest type which is allowed in the variant; it does not in any other way specify the possible types. The semantic_type alias serves to ensure that the same template argument is used for every bison variant object in the parser code.)
Because it is not a discriminated union, its destructor cannot destruct the current value; it has no way to know how to do that.
This design decision is actually mentioned in the (lamentably insufficient) documentation for the Bison "variant" type. (When reading this, remember that it was originally written before std::variant existed. These days, it would be std::variant which was being rejected as "redundant", although it is also possible that the existence of std::variant might have had the happy result of revisiting this design decision). In the chapter on C++ Variant Types, we read:
Warning: We do not use Boost.Variant, for two reasons. First, it appeared unacceptable to require Boost on the user’s machine (i.e., the machine on which the generated parser will be compiled, not the machine on which bison was run). Second, for each possible semantic value, Boost.Variant not only stores the value, but also a tag specifying its type. But the parser already “knows” the type of the semantic value, so that would be duplicating the information.
Therefore we developed light-weight variants whose type tag is external (so they are really like unions for C++ actually).
And indeed they are. So any use of a bison "variant" must have a definite type:
You can build a variant with an argument of the type to build. (This is the only case where you don't need a template parameter, because the type is deduced from the argument. You would have to use an explicit template parameter only if the argument were not of the precise type; for example, an integer of lesser rank.)
You can get a reference to the value of known type T with as<T>. (This is undefined behaviour if the value has a different type.)
You can destruct the value of known type T with destroy<T>.
You can copy or move the value from another variant of known type T with copy<T> or move<T>. (move<T> involves constructing and then destructing a T(), so you might not want to do it if T had an expensive default constructor. On the whole, I'm not convinced by the semantics of the move method. And its name conflicts semantically with std::move, but again it came first.)
You can swap the values of two variants which both have the same known type T with swap<T>.
Now, the generated parser understands all these restrictions, and it always knows the real types of the "variants" it has at its disposal. But you might come along and try to do something with one of these objects in a way that violates a constraint. Since the object really doesn't have any way to check the constraint, you'll end up with undefined behaviour which will probably have some disastrous eventual consequence.
So they also implemented an option which allows the "variant" to check the constraints. Unsurprisingly, this consists of adding a discriminator. But since the discriminator is only used to validate and not to modify behaviour, it is not a small integer which chooses between a small number of known alternatives, but rather a pointer to a std::typeid (or NULL if the variant does not yet contain a value.) (To be fair, in most cases alignment constraints mean that using a pointer for this purpose is no more expensive than using a small enum. All the same...)
So that's what you're running into. You enabled assertions with %define parse.assert; that option was provided specifically to prevent you from doing what you are trying to do, which is let the variant object's destructor run before the variant's value is explicitly destructed.
So the "correct" way to avoid the problem is to insert an explicit call at the end of the scope:
// execution comes to this point, and then, without the following
// call, the program will fail on an assertion
semantic.destroy<MyIntType*>();
}
With the parse assertion enabled, the variant object will be able to verify that the types specified as template parameters to semantic.as<T> and semantic.destroy<T> are the same types as the value stored in the object. (Without parse.assert, that too is your responsibility.)
Warning: opinion follows.
In case anyone reading this cares, my preference for using real std::variant types comes from the fact that it is actually quite common for the semantic value of an AST node to require a discriminated union. The usual solution (in C++) is to construct a type hierarchy which is, in some ways, entirely artificial, and it is quite possible that std::variant can better express the semantics.
In practice, I use the C interface and my own discriminated union implementation.
In C++, the symbols '<' and '>' are used for comparisons as well as for signifying a template argument. Thus, the code snippet
[...] Foo < Bar > [...]
might be interpreted as any of the following two ways:
An object of type Foo with template argument Bar
Compare Foo to Bar, then compare the result to whatever comes next
How does the parser for a C++ compiler efficiently decide between those two possibilities?
If Foo is known to be a template name (e.g. a template <...> Foo ... declaration is in scope, or the compiler sees a template Foo sequence), then Foo < Bar cannot be a comparison. It must be a beginning of a template instantiation (or whatever Foo < Bar > is called this week).
If Foo is not a template name, then Foo < Bar is a comparison.
In most cases it is known what Foo is, because identifiers generally have to be declared before use, so there's no problem to decide one way or the other. There's one exception though: parsing template code. If Foo<Bar> is inside a template, and the meaning of Foo depends on a template parameter, then it is not known whether Foo is a template or not. The language standard directs to treat it as a non-template unless preceded by the keyword template.
The parser might implement this by feeding context back to the lexer. The lexer recognizes Foo as different types of tokens, depending on the context provided by the parser.
The important point to remember is that C++ grammar is not context-free. I.e., when the parser sees Foo < Bar (in most cases) knows that Foo refers to a template definition (by looking it up in the symbol table), and thus < cannot be a comparison.
There are difficult cases, when you literally have to guide the parser. For example, suppose that are writing a class template with a template member function, which you want to specialize explicitly. You might have to use syntax like:
a->template foo<int>();
(in some cases; see Calling template function within template class for details)
Also, comparisons inside non-type template arguments must be surrounded by parentheses, i.e.:
foo<(A > B)>
not
foo<A > B>
Non-static data member initializers bring more fun: http://open-std.org/JTC1/SC22/WG21/docs/cwg_active.html#325
C and C++ parsers are "context sensitive", in other words, for a given token or lexeme, it is not guaranteed to be distinct and have only one meaning - it depends on the context within which the token is used.
So, the parser part of the compiler will know (by understanding "where in the source it is") that it is parsing some kind of type or some kind of comparison (This is NOT simple to know, which is why reading the source of competent C or C++ compiler is not entirely straight forward - there are lots of conditions and function calls checking "is this one of these, if so do this, else do something else").
The keyword template helps the compiler understand what is going on, but in most cases, the compiler simply knows because < doesn't make sense in the other aspect - and if it doesn't make sense in EITHER form, then it's an error, so then it's just a matter of trying to figure out what the programmer might have wanted - and this is one of the reasons that sometimes, a simple mistake such as a missing } or template can lead the entire parsing astray and result in hundreds or thousands of errors [although sane compilers stop after a reasonable number to not fill the entire universe with error messages]
Most of the answers here confuse determining the meaning of the symbol (what I call "name resolution") with parsing (defined narrowly as "can read the syntax of the program").
You can do these tasks separately..
What this means is that you can build a completely context-free parser for C++ (as my company, Semantic Designs does), and leave the issues of deciding what the meaning of the symbol is to a explicitly seperate following task.
Now, that task is driven by the possible syntax interpretations of the source code. In our parsers, these are captured as ambiguities in the parse.
What name resolution does is collect information about the declarations of names, and use that information to determine which of the ambiguous parses doesn't make sense, and simply drop those. What remains is a single valid parse, with a single valid interpretation.
The machinery to accomplish name resolution in practice is a big mess. But that's the C++ committee's fault, not the parser or name resolver. The ambiguity removal with our tool is actually done automatically, making that part actually pretty nice but if you don't look inside our tools you would not appreciate that, but we do because it means a small engineering team was able to build it.
See an example of resolution of template-vs-less than on C++s most vexing parse done by our parser.
I've been lately using templates and macros, but i have to say i have barely found information about these important types. This is my superficial understanding:
typed/expr is something that must exists previously, but you can use .immediate. to overcome them.
untyped/stmt is something that doesn't to be defined previously/one or more statements.
This is a very vague notion of the types. I'd like to have a better explanation of them, including which types should be used as return.
The goal of these different parameter types is to give you several increasing levels of precision in specifying what the compiler should accept as a parameter to the macro.
Let's imagine a hypothetical macro that can solve mathematical equations. It will be used like this:
solve(x + 10 = 25) # figures out that the correct value for x is 15
Here, the macro just cares about the structure of the supplied AST tree. It doesn't require that the same tree is a valid expression in the current scope (i.e. that x is defined and so on). The macro just takes advantage of the Nim parser that already can decode most of the mathematical equations to turn them into easier to handle AST trees. That's what untyped parameters are for. They don't get semantically checked and you get the raw AST.
On the next step in the precision ladder are the typed parameters. They allow us to write a generic kind of macro that will accept any expression, as long as it has a proper meaning in the current scope (i.e. its type can be determined). Besides catching errors earlier, this also has the advantage that we can now work with the type of the expression within the macro body (using the macros.getType proc).
We can get even more precise by requiring an expression of a specific type (either a concrete type or a type class/concept). The macro will now be able to participate in overload resolution like a regular proc. It's important to understand that the macro will still receive an AST tree, as it will accept both expressions that can be evaluated at compile-time and expressions that can only be evaluated at run-time.
Finally, we can require that the macro receives a value of specific type that is supplied at compile-time. The macro can work with this value to parametrise the code generation. This is realm of the static parameters. Within the body of the macro, they are no longer AST trees, but rather ordinary well typed values.
So far, we've only talked about expressions, but Nim's macros also accept and produce blocks and this is the second axis, which we can control. expr generally means a single expression, while stmt denotes a list of expressions (historically, its name comes from StatementList, which existed as a separate concept before expressions and statements were unified in Nim).
The distinction is most easily illustrated with the return types of templates. Consider the newException template from the system module:
template newException*(exceptn: typedesc, message: string): expr =
## creates an exception object of type ``exceptn`` and sets its ``msg`` field
## to `message`. Returns the new exception object.
var
e: ref exceptn
new(e)
e.msg = message
e
Even thought it takes several steps to construct an exception, by specifying expr as the return type of the template, we tell the compiler that only that last expression will be considered as the return value of the template. The rest of the statements will be inlined, but cleverly hidden from the calling code.
As another example, let's define a special assignment operator that can emulate the semantics of C/C++, allowing assignments within if statements:
template `:=` (a: untyped, b: typed): bool =
var a = b
a != nil
if f := open("foo"):
...
Specifying a concrete type has the same semantics as using expr. If we had used the default stmt return type instead, the compiler wouldn't have allowed us to pass a "list of expressions", because the if statement obviously expects a single expression.
.immediate. is a legacy from a long-gone past, when templates and macros didn't participate in overload resolution. When we first made them aware of the type system, plenty of code needed the current untyped parameters, but it was too hard to refactor the compiler to introduce them from the start and instead we added the .immediate. pragma as a way to force the backward-compatible behaviour for the whole macro/template.
With typed/untyped, you have a more granular control over the individual parameters of the macro and the .immediate. pragma will be gradually phased out and deprecated.
These should probably be in different questions, but they're related so...
Why do we need to write constexpr at all? Given a set of restrictions couldn't a compiler evaluate code to see if it satisfies the constexpr requirements, and treat it as constexpr if it does? As a purely documentation keyword I'm not sure it holds up because I can't think of a case where I (the user of someone else's constexpr function) should really care if it's run time or not.
Here's my logic: If it's an expensive function I think as a matter of good practice I should treat it as such regardless of whether I give it compile-time constant input or not. That might mean calling it during load time and saving off the result, instead of calling it during a critical point in the execution. The reason is because constexpr doesn't actually guarantee to me that it will not be executed in run time in the first place — so perhaps a new/different mechanism should do that.
The constexpr restrictions seem to exclude many, if not most, functions from being compile-time evaluated which logically could be. I've read this is at least in part (or perhaps wholly?) to prevent infinite looping and hanging the compiler. But, if this is the reason, is it legitimate?
Shouldn't a compiler be able to compute if, for any given constexpr function with the given inputs used, it loops infinitely? This is not solving the halting problem for any input. The input to a constexpr function is compile time constant and finite, so the compiler only has to check for infinite looping for a finite set of input: the input actually used. It should be a regular compilation error if you write a compile-time infinite loop.
I asked a very similar question, Why do we need to mark functions as constexpr?
When I pressed Richard Smith, a Clang author, he explained:
The constexpr keyword does have utility.
It affects when a function template specialization is instantiated (constexpr function template specializations may need to be instantiated if they're called in unevaluated contexts; the same is not true for non-constexpr functions since a call to one can never be part of a constant expression). If we removed the meaning of the keyword, we'd have to instantiate a bunch more specializations early, just in case the call happens to be a constant expression.
It reduces compilation time, by limiting the set of function calls that implementations are required to try evaluating during translation. (This matters for contexts where implementations are required to try constant expression evaluation, but it's not an error if such evaluation fails -- in particular, the initializers of objects of static storage duration.)
This all didn't seem convincing at first, but if you work through the details, things do unravel without constexpr. A function need not be instantiated until it is ODR-used, which essentially means used at runtime. What is special about constexpr functions is that they can violate this rule and require instantiation anyway.
Function instantiation is a recursive procedure. Instantiating a function results in instantiation of the functions and classes it uses, regardless of the arguments to any particular call.
If something went wrong while instantiating this dependency tree (potentially at significant expense), it would be difficult to swallow the error. Furthermore, class template instantiation can have runtime side-effects.
Given an argument-dependent compile-time function call in a function signature, overload resolution may incur instantiation of function definitions merely auxiliary to the ones in the overload set, including the functions that don't even get called. Such instantiations may have side effects including ill-formedness and runtime behavior.
It's a corner case to be sure, but bad things can happen if you don't require people to opt-in to constexpr functions.
As for constexpr objects, certain types can produce core constant expressions which are usable in constant expression contexts without having been declared constexpr. But you don't really want the compiler to try evaluating every single expression at compile time. That's what constant propagation is for. On the other hand it seems pretty essential to document when something needs to happen at compile time.
[Note, I totally changed my answer]
To answer your second question, there are two cases for the compiler here:
The compiler has to be able to handle any arbitrary constexpr function(s). In this case you still have the halting problem because the set of inputs is all combinations of constexpr functions and calls to them.
The compiler can handle a finite set of constexpr function(s). In this case the compiler can in fact determine whether some programs will result in infinite loops, while other programs will be uncompilable (since they aren't in the set of valid inputs).
So presumably the restrictions are in place so that it satisfies case 2 for a reasonable amount of compiler effort.
There are both technical and ideological reasons behind this decision.
Not always do we want constexpr ourselves by default - it can take
too much compiling time. That's first. Just imagine you implemented
isPrime function and you have 100 calls with big constexpr
values passed in. I think you don't (in most cases) want to make
compiler compiling this for a couple of minutes longer because it
decided that you need those values in compile-time by itself. But if
it's exactly the case - specify constexpr modifier manually. And this adds the next point:
backward compatibility - it's unwise to assume that every possible C++98 program author who converted this program to C++11 wantsconstexpr.
The second point is that deciding if the function can be constexpr
would take compiling time by itself. And if it was trying to do that for every possible function it would take some additional time overhead. Even more, often compiler
couldn't decide if the given function can be constexpr at all, so
your first assumption is not correct.