Using variant in Bison and Flex - c++

Question

I've been rewriting a parser from c to c++, and as such I am trying to use variant with my code. However, I am not sure how to integrate it with flex, and I keep getting esoteric error messages.
My bison file looks like
%require "3"
%language "c++"
%{
// declarations
%}
%define api.value.type {std::variant<double, std::string>}
%token COMMENT
%token <double> DOUBLE
%token <std::string> STRING
// grammar
and my lexer looks like
%{
#include "y.tab.h"
%}
%option noyywrap
ID [a-zA-Z][a-zA-Z0-9_]*
%%
[ \t\n ]+ ;
\-?[0-9]+ |
\-?[0-9]+\. |
\-?[0-9]+\.[0-9]+ |
\-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
// other tokens
zA-Z][\.a-zA-Z0-9_]* { yylval.emplace<std::string>(yytext); return STRING;}
%%
I am not sure of my usage of yylval, I am trying to access the variant as I would have with the %union.
I get the following error:
y.tab.h:125:18: error: ‘variant’ in namespace ‘std’ does not name a template type
typedef std::variant<double, std::string> semantic_type;
^~~~~~~
y.tab.h:197:27: error: ‘semantic_type’ does not name a type
const semantic_type& v);
^~~~~~~~~~~~~
y.tab.h:212:7: error: ‘semantic_type’ does not name a type
semantic_type value;
^~~~~~~~~~~~~
my_mdl.l: In function ‘int yylex()’:
my_mdl.l:16:3: error: ‘yylval’ was not declared in this scope
\-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
^~~~~~
my_mdl.l:16:3: note: suggested alternative: ‘yylex’
\-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
^~~~~~
yylex
my_mdl.l:16:18: error: expected primary-expression before ‘double’
\-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
^~~~~~
my_mdl.l:16:53: error: ‘DOUBLE’ was not declared in this scope
\-?\.[0-9]+ { yylval.emplace<double>(std::atof(yytext)); return DOUBLE;}
^~~~~~
my_mdl.l:18:10: error: ‘COMMENT’ was not declared in this scope
"//".* { return COMMENT;}
^~~~~~~
my_mdl.l:37:29: error: expected primary-expression before ‘>’ token
[a-zA-Z][\.a-zA-Z0-9_]* { yylval.emplace<std::string>(yytext); return STRING;}
^
my_mdl.l:37:47: error: ‘STRING’ was not declared in this scope
[a-zA-Z][\.a-zA-Z0-9_]* { yylval.emplace<std::string>(yytext); return STRING;}
^~~~~~
I also get a few hundred lines of errors from my .y file such as
my_mdl.y:88:79: error: no matching function for call to ‘MOVE::MOVE(<brace-enclosed initializer list>)’
p.add_command(Command{in_place_index<5>, MOVE( {{$2, $3, $4}}, $5)});
^
In file included from parsing/symt.h:7:0,
from my_mdl.y:10:
parsing/cmd.h:44:5: note: candidate: MOVE::MOVE(const Scalable<double, 3>&, const string&)
MOVE(const Scalable<double, 3> &params, const std::string &scaleFactorName);
^~~~
MOVE is a struct defined as
struct MOVE {
MOVE(const Scalable<double, 3> &params, const std::string &scaleFactorName);
Scalable<double, 3> params; // todo equationify
std::string scale_factor_name;
};
and it is one of the types in the variant (std::variant<MOVE, etc...> Command). The strange thing is that this works normally in my code if I write
p.add_command(Command{in_place_index<5>, MOVE{{{x, y, z}}, "asdfads"}});

Answer 1

You have not included enough of your program to allow for a precise answer. Please see the SO help page on preparing a [mcse]. But it seems likely that you get the error
y.tab.h:125:18: error: ‘variant’ in namespace ‘std’ does not name a template type
because you haven't arranged for #include <variant> to be in your flex file.
The typedef itself is from the generated code in the header file produced by bison, but bison cannot guess what #include directives it might need, so it leaves it up to you to insert them. You must ensure that all types needed by your semantic type have been defined before you #include the bison-generated header. You could insert appropriate #include directives in the prologue block in your flex file, or you could use a %code requires block in your bison file. (Since you are using bison 3, the latter is probably the best solution.)
I have no idea what SAVE means in the errors from your bison file. I assume it is a macro you have (or have not defined), so the error would be a result of the macro expansion.

Answer 2

Bison actually provides its own custom type called 'variant', which is not the C++ std::variant, and supports it nicely.
https://www.gnu.org/software/bison/manual/bison.html#C_002b_002b-Variants
Bison provides a variant based implementation of semantic values for C++. This alleviates all the limitations reported in the previous section, and in particular, object types can be used without pointers.
To enable variant-based semantic values, set the %define variable api.value.type to variant (see %define Summary). Then %union is ignored; instead of using the name of the fields of the %union to “type” the symbols, use genuine types.
For instance, instead of:
%union
{
int ival;
std::string* sval;
}
%token <ival> NUMBER;
%token <sval> STRING;
write:
%token <int> NUMBER;
%token <std::string> STRING;
STRING is no longer a pointer, which should fairly simplify the user actions in the grammar and in the scanner (in particular the memory management).
To enable it, you simply specify
%define api.value.type variant
However this will also affect your lexer. I put together a small demo C++ flex/bison skeleton using the latest flex/bison binaries under Ubuntu:
https://github.com/kfsone/flub
highlights being:
/* lexer */
{string} yylval->emplace<std::string>(yytext); return Token::STRING;
/* parser */
%language "C++"
%skeleton "lalr1.cc"
%require "3.8.2"
%define api.value.type variant
/*
%define api.value.automove // use with care
*/
%define parse.assert
%define parse.trace
%define parse.error detailed
%define parse.lac full
/* ... */
/*
NB: You now use the TYPE rather than union-member name
*/
%token <std::string> STRING
/*
// ditto for nterms, but I now prefer putting them by the production itself.
%nterm <string::string> string_literal
*/
%%
/* ... */
/* Declare the type of this non-terminal, to be done here in the
productions section, you need a trailing semicolon */
%nterm <std::string> using_file;
using_file
: "using" STRING
{ $$ = $2; }
;
You can then use a much more modern AST-building approach, one thing I found helpful was to have a wrapper type of my own that captured a value AND its location.
template<typename ValueType>
struct ParsedType
{
location mLoc; // or YourParserType::location_type
ValueType mValue;
};
using ParsedString = ParsedType<std::string>;
/* ... */
%nterm <ParsedString> using_file;
using_file
: "using" STRING
{ $$ = ParsedString(#2, $2); }