I'm writing a little compiler just for fun and I'm using Boost Spirit Qi to describe my grammar. Now I want to make a minor change in the grammar to prepare some further additions. Unfortunately these changes won't compile and I would like to understand why this is the case.
Here is a snippet from the code I want to change. I hope the provided information is enough to understand the idea. The complete code is a bit large, but if you want to look at it or even test it (Makefile and Travis CI is provided), see https://github.com/Kruecke/BFGenerator/blob/8f66aa5/bf/compiler.cpp#L433.
typedef boost::variant<
function_call_t,
variable_declaration_t,
variable_assignment_t,
// ...
> instruction_t;
struct grammar : qi::grammar<iterator, program_t(), ascii::space_type> {
grammar() : grammar::base_type(program) {
instruction = function_call
| variable_declaration
| variable_assignment
// | ...
;
function_call = function_name >> '(' > -(variable_name % ',') > ')' > ';';
// ...
}
qi::rule<iterator, instruction::instruction_t(), ascii::space_type> instruction;
qi::rule<iterator, instruction::function_call_t(), ascii::space_type> function_call;
// ...
};
So far, everything is just working fine. Now I want to move the parsing of the trailing semicolon (> ';') from the function_call rule to the instruction rule. My code now looks like this:
struct grammar : qi::grammar<iterator, program_t(), ascii::space_type> {
grammar() : grammar::base_type(program) {
instruction = (function_call > ';') // Added trailing semicolon
| variable_declaration
| variable_assignment
// | ...
;
// Removed trailing semicolon here:
function_call = function_name >> '(' > -(variable_name % ',') > ')';
// ...
}
From my understanding the rules haven't really changed because the character parser ';' doesn't yield any attribute and so it shouldn't matter where this parser is positioned. However, this change won't compile:
/usr/include/boost/spirit/home/support/container.hpp:278:13: error: no matching function for call to ‘std::basic_string<char>::insert(std::basic_string<char>::iterator, const bf::instruction::function_call_t&)’
c.insert(c.end(), val);
^
(This error comes from the instruction = ... line.)
Why is this change not compiling? I'm rather looking for an explanation to understand what's going on than a workaround.
Ok, so after looking at this closely, you are trying to insert multiple strings into your function_call_t type, which is a fusion sequence that can be converted to from a single std::string. However, you are probably going to run into issues with your function_call rule because it's attribute is actually tuple <std::string, optional <vector <std::string>>>. I'd imagine that spirit is having issues flattening that structure out and that is causing your issue, however, I don't have a compiler to test it out at the moment.
Related
I'm trying to make a JSON parser but my object rule doesn't compile...
Code (complete code here):
// AST
using Object = std::map<std::string, struct Value>; // (Value is a variant which can contain a float, a string, an Object or an Array)
// Grammar def
using ObjectType = x3::rule<struct ObjectClass, Object>;
const ObjectType obj{"object"};
const auto obj_def = '{' > ((quotedString > ':' > val) % ',') > '}';
Error (complete error here):
/usr/include/boost/spirit/home/x3/support/traits/container_traits.hpp:77:56: error: no type named 'value_type' in 'std::pair<std::basic_string<char>, Json::Value>'
: detail::remove_value_const<typename Container::value_type>
~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
The type std::pair<std::basic_string<char>, Json::Value> is good, but it must be an array (std::vector<std::pair<std::basic_string<char>, Json::Value>>, so std::map<std::basic_string<char>, Json::Value>)
What is the problem?
Your diagnosis is off the mark. You can just eliminate rules and defs until you find the culprit. The obj_def is the culprit, which you can confirm by commenting it out:
const auto obj_def = x3::eps; // '{' > ((quotedString > ':' > val) % ',') > '}';
In your grammardef.hpp you need to include
#include <boost/fusion/adapted.hpp>
so that Fusion knows how to deal with std::pair<std::string, Json::Value>.
This is a FAQ entry since early days of Spirit V2 (http://boost-spirit.com/home/articles/qi-example/parsing-a-list-of-key-value-pairs-using-spirit-qi/).
Also, bear in mind that some implementations will expect properties to be ordered (this is not actually specified) and you might want to check against duplicate keys (especially after normalizing unicode escapes).
I am using Spirit Qi as my parser, to parse mathematical expressions into an expression tree. I keep track of such things as the types of the symbols which are encountered as I parse, and which must be declared in the text I am parsing. Namely, I am parsing Bertini input files, a simple-ish example of which is here, a complicated example is here, and for completeness purposes, as below:
%input: our first input file
variable_group x,y;
function f,g;
f = x^2 - 1;
g = y^2 - 4;
END;
The grammar I have been working on will ideally
find declaration statements, and then parse the following comma-separated list of symbols of the type being declared, and store the resulting vector of symbols in the class object being parsed into; e.g. variable_group x, y;
find a previously declared symbol, which is followed by an equals sign, and is the definition of that symbol as an evaluatable mathematical object; e.g. f = x^2 - 1; This part I mostly have under control.
find a not-previously declared symbol followed by =, and parse it as a subfunction. I think I can handle this, too.
The problem I have been struggling to solve seems like it is so trivial, yet after hours of searching, I still haven't gotten there. I've read dozens of Boost Spirit mailing list posts, SO posts, the manual, and the headers for Spirit themselves, yet still don't quite grok a few critical things about Spirit Qi parsing.
Here is the problematic basic grammar definition, which would go in system_parser.hpp:
#define BOOST_SPIRIT_USE_PHOENIX_V3 1
#include <boost/spirit/include/qi_core.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <iostream>
#include <string>
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
template<typename Iterator>
struct SystemParser : qi::grammar<Iterator, std::vector<std::string>(), boost::spirit::ascii::space_type>
{
SystemParser() : SystemParser::base_type(variable_group_)
{
namespace phx = boost::phoenix;
using qi::_1;
using qi::_val;
using qi::eps;
using qi::lit;
qi::symbols<char,int> encountered_variables;
qi::symbols<char,int> declarative_symbols;
declarative_symbols.add("variable_group",0);
// wraps the vector between its appropriate declaration and line termination.
BOOST_SPIRIT_DEBUG_NODE(variable_group_);
debug(variable_group_);
variable_group_.name("variable_group_");
variable_group_ %= lit("variable_group") >> genericvargp_ >> lit(';');
// creates a vector of strings
BOOST_SPIRIT_DEBUG_NODE(genericvargp_);
debug(genericvargp_);
genericvargp_.name("genericvargp_");
genericvargp_ %= new_variable_ % ',';
// will in the future make a shared pointer to an object using the string
BOOST_SPIRIT_DEBUG_NODE(new_variable_);
debug(new_variable_);
new_variable_.name("new_variable_");
new_variable_ %= unencountered_symbol_;
// this rule gets a string.
BOOST_SPIRIT_DEBUG_NODE(unencountered_symbol_);
debug(unencountered_symbol_);
unencountered_symbol_.name("unencountered_symbol");
unencountered_symbol_ %= valid_variable_name_ - ( encountered_variables | declarative_symbols);
// get a string which fits the naming rules.
BOOST_SPIRIT_DEBUG_NODE(valid_variable_name_);
valid_variable_name_.name("valid_variable_name_");
valid_variable_name_ %= +qi::alpha >> *(qi::alnum | qi::char_('_') | qi::char_('[') | qi::char_(']') );
}
// rule declarations. these are member variables for the parser.
qi::rule<Iterator, std::vector<std::string>(), ascii::space_type > variable_group_;
qi::rule<Iterator, std::vector<std::string>(), ascii::space_type > genericvargp_;
qi::rule<Iterator, std::string(), ascii::space_type> new_variable_;
qi::rule<Iterator, std::string(), ascii::space_type > unencountered_symbol_;// , ascii::space_type
// the rule which determines valid variable names
qi::rule<Iterator, std::string()> valid_variable_name_;
};
and some code which uses it:
#include "system_parsing.hpp"
int main(int argc, char** argv)
{
std::vector<std::string> V;
std::string str = "variable_group x, y, z;";
std::string::const_iterator iter = str.begin();
std::string::const_iterator end = str.end();
SystemParser<std::string::const_iterator> S;
bool s = phrase_parse(iter, end, S, boost::spirit::ascii::space, V);
std::cout << "the unparsed string:\n" << std::string(iter,end);
return 0;
}
It compiles under Clang 4.9.x on OSX just fine. When I run it, I get:
Assertion failed: (px != 0), function operator->, file /usr/local/include/boost/smart_ptr/shared_ptr.hpp, line 648.
Alternately, if I use expectation operator > rather than >> in the definition of the variable_group_ rule, I get our dear old friend Segmentation fault: 11.
In my learning process, I've come across such excellent posts as how to tell the type spirit is trying to generate, attribute propagation, how to interact with symbols, an example of infinite left recursion which lead to a segfault, information on parsing into classes, not structs which has a link to using Customization points (yet the links contain no examples), the Nabialek trick which couples keywords to actions, and perhaps most relevant for what I am trying to do dynamic difference parsing which is certainly something I need since the set of symbols grows, and I disallow usage of them as another type later, as the set of already-encountered symbols starts empty, and grows -- that it, the rules for parsing are dynamic.
So here's where I am at. My current problem is the assert/segfault generated by this particular example. However, I am unclear on some things, and need guiding advice, which I just haven't put together from any of the sources I have consulted, and the request for which hopefully makes this SO question disjoint from others previously asked:
When is it appropriate to use lexeme? I just don't know when to use lexeme, and not.
What are some guidelines for when to use > rather than >>?
I've seen many Fusion adapt examples where there is a struct to be parsed into, and a set of rules to do so. My input files will possibly have multiple occurrences of declarations of function, variables, etc, which all need to go the same place, so I need to be able to add to fields of the terminal class object into which I am parsing, in any order, multiple times. I think I would like to use getter/setters for the class object, so that parsing is not the only pathway to object construction. Is this a problem?
Any kind advice for this beginner is most welcome.
You reference the symbols variables. But they are locals so they don't exist once the constructor returns. This invokes Undefined Behaviour. Anything can happen.
Make the symmbol tables members of the class.
Also simplifying the dance around
the skippers (see Boost spirit skipper issues). That link also answers your _"When is it appropriate to use lexeme[]. In your sample you lacked the lexeme[] around encountered_variables|declarative_symbols, for example.
the debug macros
the operator%=, and some generally unused stuff
guessing you didn't need the mapped type of the symbols<> (because the int wasn't consumed), simplified the initialization there
Demo
Live On Coliru
#define BOOST_SPIRIT_USE_PHOENIX_V3 1
#define BOOST_SPIRIT_DEBUG 1
#include <boost/spirit/include/qi_core.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <iostream>
#include <string>
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
template <typename Iterator, typename Skipper = ascii::space_type>
struct SystemParser : qi::grammar<Iterator, std::vector<std::string>(), Skipper> {
SystemParser() : SystemParser::base_type(variable_group_)
{
declarative_symbols += "variable_group";
variable_group_ = "variable_group" >> genericvargp_ >> ';';
genericvargp_ = new_variable_ % ',';
valid_variable_name_ = qi::alpha >> *(qi::alnum | qi::char_("_[]"));
unencountered_symbol_ = valid_variable_name_ - (encountered_variables|declarative_symbols);
new_variable_ = unencountered_symbol_;
BOOST_SPIRIT_DEBUG_NODES((variable_group_) (valid_variable_name_) (unencountered_symbol_) (new_variable_) (genericvargp_))
}
private:
qi::symbols<char, qi::unused_type> encountered_variables, declarative_symbols;
// rule declarations. these are member variables for the parser.
qi::rule<Iterator, std::vector<std::string>(), Skipper> variable_group_;
qi::rule<Iterator, std::vector<std::string>(), Skipper> genericvargp_;
qi::rule<Iterator, std::string()> new_variable_;
qi::rule<Iterator, std::string()> unencountered_symbol_; // , Skipper
// the rule which determines valid variable names
qi::rule<Iterator, std::string()> valid_variable_name_;
};
//#include "system_parsing.hpp"
int main() {
using It = std::string::const_iterator;
std::string const str = "variable_group x, y, z;";
SystemParser<It> S;
It iter = str.begin(), end = str.end();
std::vector<std::string> V;
bool s = phrase_parse(iter, end, S, boost::spirit::ascii::space, V);
if (s)
{
std::cout << "Parse succeeded: " << V.size() << "\n";
for (auto& s : V)
std::cout << " - '" << s << "'\n";
}
else
std::cout << "Parse failed\n";
if (iter!=end)
std::cout << "Remaining unparsed: '" << std::string(iter, end) << "'\n";
}
Prints
Parse succeeded: 3
- 'x'
- 'y'
- 'z'
I'm parsing some input that is vaguely structured like C-ish code. Like this:
Name0
{
Name1
{
//A COMMENT!!
Param0 *= 2
Param2 = "lol"
}
}
Part of that is comments, which I want to totally ignore (and it's not working). I consider two things to be a node, the named scopes (category rule) like Name0 {} and the values (param rule) like Param0 *= 2... then there is comment. I've tried setting things up like this:
typedef boost::variant<boost::recursive_wrapper<Category>, Param> Node;
qi::rule<Iterator, Node(), ascii::space_type> node;
So the node rule puts either a Category or a Param in a variant. Here are the other rules (I've omitted some rules that don't really matter for this):
qi::rule<Iterator> comment; //comment has no return type
qi::rule<Iterator, Category(), ascii::space_type> category;
qi::rule<Iterator, Param(), ascii::space_type> param;
And their actual code:
comment = "//" >> *(char_ - eol);
param %=
tagstring
>> operators
>> value;
category %=
tagstring
>> '{'
>> *node
> '}';
node %= comment | category | param;
comment is setup to use = instead of %=, and it has no return type. However, comments end up creating null Categorys in my output Nodes wherever they show up. I've tried moving comment out of the node rule and into category like this:
category %=
tagstring
>> '{'
>> *(comment | node)
> '}';
And various other things, but those null entries keep popping up. I had to make comment output a string and put std::string in my Node variant just to sorta catch them, but that messes up my ability to stick in commenting in other parts of my rules (unless I actually grab the string in every location).
How can I completely ignore the comment and have it not show up in any output in any way?
edit: You'd think omit would do it, but didn't seem to change anything...
edit 2: Referencing this SO answer, I have a shaky solution in this:
node %= category | param;
category %=
tagstring
>> '{'
>> *comment >> *(node >> *comment)
> '}';
However, I want to try to stick comments into all sorts of places (between tagstring and {, in my unshown root rule between root categorys, etc). Is there a simpler method than this? I was hoping it could be done with a simple >> commentwrapper plugged into wherever I wanted...
Alright, so making your own skipper isn't too bad. And it elegantly solves this commenting problem, just as Mike M said. I define my rules in a struct called Parser that is templated with an Iterator. Had to make some adjustments to use the skipper. First, here is the skipper which is defined in Parser with all my other rules:
typedef qi::rule<Iterator> Skipper;
Skipper skipper;
So skipper is a rule of type Skipper. Here is what my Parser struct looked like originally, where it was using the ascii::space rule of type ascii::space_type as its skipper, which IS NOT the same type as qi::rule<Iterator> that skipper is based on!
struct Parser : qi::grammar<Iterator, std::vector<Category>(), ascii::space_type>
{
qi::rule<Iterator, std::vector<Category>(), ascii::space_type> root;
...
So every instance of ascii::space_type in the rule templates must be replaced with Skipper! That includes other rules besides the root that is shown here, such as param and category from my question. Leaving any remnant of the old ascii::space_type behind gives cryptic compiler errors.
struct Parser : qi::grammar<Iterator, std::vector<Category>(), qi::rule<Iterator>>
{
typedef qi::rule<Iterator> Skipper;
Skipper skipper;
qi::rule<Iterator, std::vector<Category>(), Skipper> root;
...
The original skipper was merely space, mine is now an alternative of space and comment. No old functionality (space skipping) is lost.
skipper = space | comment;
Then the phrase_parse call needs to be adjusted from this old version that used ascii::space:
bool r = phrase_parse(iter, end, parser, ascii::space, result);
to
bool r = phrase_parse(iter, end, parser, parser.skipper, result);
And now comments disappear as easily as white space. Awesome.
I'd like to parse a sequence of integers into an std::vector<int>, using boost::spirit. The integers may be separated by a semicolon or a newline.
But this grammar doesn't compile:
typedef std::vector<int> IntVec;
template <typename Iterator, typename Skipper>
struct MyGrammar : qi::grammar<Iterator, IntVec(), Skipper> {
MyGrammar() : MyGrammar::base_type(start) {
start = +(qi::int_
>> (";" | qi::no_skip(qi::eol)));
}
qi::rule<Iterator, IntVec(), Skipper> start;
};
To be clear, I want to parse the following input, for example,
1; 2; 3
4 ; 5
into one vector (1,2,3,4,5). How can I do that and why does my version not compile?
Can I somehow write the separator ("semicolon or newline") as its own rule? What would its return type be? Some kind of null value?
It looks like the skipper is being applied when checking the semicolon, and so the skip characters (including newline) have already been consumed once qi::no_skip[qi::eol] is reached. The following is working for me, with the no_skip token first:
start = qi::int_ % (qi::no_skip[qi::eol] | ';');
I'm using % so that the final integer does not need to be followed by a semicolon or end-of-line.
I have a simple grammar consisting of mixed variables ($(name)) and variable-value pairs ($(name:value)). I have a hand-coded recursive parser, but am interested in using it as an exercise to learn Spirit, which I'll need for more complex grammars eventually(/soon).
Anyway, the set of possible forms I'm working with (simplified from the full grammar) is:
$(variable) // Uses simple look-up, recursion and inline replace
$(name:value) // Inserts a new variable into the local lookup table
My current rules look something like:
typedef std::map<std::string, std::string> dictionary;
template <typename Iterator>
bool parse_vars(Iterator first, Iterator last, dictionary & vars, std::string & output)
{
using qi::phrase_parse;
using qi::_1;
using ascii::char_;
using ascii::string;
using ascii::space;
using phoenix::insert;
dictionary statevars;
typedef qi::rule<Iterator, std::string()> string_rule;
typedef qi::rule<Iterator, std::pair<std::string, std::string>()> pair_rule;
string_rule state = string >> ':' >> string; // Error 3
pair_rule variable =
(
char_('$') >> '(' >>
(
state[insert(phoenix::ref(statevars), _1)] |
string[output += vars[_1]] // Error 1, will eventually need to recurse
) >> ')'
); // Error 2
bool result = phrase_parse
(
first, last,
(
variable % ','
),
space
);
return r;
}
If it wasn't obvious, I have no idea how Spirit works and the docs have everything but actual explanations, so this is about an hour of throwing examples together.
The parts I particularly question are the leading char_('$') in the variable rule, but removing this causes a shift operator error (the compiler interprets '$' >> '(' as a right-shift).
When compiling, I get errors related to the state rule, particularly creating the pair, and the lookup:
error C2679: binary '[' : no operator found which takes a right-hand operand of type 'const boost::spirit::_1_type' (or there is no acceptable conversion)
error C2512: 'boost::spirit::qi::rule::rule' : no appropriate default constructor available
Changing the lookup (vars[_1]) to a simple += gives:
3. error C2665: 'boost::spirit::char_class::classify::is' : none of the 15 overloads could convert all the argument types
Error 1 seems to relate to the type (attribute?) of the _1 placeholder, but that should be a string, and is when used for printing or concatenation to the output string. 2 appears to be noise caused by 1.
Error 3, digging down the stack of template errors, seems to relate to not being able to turn the state rule into a pair, which seems odd as it almost exactly matches one of the rules from this example.
How can I modify the variable rule to properly handle both input forms?
A few things to note:
To adapt std::pair (so you can use it with maps) you should include (at least)
#include <boost/fusion/adapted/std_pair.hpp>
It looks like you are trying to create a symbol table. You could use qi::symbols for that
avoid mixing output generation with parsing, it complicates matters unduly
I haven't 'fixed' all the above (due to lack of context), but I'd happy to help out with any other questions arising from those.
Here is a fixed code version staying pretty close to the OP. Edit have tested it too now, output below:
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/fusion/adapted/std_pair.hpp>
#include <map>
namespace qi = boost::spirit::qi;
namespace phx = boost::phoenix;
typedef std::map<std::string, std::string> dictionary;
template <typename Iterator, typename Skipper = qi::space_type>
struct parser : qi::grammar<Iterator, Skipper>
{
parser(dictionary& statevars, std::string& output) : parser::base_type(start)
{
using namespace qi;
using phx::insert;
with_initializer = +~char_(":)") >> ':' >> *~char_(")");
simple = +~char_(")");
variable =
"$(" >> (
with_initializer [ insert(phx::ref(statevars), qi::_1) ]
| simple [ phx::ref(output) += phx::ref(statevars)[_1] ]
) >> ')';
start = variable % ',';
BOOST_SPIRIT_DEBUG_NODE(start);
BOOST_SPIRIT_DEBUG_NODE(variable);
BOOST_SPIRIT_DEBUG_NODE(simple);
BOOST_SPIRIT_DEBUG_NODE(with_initializer);
}
private:
qi::rule<Iterator, std::pair<std::string, std::string>(), Skipper> with_initializer;
qi::rule<Iterator, std::string(), Skipper> simple;
qi::rule<Iterator, Skipper> variable;
qi::rule<Iterator, Skipper> start;
};
template <typename Iterator>
bool parse_vars(Iterator &first, Iterator last, dictionary & vars, std::string & output)
{
parser<Iterator> p(vars, output);
return qi::phrase_parse(first, last, p, qi::space);
}
int main()
{
const std::string input = "$(name:default),$(var),$(name)";
std::string::const_iterator f(input.begin());
std::string::const_iterator l(input.end());
std::string output;
dictionary table;
if (!parse_vars(f,l,table,output))
std::cerr << "oops\n";
if (f!=l)
std::cerr << "Unparsed: '" << std::string(f,l) << "'\n";
std::cout << "Output: '" << output << "'\n";
}
Output:
Output: 'default'
you have to have char_('$') otherwise the >> is 'char' on both sides - you need to have at least one spirit type in there to get the overloaded operator >>.
You may also need to use _1 from phoenix.
Also take a look at:
http://boost-spirit.com/home/articles/qi-example/parsing-a-list-of-key-value-pairs-using-spirit-qi/