I have a more complex boost::spirit grammar that doesn't match like I expected.
I was able to break it down to this minimal example: http://ideone.com/oPu2e7 (doesn't compile there, but compiles with VS2010)
Basically this is my grammar:
my_grammar() : my_grammar::base_type(start)
{
start %=
(+alpha | +alnum)
;
}
qi::rule<Iterator, std::string(), ascii::space_type> start;
It matches foobar, 123foo but doesn't match foo123. Why? I would expect it to match all three.
PEG parsers match greedy, left-to-right. That should be enough to explain.
But lets look at foo123: it matches "1 or more +alpha, so the first branch is taken. The second branch is not taken, so the numerics 123 remain unparsed.
There's no "inherent" backtracking on the kleen operators. You /can/ employ backtracking if you know e.g. that you need to parse the full input:
(+alpha >> eoi | +alnum >> eoi)
Related
I want to create a parser that will match exactly two alphanumeric words from a string, such as:
message1 message2
and then save that into two variables of type std::string.
I've read this previous answer which seems to work for an endless amount of repetitions, which uses the following parser:
+qi::alnum % +qi::space
However when I try to do this:
bool const result = qi::phrase_parse(
input.begin(), input.end(),
+qi::alnum >> +qi::alnum,
+qi::space,
words
);
the words vector contains every single letter in a different string:
't'
'h'
'i'
's'
'i'
's'
This is extremely counter-intuitive, and I'm not sure as to why it's happening. Could someone please explain that?
Also, can I have two predefined strings to be populated instead of a std::vector?
Final note: I would like to avoid the using statement, as I would like to have every namespace clearly defined to help me understand how Spirit works.
Yes, but the skipper ignores the whitespace before you can act on it.
Use lexeme to control the skipper:
bool const result = qi::phrase_parse(
input.begin(), input.end(),
qi::lexeme [+qi::alnum] >> qi::lexeme [+qi::alnum],
qi::space,
words
);
Note the skipper should be qi::space instead of +qi::space.
See also Boost spirit skipper issues
I'm writing a little compiler just for fun and I'm using Boost Spirit Qi to describe my grammar. Now I want to make a minor change in the grammar to prepare some further additions. Unfortunately these changes won't compile and I would like to understand why this is the case.
Here is a snippet from the code I want to change. I hope the provided information is enough to understand the idea. The complete code is a bit large, but if you want to look at it or even test it (Makefile and Travis CI is provided), see https://github.com/Kruecke/BFGenerator/blob/8f66aa5/bf/compiler.cpp#L433.
typedef boost::variant<
function_call_t,
variable_declaration_t,
variable_assignment_t,
// ...
> instruction_t;
struct grammar : qi::grammar<iterator, program_t(), ascii::space_type> {
grammar() : grammar::base_type(program) {
instruction = function_call
| variable_declaration
| variable_assignment
// | ...
;
function_call = function_name >> '(' > -(variable_name % ',') > ')' > ';';
// ...
}
qi::rule<iterator, instruction::instruction_t(), ascii::space_type> instruction;
qi::rule<iterator, instruction::function_call_t(), ascii::space_type> function_call;
// ...
};
So far, everything is just working fine. Now I want to move the parsing of the trailing semicolon (> ';') from the function_call rule to the instruction rule. My code now looks like this:
struct grammar : qi::grammar<iterator, program_t(), ascii::space_type> {
grammar() : grammar::base_type(program) {
instruction = (function_call > ';') // Added trailing semicolon
| variable_declaration
| variable_assignment
// | ...
;
// Removed trailing semicolon here:
function_call = function_name >> '(' > -(variable_name % ',') > ')';
// ...
}
From my understanding the rules haven't really changed because the character parser ';' doesn't yield any attribute and so it shouldn't matter where this parser is positioned. However, this change won't compile:
/usr/include/boost/spirit/home/support/container.hpp:278:13: error: no matching function for call to ‘std::basic_string<char>::insert(std::basic_string<char>::iterator, const bf::instruction::function_call_t&)’
c.insert(c.end(), val);
^
(This error comes from the instruction = ... line.)
Why is this change not compiling? I'm rather looking for an explanation to understand what's going on than a workaround.
Ok, so after looking at this closely, you are trying to insert multiple strings into your function_call_t type, which is a fusion sequence that can be converted to from a single std::string. However, you are probably going to run into issues with your function_call rule because it's attribute is actually tuple <std::string, optional <vector <std::string>>>. I'd imagine that spirit is having issues flattening that structure out and that is causing your issue, however, I don't have a compiler to test it out at the moment.
Why this parser leave 'b' in attributes, even if option wasn't matched?
using namespace boost::spirit::qi;
std::string str = "abc";
auto a = char_("a");
auto b = char_("b");
qi::rule<std::string::iterator, std::string()> expr;
expr = +a >> -(b >> +a);
std::string res;
bool r = qi::parse(
str.begin(),
str.end(),
expr >> lit("bc"),
res
);
It parses successfully, but res is "ab".
If parse "abac" with expr alone, option is matched and attribute is "aba".
Same with "aac", option doesn't start to match and attribute is "aa".
But with "ab", attribute is "ab", even though b gets backtracked, and, as in example, matched with next parser.
UPD
With expr.name("expr"); and debug(expr); I got
<expr>
<try>abc</try>
<success>bc</success>
<attributes>[[a, b]]</attributes>
</expr>
Firstly, it's UB to use the auto variables to keep the expression templates, because they hold references to the temporaries "a" and "b" [1].
Instead write
expr = +qi::char_("a") >> -(qi::char_("b") >> +qi::char_("a"));
or, if you insist:
auto a = boost::proto::deep_copy(qi::char_("a"));
auto b = boost::proto::deep_copy(qi::char_("b"));
expr = +a >> -(b >> +a);
Now noticing the >> lit("bc") part hiding in the parse call, suggests you may expect backtracking to on succesfully matched tokens when a parse failure happens down the road.
That doesn't happen: Spirit generates PEG grammars, and always greedily matches from left to right.
On to the sample given, ab results, even though backtracking does occur, the effects on the attribute are not rolled back without qi::hold: Live On Coliru
Container attributes are passed along by ref and the effects of previous (successful) expressions is not rolled back, unless you tell Spirit too. This way, you can "pay for what you use" (as copying temporaries all the time would be costly).
See e.g.
boost::spirit::qi duplicate parsing on the output
Understanding Boost.spirit's string parser
Boost spirit revert parsing
<a>
<try>abc</try>
<success>bc</success>
<attributes>[a]</attributes>
</a>
<a>
<try>bc</try>
<fail/>
</a>
<b>
<try>bc</try>
<success>c</success>
<attributes>[b]</attributes>
</b>
<a>
<try>c</try>
<fail/>
</a>
<bc>
<try>bc</try>
<success></success>
<attributes>[]</attributes>
</bc>
Success: 'ab'
[1] see here:
Assigning parsers to auto variables
Generating Spirit parser expressions from a variadic list of alternative parser expressions
boost spirit V2 qi bug associated with optimization level
Quoting #sehe from this SO question
A string attribute is a container attribute and many elements could be
assigned into it by different parser subexpressions. Now for
efficiency reasons, Spirit doesn't rollback the values of emitted
attributes on backtracking.
So, I've put optional parser on hold, and it's done.
expr = +qi::char_("a") >> -(qi::hold[qi::char_("b") >> +qi::char_("a")]);
For more information see mentioned question and hold docs
I'm parsing some input that is vaguely structured like C-ish code. Like this:
Name0
{
Name1
{
//A COMMENT!!
Param0 *= 2
Param2 = "lol"
}
}
Part of that is comments, which I want to totally ignore (and it's not working). I consider two things to be a node, the named scopes (category rule) like Name0 {} and the values (param rule) like Param0 *= 2... then there is comment. I've tried setting things up like this:
typedef boost::variant<boost::recursive_wrapper<Category>, Param> Node;
qi::rule<Iterator, Node(), ascii::space_type> node;
So the node rule puts either a Category or a Param in a variant. Here are the other rules (I've omitted some rules that don't really matter for this):
qi::rule<Iterator> comment; //comment has no return type
qi::rule<Iterator, Category(), ascii::space_type> category;
qi::rule<Iterator, Param(), ascii::space_type> param;
And their actual code:
comment = "//" >> *(char_ - eol);
param %=
tagstring
>> operators
>> value;
category %=
tagstring
>> '{'
>> *node
> '}';
node %= comment | category | param;
comment is setup to use = instead of %=, and it has no return type. However, comments end up creating null Categorys in my output Nodes wherever they show up. I've tried moving comment out of the node rule and into category like this:
category %=
tagstring
>> '{'
>> *(comment | node)
> '}';
And various other things, but those null entries keep popping up. I had to make comment output a string and put std::string in my Node variant just to sorta catch them, but that messes up my ability to stick in commenting in other parts of my rules (unless I actually grab the string in every location).
How can I completely ignore the comment and have it not show up in any output in any way?
edit: You'd think omit would do it, but didn't seem to change anything...
edit 2: Referencing this SO answer, I have a shaky solution in this:
node %= category | param;
category %=
tagstring
>> '{'
>> *comment >> *(node >> *comment)
> '}';
However, I want to try to stick comments into all sorts of places (between tagstring and {, in my unshown root rule between root categorys, etc). Is there a simpler method than this? I was hoping it could be done with a simple >> commentwrapper plugged into wherever I wanted...
Alright, so making your own skipper isn't too bad. And it elegantly solves this commenting problem, just as Mike M said. I define my rules in a struct called Parser that is templated with an Iterator. Had to make some adjustments to use the skipper. First, here is the skipper which is defined in Parser with all my other rules:
typedef qi::rule<Iterator> Skipper;
Skipper skipper;
So skipper is a rule of type Skipper. Here is what my Parser struct looked like originally, where it was using the ascii::space rule of type ascii::space_type as its skipper, which IS NOT the same type as qi::rule<Iterator> that skipper is based on!
struct Parser : qi::grammar<Iterator, std::vector<Category>(), ascii::space_type>
{
qi::rule<Iterator, std::vector<Category>(), ascii::space_type> root;
...
So every instance of ascii::space_type in the rule templates must be replaced with Skipper! That includes other rules besides the root that is shown here, such as param and category from my question. Leaving any remnant of the old ascii::space_type behind gives cryptic compiler errors.
struct Parser : qi::grammar<Iterator, std::vector<Category>(), qi::rule<Iterator>>
{
typedef qi::rule<Iterator> Skipper;
Skipper skipper;
qi::rule<Iterator, std::vector<Category>(), Skipper> root;
...
The original skipper was merely space, mine is now an alternative of space and comment. No old functionality (space skipping) is lost.
skipper = space | comment;
Then the phrase_parse call needs to be adjusted from this old version that used ascii::space:
bool r = phrase_parse(iter, end, parser, ascii::space, result);
to
bool r = phrase_parse(iter, end, parser, parser.skipper, result);
And now comments disappear as easily as white space. Awesome.
I'd like to parse a sequence of integers into an std::vector<int>, using boost::spirit. The integers may be separated by a semicolon or a newline.
But this grammar doesn't compile:
typedef std::vector<int> IntVec;
template <typename Iterator, typename Skipper>
struct MyGrammar : qi::grammar<Iterator, IntVec(), Skipper> {
MyGrammar() : MyGrammar::base_type(start) {
start = +(qi::int_
>> (";" | qi::no_skip(qi::eol)));
}
qi::rule<Iterator, IntVec(), Skipper> start;
};
To be clear, I want to parse the following input, for example,
1; 2; 3
4 ; 5
into one vector (1,2,3,4,5). How can I do that and why does my version not compile?
Can I somehow write the separator ("semicolon or newline") as its own rule? What would its return type be? Some kind of null value?
It looks like the skipper is being applied when checking the semicolon, and so the skip characters (including newline) have already been consumed once qi::no_skip[qi::eol] is reached. The following is working for me, with the no_skip token first:
start = qi::int_ % (qi::no_skip[qi::eol] | ';');
I'm using % so that the final integer does not need to be followed by a semicolon or end-of-line.