confusing output when parsing a struct in alternate ways using spirit - c++

Question

This is a greatly reduced case of something I'm trying to do in the best way possible. (Certainly though, the question is also about, me trying to understand how best to use spirit though.)
I need to parse data into a struct with several members. The members are simply listed as key-value pairs, so this is straightforward -- however, if some of the keys are different, then in the data that I'm parsing, different values may appear later, or some keys may be omitted. Nevertheless, the data structure that I ultimately parse has a fixed form.
In the example code, my_struct is a struct like this:
struct my_struct {
std::string a;
std::string b;
std::string c;
std::string d;
};
and grammar1 is a grammar which parses strings like this
"a: x b: y c: z d: w"
into structs like this
my_struct{ "x", "y", "z", "w" }
I would like to additionally parse strings like this:
"a: x b: y d-no-c: w"
into structs like this
my_struct{ "x", "y", "", "w" }
and I would ideally like to do this in as simple a way as I can without making unnecessary copies of strings along the way.
My first thought was, the main rule should be rewritten so that it parses "a" and "b", and then selects between two alternatives depending on whether "c" is present or not. This is easy to work out as a grammar, but when we try to get the data-types right for the attributed grammar part of it, I can't seem to get it to work. I tried using std::pair<std::string, std::string> and also fusion::vector for the alternative types, but this cannot apparently be streamed into my struct using qi operator <<. (grammar2 tests are commented out because it doesn't compile.)
My next thought was, we can simply have two alternative forms of the main rule, which are attributed with type my_struct to make sure the attributed parsing works out. Surprisingly though, this implementation is actually broken -- it seems that when the grammar backtracks, it duplicates the a and b fields inside the resulting structure. I did not expect this and I don't know why it happens, do you know? (This is grammar3).
grammar3 has the problem that, even if it worked like I think it should (the tests passed), when the alternative part backtracks, it will have to reparse a and b which is some inefficiency. If we are willing to change our target struct from my_struct to a different struct, then we can use grammar4, which has the same plan as grammar2, but targets a struct in which one of the elements is a std::pair. Then we have move all the strings out of this temporary struct into the format we really wanted.
So, the questions are:
grammar4 works, but is there a way to do something along the lines of grammar2 which is presumably more efficient?
Why does grammar3 fail the tests?
Complete listing:
#define SPIRIT_USE_PHOENIX_V3
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/adapted/struct/define_struct.hpp>
#include <boost/fusion/include/define_struct.hpp>
#include <boost/fusion/include/std_pair.hpp>
#include <iostream>
#include <string>
#include <utility>
namespace qi = boost::spirit::qi;
BOOST_FUSION_DEFINE_STRUCT(
/**/
,
my_struct,
(std::string, a)
(std::string, b)
(std::string, c)
(std::string, d))
template<typename Iterator>
class grammar1 : public qi::grammar<Iterator, my_struct()> {
public:
qi::rule<Iterator, std::string()> id;
qi::rule<Iterator, my_struct()> main;
grammar1() : grammar1::base_type(main) {
using qi::lit;
using qi::char_;
using qi::omit;
using qi::space;
id = omit[ *space ] >> *char_("A-Za-z_") >> omit [ *space ];
main = lit("a:") >> id >> lit("b:") >> id >> lit("c:") >> id >> lit("d:") >> id;
}
};
//typedef std::pair<std::string, std::string> second_part_type;
typedef boost::fusion::vector<std::string, std::string> second_part_type;
template<typename Iterator>
class grammar2 : public qi::grammar<Iterator, my_struct()> {
public:
qi::rule<Iterator, std::string()> id;
qi::rule<Iterator, second_part_type()> with_c;
qi::rule<Iterator, second_part_type()> without_c;
qi::rule<Iterator, my_struct()> main;
grammar2() : grammar2::base_type(main) {
using qi::lit;
using qi::char_;
using qi::omit;
using qi::space;
using qi::attr;
id = omit[ *space ] >> *char_("A-Za-z_") >> omit [ *space ];
with_c = lit("c:") >> id >> lit("d:") >> id;
without_c = attr("") >> lit("d-no-c:") >> id;
main = lit("a:") >> id >> lit("b:") >> id >> (with_c | without_c);
}
};
template<typename Iterator>
class grammar3 : public qi::grammar<Iterator, my_struct()> {
public:
qi::rule<Iterator, std::string()> id;
qi::rule<Iterator, my_struct()> with_c;
qi::rule<Iterator, my_struct()> without_c;
qi::rule<Iterator, my_struct()> main;
grammar3() : grammar3::base_type(main) {
using qi::lit;
using qi::char_;
using qi::omit;
using qi::space;
using qi::attr;
id = omit[ *space ] >> *char_("A-Za-z_") >> omit [ *space ];
with_c = lit("a:") >> id >> lit("b:") >> id >> lit("c:") >> id >> lit("d:") >> id;
without_c = lit("a:") >> id >> lit("b:") >> id >> attr("") >> lit("d-no-c:") >> id;
main = with_c | without_c;
}
};
/***
* Alternate approach
*/
typedef std::pair<std::string, std::string> spair;
BOOST_FUSION_DEFINE_STRUCT(
/**/
,
my_struct2,
(std::string, a)
(std::string, b)
(spair, cd))
template<typename Iterator>
class grammar4 : public qi::grammar<Iterator, my_struct2()> {
public:
qi::rule<Iterator, std::string()> id;
qi::rule<Iterator, spair()> with_c;
qi::rule<Iterator, spair()> without_c;
qi::rule<Iterator, my_struct2()> main;
grammar4() : grammar4::base_type(main) {
using qi::lit;
using qi::char_;
using qi::omit;
using qi::space;
using qi::attr;
id = omit[ *space ] >> *char_("A-Za-z_") >> omit [ *space ];
with_c = lit("c:") >> id >> lit("d:") >> id;
without_c = attr("") >> lit("d-no-c:") >> id;
main = lit("a:") >> id >> lit("b:") >> id >> (with_c | without_c);
}
};
my_struct convert_struct(my_struct2 && s) {
return { std::move(s.a), std::move(s.b), std::move(s.cd.first), std::move(s.cd.second) };
}
/***
* Testing
*/
void check_strings_eq(const std::string & a, const std::string & b, const char * label, int line = 0) {
if (a != b) {
std::cerr << "Mismatch '" << label << "' ";
if (line) { std::cerr << "at line " << line << " "; }
std::cerr << "\"" << a << "\" != \"" << b << "\"\n";
}
}
void check_eq(const my_struct & s, const my_struct & t, int line = 0) {
check_strings_eq(s.a, t.a, "a", line);
check_strings_eq(s.b, t.b, "b", line);
check_strings_eq(s.c, t.c, "c", line);
check_strings_eq(s.d, t.d, "d", line);
}
template<template<typename> class Grammar>
void test_grammar(const std::string & input, const my_struct & expected, int line = 0) {
auto it = input.begin();
auto end = input.end();
Grammar<decltype(it)> grammar;
my_struct result;
if (!qi::parse(it, end, grammar, result)) {
std::cerr << "Failed to parse! ";
if (line) { std::cerr << "line = " << line; }
std::cerr << "\n";
std::cerr << "Stopped at:\n" << input << "\n";
for (auto temp = input.begin(); temp != it; ++temp) { std::cerr << " "; }
std::cerr << "^\n";
} else {
check_eq(result, expected, line);
}
}
int main() {
test_grammar<grammar1> ( "a: x b: y c: z d: w", my_struct{ "x", "y", "z", "w" }, __LINE__);
test_grammar<grammar1> ( "a: asdf b: jkl c: foo d: bar", my_struct{ "asdf", "jkl", "foo", "bar" }, __LINE__ );
//test_grammar<grammar2> ( "a: asdf b: jkl c: foo d: bar", my_struct{ "asdf", "jkl", "foo", "bar" }, __LINE__ );
//test_grammar<grammar2> ( "a: asdf b: jkl d-no-c: bar", my_struct{ "asdf", "jkl", "", "bar" }, __LINE__ );
test_grammar<grammar3> ( "a: asdf b: jkl c: foo d: bar", my_struct{ "asdf", "jkl", "foo", "bar" }, __LINE__);
test_grammar<grammar3> ( "a: asdf b: jkl d-no-c: bar", my_struct{ "asdf", "jkl", "", "bar" }, __LINE__ );
// Test 4th grammar
{
std::string input = "a: asdf b: jkl c: foo d: bar";
auto it = input.begin();
auto end = input.end();
grammar4<decltype(it)> grammar;
my_struct2 result;
if (!qi::parse(it, end, grammar, result)) {
std::cerr << "Failed to parse! Line = " << __LINE__ << std::endl;
} else {
check_eq(convert_struct(std::move(result)), my_struct{ "asdf", "jkl", "foo", "bar" }, __LINE__);
}
}
{
std::string input = "a: asdf b: jkl d-no-c: bar";
auto it = input.begin();
auto end = input.end();
grammar4<decltype(it)> grammar;
my_struct2 result;
if (!qi::parse(it, end, grammar, result)) {
std::cerr << "Failed to parse! Line = " << __LINE__ << std::endl;
} else {
check_eq(convert_struct(std::move(result)), my_struct{ "asdf", "jkl", "", "bar" }, __LINE__);
}
}
}

Answer 1

My suggestion here would be to use a permutation parser indeed.
It is quite a bit more flexible though, so you might wish to add a validation constraint in a semantic action:
Live On Coliru
//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/tuple/tuple_comparison.hpp>
#include <iostream>
#include <string>
namespace qi = boost::spirit::qi;
struct my_struct {
std::string a,b,c,d;
};
BOOST_FUSION_ADAPT_STRUCT(my_struct, a, b, c, d)
template<typename Iterator>
class grammar : public qi::grammar<Iterator, my_struct()> {
public:
grammar() : grammar::base_type(start) {
using namespace qi;
id = +char_("A-Za-z_");
part = lexeme[lit(_r1) >> ':'] >> id;
main = part(+"a")
^ part(+"b")
^ part(+"c")
^ (part(+"d") | part(+"d-no-c"));
;
start = skip(space) [ main ];
BOOST_SPIRIT_DEBUG_NODES((main)(part))
}
private:
qi::rule<Iterator, std::string()> id;
qi::rule<Iterator, std::string(const char*), qi::space_type> part;
qi::rule<Iterator, my_struct(), qi::space_type> main;
//
qi::rule<Iterator, my_struct()> start;
};
/***
* Testing
*/
void check_strings_eq(const std::string & a, const std::string & b, const char * label) {
if (a != b) {
std::cerr << "Mismatch '" << label << "' \"" << a << "\" != \"" << b << "\"\n";
}
}
void check_eq(const my_struct & s, const my_struct & t) {
check_strings_eq(s.a, t.a, "a");
check_strings_eq(s.b, t.b, "b");
check_strings_eq(s.c, t.c, "c");
check_strings_eq(s.d, t.d, "d");
if (boost::tie(s.a,s.b,s.c,s.d) == boost::tie(t.a,t.b,t.c,t.d))
std::cerr << "struct data matches\n";
}
template<template<typename> class Grammar>
void test_grammar(const std::string &input, const my_struct &expected) {
auto it = input.begin();
auto end = input.end();
Grammar<decltype(it)> grammar;
my_struct result;
if (!qi::parse(it, end, grammar, result)) {
std::cerr << "Failed to parse!\n";
std::cerr << "Stopped at:\n" << input << "\n";
for (auto temp = input.begin(); temp != it; ++temp) {
std::cerr << " ";
}
std::cerr << "^\n";
} else {
check_eq(result, expected);
}
}
int main() {
for (auto&& p : std::vector<std::pair<std::string, my_struct> > {
{"a: x b: y c: z d: w", my_struct{ "x", "y", "z", "w" }},
{"a: x c: z d: w", my_struct{ "x", "" , "z", "w" }},
{"a: x c: z" , my_struct{ "x", "" , "z", "" }},
{" b: y c: z d: w", my_struct{ "" , "y", "z", "w" }},
{"b: y c: z a: x d: w", my_struct{ "x", "y", "z", "w" }},
// if you really need:
{"a: x b: y d-no-c: w", my_struct{ "x", "y", "" , "w" }},
})
{
auto const& input = p.first;
auto const& expected = p.second;
std::cout << "----\nParsing '" << input << "'\n";
test_grammar<grammar> (input, expected);
}
}
Prints
----
Parsing 'a: x b: y c: z d: w'
struct data matches
----
Parsing 'a: x c: z d: w'
struct data matches
----
Parsing 'a: x c: z'
struct data matches
----
Parsing ' b: y c: z d: w'
struct data matches
----
Parsing 'b: y c: z a: x d: w'
struct data matches
----
Parsing 'a: x b: y d-no-c: w'
struct data matches