Zero-filled results for unused (but matching) rules [duplicate] - c++

I want to parse special constructs and throw the rest away. But I don't want to use a skipper.
I want to get a vector of these constructs, so I use a Kleene Star parser as main rule. But, everytime something gets thrown away, a default constructed element is inserted into the vector.
Here is a made up example. It just looks for the string Test and throws the rest away, at least this is the plan. But every time the rule garbage succeeds it adds a default constructed item to the vector in the rule all, giving an output of 7 insteat of 1. How can I tell Spirit to just add to the vector if the rule item succeeds?
#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <iostream>
#include <string>
#include <vector>
namespace qi = boost::spirit::qi;
struct container {
std::string name;
bool dummy;
};
BOOST_FUSION_ADAPT_STRUCT(::container,
(std::string, name)
(bool, dummy))
int main() {
typedef std::string::const_iterator iterator;
qi::rule<iterator, std::vector<container>()> all;
qi::rule<iterator, container()> item;
qi::rule<iterator, std::string()> string_rule;
qi::rule<iterator> garbage;
all = *(garbage | item);
garbage = qi::char_ - qi::lit("Test");
string_rule = qi::string("Test");
item = string_rule >> qi::attr(true);
std::vector<container> ast;
std::string input = "blaTestbla";
iterator first = input.begin();
iterator last = input.end();
bool result = qi::parse(first, last, all, ast);
if (result) {
result = first == last;
}
if (result) {
std::cout << "Parsed " << ast.size() << " element(s)" << std::endl;
} else {
std::cout << "failure" << std::endl;
}
}

Since sehe's answer was more or less for educational purposes, we have now several solutions:
*garbage >> -(item % *garbage) >> *garbage
*garbage >> *(item >> *garbage)
all = *(garbage | item[phx::push_back(qi::_val,qi::_1)]);
And the solution from cv_and_he:
#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <iostream>
#include <string>
#include <vector>
namespace qi = boost::spirit::qi;
struct container {
std::string name;
bool dummy;
};
BOOST_FUSION_ADAPT_STRUCT(::container,
(std::string, name)
(bool, dummy))
struct container_vector { //ADDED
std::vector<container> data;
};
namespace boost{ namespace spirit{ namespace traits //ADDED
{
template <>
struct is_container<container_vector> : boost::mpl::true_ {};
template <>
struct container_value<container_vector> {
typedef optional<container> type;
};
template <>
struct push_back_container<container_vector,optional<container> > {
static bool call(container_vector& cont, const optional<container>& val) {
if(val)
cont.data.push_back(*val);
return true;
}
};
}}}
int main() {
typedef std::string::const_iterator iterator;
qi::rule<iterator, container_vector()> all; //CHANGED
qi::rule<iterator, container()> item;
qi::rule<iterator, std::string()> string_rule;
qi::rule<iterator> garbage;
all = *(garbage | item);
garbage = qi::char_ - qi::lit("Test");
string_rule = qi::string("Test");
item = string_rule >> qi::attr(true);
container_vector ast; //CHANGED
std::string input = "blaTestbla";
iterator first = input.begin();
iterator last = input.end();
bool result = qi::parse(first, last, all, ast);
if (result) {
result = first == last;
}
if (result) {
std::cout << "Parsed " << ast.data.size() << " element(s)" << std::endl; //CHANGED
} else {
std::cout << "failure" << std::endl;
}
}
Although I didn't want to use a skipper I ended up with:
start = qi::skip(garbage.alias())[*item];
This last solution was the fastest (by 1-2%) in my unscientific tests using the c-files of the Linux kernel with my production rules.

A quick fix (not necessarily most performant) would be
all = -(item - garbage) % +garbage;
It prints:
Parsed 3 element(s)
See it Live on Coliru

Related

boost spirit: copy the result in a vector of strings

I want to parse a function (with an arbitrary name and an arbitrary numbers af arguments) in this form:
function(bye, 1, 3, 4, foo)
The arguments could be generic strings comma separated.
And I want to copy the name of the function and the arguments in a vector of strings.
like this
std::vector<std::string> F;
std::string fun = "function(bye, 1, 3, 4, foo)";
// The parser must produce this vector from the example
F[0] == "function"
F[1] == "1"
F[2] == "3"
F[3] == "4"
F[4] == "foo"
I've written the following code by after reading some tutorial but it does not work (In the sense that it not compile).
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_object.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/include/io.hpp>
#include <iostream>
#include <string>
namespace client
{
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
///////////////////////////////////////////////////////////////////////////////
template <typename Iterator>
struct command_parser : qi::grammar<Iterator, std::vector<std::string>(), ascii::space_type>
{
command_parser() : command_parser::base_type(start)
{
using qi::int_;
using qi::lit;
using qi::double_;
using qi::lexeme;
using ascii::char_;
fn_name = +qi::char_("a-zA-Z");
string = +qi::char_("a-zA-Z_0-9");
rec = *( lit(",") >> string );
start %= fn_name >> lit("(") >> string >> rec >> lit(")") ;
}
qi::rule<Iterator, std::string(), ascii::space_type> fn_name;
qi::rule<Iterator, std::string(), ascii::space_type> string;
qi::rule<Iterator, std::string(), ascii::space_type> rec;
qi::rule<Iterator, std::vector<std::string>, ascii::space_type> start;
};
}
////////////////////////////////////////////////////////////////////////////
// Main program
////////////////////////////////////////////////////////////////////////////
int
main()
{
namespace qi = boost::spirit::qi;
std::cout << "/////////////////////////////////////////////////////////\n\n";
client::command_parser<std::string::iterator> CP;
std::string cmd("fun(1,2,3,4 , 5, foo) ");
std::vector<std::string> VV;
bool result = qi::parse(cmd.begin(), cmd.end(), CP, VV);
if (result) {
for ( auto sss : VV ){
std::cout << sss << std::endl;
}
} else {
std::cout << "Fail" << std::endl;
}
return 0 ;
}
Just for fun, here's my minimalist take on this grammar:
using CallList = std::vector<std::string>;
struct ParseError : std::runtime_error {
ParseError() : std::runtime_error("ParseError") {}
};
// The parse implementation
CallList parse_function_call(std::string const& fun) {
CallList elements;
using namespace boost::spirit::qi;
using It = decltype(begin(fun));
static const rule<It, std::string()> identifier = alpha >> +(alnum | char_('_'));
if (!phrase_parse(begin(fun), end(fun),
identifier >> '(' >> -(lexeme[+~char_(",)")] % ",") >> ')' >> eoi,
space, elements))
throw ParseError{};
return elements;
}
With a little bit of plumbing
// just for test output
using TestResult = std::variant<CallList, ParseError>;
// exceptions are equivalent
static constexpr bool operator==(ParseError const&, ParseError const&)
{ return true; }
static inline std::ostream& operator<<(std::ostream& os, TestResult const& tr) {
using namespace std;
if (holds_alternative<ParseError>(tr)) {
return os << "ParseError";
} else {
auto& list = get<CallList>(tr);
copy(begin(list), end(list), std::experimental::make_ostream_joiner(os << "{", ","));
return os << "}";
}
}
TestResult try_parse(std::string const& fun) {
try { return parse_function_call(fun); }
catch(ParseError const& e) { return e; }
}
Here's a test runner:
for (auto const& [input, expected]: {
Case("function(bye, 1, 3, 4, foo)", CallList{"function", "1", "3", "4", "foo"}),
{"liar(pants on fire)", CallList{"liar", "pants on fire"}},
{"liar('pants on fire')", CallList{"liar", "'pants on fire'"}},
{"nullary()", CallList{"nullary"}},
{"nullary( )", CallList{"nullary"}},
{"zerolength(a,,b)", ParseError{}},
{"zerolength(a, ,b)", ParseError{}},
{"noarglust", ParseError{}},
{"", ParseError{}},
{"()", ParseError{}},
{"1(invalidfunctionname)", ParseError{}},
{"foo(bar) BOGUS", ParseError{}},
})
{
auto const actual = try_parse(input);
bool const ok = (actual == expected);
cout << std::quoted(input) << ": " << (ok? "PASS":"FAIL") << "\n";
if (!ok) {
std::cout << " -- expected: " << expected << "\n";
std::cout << " -- actual: " << actual << "\n";
}
}
Which prints Live On Coliru
"function(bye, 1, 3, 4, foo)": FAIL
-- expected: {function,1,3,4,foo}
-- actual: {function,bye,1,3,4,foo}
"liar(pants on fire)": PASS
"liar('pants on fire')": PASS
"nullary()": PASS
"nullary( )": PASS
"zerolength(a,,b)": PASS
"zerolength(a, ,b)": PASS
"noarglust": PASS
"": PASS
"()": PASS
"1(invalidfunctionname)": PASS
"foo(bar) BOGUS": PASS
Note that your example test-case doesn't pass, but I think that was a mistake in the test case.
Full Listing
Live On Coliru
//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <experimental/iterator>
#include <variant>
#include <iomanip>
using CallList = std::vector<std::string>;
struct ParseError : std::runtime_error {
ParseError() : std::runtime_error("ParseError") {}
};
// The parse implementation
CallList parse_function_call(std::string const& fun) {
CallList elements;
using namespace boost::spirit::qi;
using It = decltype(begin(fun));
static const rule<It, std::string()> identifier = alpha >> +(alnum | char_('_'));
if (!phrase_parse(begin(fun), end(fun),
identifier >> '(' >> -(lexeme[+~char_(",)")] % ",") >> ')' >> eoi,
space, elements))
throw ParseError{};
return elements;
}
// just for test output
using TestResult = std::variant<CallList, ParseError>;
// exceptions are equivalent
static constexpr bool operator==(ParseError const&, ParseError const&)
{ return true; }
static inline std::ostream& operator<<(std::ostream& os, TestResult const& tr) {
using namespace std;
if (holds_alternative<ParseError>(tr)) {
return os << "ParseError";
} else {
auto& list = get<CallList>(tr);
copy(begin(list), end(list), std::experimental::make_ostream_joiner(os << "{", ","));
return os << "}";
}
}
TestResult try_parse(std::string const& fun) {
try { return parse_function_call(fun); }
catch(ParseError const& e) { return e; }
}
int main() {
using namespace std;
using Case = pair<std::string, TestResult>;
for (auto const& [input, expected]: {
Case("function(bye, 1, 3, 4, foo)", CallList{"function", "1", "3", "4", "foo"}),
{"liar(pants on fire)", CallList{"liar", "pants on fire"}},
{"liar('pants on fire')", CallList{"liar", "'pants on fire'"}},
{"nullary()", CallList{"nullary"}},
{"nullary( )", CallList{"nullary"}},
{"zerolength(a,,b)", ParseError{}},
{"zerolength(a, ,b)", ParseError{}},
{"noarglust", ParseError{}},
{"", ParseError{}},
{"()", ParseError{}},
{"1(invalidfunctionname)", ParseError{}},
{"foo(bar) BOGUS", ParseError{}},
})
{
auto const actual = try_parse(input);
bool const ok = (actual == expected);
cout << std::quoted(input) << ": " << (ok? "PASS":"FAIL") << "\n";
if (!ok) {
std::cout << " -- expected: " << expected << "\n";
std::cout << " -- actual: " << actual << "\n";
}
}
}
I'm correcting my answer per suggestions made by #sehe. All the credit for these corrections go to him. I am referencing your line numbers below. So the first error is from spirit and it says:
incompatible_start_rule:
// If you see the assertion below failing then the start rule
// passed to the constructor of the grammar is not compatible with
// the grammar (i.e. it uses different template parameters).
The signature of the start parser does not match that of the parser deceleration.
22. struct command_parser : qi::grammar<Iterator, std::vector<std::string>(), ascii::space_type>
43. qi::rule<Iterator, std::vector<std::string>, ascii::space_type> start;
I googled this and could not find an explanation but using an object rather than a type is preferable. I did it the other way in my first answer. The proper fix is at line 43:
43. qi::rule<Iterator, std::vector<std::string>(), ascii::space_type> start;
The next spirit error is:
The rule was instantiated with a skipper type but you have not pass
any. Did you use parse instead of phrase_parse?");
So a phrase_parse is required with a skipper. Note that we need a skipper to pass along.
64. using qi::ascii::space;
65. bool result = qi::phrase_parse(cmd.begin(), cmd.end(), CP, space, VV);
Now it compiles and the output is:
fun
1
2345foo
I see that won't do and you are looking to stuff the vector with each of the passed parameters. So you need a rule that is compatible with your attribute and intention. The kleene operator working with a std::string will put all the data into one string. So use your attribute:
41. qi::rule<Iterator, std::vector<std::string>(), ascii::space_type> rec;``
Now as #sehe points out, the skipper with fn_name and string will just concatenate names with spaces and newlines. So don't use skippers there.
39. qi::rule<Iterator, std::string()> fn_name;
40. qi::rule<Iterator, std::string()> string;
The other error I made was to see the %= and call it a list operator. From here, it is a definition operator. I'm not sure why there are two but playing around, it seems you need to use %= with semantic action. Here is the corrected code:
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_object.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/include/io.hpp>
#include <iostream>
#include <string>
namespace client
{
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
template <typename Iterator>
struct command_parser : qi::grammar<Iterator, std::vector<std::string>(), ascii::space_type>
{
command_parser() : command_parser::base_type(start)
{
using qi::int_;
using qi::lit;
using qi::double_;
using qi::lexeme;
using ascii::char_;
fn_name = +qi::char_("a-zA-Z");
string = +qi::char_("a-zA-Z_0-9");
rec = *(lit(",") >> string);
start %= fn_name >> lit("(") >> string >> rec >> lit(")");
}
qi::rule<Iterator, std::string()> fn_name;
qi::rule<Iterator, std::string()> string;
qi::rule<Iterator, std::vector<std::string>(), ascii::space_type> rec;
qi::rule<Iterator, std::vector<std::string>(), ascii::space_type> start;
};
}
int main()
{
namespace qi = boost::spirit::qi;
client::command_parser<std::string::iterator> CP;
std::string cmd("function(1,2,3,4 , 5, foo) ");
std::vector<std::string> VV;
bool result = qi::phrase_parse(cmd.begin(), cmd.end(), CP, qi::ascii::space, VV);
if (result) {
for (auto sss : VV) {
std::cout << sss << std::endl;
}
}
else {
std::cout << "Fail" << std::endl;
}
return 0;
}
And here is an example using X3:
#include <boost/spirit/home/x3.hpp>
#include <iostream>
#include <vector>
//your attribute, could be more complex, might use namespace
using attr = std::vector<std::string>;
namespace parser {
namespace x3 = boost::spirit::x3;
const auto fn_name = +x3::char_("a-zA-Z");
const auto string = +x3::char_("a-zA-Z_0-9");
const auto start = x3::rule<struct _, attr>() = fn_name >> "(" >> string % ',' >> ")";
}
int main()
{
namespace x3 = boost::spirit::x3;
std::string cmd("fun(1,.2,3,4 , 5, foo) ");
attr VV;
auto it = cmd.begin();
bool result = phrase_parse(it, cmd.end(), parser::start, x3::space, VV);
if (result) {
for (auto sss : VV) {
std::cout << "-> " << sss << std::endl;
}
}
else
std::cout << "Fail at" << std::endl;
return 0;
}

Spirit Qi First Parser

What did I mess up here? I'm getting 'start': undeclared identifier but I stuck pretty closely to the tutorial, so I'm not sure where I made a typo, or what I did wrong. Any hints? You all see the same thing, right?
#include <iostream>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_object.hpp>
#include <iostream>
#include <string>
#include <boost/array.hpp>
#include <boost/fusion/include/io.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/spirit/include/qi_no_skip.hpp>
#include <boost/spirit/include/phoenix.hpp>
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
using qi::lit;
using qi::int_;
using qi::double_;
using ascii::char_;
using boost::spirit::qi::phrase_parse;
using boost::spirit::qi::no_skip;
using qi::eoi;
struct LETTER
{
char hi;
// int fourtytwo;
// char mom;
};
BOOST_FUSION_ADAPT_STRUCT(
LETTER,
(char, hi)
// (int, fourtytwo)
// (char, mom)
)
template <typename Iterator>
struct LETTERParser : qi::grammar<Iterator, LETTER(), ascii::space_type>
{
LETTERParser(): LETTERParser::base_type(start)
{
start %= lit("LETTER") >> char_;
// >> char_
// >> int_
// >> char_
// >> eoi
// ;
}
};
const std::string wat("Z");
int main()
{
LETTERParser<std::string::const_iterator> f;
LETTER example;
phrase_parse(wat.begin(), wat.end(), f, no_skip, example);
return 0;
}
There are a number of issues, one of which is non obvious
where's no_skip? Why are you passing it to a grammar that requires ascii::space_type?
where is the start rule declared?
don't pollute global namespace - it creates hard problems in generic code
handle errors
the grammar starts with a mandatory character sequence, which doesn't match the input
the non-obvious one: single-element structs interfere in unfortunate ways in Spirit/Fusion land.
Simplify:
Fixing the above and modernizing (c++11) the fusion adaptation:
live On Coliru
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/spirit/include/qi.hpp>
#include <iostream>
namespace qi = boost::spirit::qi;
struct LETTER {
char hi;
int fourtytwo;
char mom;
};
BOOST_FUSION_ADAPT_STRUCT(LETTER, hi, fourtytwo, mom)
template <typename Iterator> struct LETTERParser : qi::grammar<Iterator, LETTER(), qi::ascii::space_type> {
LETTERParser() : LETTERParser::base_type(start) {
using qi::char_;
using qi::int_;
start = "LETTER" >> char_ >> int_ >> char_;
}
private:
qi::rule<Iterator, LETTER(), qi::ascii::space_type> start;
};
int main() {
const std::string input("LETTER Z 42m");
using It = std::string::const_iterator;
LETTERParser<It> parser;
LETTER example;
It f = input.begin(), l = input.end();
if (phrase_parse(f, l, parser, qi::ascii::space, example)) {
std::cout << "parsed: " << boost::fusion::as_vector(example) << "\n";
} else {
std::cout << "couldn't parse '" << input << "'\n";
}
if (f != l)
std::cout << "Remaining unparsed input: '" << std::string(f,l) << "'\n";
}
Prints
parsed: (Z 42 m)
Single Element:
You're in, luck it doesn't bite in your case:
Live On Coliru
Prints
parsed: (Z)
Remaining unparsed input: '42m'
as expected. If it strikes in the future, refer here e.g. Size of struct with a single element
Bonus
Consider encapsulating the choice of skipper. The caller should probably never be able to override it Live On Coliru - see also Boost spirit skipper issues

making a vector of shared pointers from Spirit Qi

This is a followup question from a previous question.
I can parse into vectors of strings from my grammar, but I cannot seem to parse into a vector of shared pointers to strings; i.e. std::vector<std::shared_ptr<std::string> >, and need a bit of help.
My compiling header:
#define BOOST_SPIRIT_USE_PHOENIX_V3 1
#include <boost/spirit/include/qi_core.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <iostream>
#include <string>
#include <boost/spirit/include/phoenix_stl.hpp>
#include <boost/phoenix/bind/bind_member_function.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
// this solution for lazy make shared comes from the SO forum, user sehe.
// https://stackoverflow.com/questions/21516201/how-to-create-boost-phoenix-make-shared
// post found using google search terms `phoenix construct shared_ptr`
// changed from boost::shared_ptr to std::shared_ptr
namespace {
template <typename T>
struct make_shared_f
{
template <typename... A> struct result
{ typedef std::shared_ptr<T> type; };
template <typename... A>
typename result<A...>::type operator()(A&&... a) const {
return std::make_shared<T>(std::forward<A>(a)...);
}
};
template <typename T>
using make_shared_ = boost::phoenix::function<make_shared_f<T> >;
}
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
template<typename Iterator, typename Skipper = ascii::space_type>
struct SystemParser : qi::grammar<Iterator, std::vector<std::shared_ptr<std::string> >(), Skipper>
{
SystemParser() : SystemParser::base_type(variable_group_)
{
namespace phx = boost::phoenix;
using qi::_1;
using qi::_val;
using qi::eps;
using qi::lit;
var_counter = 0;
declarative_symbols.add("variable_group",0);
variable_group_ = "variable_group" > genericvargp_ > ';';
genericvargp_ = new_variable_ % ','; //
new_variable_ = unencountered_symbol_ [_val = make_shared_<std::string>() (_1)];
unencountered_symbol_ = valid_variable_name_ - ( encountered_variables | declarative_symbols );
valid_variable_name_ = +qi::alpha >> *(qi::alnum | qi::char_("[]_") );
// debug(variable_group_); debug(unencountered_symbol_); debug(new_variable_); debug(genericvargp_);
// BOOST_SPIRIT_DEBUG_NODES((variable_group_) (valid_variable_name_) (unencountered_symbol_) (new_variable_) (genericvargp_))
}
// rule declarations. these are member variables for the parser.
qi::rule<Iterator, std::vector<std::shared_ptr<std::string> >(), Skipper > variable_group_;
qi::rule<Iterator, std::vector<std::shared_ptr<std::string> >(), Skipper > genericvargp_;
qi::rule<Iterator, std::shared_ptr<std::string()> > new_variable_;
qi::rule<Iterator, std::string()> unencountered_symbol_;
qi::rule<Iterator, std::string()> valid_variable_name_;
unsigned var_counter;
qi::symbols<char,int> encountered_variables;
qi::symbols<char,int> declarative_symbols;
};
with driver code:
int main(int argc, char** argv)
{
std::vector<std::shared_ptr<std::string> > V;
std::string str = "variable_group x, y, z; ";
std::string::const_iterator iter = str.begin();
std::string::const_iterator end = str.end();
SystemParser<std::string::const_iterator> S;
bool s = phrase_parse(iter, end, S, boost::spirit::ascii::space, V);
if (s)
{
std::cout << "Parse succeeded: " << V.size() << " variables\n";
for (auto& s : V)
std::cout << " - '" << s << "'\n";
}
else
std::cout << "Parse failed\n";
if (iter!=end)
std::cout << "Remaining unparsed: '" << std::string(iter, end) << "'\n";
return 0;
}
The text is parsed correctly, but the resulting vector is of length 0, while it should be of length 3. Somehow, the std::shared_ptr<string> is not pushed onto the back of the vector resulting from the rule genericvargp_.
I've tried many things, including reading all the debug information from a test parse, and placement of the %= signs for rule definitions, which should be used for rules for which there is a semantic action that does not assign _val unless I am mistaken. I've also played all night and day with using phx::bind to manually push onto the back of _val, but got nowhere. I've further verified that the make_shared_ provided by sehe in another answer is in fact lazy for std::shared_ptr.
As an aside, I have also struggled with getting the result of an unencountered_symbol_ to add to encountered_variables so as to enforce uniqueness of variable names...
The problem seems to be the propagation of the result of the new_variable_ rule onto the desired vector of shared pointers in the genericvargp_ rule.
This declaration
qi::rule<Iterator, std::shared_ptr<std::string()> > new_variable_;
Doesn't match the desired type:
qi::rule<Iterator, std::shared_ptr<std::string>() > new_variable_;
Sadly, in old SpiritV2 this attribute is silently ignored and no attribute propagation is done. This also explains why it didn't error out on compile time.
Live On Coliru
#define BOOST_SPIRIT_USE_PHOENIX_V3 1
#define BOOST_SPIRIT_DEBUG 1
#include <boost/spirit/include/qi_core.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <iostream>
#include <string>
#include <boost/spirit/include/phoenix_stl.hpp>
#include <boost/phoenix/bind/bind_member_function.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
// this solution for lazy make shared comes from the SO forum, user sehe.
// https://stackoverflow.com/questions/21516201/how-to-create-boost-phoenix-make-shared
// post found using google search terms `phoenix construct shared_ptr`
// changed from boost::shared_ptr to std::shared_ptr
namespace {
template <typename T> struct make_shared_f {
template <typename... A> struct result { typedef std::shared_ptr<T> type; };
template <typename... A> typename result<A...>::type operator()(A &&... a) const {
return std::make_shared<T>(std::forward<A>(a)...);
}
};
template <typename T> using make_shared_ = boost::phoenix::function<make_shared_f<T> >;
}
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
template <typename Iterator, typename Skipper = ascii::space_type>
struct SystemParser : qi::grammar<Iterator, std::vector<std::shared_ptr<std::string> >(), Skipper> {
SystemParser() : SystemParser::base_type(variable_group_) {
namespace phx = boost::phoenix;
using qi::_1;
using qi::_val;
using qi::eps;
using qi::lit;
var_counter = 0;
declarative_symbols.add("variable_group", 0);
variable_group_ = "variable_group" > genericvargp_ > ';';
genericvargp_ = new_variable_ % ','; //
new_variable_ = unencountered_symbol_ [_val = make_shared_<std::string>()(_1)];
unencountered_symbol_ = valid_variable_name_ - (encountered_variables | declarative_symbols);
valid_variable_name_ = +qi::alpha >> *(qi::alnum | qi::char_("[]_"));
BOOST_SPIRIT_DEBUG_NODES((variable_group_) (valid_variable_name_) (unencountered_symbol_) (new_variable_) (genericvargp_))
}
// rule declarations. these are member variables for the parser.
qi::rule<Iterator, std::vector<std::shared_ptr<std::string> >(), Skipper> variable_group_;
qi::rule<Iterator, std::vector<std::shared_ptr<std::string> >(), Skipper> genericvargp_;
qi::rule<Iterator, std::shared_ptr<std::string>() > new_variable_;
qi::rule<Iterator, std::string()> unencountered_symbol_;
qi::rule<Iterator, std::string()> valid_variable_name_;
unsigned var_counter;
qi::symbols<char, qi::unused_type> encountered_variables;
qi::symbols<char, qi::unused_type> declarative_symbols;
};
int main()
{
std::vector<std::shared_ptr<std::string> > V;
std::string str = "variable_group x, y, z; ";
std::string::const_iterator iter = str.begin();
std::string::const_iterator end = str.end();
SystemParser<std::string::const_iterator> S;
bool s = phrase_parse(iter, end, S, boost::spirit::ascii::space, V);
if (s)
{
std::cout << "Parse succeeded: " << V.size() << " variables\n";
for (auto& s : V)
std::cout << " - '" << *s << "'\n";
}
else
std::cout << "Parse failed\n";
if (iter!=end)
std::cout << "Remaining unparsed: '" << std::string(iter, end) << "'\n";
}
Prints
Parse succeeded: 3 variables
- 'x'
- 'y'
- 'z'
As well as a lot of debug information

Getting compilation errors while binding a rule production to my struct members

Writing Qi grammar with Phoenix bind I got a compilation error like
boost/spirit/home/support/context.hpp(180): error C2338: index_is_out_of_bounds
here
>> ruleHandId_[phx::bind(&parseContext::handId_, qi::_r1) = qi::_1];
I just havent too much expirience with phoenix binding but perv bind in the line
ruleStart_ = ruleEncoding_[phx::bind(&parseContext::encoding_, qi::_r1) = qi::_1]
works good without compilation errors
It's all under MSVC from VS2013 with boost 1.56 x86
Whats I do wrong under code with the compilation error?
Source Code
#include <boost/spirit/include/qi.hpp>
#include <boost/phoenix/phoenix.hpp>
#include <boost/shared_ptr.hpp>
#include <sstream>
namespace sp = boost::spirit;
namespace qi = boost::spirit::qi;
namespace phx = boost::phoenix;
struct parseContext {
std::string encoding_;
uint64_t handId_;
};
typedef boost::shared_ptr<parseContext> parseContextShPtr;
template <typename Iterator>
struct parseGrammar : qi::grammar<Iterator, void(parseContext&)> {
parseGrammar() : parseGrammar::base_type(ruleStart_)
{
ruleStart_ = ruleEncoding_[phx::bind(&parseContext::encoding_, qi::_r1) = qi::_1]
>> ruleHandHeader_;
ruleEncoding_ = qi::lit("ABC");
ruleHandHeader_ = qi::lit("DEF") >> qi::space
>> qi::lit("XYZ #")
>> ruleHandId_[phx::bind(&parseContext::handId_, qi::_r1) = qi::_1];
ruleHandId_ = qi::long_long;
}
// Rules
qi::rule<Iterator, void(parseContext&)> ruleStart_;
qi::rule<Iterator, std::string()> ruleEncoding_;
qi::rule<Iterator> ruleHandHeader_;
qi::rule<Iterator, uint64_t> ruleHandId_;
};
void test()
{
std::string s("ABCDEF XYZ #555: PQI #777");
std::stringstream sb;
sb.unsetf(std::ios::skipws);
sb << s;
const parseGrammar<sp::istream_iterator> p;
sp::istream_iterator b(sb);
sp::istream_iterator e;
parseContextShPtr ctx(new parseContext);
bool r = qi::parse(b, e, p(phx::ref(*ctx.get())));
if (r) {
std::cout << "Success" << std::endl;
}
else {
std::cout << "Failure" << std::endl;
}
std::cout << std::string(b, e).substr(0, 32) << std::endl;
}
Some of the placeholders cannot be bound.
This could be because ruleEncoding_ doesn't expose an attribute (for _1) (unlikely) or ruleStart_ doesn't have the inherited attribute (_r1).
That's all I can tell you right now.
Edit It was the latter. ruleHandHeader doesn't declare any attributes, let alone an inherited attribute to bind to _r1
Update To the comment.
Here are some suggestions. Much in the vein of my oft-repeated advice to avoid semantic actions (Boost Spirit: "Semantic actions are evil"?), I'd adapt the structure as a fusion sequence:
And use much simplified grammar rules:
ruleStart_ = ruleEncoding_ >> ruleHandHeader_;
ruleEncoding_ = "ABC";
ruleHandId_ = qi::long_long;
ruleHandHeader_ = "DEF XYZ #" >> ruleHandId_;
Now, adding in BOOST_SPIRIT_DEBUG macros and fixing uint64_t to uint64_t() in the rule definition:
Live On Coliru
#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/shared_ptr.hpp>
#include <sstream>
namespace qi = boost::spirit::qi;
struct parseContext {
std::string encoding_;
uint64_t handId_;
};
BOOST_FUSION_ADAPT_STRUCT(parseContext, (std::string, encoding_)(uint64_t,handId_))
typedef boost::shared_ptr<parseContext> parseContextShPtr;
template <typename Iterator>
struct parseGrammar : qi::grammar<Iterator, parseContext()> {
parseGrammar() : parseGrammar::base_type(ruleStart_)
{
ruleStart_ = ruleEncoding_ >> ruleHandHeader_;
ruleEncoding_ = "ABC";
ruleHandId_ = qi::long_long;
ruleHandHeader_ = "DEF XYZ #" >> ruleHandId_;
BOOST_SPIRIT_DEBUG_NODES((ruleStart_)(ruleEncoding_)(ruleHandId_)(ruleHandHeader_))
}
// Rules
qi::rule<Iterator, parseContext()> ruleStart_;
qi::rule<Iterator, std::string()> ruleEncoding_;
qi::rule<Iterator, uint64_t()> ruleHandId_, ruleHandHeader_;
};
void test()
{
std::stringstream sb("ABCDEF XYZ #555: PQI #777");
sb.unsetf(std::ios::skipws);
typedef boost::spirit::istream_iterator It;
const parseGrammar<It> p;
It b(sb), e;
parseContextShPtr ctx(new parseContext);
bool r = qi::parse(b, e, p, *ctx);
if (r) {
std::cout << "Success: " << ctx->encoding_ << ", " << ctx->handId_ << std::endl;
}
else {
std::cout << "Failure" << std::endl;
}
if (b!=e)
std::cout << "Remaining: '" << std::string(b, e).substr(0, 32) << "'...\n";
}
int main()
{
test();
}
Prints
Success: ABC, 555
Remaining: ': PQI #777'...

Boost Spirit Qi: Omit element in Kleene Star parser

I want to parse special constructs and throw the rest away. But I don't want to use a skipper.
I want to get a vector of these constructs, so I use a Kleene Star parser as main rule. But, everytime something gets thrown away, a default constructed element is inserted into the vector.
Here is a made up example. It just looks for the string Test and throws the rest away, at least this is the plan. But every time the rule garbage succeeds it adds a default constructed item to the vector in the rule all, giving an output of 7 insteat of 1. How can I tell Spirit to just add to the vector if the rule item succeeds?
#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <iostream>
#include <string>
#include <vector>
namespace qi = boost::spirit::qi;
struct container {
std::string name;
bool dummy;
};
BOOST_FUSION_ADAPT_STRUCT(::container,
(std::string, name)
(bool, dummy))
int main() {
typedef std::string::const_iterator iterator;
qi::rule<iterator, std::vector<container>()> all;
qi::rule<iterator, container()> item;
qi::rule<iterator, std::string()> string_rule;
qi::rule<iterator> garbage;
all = *(garbage | item);
garbage = qi::char_ - qi::lit("Test");
string_rule = qi::string("Test");
item = string_rule >> qi::attr(true);
std::vector<container> ast;
std::string input = "blaTestbla";
iterator first = input.begin();
iterator last = input.end();
bool result = qi::parse(first, last, all, ast);
if (result) {
result = first == last;
}
if (result) {
std::cout << "Parsed " << ast.size() << " element(s)" << std::endl;
} else {
std::cout << "failure" << std::endl;
}
}
Since sehe's answer was more or less for educational purposes, we have now several solutions:
*garbage >> -(item % *garbage) >> *garbage
*garbage >> *(item >> *garbage)
all = *(garbage | item[phx::push_back(qi::_val,qi::_1)]);
And the solution from cv_and_he:
#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <iostream>
#include <string>
#include <vector>
namespace qi = boost::spirit::qi;
struct container {
std::string name;
bool dummy;
};
BOOST_FUSION_ADAPT_STRUCT(::container,
(std::string, name)
(bool, dummy))
struct container_vector { //ADDED
std::vector<container> data;
};
namespace boost{ namespace spirit{ namespace traits //ADDED
{
template <>
struct is_container<container_vector> : boost::mpl::true_ {};
template <>
struct container_value<container_vector> {
typedef optional<container> type;
};
template <>
struct push_back_container<container_vector,optional<container> > {
static bool call(container_vector& cont, const optional<container>& val) {
if(val)
cont.data.push_back(*val);
return true;
}
};
}}}
int main() {
typedef std::string::const_iterator iterator;
qi::rule<iterator, container_vector()> all; //CHANGED
qi::rule<iterator, container()> item;
qi::rule<iterator, std::string()> string_rule;
qi::rule<iterator> garbage;
all = *(garbage | item);
garbage = qi::char_ - qi::lit("Test");
string_rule = qi::string("Test");
item = string_rule >> qi::attr(true);
container_vector ast; //CHANGED
std::string input = "blaTestbla";
iterator first = input.begin();
iterator last = input.end();
bool result = qi::parse(first, last, all, ast);
if (result) {
result = first == last;
}
if (result) {
std::cout << "Parsed " << ast.data.size() << " element(s)" << std::endl; //CHANGED
} else {
std::cout << "failure" << std::endl;
}
}
Although I didn't want to use a skipper I ended up with:
start = qi::skip(garbage.alias())[*item];
This last solution was the fastest (by 1-2%) in my unscientific tests using the c-files of the Linux kernel with my production rules.
A quick fix (not necessarily most performant) would be
all = -(item - garbage) % +garbage;
It prints:
Parsed 3 element(s)
See it Live on Coliru