Related
I seem to be experiencing some mental block with Boost Spirit I just cannot get by. I have a fairly simple grammar I need to handle, where I would like to put the values into a struct, that contains a std::map<> as one of it's members. The key names for the pairs are known up front, so that only those are allowed. There could be one to many keys in the map, in any order with each key name validated via qi.
The grammar looks something like this, as an example.
test .|*|<hostname> add|modify|save ( key [value] key [value] ... ) ;
//
test . add ( a1 ex00
a2 ex01
a3 "ex02,ex03,ex04" );
//
test * modify ( m1 ex10
m2 ex11
m3 "ex12,ex13,ex14"
m4 "abc def ghi" );
//
test 10.0.0.1 clear ( c1
c2
c3 );
In this example the keys for “add” being a1, a2 and a3, likewise for “modify” m1, m2, m3 and m4 and each must contain a value. For “clear” the keys of the map c1, c2 and c3 may not contain a value. Also, let's say for this example you can have up to 10 keys (a1 ... a11, m1 ... m11 and c1 ... c11) any combination of them could be used, in any order, for their corresponding action. Meaning that you cannot use the known key cX for the "add" or mX for "clear"
The structure follows this simple pattern
//
struct test
{
std::string host;
std::string action;
std::map<std::string,std::string> option;
}
So from the above examples, I would expect to have the struct contain ...
// add ...
test.host = .
test.action = add
test.option[0].first = a1
test.option[0].second = ex00
test.option[1].first = a2
test.option[1].second = ex01
test.option[2].first = a3
test.option[2].second = ex02,ex03,ex04
// modify ...
test.host = *
test.action = modify
test.option[0].first = m1
test.option[0].second = ex10
test.option[1].first = m2
test.option[1].second = ex11
test.option[2].first = m3
test.option[2].second = ex12,ex13,ex14
test.option[2].first = m3
test.option[2].second = abc def ghi
// clear ...
test.host = *
test.action = 10.0.0.1
test.option[0].first = c1
test.option[0].second =
test.option[1].first = c2
test.option[1].second =
test.option[2].first = c3
test.option[2].second =
I can get each indivudal part working, standalone, but I cannot seem to them working together. For example I have the host and action working without the map<>.
I’ve adapted a previously posted example from Sehe (here) trying to get this to work (BTW: Sehe has some awesome examples, which I’ve been using as much as the documentation).
Here is an excerpt (obviously not working), but at least shows where I’m trying to go.
namespace ast {
namespace qi = boost::spirit::qi;
//
using unused = qi::unused_type;
//
using string = std::string;
using strings = std::vector<string>;
using list = strings;
using pair = std::pair<string, string>;
using map = std::map<string, string>;
//
struct test
{
using preference = std::map<string,string>;
string host;
string action;
preference option;
};
}
//
BOOST_FUSION_ADAPT_STRUCT( ast::test,
( std::string, host )
( std::string, action ) )
( ast::test::preference, option ) )
//
namespace grammar
{
//
template <typename It>
struct parser
{
//
struct skip : qi::grammar<It>
{
//
skip() : skip::base_type( text )
{
using namespace qi;
// handle all whitespace (" ", \t, ...)
// along with comment lines/blocks
//
// comment blocks: /* ... */
// // ...
// -- ...
// # ...
text = ascii::space
| ( "#" >> *( char_ - eol ) >> ( eoi | eol ) ) // line comment
| ( "--" >> *( char_ - eol ) >> ( eoi | eol ) ) // ...
| ( "//" >> *( char_ - eol ) >> ( eoi | eol ) ) // ...
| ( "/*" >> *( char_ - "*/" ) >> "*/" ); // block comment
//
BOOST_SPIRIT_DEBUG_NODES( ( text ) )
}
//
qi::rule<It> text;
};
//
struct token
{
//
token()
{
using namespace qi;
// common
string = '"' >> *("\\" >> char_ | ~char_('"')) >> '"';
identity = char_("a-zA-Z_") >> *char_("a-zA-Z0-9_");
real = double_;
integer = int_;
//
value = ( string | identity );
// ip target
any = '*';
local = ( char_('.') | fqdn );
fqdn = +char_("a-zA-Z0-9.\\-" ); // consession
ipv4 = +as_string[ octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] >> '.'
>> octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] >> '.'
>> octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] >> '.'
>> octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] ];
//
target = ( any | local | fqdn | ipv4 );
//
pair = identity >> -( attr( ' ' ) >> value );
map = pair >> *( attr( ' ' ) >> pair );
list = *( value );
//
BOOST_SPIRIT_DEBUG_NODES( ( string )
( identity )
( value )
( real )
( integer )
( any )
( local )
( fqdn )
( ipv4 )
( target )
( pair )
( keyval )
( map )
( list ) )
}
//
qi::rule<It, std::string()> string;
qi::rule<It, std::string()> identity;
qi::rule<It, std::string()> value;
qi::rule<It, double()> real;
qi::rule<It, int()> integer;
qi::uint_parser<unsigned, 10, 1, 3> octet;
qi::rule<It, std::string()> any;
qi::rule<It, std::string()> local;
qi::rule<It, std::string()> fqdn;
qi::rule<It, std::string()> ipv4;
qi::rule<It, std::string()> target;
//
qi::rule<It, ast::map()> map;
qi::rule<It, ast::pair()> pair;
qi::rule<It, ast::pair()> keyval;
qi::rule<It, ast::list()> list;
};
//
struct test : token, qi::grammar<It, ast::test(), skip>
{
//
test() : test::base_type( command_ )
{
using namespace qi;
using namespace qr;
auto kw = qr::distinct( copy( char_( "a-zA-Z0-9_" ) ) );
// not sure how to enforce the "key" names!
key_ = *( '(' >> *value >> ')' );
// tried using token::map ... didn't work ...
//
add_ = ( ( "add" >> attr( ' ' ) ) [ _val = "add" ] );
modify_ = ( ( "modify" >> attr( ' ' ) ) [ _val = "modify" ] );
clear_ = ( ( "clear" >> attr( ' ' ) ) [ _val = "clear" ] );
//
action_ = ( add_ | modify_ | clear_ );
/* *** can't get from A to B here ... not sure what to do *** */
//
command_ = kw[ "test" ]
>> target
>> action_
>> ';';
BOOST_SPIRIT_DEBUG_NODES( ( command_ )
( action_ )
( add_ )
( modify_ )
( clear_ ) )
}
//
private:
//
using token::value;
using token::target;
using token::map;
qi::rule<It, ast::test(), skip> command_;
qi::rule<It, std::string(), skip> action_;
//
qi::rule<It, std::string(), skip> add_;
qi::rule<It, std::string(), skip> modify_;
qi::rule<It, std::string(), skip> clear_;
};
...
};
}
I hope this question isn't too ambiguous and if you need a working example of the problem, I can certainly provide that. Any and all help is greatly appreciated, so thank you in advance!
Notes:
with this
add_ = ( ( "add" >> attr( ' ' ) ) [ _val = "add" ] );
modify_ = ( ( "modify" >> attr( ' ' ) ) [ _val = "modify" ] );
clear_ = ( ( "clear" >> attr( ' ' ) ) [ _val = "clear" ] );
did you mean to require a space? Or are you really just trying to force the struct action field to contain a trailing space (that's what will happen).
If you meant the latter, I'd do that outside of the parser¹.
If you wanted the first, use the kw facility:
add_ = kw["add"] [ _val = "add" ];
modify_ = kw["modify"] [ _val = "modify" ];
clear_ = kw["clear"] [ _val = "clear" ];
In fact, you can simplify that (again, ¹):
add_ = raw[ kw["add"] ];
modify_ = raw[ kw["modify"] ];
clear_ = raw[ kw["clear"] ];
Which also means that you can simplify to
action_ = raw[ kw[lit("add")|"modify"|"clear"] ];
However, getting a bit close to your question, you could also use a symbol parser:
symbols<char> action_sym;
action_sym += "add", "modify", "clear";
//
action_ = raw[ kw[action_sym] ];
Caveat: the symbols needs to be a member so its lifetime extends beyond the constructor.
If you meant to capture the input representation of ipv4 addresses with
ipv4 = +as_string[ octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] >> '.'
>> octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] >> '.'
>> octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] >> '.'
>> octet[ _pass = ( _1 >= 0 && _1 <= 255 ) ] ];
Side note I'm assuming +as_string is a simple mistake and you meant as_string instead.
Simplify:
qi::uint_parser<uint8_t, 10, 1, 3> octet;
This obviates the range checks (see ¹ again):
ipv4 = as_string[ octet >> '.' >> octet >> '.' >> octet >> '.' >> octet ];
However, this would build a 4-char binary string representation of the address. If you wanted that, fine. I doubt it (because you'd have written std::array<uint8_t, 4> or uint64_t, right?). So if you wanted the string, again use raw[]:
ipv4 = raw[ octet >> '.' >> octet >> '.' >> octet >> '.' >> octet ];
Same issue as with number 1.:
pair = identity >> -( attr(' ') >> value );
This time, the problem betrays that the productions should not be in token; Conceptually token-izing precedes parsing and hence I'd keep the tokens skipper-less. kw doesn't really do a lot of good in that context. Instead, I'd move pair, map and list (unused?) into the parser:
pair = kw[identity] >> -value;
map = +pair;
list = *value;
Some examples
There's a very recent example I made about using symbols to parse (here), but this answer comes a lot closer to your question:
How to provider user with autocomplete suggestions for given boost::spirit grammar?
It goes far beyond the scope of your parser because it does all kinds of actions in the grammar, but what it does show is to have generic "lookup-ish" rules that can be parameterized with a particular "symbol set": see the Identifier Lookup section of the answer:
Identifier Lookup
We store "symbol tables" in Domain members _variables and
_functions:
using Domain = qi::symbols<char>; Domain _variables, _functions;
Then we declare some rules that can do lookups on either of them:
// domain identifier lookups
qi::_r1_type _domain;
qi::rule<It, Ast::Identifier(Domain const&)> maybe_known, known,
unknown;
The corresponding declarations will be shown shortly.
Variables are pretty simple:
variable = maybe_known(phx::ref(_variables));
Calls are trickier. If a name is unknown we don't want to assume it
implies a function unless it's followed by a '(' character.
However, if an identifier is a known function name, we want even to
imply the ( (this gives the UX the appearance of autocompletion
where when the user types sqrt, it suggests the next character to be
( magically).
// The heuristics: // - an unknown identifier followed by (
// - an unclosed argument list implies ) call %= (
known(phx::ref(_functions)) // known -> imply the parens
| &(identifier >> '(') >> unknown(phx::ref(_functions))
) >> implied('(') >> -(expression % ',') >> implied(')');
It all builds on known, unknown and maybe_known:
///////////////////////////////
// identifier loopkup, suggesting
{
maybe_known = known(_domain) | unknown(_domain);
// distinct to avoid partially-matching identifiers
using boost::spirit::repository::qi::distinct;
auto kw = distinct(copy(alnum | '_'));
known = raw[kw[lazy(_domain)]];
unknown = raw[identifier[_val=_1]] [suggest_for(_1, _domain)];
}
I think you can use the same approach constructively here. One additional gimmick could be to validate that properties supplied are, in fact, unique.
Demo Work
Combining all the hints above makes it compile and "parse" the test commands:
Live On Coliru
#include <string>
#include <map>
#include <vector>
namespace ast {
//
using string = std::string;
using strings = std::vector<string>;
using list = strings;
using pair = std::pair<string, string>;
using map = std::map<string, string>;
//
struct command {
string host;
string action;
map option;
};
}
#include <boost/fusion/adapted.hpp>
BOOST_FUSION_ADAPT_STRUCT(ast::command, host, action, option)
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/spirit/repository/include/qi_distinct.hpp>
namespace grammar
{
namespace qi = boost::spirit::qi;
namespace qr = boost::spirit::repository::qi;
template <typename It>
struct parser
{
struct skip : qi::grammar<It> {
skip() : skip::base_type(text) {
using namespace qi;
// handle all whitespace along with line/block comments
text = ascii::space
| (lit("#")|"--"|"//") >> *(char_ - eol) >> (eoi | eol) // line comment
| "/*" >> *(char_ - "*/") >> "*/"; // block comment
//
BOOST_SPIRIT_DEBUG_NODES((text))
}
private:
qi::rule<It> text;
};
//
struct token {
//
token() {
using namespace qi;
// common
string = '"' >> *("\\" >> char_ | ~char_('"')) >> '"';
identity = char_("a-zA-Z_") >> *char_("a-zA-Z0-9_");
value = string | identity;
// ip target
any = '*';
local = '.' | fqdn;
fqdn = +char_("a-zA-Z0-9.\\-"); // concession
ipv4 = raw [ octet >> '.' >> octet >> '.' >> octet >> '.' >> octet ];
//
target = any | local | fqdn | ipv4;
//
BOOST_SPIRIT_DEBUG_NODES(
(string) (identity) (value)
(any) (local) (fqdn) (ipv4) (target)
)
}
protected:
//
qi::rule<It, std::string()> string;
qi::rule<It, std::string()> identity;
qi::rule<It, std::string()> value;
qi::uint_parser<uint8_t, 10, 1, 3> octet;
qi::rule<It, std::string()> any;
qi::rule<It, std::string()> local;
qi::rule<It, std::string()> fqdn;
qi::rule<It, std::string()> ipv4;
qi::rule<It, std::string()> target;
};
//
struct test : token, qi::grammar<It, ast::command(), skip> {
//
test() : test::base_type(command_)
{
using namespace qi;
auto kw = qr::distinct( copy( char_( "a-zA-Z0-9_" ) ) );
//
action_sym += "add", "modify", "clear";
action_ = raw[ kw[action_sym] ];
//
command_ = kw["test"]
>> target
>> action_
>> '(' >> map >> ')'
>> ';';
//
pair = kw[identity] >> -value;
map = +pair;
list = *value;
BOOST_SPIRIT_DEBUG_NODES(
(command_) (action_)
(pair) (map) (list)
)
}
private:
using token::target;
using token::identity;
using token::value;
qi::symbols<char> action_sym;
//
qi::rule<It, ast::command(), skip> command_;
qi::rule<It, std::string(), skip> action_;
//
qi::rule<It, ast::map(), skip> map;
qi::rule<It, ast::pair(), skip> pair;
qi::rule<It, ast::list(), skip> list;
};
};
}
#include <fstream>
int main() {
using It = boost::spirit::istream_iterator;
using Parser = grammar::parser<It>;
std::ifstream input("input.txt");
It f(input >> std::noskipws), l;
Parser::skip const s{};
Parser::test const p{};
std::vector<ast::command> data;
bool ok = phrase_parse(f, l, *p, s, data);
if (ok) {
std::cout << "Parsed " << data.size() << " commands\n";
} else {
std::cout << "Parsed failed\n";
}
if (f != l) {
std::cout << "Remaining unparsed input: '" << std::string(f,l) << "'\n";
}
}
Prints
Parsed 3 commands
Let's restrict the Keys
Like in the linked answer above, let's pass the map, pair rules the actual key set to get their allowed values from:
using KeySet = qi::symbols<char>;
using KeyRef = KeySet const*;
//
KeySet add_keys, modify_keys, clear_keys;
qi::symbols<char, KeyRef> action_sym;
qi::rule<It, ast::pair(KeyRef), skip> pair;
qi::rule<It, ast::map(KeyRef), skip> map;
Note A key feature used is the associated attribute value with a symbols<> lookup (in this case we associate a KeyRef with an action symbol):
//
add_keys += "a1", "a2", "a3", "a4", "a5", "a6";
modify_keys += "m1", "m2", "m3", "m4";
clear_keys += "c1", "c2", "c3", "c4", "c5";
action_sym.add
("add", &add_keys)
("modify", &modify_keys)
("clear", &clear_keys);
Now the heavy lifting starts.
Using qi::locals<> and inherited attributes
Let's give command_ some local space to store the selected keyset:
qi::rule<It, ast::command(), skip, qi::locals<KeyRef> > command_;
Now we can in principle assignt to it (using the _a placeholder). However, there's some details:
//
qi::_a_type selected;
Always prefer descriptive names :) _a and _r1 get old pretty quick. Things are confusing enough as it is.
command_ %= kw["test"]
>> target
>> raw[ kw[action_sym] [ selected = _1 ] ]
>> '(' >> map(selected) >> ')'
>> ';';
Note: the subtlest detail here is %= instead of = to avoid the suppression of automatic attribute propagation when a semantic action is present (yeah, see ¹ again...)
But all in all, that doesn't read so bad?
//
qi::_r1_type symref;
pair = raw[ kw[lazy(*symref)] ] >> -value;
map = +pair(symref);
And now at least things parse
Almost there
Live On Coliru
//#define BOOST_SPIRIT_DEBUG
#include <string>
#include <map>
#include <vector>
namespace ast {
//
using string = std::string;
using strings = std::vector<string>;
using list = strings;
using pair = std::pair<string, string>;
using map = std::map<string, string>;
//
struct command {
string host;
string action;
map option;
};
}
#include <boost/fusion/adapted.hpp>
BOOST_FUSION_ADAPT_STRUCT(ast::command, host, action, option)
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/spirit/repository/include/qi_distinct.hpp>
namespace grammar
{
namespace qi = boost::spirit::qi;
namespace qr = boost::spirit::repository::qi;
template <typename It>
struct parser
{
struct skip : qi::grammar<It> {
skip() : skip::base_type(rule_) {
using namespace qi;
// handle all whitespace along with line/block comments
rule_ = ascii::space
| (lit("#")|"--"|"//") >> *(char_ - eol) >> (eoi | eol) // line comment
| "/*" >> *(char_ - "*/") >> "*/"; // block comment
//
//BOOST_SPIRIT_DEBUG_NODES((skipper))
}
private:
qi::rule<It> rule_;
};
//
struct token {
//
token() {
using namespace qi;
// common
string = '"' >> *("\\" >> char_ | ~char_('"')) >> '"';
identity = char_("a-zA-Z_") >> *char_("a-zA-Z0-9_");
value = string | identity;
// ip target
any = '*';
local = '.' | fqdn;
fqdn = +char_("a-zA-Z0-9.\\-"); // concession
ipv4 = raw [ octet >> '.' >> octet >> '.' >> octet >> '.' >> octet ];
//
target = any | local | fqdn | ipv4;
//
BOOST_SPIRIT_DEBUG_NODES(
(string) (identity) (value)
(any) (local) (fqdn) (ipv4) (target)
)
}
protected:
//
qi::rule<It, std::string()> string;
qi::rule<It, std::string()> identity;
qi::rule<It, std::string()> value;
qi::uint_parser<uint8_t, 10, 1, 3> octet;
qi::rule<It, std::string()> any;
qi::rule<It, std::string()> local;
qi::rule<It, std::string()> fqdn;
qi::rule<It, std::string()> ipv4;
qi::rule<It, std::string()> target;
};
//
struct test : token, qi::grammar<It, ast::command(), skip> {
//
test() : test::base_type(start_)
{
using namespace qi;
auto kw = qr::distinct( copy( char_( "a-zA-Z0-9_" ) ) );
//
add_keys += "a1", "a2", "a3", "a4", "a5", "a6";
modify_keys += "m1", "m2", "m3", "m4";
clear_keys += "c1", "c2", "c3", "c4", "c5";
action_sym.add
("add", &add_keys)
("modify", &modify_keys)
("clear", &clear_keys);
//
qi::_a_type selected;
command_ %= kw["test"]
>> target
>> raw[ kw[action_sym] [ selected = _1 ] ]
>> '(' >> map(selected) >> ')'
>> ';';
//
qi::_r1_type symref;
pair = raw[ kw[lazy(*symref)] ] >> -value;
map = +pair(symref);
list = *value;
start_ = command_;
BOOST_SPIRIT_DEBUG_NODES(
(start_) (command_)
(pair) (map) (list)
)
}
private:
using token::target;
using token::identity;
using token::value;
using KeySet = qi::symbols<char>;
using KeyRef = KeySet const*;
//
qi::rule<It, ast::command(), skip> start_;
qi::rule<It, ast::command(), skip, qi::locals<KeyRef> > command_;
//
KeySet add_keys, modify_keys, clear_keys;
qi::symbols<char, KeyRef> action_sym;
qi::rule<It, ast::pair(KeyRef), skip> pair;
qi::rule<It, ast::map(KeyRef), skip> map;
qi::rule<It, ast::list(), skip> list;
};
};
}
#include <fstream>
int main() {
using It = boost::spirit::istream_iterator;
using Parser = grammar::parser<It>;
std::ifstream input("input.txt");
It f(input >> std::noskipws), l;
Parser::skip const s{};
Parser::test const p{};
std::vector<ast::command> data;
bool ok = phrase_parse(f, l, *p, s, data);
if (ok) {
std::cout << "Parsed " << data.size() << " commands\n";
} else {
std::cout << "Parsed failed\n";
}
if (f != l) {
std::cout << "Remaining unparsed input: '" << std::string(f,l) << "'\n";
}
}
Prints
Parsed 3 commands
HOLD ON, NOT SO FAST! It's wrong
Yeah. If you enable debug, you'll see it parses things oddly:
<attributes>[[[1, 0, ., 0, ., 0, ., 1], [c, l, e, a, r], [[[c, 1], [c, 2]], [[c, 3], []]]]]</attributes>
This is actually "merely" a problem with the grammar. If the grammar cannot see the difference between a key and value then obviously c2 is going to be parsed as the value of property with key c1.
It's up to you to disambiguate the grammar. For now, I'm going to demonstrate a fix using a negative assertion: we only accept values that are not known keys. It's a bit dirty, but might be useful to you for instructional purposes:
key = raw[ kw[lazy(*symref)] ];
pair = key(symref) >> -(!key(symref) >> value);
map = +pair(symref);
Note I factored out the key rule for readability:
Live On Coliru
Parses
<attributes>[[[1, 0, ., 0, ., 0, ., 1], [c, l, e, a, r], [[[c, 1], []], [[c, 2], []], [[c, 3], []]]]]</attributes>
Just what the doctor ordered!
¹ Boost Spirit: "Semantic actions are evil"?
Once again, I find myself reaching for boost spirit. Once again I find myself defeated by it.
A HTTP header value takes the general form:
text/html; q=1.0, text/*; q=0.8, image/gif; q=0.6, image/jpeg; q=0.6, image/*; q=0.5, */*; q=0.1
i.e. value *OWS [; *OWS name *OWS [= *OWS possibly_quoted_value] *OWS [...]] *OWS [ , <another value> ...]
so in my mind, this header decodes to:
value[0]:
text/html
params:
name : q
value : 1.0
value[1]:
text/*
params:
name : q
value : 0.8
...
and so on.
I am certain that to anyone who knows how, the boost::spirit::qi syntax for this is trivial.
I humbly ask for your assistance.
for example, here's the outline of the code which decodes the Content-Type header, which is limited to one value of the form type/subtype, with any number of parameters of the form <sp> ; <sp> token=token|quoted_string
template<class Iter>
void parse(ContentType& ct, Iter first, Iter last)
{
ct.mutable_type()->append(to_lower(consume_token(first, last)));
consume_lit(first, last, '/');
ct.mutable_subtype()->append(to_lower(consume_token(first, last)));
while (first != last) {
skipwhite(first, last);
if (consume_char_if(first, last, ';'))
{
auto p = ct.add_parameters();
skipwhite(first, last);
p->set_name(to_lower(consume_token(first, last)));
skipwhite(first, last);
if (consume_char_if(first, last, '='))
{
skipwhite(first, last);
p->set_value(consume_token_or_quoted(first, last));
}
else {
// no value on this parameter
}
}
else if (consume_char_if(first, last, ','))
{
// normally we should get the next value-token here but in the case of Content-Type
// we must barf
throw std::runtime_error("invalid use of ; in Content-Type");
}
}
}
ContentType& populate(ContentType& ct, const std::string& header_value)
{
parse(ct, header_value.begin(), header_value.end());
return ct;
}
OK, after an heroic 24 hours of struggle (well, not really - more like reading the manual over and over again...), I've found a way that works.
I am by no means competent with boost::spirit. If someone out there can improve on this answer, please do post it.
This spirit state machine takes the value of a header (with one, optionally parameterised, value) and turns it into a content_type structure.
My amateur reading of the HTTP standard indicates that some headers have the form (spaces here indicate any amount of white space, values may be quoted or not:
Header-Name: tokena/tokenb [; param1 = "value" [; param2 = value]...]
whereas others have the more general form:
Header-Name: token [; param1 = "value"[; param2 = value]...] [ , token ...]
This code covers the first case - i.e. the HTTP Content-Type header value. I will need to extend it to cater for the Accept header (which can advertise multiple values with parameters) - that will come later.
So here's the code. Please by all means show me how to improve it!!
#define BOOST_SPIRIT_DEBUG
#include <gtest/gtest.h>
#include <boost/spirit/include/qi.hpp>
#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/qi_char.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
#include <boost/spirit/include/phoenix_stl.hpp>
#include <boost/fusion/include/adapt_struct.hpp>
#include <boost/fusion/include/std_pair.hpp>
#include <utility>
#include <vector>
#include <string>
#include <boost/variant.hpp>
namespace qi = boost::spirit::qi;
namespace ascii = boost::spirit::ascii;
using unary_parameter = std::string;
struct binary_parameter
{
std::string name;
std::string value;
};
BOOST_FUSION_ADAPT_STRUCT(binary_parameter,
(std::string, name)
(std::string, value))
using parameter = boost::variant<unary_parameter, binary_parameter>;
struct type_subtype
{
std::string type;
std::string subtype;
};
BOOST_FUSION_ADAPT_STRUCT(type_subtype,
(std::string, type)
(std::string, subtype))
using content_type_pair = std::pair<std::string, std::string>;
struct content_type
{
type_subtype type;
std::vector<parameter> params;
};
BOOST_FUSION_ADAPT_STRUCT(content_type,
(type_subtype, type)
(std::vector<parameter>, params))
template<class Iterator>
struct token_grammar : qi::grammar<Iterator, content_type()>
{
token_grammar() : token_grammar::base_type(content_type_rule)
{
using ascii::char_;
using qi::omit;
using qi::eoi;
CR = char_('\r');
LF = char_('\n');
CRLF = CR >> LF;
SP = char_(' ');
HT = char_('\t');
LWS = -CRLF >> +(SP | HT);
UPALPHA = char_('A', 'Z');
LOALPHA = char_('a', 'z');
ALPHA = UPALPHA | LOALPHA;
DIGIT = char_('0', '9');
CTL = char_(0, 31) | char_(127);
QUOT = char_('"');
TEXT = (char_ - CTL) | HT;
separator = char_('(') | ')' | '<' | '>' | '#'
| ',' | ';' | ':' | '\\' | '"'
| '/' | '[' | ']' | '?' | '='
| '{' | '}' | SP | HT;
end_sequence = separator | space;
token = +(char_ - separator);
qdtext = char_ - char_('"') - '\\';
quoted_pair = omit[char_('\\')] >> char_;
quoted_string = omit[char_('"')] >> *(qdtext | quoted_pair) >> omit[char_('"')];
value = quoted_string | token ;
type_subtype_rule = token >> '/' >> token;
name_only = token;
nvp = token >> omit[*SP] >> omit['='] >> omit[*SP] >> value;
any_parameter = omit[*SP] >> omit[char_(';')] >> omit[*SP] >> (nvp | name_only);
content_type_rule = type_subtype_rule >> *any_parameter;
BOOST_SPIRIT_DEBUG_NODES((qdtext)(quoted_pair)(quoted_string)(value)(token)(separator));
}
qi::rule<Iterator, void()> CR, LF, CRLF, SP, HT, LWS, CTL, QUOT;
qi::rule<Iterator, char()> UPALPHA, LOALPHA, ALPHA, DIGIT, TEXT, qdtext, quoted_pair;
qi::rule<Iterator, void()> separator, space, end_sequence;
qi::rule<Iterator, std::string()> quoted_string, token, value;
qi::rule<Iterator, type_subtype()> type_subtype_rule;
qi::rule<Iterator, unary_parameter()> name_only;
qi::rule<Iterator, binary_parameter()> nvp;
qi::rule<Iterator, parameter()> any_parameter;
qi::rule<Iterator, content_type()> content_type_rule;
};
TEST(spirit_test, test1)
{
token_grammar<std::string::const_iterator> grammar{};
std::string test = R"__test(application/json )__test";
content_type ct;
bool r = qi::parse(test.cbegin(), test.cend(), grammar, ct);
EXPECT_EQ("application", ct.type.type);
EXPECT_EQ("json", ct.type.subtype);
EXPECT_EQ(0, ct.params.size());
ct = {};
test = R"__test(text/html ; charset = "ISO-8859-5")__test";
qi::parse(test.cbegin(), test.cend(), grammar, ct);
EXPECT_EQ("text", ct.type.type);
EXPECT_EQ("html", ct.type.subtype);
ASSERT_EQ(1, ct.params.size());
ASSERT_EQ(typeid(binary_parameter), ct.params[0].type());
auto& x = boost::get<binary_parameter>(ct.params[0]);
EXPECT_EQ("charset", x.name);
EXPECT_EQ("ISO-8859-5", x.value);
}
I've taken the code as posted by OP and given it a review.
there's no need to specify void(). In fact it's preferable to use qi::unused_type in such cases, which is what rules will default to if no attribute type is declared.
there no need for char_ if you don't wish to expose the attribute. Use lit instead.
there is no need to wrap every char parser in a rule. That hurts performance. It's best to leave the proto expression tree un-evaluated as long so Qi can optimize parser expressions more, and the compiler can inline more.
Also, Qi doesn't have move semantics on attributes, so avoiding redundant rules eliminates redundant copies of sub-attributes that get concatenated in the containing rules.
Sample alternative spelling (caution, see Assigning parsers to auto variables)
auto CR = qi::lit('\r');
auto LF = qi::lit('\n');
auto CRLF = qi::lit("\r\n");
auto HT = qi::lit('\t');
auto SP = qi::lit(' ');
auto LWS = qi::copy(-CRLF >> +(SP | HT)); // deepcopy
UPALPHA = char_('A', 'Z');
LOALPHA = char_('a', 'z');
ALPHA = UPALPHA | LOALPHA;
DIGIT = char_('0', '9');
//CTL = char_(0, 31) | char_(127);
TEXT = char_("\t\x20-\x7e\x80-\xff");
Since you didn't have to use char_, you also don't have kill the attribute using qi::omit[].
When you are in a Qi domain expression template, raw string/char literals are implicitly wrapped in a qi::lit so, you can simply things like
quoted_pair = omit[char_('\\')] >> char_;
quoted_string = omit[char_('"')] >> *(qdtext | quoted_pair) >> omit[char_('"')];
to just
quoted_pair = '\\' >> char_;
quoted_string = '"' >> *(qdtext | quoted_pair) >> '"';
instead of spelling out skipping spaces with omit[*SP] all the time, just declare the rule with a skipper. Now, you can simplify
nvp = token >> omit[*SP] >> omit['='] >> omit[*SP] >> value;
any_parameter = omit[*SP] >> omit[char_(';')] >> omit[*SP] >> (nvp | name_only);
content_type_rule = type_subtype_rule >> *any_parameter;
to just
nvp = token >> '=' >> value;
any_parameter = ';' >> (nvp | name_only);
content_type_rule = type_subtype_rule >> qi::skip(spaces)[*any_parameter];
Note that any subrule invocations of rules that are declared without a skipper are implicitly lexeme: Boost spirit skipper issues
there were many redundant/unused headers
recent compilers + boost versions make BOOST_FUSION_ADAPT_STRUCT much simpler by using decltype
The results of simplifying are much less noisy:
//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/include/adapted.hpp>
struct parameter {
boost::optional<std::string> name;
std::string value;
};
struct type_subtype {
std::string type;
std::string subtype;
};
struct content_type {
type_subtype type;
std::vector<parameter> params;
};
BOOST_FUSION_ADAPT_STRUCT(type_subtype, type, subtype)
BOOST_FUSION_ADAPT_STRUCT(content_type, type, params)
template<class Iterator>
struct token_grammar : qi::grammar<Iterator, content_type()>
{
token_grammar() : token_grammar::base_type(content_type_rule)
{
using qi::ascii::char_;
spaces = char_(' ');
token = +~char_( "()<>#,;:\\\"/[]?={} \t");
quoted_string = '"' >> *('\\' >> char_ | ~char_('"')) >> '"';
value = quoted_string | token;
type_subtype_rule = token >> '/' >> token;
name_only = token;
nvp = token >> '=' >> value;
any_parameter = ';' >> (nvp | name_only);
content_type_rule = type_subtype_rule >> qi::skip(spaces) [*any_parameter];
BOOST_SPIRIT_DEBUG_NODES((nvp)(any_parameter)(content_type_rule)(quoted_string)(token)(value)(type_subtype_rule))
}
private:
using Skipper = qi::space_type;
Skipper spaces;
qi::rule<Iterator, binary_parameter(), Skipper> nvp;
qi::rule<Iterator, parameter(), Skipper> any_parameter;
qi::rule<Iterator, content_type()> content_type_rule;
// lexemes
qi::rule<Iterator, std::string()> quoted_string, token, value;
qi::rule<Iterator, type_subtype()> type_subtype_rule;
qi::rule<Iterator, unary_parameter()> name_only;
};
See it Live On Coliru (with the same test cases)
BONUS
I'd prefer a simpler AST in a case like this. By injecting some attribute values using qi::attr you can avoid using boost::variant and/or even avoid boost::optional:
struct parameter {
bool have_name;
std::string name;
std::string value;
};
struct type_subtype {
std::string type;
std::string subtype;
};
struct content_type {
type_subtype type;
std::vector<parameter> params;
};
BOOST_FUSION_ADAPT_STRUCT(parameter, have_name, name, value)
BOOST_FUSION_ADAPT_STRUCT(type_subtype, type, subtype)
BOOST_FUSION_ADAPT_STRUCT(content_type, type, params)
namespace qi = boost::spirit::qi;
template<class Iterator>
struct token_grammar : qi::grammar<Iterator, content_type()>
{
token_grammar() : token_grammar::base_type(content_type_rule)
{
using qi::ascii::char_;
spaces = char_(' ');
token = +~char_( "()<>#,;:\\\"/[]?={} \t");
quoted_string = '"' >> *('\\' >> char_ | ~char_('"')) >> '"';
value = quoted_string | token;
type_subtype_rule = token >> '/' >> token;
name_only = qi::attr(false) >> qi::attr("") >> token;
nvp = qi::attr(true) >> token >> '=' >> value;
any_parameter = ';' >> (nvp | name_only);
content_type_rule = type_subtype_rule >> qi::skip(spaces) [*any_parameter];
BOOST_SPIRIT_DEBUG_NODES((nvp)(any_parameter)(content_type_rule)(quoted_string)(token)(value)(type_subtype_rule))
}
private:
using Skipper = qi::space_type;
Skipper spaces;
qi::rule<Iterator, parameter(), Skipper> nvp, name_only, any_parameter;
qi::rule<Iterator, content_type()> content_type_rule;
// lexemes
qi::rule<Iterator, std::string()> quoted_string, token, value;
qi::rule<Iterator, type_subtype()> type_subtype_rule;
};
I want to create vector and append values to it (if any) in one spirit rule. Is it possible? I tried something like below but with no success. Read code comments for details please. Thanks.
typedef std::vector<double> number_array;
typedef std::vector<std::string> string_array;
typedef boost::variant<number_array, string_array> node
template<typename Iterator>
struct parser
: qi::grammar<Iterator, node(), ascii::space_type> {
parser(parser_impl* p)
: parser::base_type(expr_, ""),
error_handler(ErrorHandler(p)) {
// Here I want to create vector on the fly
// and append values to newly created vector.
// but this doesn't compile, specifically phoenix::push_back(...)
number_array_ = qi::lit('n[')[qi::_val = construct<number_array>()] >>
-(qi::double_ % ',')[phoenix::push_back(phoenix::ref(qi::_val), qi::_1)] >> ']';
// this doesn't compile too
string_array_ = qi::lit('s[')[qi::_val = construct<string_array>()] >>
-(quoted_string % ',')[phoenix::push_back(phoenix::ref(qi::_val), qi::_1)] >> ']';
quoted_string %= "'" > qi::lexeme[*(qi::char_ - "'")] > "'";
expr_ = number_array_[qi::_val = qi::_1] | string_array_[[qi::_val = qi::_1]];
}
qi::rule<Iterator, number_array(), ascii::space_type> number_array_;
qi::rule<Iterator, string_array(), ascii::space_type> string_array_;
qi::rule<Iterator, std::string(), ascii::space_type> quoted_string;
qi::rule<Iterator, node(), ascii::space_type> expr_;
};
Most important note here is that I think you can just do without all the semantic actions.
They only do what the default attribute rules already do (_val = _1 for scalar attributes, insert(_val, end(_val), _1) for conainer attributes, basically).
This means you can just write the whole shebang as
number_array_ = "n[" >> -(qi::double_ % ',') >> ']';
string_array_ = "s[" >> -(quoted_string % ',') >> ']';
quoted_string = "'" > qi::lexeme[*(qi::char_ - "'")] > "'";
expr_ = number_array_ | string_array_;
And this will work. Note that I fixed the multibyte literals 'n[' and 's[n'.
See also Boost Spirit: "Semantic actions are evil"?
I have 2 type of expressions that I want parse and calculate the results.
Artimetic expressions: +,-,*,/ and sqrt() function;
Ex: "2 + 3 * sqrt(100*25)" -> should be calculated as 152
Functions: GetSubString() and ConcatenateStrings()
Ex: "GetSubString('100str1', 0, 3)" -> should be calculated as 100
I have 2 seperate grammars to parse these expression types. Now I want to combine these 2 grammars and make it possible to define these expressions together.
Ex:
"GetSubString('100str1', 0, 2+1) + sqrt(9)" -> result= 103
"2 + 3 * sqrt(GetSubString('100str1', 0, 2+1))" -> result= 32
I have tried to combine 2 grammars as below by using permutation operator. But it doesnt compile.
expr_ =
( *( (function_call_ ^ arithmeticexpression_)| string_ ));
So is this a right way to combine my function_call_ and arithmeticexpression_ rules or how should I do this?
typedef boost::variant<int, float, double, std::wstring> RetValue;
RetValue CTranslationFunctions::GetSubString(RetValue const& str, RetValue position, RetValue len)
{
std::wstring strToCut;
size_t posInt = 0;
size_t lenInt = 0;
try
{
strToCut = boost::get<std::wstring>(str);
posInt = boost::get<int>(position);
lenInt = boost::get<int>(len);
}
catch (const boost::bad_get&)
{
throw;
}
return strToCut.substr(posInt, lenInt);
}
RetValue CTranslationFunctions::ConcatenateStrings(RetValue const& a, RetValue const& b)
{
wostringstream woss;
woss << a << b;
return woss.str();
}
double CTranslationFunctions::Negate(double num)
{
return -num;
}
double CTranslationFunctions::Add(double num1 , const double num2)
{
return num1 + num2;
};
double CTranslationFunctions::Subtruct(double num1 , double num2)
{
return num1 - num2;
};
double CTranslationFunctions::Multiply(double num1 , double num2)
{
return num1 * num2;
};
double CTranslationFunctions::Divide(double num1 , double num2)
{
return num1 / num2;
};
double CTranslationFunctions::Sqrt(double num)
{
return sqrt(num);
}
class InvalidParamEx{};
double CTranslationFunctions::ConvertStringToDouble(RetValue val)
{
wostringstream wss;
double dNum;
wss << val;
std::wistringstream iss;
iss.str(wss.str());
try
{
iss >> dNum;
}
catch (...)
{
throw InvalidParamEx();
}
return dNum;
}
BOOST_PHOENIX_ADAPT_FUNCTION(RetValue, ConcatenateStrings_, ConcatenateStrings, 2)
BOOST_PHOENIX_ADAPT_FUNCTION(RetValue, GetContainerId_, GetContainerId, 2)
BOOST_PHOENIX_ADAPT_FUNCTION(double, Add_, Add, 2)
BOOST_PHOENIX_ADAPT_FUNCTION(double, Subtruct_, Subtruct, 2)
BOOST_PHOENIX_ADAPT_FUNCTION(double, Multiply_, Multiply, 2)
BOOST_PHOENIX_ADAPT_FUNCTION(double, Divide_, Divide, 2)
BOOST_PHOENIX_ADAPT_FUNCTION(double, Negate_, Negate, 1)
BOOST_PHOENIX_ADAPT_FUNCTION(double, Sqrt_, Sqrt, 1)
BOOST_PHOENIX_ADAPT_FUNCTION(double, ConvertStringToDouble_, ConvertStringToDouble, 1)
// Grammar to parse map functions
template <typename It, typename Skipper = qi::space_type >
struct MapFunctionParser : qi::grammar<It, RetValue(), Skipper, qi::locals<char> >
{
MapFunctionParser() : MapFunctionParser::base_type(expr_)
{
using namespace qi;
function_call_ =
| (lit(L"GetSubString") > '(' > expr_ > ',' > expr_ > ',' > expr_ > ')')
[ _val = GetSubString_(_1, _2, _3) ]
| (lit(L"ConcatenateStrings") > '(' > expr_ > lit(',') > expr_ > ')')
[ _val = ConcatenateStrings_(_1, _2) ];
string_ = as_wstring[omit [ char_("'\"") [_a =_1] ]
>> no_skip [ *(char_ - char_(_a)) ]
>> lit(_a)];
arithmeticexpression_ =
term_ [_val = _1]
>> *( ('+' >> term_ [_val = Add_(_val,_1)])
| ('-' >> term_ [_val = Subtruct_(_val, _1)])
);
term_ =
factor_ [_val = _1]
>> *( ('*' >> factor_ [_val = Multiply_(_val, _1)])
| ('/' >> factor_ [_val = Divide_(_val, _1)])
);
factor_ =
double_ [_val = _1]
string_ [_val = ConvertStringToDouble(_1)]
| ('-' >> factor_ [_val = Negate_(_1)])
| ('+' >> factor_ [_val = _1])
| (L"Sqrt" > '(' > double_ > ')' ) [_val = Sqrt_(_1)]);
expr_ =
( *( (function_call_ ^ arithmeticexpression_)| string_ ));
on_error<fail> ( expr_, std::cout
<< phx::val("Error! Expecting ") << _4 << phx::val(" here: \"")
<< phx::construct<std::string>(_3, _2) << phx::val("\"\n"));
BOOST_SPIRIT_DEBUG_NODE(function_call_);
BOOST_SPIRIT_DEBUG_NODE(expr_);
BOOST_SPIRIT_DEBUG_NODE(string_);
BOOST_SPIRIT_DEBUG_NODE(funcparameter_);
BOOST_SPIRIT_DEBUG_NODE(arithmeticexpression_);
BOOST_SPIRIT_DEBUG_NODE(factor_);
BOOST_SPIRIT_DEBUG_NODE(term_);
}
private:
qi::rule<It, RetValue(), Skipper, qi::locals<char> > function_call_, expr_, funcparameter_;
qi::rule<It, wstring(), Skipper, qi::locals<char> > string_;
qi::rule<It, double(), Skipper> arithmeticexpression_, factor_, term_;
};
Edit Moved my early response to the bottom
BIG UPDATE
That took a while. Mostly it was because the code shown has strange problems:
several rules contain syntax errors (function_call and factor_
there is a reference to GetContainerId and GetSubstring was never Phoenix-adapted
The type CTranslationFunctions didn't exist, and member functions were being declared
however the ADAPT_FUNCTION macros still referenced the member function names as if they were supposed to be in the enclosing namespace (?!)
So what I basically ended up doing was a re-write. Yeah I know. I'm crazy. Nevertheless, let me walk you through it, explaining some of the things I changed and why.
#define BOOST_SPIRIT_USE_PHOENIX_V3
// #define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/phoenix/function/adapt_function.hpp>
#include <boost/lexical_cast.hpp>
namespace qi = boost::spirit::qi;
namespace phx = boost::phoenix;
typedef boost::variant<int, double> NumValue;
typedef boost::variant<int, double, std::wstring> GenericValue;
Right away, I split the concept of numeric and generic values. This is because the distinction is important to certain expressions (mainly the arithmetic expressions). I could have still used GenericValue everywhere, but we'll see later how NumValue makes handling the arithmetic evaluations simpler.
struct InvalidParamEx : public virtual std::exception
{
const char* what() const noexcept { return "Invalid type of operand/parameter"; }
};
There's your exception type, showing some good practices. We throw it when a numeric value was expected, but the GenericValue contained something incompatible. How? Let's see:
struct AsNumValue : boost::static_visitor<NumValue>
{
int operator()(int i) const { return i; }
double operator()(double d) const { return d; }
NumValue operator()(std::wstring const& s) const
{
try { return boost::lexical_cast<int>(s); } catch(...) {}
try { return boost::lexical_cast<double>(s); } catch(...) {}
throw InvalidParamEx();
}
};
class CTranslationFunctions
{
// helper
NumValue static num(GenericValue const& v) { return boost::apply_visitor(AsNumValue(), v); }
There. I defined the missing class for you, and right away added the helper that converts GenericValue → NumValue. As you can see, I used boost::lexical_cast because there is no use in reinventing the wheel. Note that your earlier approach with ConvertStringToDouble had several big problems:
it would always result in a double value, whereas your functions may require int
it would treate '100str1' as the value 100 without warning
it happened at the wrong time: any string a simple term would be converted to double, even if it really was a string. (Why this was relevant, will become clear when you see the modified expr_ and term_ rules.)
Let's move on:
public:
static GenericValue GetSubString(GenericValue const& str, GenericValue position, GenericValue len);
static GenericValue ConcatenateStrings(GenericValue const& a, GenericValue const& b);
Yup, we'll define them later. Now, brace yourself for the arithmetic operation functions:
#define DEFUNOP(name, expr) private: struct do_##name : boost::static_visitor<NumValue> { \
template <typename T1> NumValue operator()(T1 const& a) const { return expr; } \
}; \
public: static NumValue name(GenericValue const& a) { auto na=num(a); return boost::apply_visitor(do_##name(), na); }
#define DEFBINOP(name, infix) struct do_##name : boost::static_visitor<NumValue> { \
template <typename T1, typename T2> NumValue operator()(T1 const&a, T2 const&b) const\
{ return a infix b; } \
}; \
public: static NumValue name(GenericValue const& a, GenericValue const& b) { auto na=num(a), nb=num(b); return boost::apply_visitor(do_##name(), na, nb); }
// define the operators polymorphically, so `int` + `double` becomes `double`, but `int` * `int` stays `int`
DEFBINOP(Add , +);
DEFBINOP(Subtruct, -);
DEFBINOP(Multiply, *);
DEFBINOP(Divide , /);
DEFUNOP (Negate , -a);
DEFUNOP (Sqrt , sqrt(a));
};
Whoaaaaah What happened there? Well, the comment says it all:
You needed to distinguish between int+int vs. double+int etc. This is known as polymorphic evaluation. Example: GetSubString('100str1', 0, 2+1) could never work, because 2+1 needs to evaluate to an int(3), but your double Add(double,double) always produced a double.
I've used MACROs to remove the tedious work from creating a polymorphic function object for each operator
I've let decltype detect the resultant types in mixed cases
Here's where NumValue has merit above GenericValue: because NumValue can only be int or double, we know that the generic operator() implementation covers all legal combinations.
To ensure that all parameters are actually NumValues, they are passed through asNumeric before calling the function object.
This thorughly solves your arithmetic operations, and has another bonus: it removes the 'need' for ConvertStringToDouble, since you get conversion to NumValue when it's needed, namely on evaluation of arithmetic operations. This is an important thing, down the road when we fix your grammar to support your desired input expressions.
If you've come this far, you've seen the rough parts. The rest is plain sailing.
GenericValue CTranslationFunctions::GetSubString(GenericValue const& str, GenericValue position, GenericValue len)
{
using boost::get;
return get<std::wstring>(str).substr(get<int>(position), get<int>(len));
}
Yeah, I shortened it a bit.
GenericValue CTranslationFunctions::ConcatenateStrings(GenericValue const& a, GenericValue const& b)
{
std::wostringstream woss;
woss << a << b;
return woss.str();
}
BOOST_PHOENIX_ADAPT_FUNCTION(GenericValue, ConcatenateStrings_, CTranslationFunctions::ConcatenateStrings, 2)
BOOST_PHOENIX_ADAPT_FUNCTION(GenericValue, GetSubString_ , CTranslationFunctions::GetSubString , 3)
BOOST_PHOENIX_ADAPT_FUNCTION(NumValue , Add_ , CTranslationFunctions::Add , 2)
BOOST_PHOENIX_ADAPT_FUNCTION(NumValue , Subtruct_ , CTranslationFunctions::Subtruct , 2)
BOOST_PHOENIX_ADAPT_FUNCTION(NumValue , Multiply_ , CTranslationFunctions::Multiply , 2)
BOOST_PHOENIX_ADAPT_FUNCTION(NumValue , Divide_ , CTranslationFunctions::Divide , 2)
BOOST_PHOENIX_ADAPT_FUNCTION(NumValue , Negate_ , CTranslationFunctions::Negate , 1)
BOOST_PHOENIX_ADAPT_FUNCTION(NumValue , Sqrt_ , CTranslationFunctions::Sqrt , 1)
Yawn. We know how to adapt functions for Phoenix, already; let's get to the grammar definition!
// Grammar to parse map functions
template <typename It, typename Skipper = qi::space_type >
struct MapFunctionParser : qi::grammar<It, GenericValue(), Skipper>
{
MapFunctionParser() : MapFunctionParser::base_type(expr_)
{
using namespace qi;
function_call_ =
(no_case["GetSubString"] > '(' > expr_ > ',' > expr_ > ',' > expr_ > ')') [ _val = GetSubString_(_1, _2, _3) ]
| (no_case["ConcatenateStrings"] > '(' > expr_ > ',' > expr_ > ')') [ _val = ConcatenateStrings_(_1, _2) ]
| (no_case["Sqrt"] > '(' > expr_ > ')') [ _val = Sqrt_(_1) ]
;
string_ = // keep it simple, silly (KISS)
(L'"' > *~char_('"') > L'"')
| (L"'" > *~char_("'") > L"'");
arithmeticexpression_ =
term_ [ _val = _1 ]
>> *( ('+' >> term_ [ _val = Add_(_val,_1) ])
| ('-' >> term_ [ _val = Subtruct_(_val, _1) ])
);
term_ =
factor_ [ _val = _1 ]
>> *( ('*' >> factor_ [ _val = Multiply_(_val, _1) ])
| ('/' >> factor_ [ _val = Divide_(_val, _1) ])
);
factor_ =
int_ [ _val = _1 ]
| double_ [ _val = _1 ]
| string_ [ _val = _1 ]
| ('-' >> factor_) [ _val = Negate_(_1) ]
| ('+' >> factor_) [ _val = _1 ]
| function_call_ [ _val = _1 ]
;
expr_ = arithmeticexpression_;
on_error<fail> ( expr_, std::cout
<< phx::val("Error! Expecting ") << _4 << phx::val(" here: \"")
<< phx::construct<std::string>(_3, _2) << phx::val("\"\n"));
BOOST_SPIRIT_DEBUG_NODES((function_call_) (expr_) (string_) (funcparameter_) (arithmeticexpression_) (factor_) (term_))
}
private:
qi::rule<It, std::wstring()>
string_; // NO SKIPPER (review)
qi::rule<It, GenericValue(), Skipper>
function_call_, expr_, funcparameter_, // NO LOCALS (review)
arithmeticexpression_, term_, factor_;
};
Well. What have we here. What changed?
I removed qi::locals which was only ever used in the string_ rule anyways, and I rewrote that to honour the KISS principle
I also fixed the problem with whitespace in strings (your parser would have parsed " oops " identical to "oops"). I did so by removing the Skipper from the string_ declaration. This has the same effect as enclosing the whole rule in qi::lexeme[].
I moved Sqrt to the function_call_ rule, because, well, it's a function call.
I tweaked the function names to be no_case[] case insensitive, since your examples suggested that sqrt(9) should work
Note that Sqrt now takes any expression whereas the old situation had
| (L"Sqrt" > '(' > double_ > ')') // Wait, whaaat?
Yeah, this was never going to parse your second example, really :|
Now the real meat of the operation comes. In order to let sqrt(GetSubstring(....)) parse, we'll have to let function_call_ be a possible value for a term_. Once that's the case, we don't have to anything more in expr_ since expr_ might consist of a single factor_ containing a single term_ representing a function_call_ already, so
expr_ = ( *( (function_call_ ^ arithmeticexpression_)| string_ ));
evaporates into
expr_ = arithmeticexpression_;
What happened to string_ there? Well, it's still in term_, where it was, but the ConvertStringToDouble was removed there. Strings will just happily be strings, unless they are required in the context of an arithmetic operation that requires NumValues. That's when they will be coerced into a number, and no earlier (as shown above).
int main()
{
static const MapFunctionParser<std::wstring::const_iterator> p;
std::wstring input;
while (std::getline(std::wcin, input))
{
std::wstring::const_iterator f(begin(input)), l(end(input));
GenericValue value;
assert(qi::phrase_parse(f, l, p, qi::space, value));
if (f!=l)
std::wcout << L"remaining unparsed: '" << std::wstring(f,l) << L"'\n";
std::wcout << input << " --> " << value << std::endl;
}
}
When I fed this little test program the two lines from your question, it dutifully churned out the following text:
GetSubString('100str1', 0, 2+1) + sqrt(9) --> 103
2 + 3 * sqrt(GetSubString('100str1', 0, 2+1)) --> 32
You can see the full code on Coliru (sadly, it takes too long to compile).
Originally this answer started with the following:
Q. I have tried to combine 2 grammars as below by using permutation operator. But it doesnt compile
What did you expect the permutation operator to do? The documentation states:
The permutation operator, a ^ b, matches one or more operands (a, b, ... etc.) in any order...
As you can see it would result in an attribute
boost::variant<
fusion::vector2<optional<RetValue>, optional<double>>,
std::wstring>
which clearly is not compatible. Now, I assume you just want either/or semantics, so
expr_ = string_ | function_call_ | arithmeticexpression_;
should do nicely, resulting in boost::variant<RetValue, double, std::wstring>
which is assignable to a RetValue.
Now after jumping through a dozen hoops to make your sample code compile (why...) here's a fix:
I'm using spirit first time. I'm trying to write a boolean expression (with only &, | and ! operators) parser. I've defined my grammar like following:
template <typename Iterator>
struct boolean_expression_parser : qi::grammar<Iterator, std::string(), ascii::space_type>
{
boolean_expression_parser() : boolean_expression_parser::base_type(expr)
{
using namespace qi;
using ascii::char_;
using boost::spirit::ascii::alnum;
using namespace qi::labels;
using phoenix::construct;
using phoenix::val;
operand %= lexeme[+(alnum)];
simple_expr %= ('(' > expr > ')') | operand;
unary_expr %= ('!' > simple_expr ) ;
and_expr %= ( expr > '*' > expr);
or_expr %= (expr > '|' > expr);
expr %= simple_expr | unary_expr | *and_expr | *or_expr;
// on_error<fail>
// (
// unary_expr,
// std::cout
// << val("Error! Expecting ")
// << _4 // what failed?
// << val(" here: \"")
// << construct<std::string>(_3, _2) // iterators to error-pos, end
// << val("\"")
// << std::endl
// );
}
qi::rule<Iterator, std::string(), ascii::space_type> operand;
qi::rule<Iterator, std::string(), ascii::space_type> simple_expr;
qi::rule<Iterator, std::string(), ascii::space_type> unary_expr;
qi::rule<Iterator, std::string(), ascii::space_type> and_expr;
qi::rule<Iterator, std::string(), ascii::space_type> or_expr;
qi::rule<Iterator, std::string(), ascii::space_type> expr;
};
I'm facing few hurdles here:
It's not working for any binary expression (like 'A + B'). It's working fine for unary expressions (like '!(A)' or '(!A)'.
Can someone point me what I'm doing wrong?
I want store it in tree form (as I want to build a BDD out of it). Can someone point me how to do that?
Also, why on_error<> is not working even when I enable it?
I'm using boost 1.49 and gcc-4.2.2.
Regards,
~ Soumen
There are quite a lot problems with your parser. First of all, you meet a left-side recursion here, so the parser will crash with Stack Overflow. Your grammar should look like this:
expr = or_expr;
or_expr = and_expr >> -('|' > expr);
and_expr = unary_expr >> -('*' > expr);
unary_expr = ('!' > expr) | operand | ('(' >> expr > ')');
In this case you don't have left-recursion and everything parses.
Why your approach was failing? In your case:
input: A * B
1: expr
1.1: check simple_expr
-> fail at '(', try next
-> operand matches, return from simple_expr
matched, return.
So it should parse only A, and return without fail, but with input not fully parsed.
Also, the operator > you overused. Its purpose is to fail parsing if there is no match after it. On the other hand the operator >> returns and lets the parser check other possibilities.