Using semantic action together with attribute propagation in spirit - c++

I played a little with code at link and I have another question. I added semantic action to:
action = actions_ >> '(' >> parameters >> ')'[ /* semantic action placed here */];
so I can reuse the rule together with verification at multiple places. The problem is that then the spirit stops propagate my attribute type to the upper rules (which uses the action as parser). I read at link that operator %= should be used to enable it again (to have semantic actions and attribute propagation). But then I am getting compiler error that it is not possible to convert boost::fuction::vector2<ast::actionid, ast::parameters> to ast::action. Is there any macro in fusion to enable assignment in another direction? Or what should I do that the rule still exposes the same attribute as it is passed to the semantic action instead of having fusion vector there?
Sample code:
#include "stdafx.h"
// boost
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/repository/include/qi_iter_pos.hpp>
#include <boost/bind.hpp>
#include <boost/phoenix.hpp>
// std
#include <string>
#include <vector>
namespace bsqi = boost::spirit::qi;
namespace bsqi_coding = boost::spirit::standard_wide;
namespace bsqi_repos = boost::spirit::repository::qi;
//////////////////////////////////////////////////////////////////////////
enum class Action : uint8_t
{
eAction0 = 0,
eAction1,
eAction2
};
//////////////////////////////////////////////////////////////////////////
struct ActionSymbols : public boost::spirit::qi::symbols<wchar_t, Action>
{
ActionSymbols()
{
add
(L"action0", Action::eAction0)
(L"action1", Action::eAction1)
(L"action2", Action::eAction2)
;
}
} actionSymbols;
//////////////////////////////////////////////////////////////////////////
using ParameterValue = boost::variant<int, std::wstring>;
struct Parameter
{
std::wstring::const_iterator source; ///< position within the input where parameter begins
ParameterValue value; ///< type and value of the parameter
};
//////////////////////////////////////////////////////////////////////////
using Parameters = std::vector<Parameter>;
//////////////////////////////////////////////////////////////////////////
struct ActionParameters
{
Action action;
Parameters parameters;
};
//////////////////////////////////////////////////////////////////////////
BOOST_FUSION_ADAPT_STRUCT(Parameter, (std::wstring::const_iterator, source), (ParameterValue, value));
BOOST_FUSION_ADAPT_STRUCT(ActionParameters, (Action, action), (Parameters, parameters));
//////////////////////////////////////////////////////////////////////////
class SyntaxError : public std::runtime_error
{
public:
SyntaxError()
: std::runtime_error("Syntax error!")
{ }
};
//////////////////////////////////////////////////////////////////////////
template<typename IteratorT>
struct ScriptGrammar : bsqi::grammar<IteratorT, std::vector<ActionParameters>, bsqi_coding::space_type>
{
/// helper type to define all rules
template<typename T>
using RuleT = bsqi::rule<iterator_type, T, bsqi_coding::space_type>;
using result_type = std::vector<ActionParameters>;
explicit ScriptGrammar()
: base_type(start, "script")
{
// supported parameter types (int or quoted strings)
// note: iter_pos is used for saving the iterator for the parameter to enable generating more detailed error reports
parameter = bsqi_repos::iter_pos >> (bsqi::int_ | bsqi::lexeme[L'"' > *(bsqi_coding::char_ - L'"') > L'"']);
parameter.name("parameter");
// comma separator list of parameters (or no parameters)
parameters = -(parameter % L',');
parameters.name("parameters");
// action with parameters
action = (actionSymbols > L'(' > parameters > L')')[bsqi::_pass = boost::phoenix::bind(&ScriptGrammar::ValidateAction, this, bsqi::_1, bsqi::_2)];
action.name("action");
// action(..) [-> event(..) -> event(..) -> ..]
// eps = force to use this rule for parsing
// eoi = the rule must consume whole input
start = bsqi::eps > (action % L';') > L';' > bsqi::eoi;
}
private:
bool ValidateAction(Action action, const Parameters& parameters)
{
return true;
}
RuleT<Parameter> parameter;
RuleT<Parameters> parameters;
RuleT<ActionParameters> action;
RuleT<std::vector<ActionParameters>> start;
};
//////////////////////////////////////////////////////////////////////////
int _tmain(int argc, _TCHAR* argv[])
{
using ScriptParser = ScriptGrammar<std::wstring::const_iterator>;
ScriptParser parser;
auto input = std::wstring(L"\taction0(1, 2, 3); action1(\"str1\", \"str2\"); action2(\"strmix\", 0);\t\t");
auto it = input.begin();
ScriptParser::result_type output;
try
{
if(!phrase_parse(it, input.end(), parser, bsqi_coding::space, output))
throw SyntaxError();
}
catch(bsqi::expectation_failure<ScriptParser::iterator_type>& e)
{
std::cout << "Error! Expecting " << e.what_ << " here: \"" << std::string(e.first, e.last) << "\"";
}
catch(SyntaxError& e)
{
std::cout << e.what() << "\n";
}
return 0;
}
I try to get an attribute from start rule. I get correctly parsed values in my semantic action (ValidateAction) but the attribute from start rule receive only uninitialized values (size of the output std::vector is 3 but values are uninitialized). I tried to replace the initialization of the rules with %= instead of simple = but then the mentioned compilation error pops.

There is BOOST_SPIRIT_ACTIONS_ALLOW_ATTR_COMPAT which is supposed to allow attribute compatibility rules to work inside semantic actions like they work during automation attribute propagation.
However, the superior solution is to specify the conversions you wish, when you wish them.
The most obvious approaches are
wrap the intermediate into a qi::rule<..., T()>
Incidentally, I already solved your particular issue that way here boost spirit reporting semantic error in your previous question.
Actually, I suppose you would like to have a stateful validator working on the fly, and you can use Attribute Traits to transform your intermediates to the desired AST (e.g. if you don't want to actually store the iterators in your AST)
wrap the sub-expression in a qi::transform_attribute<T>()[p] directive.
Beware of a bug in some versions of Boost Spirit that requires you to explicitly deep-copy the subexpression in transform_attribute (use qi::copy(p))

Related

x3 modify parser at runtime

I wonder if it is possible to change the parser at runtime given it does not change the compound attribute.
Lets say I want to be able to modify at runtime the character of my parser that detects whether I have to join a line from ; to ~. Both are just characters and since the c++ types and the template instantiations dont vary (in both cases we are talking about a char) I think there must be some way, but I dont find it. So is this possible?
My concrete situation is that I am calling the X3 parser via C++/CLI and have the need that the character shall be adjustable from .NET. I hope the following example is enough to be able to understand my problem.
http://coliru.stacked-crooked.com/a/1cc2f2836dbfaa46
Kind regards
You cannot change the parser at runtime (except a DSO trick I described under your other question https://stackoverflow.com/a/56135824/3621421), but you can make your parser context-sensitive via semantic actions and/or stateful parsers (like x3::symbols).
The state for semantic actions (or probably for your custom parser) can also be stored in a parser context. However, usually I see that folks use global or function local variables for this purpose.
A simple example:
#include <boost/spirit/home/x3.hpp>
#include <iostream>
namespace x3 = boost::spirit::x3;
int main()
{
char const* s = "sep=,\n1,2,3", * e = s + std::strlen(s);
auto p = "sep=" >> x3::with<struct sep_tag, char>('\0')[
x3::char_[([](auto& ctx) { x3::get<struct sep_tag>(ctx) = _attr(ctx); })] >> x3::eol
>> x3::int_ % x3::char_[([](auto& ctx) { _pass(ctx) = x3::get<struct sep_tag>(ctx) == _attr(ctx); })]
];
if (parse(s, e, p) && s == e)
std::cout << "OK\n";
else
std::cout << "Failed\n";
}

Boost spirit changing variable value in semantic action

I want to change a local variable value in semantic action, like following:
#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <string>
namespace qi = boost::spirit::qi;
namespace spirit = boost::spirit;
namespace ascii = boost::spirit::ascii;
using boost::phoenix::ref;
using boost::phoenix::bind;
void dummy(const std::vector<char>& v, int& var)
{
var = 7;
}
template <typename Iterator>
struct x_grammar : public qi::grammar<Iterator, std::string(), ascii::space_type>
{
public:
x_grammar() : x_grammar::base_type(start_rule, "x_grammar")
{
using namespace qi;
int local_var = 0;
start_rule = (+(char_ - ";"))[bind(dummy, _1, ref(local_var))];
//repeat(ref(local_var))[some_rule];
}
private:
qi::rule<Iterator, std::string(), ascii::space_type> start_rule;
};
int main()
{
typedef std::string::const_iterator iter;
std::string storage("string;aaa");
iter it_begin(storage.begin());
iter it_end(storage.end());
std::string read_data;
using boost::spirit::ascii::space;
x_grammar<iter> g;
try {
bool r = qi::phrase_parse(it_begin, it_end, g, space, read_data);
std::cout << "Pass!\n";
} catch (const qi::expectation_failure<iter>& x) {
std::cout << "Error!\n";
}
}
I am getting some annoying compile errors using GCC 4.6.1 with boost 1.55.
I can't help but note that if compiler errors annoy you, then perhaps you should write valid code :/
Instructive Hat On...
While that's of course a flippant remark, it's also somewhat enlightening.
I've told you twice now that the whole idea of using constructor local variables in your grammar is fundamentally broken:
Boost spirit semantic action not invoked
Boost spirit using local variables
What you want is
inherited attributes
qi::locals
maayyyyybe, maaaayyybe grammar member variables; with the caveat that they make your rules non-re-entrant.
The important thing here to really get inside your head is
Boost Spirit generates parser from expression templates. Expression templates are 90% static information (type only), and get "compiled" (.compile()) into "invokable" (.parse()) form.
Most importantly, while you can write control flow in your semantic actions, none of this actually executed at the definition site. It's "compiled" into a lazy actor that can later be invoked.
The generated parse will conditionally invoke the lazy actor when the corresponding parse expression matches
Constructive Hat On...
It looks like you just want to transform attributes using a function.
Here's what you can do:
transform as part of the semantic action, placing the result into the regular attribute (maintaining 'functional' semantics for parser composition):
qi::rule<Iterator, exposed(), Skipper> myrule;
myrule = int_ [ _val = phx::bind(custom_xform, _1) ];
Where custom_xform is any old-school calleable (including polymorphic ones):
exposed custom_xform(int i) { return make_new_exposed(i); }
// or
struct custom_xfrom_t {
template <typename> struct result { typedef exposed type; };
template <typename Int>
exposed operator()(Int i) const {
return make_new_exposed(i);
}
};
static const custom_xform_t custom_xform;
You can add some syntactic sugar [1]
qi::rule<Iterator, exposed(), Skipper> myrule;
myrule = int_ [ _val = custom_xform(_1) ];
This requires custom_xform is defined as a lazy actor:
phx::function<custom_xform_t> custom_xform; // `custom_xform_t` again the (polymorphic) functor
You may note this wouldn't work for a regular function. You could wrap it in a calleable object, or use the BOOST_PHOENIX_ADAPT_FUNCTION macro to do just that for you
If you have some more involved transformations that you want to apply more often, consider using the Spirit Customization Points:
Customization of Spirit's Attribute Handling, specifically:
Transform an Attribute to a Different Type
Store a Parsed Attribute Value
These work most smoothly if you choose specific types for your attributes (e.g. Ast::Multiplicity or Ast::VelocityRanking, instead of int or double
[1] using BOOST_SPIRIT_USE_PHOENIX_V3
The code compiles with C++03. However, when using GCC 4.6's C++11 support, the code fails to compile. Here are the relevant excerpts from the error:
/usr/local/include/boost/spirit/home/support/action_dispatch.hpp: In static
member function 'static void boost::spirit::traits::action_dispatch<
Component>::caller(F&&, A&& ...) [with F =
const std::_Bind<with Boost.Phoenix actors>]'
...
main.cpp:25:9: instantiated from 'x_grammar<Iterator>::x_grammar() [...]
/usr/local/include/boost/spirit/home/support/action_dispatch.hpp:142:13: error:
no matching function for call to 'boost::spirit::traits::
action_dispatch<...>::do_call(const std::_Bind<with Boost.Phoenix actors>)'
Despite the using boost::phoenix::bind directive, the unqualified call to bind() is resolving to std::bind() rather than boost::phoenix::bind(), but the arguments are resolving to Boost.Phoenix actors. The Boost.Spirit documentation specifically warns about not mixing placeholders from different libraries:
You have to make sure not to mix placeholders with a library they don't belong to and not to use different libraries while writing a semantic action.
Hence, the compilation problem can be resolved by being explicit when defining the semantic action. Use either:
std::bind(dummy, std::placeholders::_1, std::ref(local_var))
or:
boost::phoenix::bind(dummy, _1, ref(local_var))
While that resolves the compiler error, it is worth noting that the ref(local_var) object will maintain a dangling reference, as its lifetime extends beyond that of local_var. Here is a working example where local_var's lifetime is extend to beyond the scope of the constructor by making it static.

How to tel a boost::karma::rule not to consume its attribute without providing a valid generator?

Say we have the following source code:
#include <iostream>
#include <string>
#include <iterator>
#include <boost/spirit/include/karma.hpp>
namespace karma = boost::spirit::karma;
template <typename OutputIterator> struct grammar : karma::grammar<OutputIterator, std::nullptr_t()> {
grammar() : grammar::base_type(query) {
query = "yeah";
}
karma::rule<OutputIterator, std::nullptr_t()> query;
};
int main(void) {
typedef std::back_insert_iterator<std::string> iterator_type;
std::string generated;
iterator_type output_it(generated);
//keys_and_values<sink_type> g;
grammar<iterator_type> g;
bool result = karma::generate(output_it, g, nullptr);
std::cout << result << ":" << generated << std::endl;
return 0;
}
This fails to compile because karma lacks some traits for std::nullptr_t (those are boost::spirit::traits::extract_c_string and boost::spirit::traits::char traits). More specifically, it fails because karma is not able to find a generator for an attribute of type std::nullptr_t.
I see several ways to cope with that:
Replace std::nullptr_t by karma::unused_type in the grammar definition : It works on this example but may introduce ambiguity in a more complex grammar.
Defining the traits specialization : In my opinion, this is dirty and not generic. Plus, it exposes my specialization of a standard type for everyone, leading to potential conflicts.
Specializing an attribute transform : Same problem of specializing a standard type just for me.
Write a custom generator : The best candidate so far, but it makes a serious load of highly templated code lines compared to the task complexity.
Put a intermediate rule with a karma::unused_type attribute. A quick fix that works but have no sense.
Question : How can I tell the karma::rule to generate a simple literal and not care about having or not a generator for its attribute ?
You seem to have stumbled on the inverse of the infamous single-element fusion sequence conundrum[1] :(
I noticed, because the error emanates from the code trying to verify that the input string matches the attribute (lit.hpp):
// fail if attribute isn't matched by immediate literal
typedef typename attribute<Context>::type attribute_type;
typedef typename spirit::result_of::extract_from<attribute_type, Attribute>::type
extracted_string_type;
using spirit::traits::get_c_string;
if (!detail::string_compare(
get_c_string(
traits::extract_from<attribute_type>(attr, context))
, get_c_string(str_), char_encoding(), Tag()))
{
return false;
}
However, that makes no sense at all, since the docs state:
lit, like string, also emits a string of characters. The main difference is that lit does not consumes [sic] an attribute. A plain string like "hello" or a std::basic_string is equivalent to a lit
So I just... on a whim thought to coerce things a little, by using the same workaround that works for single-element fusion sequences on the Qi side:
query = karma::eps << "yeah";
And, voilĂ : it works: Live On Coliru
[1] See
How do I use a class with only one attribute in a AST with Boost Spirit?
Spirit Qi attribute propagation issue with single-member struct
Compiler error when adapting struct with BOOST_FUSION_ADAPT_STRUCT
Adapt class containing a string member as synthesized attribute
Etc. This is a sad flaw that will probably need to be worked around for SpiritV2.
Possible answer : After posting this, I found a solution which satisfies me for the moment. That is : introduce a intermediate rule.
template <typename OutputIterator> struct grammar : karma::grammar<OutputIterator, std::nullptr_t()> {
grammar() : grammar::base_type(query) {
query = null_rule;
null_rule = "null";
}
karma::rule<OutputIterator, std::nullptr_t()> query;
karma::rule<OutputIterator, karma::unused_type()> null_rule;
};
I'm still interested in any comment, reproach or other solution.

C++ Scope of Static Variable In a Static Member Function

I have a simple object which does some parsing. Inside, there is a parse function, containing a static variable that is used to limit number of error messages to print to the user:
struct CMYParsePrimitive {
static bool Parse(const std::string &s_line)
{
// do the parsing
static bool b_warned = false;
if(!b_warned) {
b_warned = true;
std::cerr << "error: token XYZ is deprecated" << std::endl;
}
// print a warning about using this token (only once)
return true;
}
};
Now these parse primitives are passed in a typelist to a parser specialization. There is some other interface which tells parser which token types should be parsed using which parse primitives.
My issue is that the warning should be displayed up to once per application run. But in my case, it is sometimes displayed multiple times, seems to be per parser instance rather than application instance.
I'm using Visual Studio 2008, I imagine this might be some bug or a deviation from the standard? Does anyone have any idea as to why this happens?
I failed to notice that the function is also a template. My bad. It is instantiated twice in the code with different parameters - hence the warning is sometimes printed twice. The real code looks more like this:
struct CMYParsePrimitive {
template <class CSink>
static bool Parse(const std::string &s_line, CSink &sink)
{
// do the parsing, results passed down to "sink"
static bool b_warned = false;
if(!b_warned) {
b_warned = true;
std::cerr << "error: token XYZ is deprecated" << std::endl;
}
// print a warning about using this token (only once)
return true;
}
};
So then there are e.g. CMYParsePrimitive::Parse<PreviewParser>::b_warned, which can print the warning once when used by PreviewParser, and then also CMYParsePrimitive::Parse<Parser>::b_warned which can print the warning when used by Parser.

Boost property_tree: multiple values per key

Boost property tree seems like an excellent library to use for parsing config files. However, I can't figure out how to handle situations where there are multiple values per key. For example, let's say I was specifying a box like this:
box
{
x -1 1
y -1 1
z -1 1
}
where x, y, and z are the bounds of the box on the x, y, and z axes respectively, specified using property_tree's INFO format. I see mention in the manual of using quotes for values that use spaces, but then I don't see that I could import those values as numbers. I'd have to parse the string into numbers, which seems to defeat the purpose of using property_tree in the first place. I could certainly give each number a key:
box
{
xlo -1
xhi 1
ylo -1
yhi 1
zlo -1
zhi 1
}
but that seems cumbersome, and will inflate my config file. I also noted that I could handle this situation in program_options, but I lose the nested config file capabilities (yeah, I know I can use dot notation to "nest", but it's not the same).
Is there a way to import e.g. x as a list of numbers like this?
The standard property_tree handles only one string value per key since it is defined as:
typedef basic_ptree<std::string, std::string> ptree;
So, the only option is to use strings and parse them. I think the best method is to define a new class that stores the low and high values and then create a translator class for the get and set methods. For example:
struct low_high_value
{
low_high_value() : m_low(0), m_high(0) { }
low_high_value(double low, double high) : m_low(low), m_high(high) { }
double m_low;
double m_high;
};
The translator would be:
struct low_high_value_translator
{
typedef std::string internal_type;
typedef low_high_value external_type;
// Get a low_high_value from a string
boost::optional<external_type> get_value(const internal_type& str)
{
if (!str.empty())
{
low_high_value val;
std::stringstream s(str);
s >> val.m_high >> val.m_low;
return boost::optional<external_type>(val);
}
else
return boost::optional<external_type>(boost::none);
}
// Create a string from a low_high_value
boost::optional<internal_type> put_value(const external_type& b)
{
std::stringstream ss;
ss << b.m_low << " " << b.m_high;
return boost::optional<internal_type>(ss.str());
}
};
The previous get_value method is very simple. It should be improved if the file could be written by the user.
This class should be registered using:
namespace boost { namespace property_tree
{
template<typename Ch, typename Traits, typename Alloc>
struct translator_between<std::basic_string< Ch, Traits, Alloc >, low_high_value>
{
typedef low_high_value_translator type;
};
} }
After you include the previous code, you can use property_tree as:
pt.get<low_high_value>("box.x")
pt.put("box.u", low_high_value(-110, 200));