I have a problem similar to that described here: C++ Mutually Recursive Variant Type
I am trying to create a JSON representation in C++. Many libraries already offer excellent JSON representations and parsers that are very fast, but I am not reinventing this wheel. I need to create a C++ JSON representation that supports certain space optimizations under specific conditions. In short, if and only if a JSON array contains homogenous data, rather than storing every element as bloated variant types, I need compact storage of native types. I also need to support heterogeneous arrays and standard nested JSON objects.
The following is the "if wishes were horses, beggars would ride" version of the code, which is meant to clearly illustrate intent, but is obviously broken because types are used before any declaration exists. I want to avoid specifying the same information multiple times in types (i.e. Array, Object, and Value should not require duplicated type specifications). I also want to avoid any unnecessarily high run-time costs.
#include <string>
#include <unordered_map>
#include <vector>
#include <boost/variant.hpp>
#include <boost/variant/variant.hpp>
#include <boost/variant/recursive_wrapper.hpp>
class JSONDocument {
public:
using String = std::string;
using Integer = long;
using Float = double;
using Boolean = bool;
using Null = void *;
using Key = std::string;
using Path = std::string;
using Value = boost::variant<
Null,
String,
Integer,
Float,
Boolean,
Object,
Array
>;
using Object = std::unordered_map<Key,Value>;
using Array = boost::variant<
std::vector<Null>,
std::vector<String>,
std::vector<Integer>,
std::vector<Float>,
std::vector<Boolean>,
std::vector<Value> >;
private:
Value root;
class value_traversal_visitor : public boost::static_visitor<Value> {
public:
value_traversal_visitor( Path path ) : path(path) {}
Value operator()( Null x ) const {
if( path.empty() ) {
return x;
}
// otherwise throw ...
}
Value operator()( String x ) const {
if( path.empty() ) {
return x;
}
}
...
// special handling for Array and Object types
private:
Path path;
};
public:
Value get( Path path ) {
return boost::apply_visitor( value_traversal_visitor( path ), root );
}
...
};
As you can see, I am including the recursive_wrapper header. I have tried various invocations of boost::make_recursive_variant and boost::recursive_wrapper, but I always get compiler errors. I do not see how the answer from C++ Mutually Recursive Variant Type solves this, because in every attempt, I get compiler errors (from both gcc++ 5.3 and LLVM/clang++ 3.8) that almost exclusively reference Boost that essentially boil down to types not being convertible or declarations either conflicting or not existing. I would put one of my attempts along with specific compiler error messages here, but I wouldn't know which of the many attempts to use.
I'm hoping somebody can set me on the right path...
Thanks in advance!
Edit
Just to build on the accepted answer below, here is an example of a working skeleton for the types and their usages.
#include <string>
#include <unordered_map>
#include <vector>
#include <boost/variant.hpp>
#include <boost/variant/variant.hpp>
#include <boost/variant/recursive_wrapper.hpp>
using String = std::string;
using Integer = long;
using Float = double;
using Boolean = bool;
using Key = std::string;
using Value = boost::make_recursive_variant<
String,
Integer,
Float,
Boolean,
std::unordered_map<Key, boost::recursive_variant_>,
boost::variant<std::vector<String>,std::vector<Integer>,std::vector<Float>,std::vector<Boolean>,std::vector<boost::recursive_variant_> >
>::type;
using Object = std::unordered_map<Key, Value>;
using Array = boost::variant<std::vector<String>,std::vector<Integer>,std::vector<Float>,std::vector<Boolean>,std::vector<Value> >;
int main( int argc, char* argv[] ) {
Value v;
v = static_cast<Integer>( 7 );
Object o;
v = o;
Array a = std::vector<Integer>( 3 );
v = a;
return 0;
}
You could just use recursive_variant_ placeholder with make_recursive_variant.
Here's the gist:
using Value = boost::make_recursive_variant<
Null,
String,
Integer,
Float,
Boolean,
std::unordered_map<Key, boost::recursive_variant_>, // Object
std::vector<boost::recursive_variant_> // Array
>::type;
using Object = std::unordered_map<Key, Value>;
using Array = boost::variant<Value>;
Live Demo
Live On Coliru
As you can see there's unimplemented bits in the code (never write functions missing return statements!). Also note the simplifications in control flow for get and the private visitor implementation.
#include <boost/variant.hpp>
#include <boost/variant/recursive_wrapper.hpp>
#include <boost/variant/variant.hpp>
#include <string>
#include <unordered_map>
#include <vector>
class JSONDocument {
public:
struct Null { constexpr bool operator==(Null) const { return true; } };
using String = std::string;
using Integer = long;
using Float = double;
using Boolean = bool;
using Key = std::string;
using Path = std::string;
using Value = boost::make_recursive_variant<
Null,
String,
Integer,
Float,
Boolean,
std::unordered_map<Key, boost::recursive_variant_>, // Object
std::vector<boost::recursive_variant_> // Array
>::type;
using Object = std::unordered_map<Key, Value>;
using Array = boost::variant<Value>;
private:
Value root;
struct value_traversal_visitor {
Path path;
using result_type = Value;
result_type operator()(Value const &x) const {
if (path.empty()) {
return x;
}
return boost::apply_visitor(*this, x);
}
result_type operator()(Null) const { throw std::invalid_argument("null not addressable"); }
result_type operator()(String const &) const { throw std::invalid_argument("string not addressable"); }
// special handling for Array and Object types TODO
template <typename T> result_type operator()(T &&) const { return Null{}; }
};
public:
Value get(Path path) { return value_traversal_visitor{path}(root); }
};
int main() {}
CAVEATS
Note that you should NOT use void* for Null because all manner of unwanted implicit conversions
Note that you should probably not use unordered_map because
some JSON implementations allow duplicate property names
some JSON applications depend on the ordering of the properties
See also https://github.com/sehe/spirit-v2-json/blob/master/json.hpp#L37
Not a solution per se, but Here's a way to achieve variant recursivity using std::variant. I thought this might be of interest, since the stl doesn't provide any api for recursive, nor forward-declared types. Compiles using gcc 7.2 -std=c++17
#include <variant>
#include <vector>
#include <iostream>
#include <algorithm>
using namespace std;
struct Nil {};
struct vector1;
using var_t1 = variant<Nil, int, vector1>;
using var_t2 = variant<Nil, double, float, int, var_t1>;
struct vector1 {
vector<var_t2> v_;
};
struct print_var_t2;
struct print_var_t1 {
void operator()(const vector1& v);
void operator()(int) { cout << "int\n"; }
void operator()(const Nil&) { cout << "nil\n"; }
};
struct print_var_t2 {
void operator()(const Nil&) { cout << "Nil\n"; }
void operator()(int) { cout << "int\n"; }
void operator()(double) { cout << "double\n"; }
void operator()(float) { cout << "float\n"; }
void operator()(const var_t1& v);
};
void print_var_t1::operator()(const vector1& v) {
for_each(v.v_.begin(), v.v_.end(), [](const var_t2& x)
{
visit(print_var_t2{}, x);
});
}
void print_var_t2::operator()(const var_t1& v) {
visit(print_var_t1{}, v);
}
int main()
{
vector1 v1;
v1.v_.push_back(.1);
v1.v_.push_back(2.f);
v1.v_.push_back(3);
v1.v_.push_back(var_t2{3});
var_t1 var1 = v1;
std::visit(print_var_t1{}, var1);
return 0;
}
Related
It is one of the first times I am using boost and I am getting an error saying
BaseKey boost::bimaps::container_adaptor::detail::key_to_base_identity<BaseKey,KeyType>::operator ()(Key &) const': cannot convert argument 1 from 'const CompatibleKey' to 'Key &
and
boost::multi_index::detail::ordered_index_impl<KeyFromValue,Compare,SuperMeta,TagList,Category, AugmentPolicy>::find': no matching overloaded function found
I know most of the STL errors or at least where could they come from, but I am not experienced enough with boost to know what could be going on here. The code I have is the following, it is used to convert the values from an enum to strings and vice versa.
file.h
namespace FOO_NS::BAR_NS
{
class FooClass
{
public:
enum class Enum
{
Enum1, Enum2, Enum3, Enum4
};
...
};
namespace
{
using results_bimap = boost::bimap<FooClass::Enum, std::string>;
using position = results_bimap::value_type;
const auto EnumsAsStrings = []() {
results_bimap result;
result.insert(position(FooClass::Enum::Enum1, "Enum1"));
result.insert(position(FooClass::Enum::Enum2, "Enum2"));
result.insert(position(FooClass::Enum::Enum3, "Enum3"));
result.insert(position(FooClass::Enum::Enum4, "Enum4"));
return result;
};
} // namespace
}//namespace FOO_NS::BAR_NS
file.cpp
using namespace FOO_NS::BAR_NS;
void doSmth()
{
...
std::string enumString = EnumsAsStrings().left.at(FooClass::Enum::Enum1); // Expected string "Enum1"
}
Do you see any misconception or misusage I have in this code so that this mentioned error happens?
You don't show enough code. Here's
assuming enum Enum{...}: http://coliru.stacked-crooked.com/a/20455e28883f93be
assuming enum class Enum{...}:
All I can think of is that you might have FooClass defined in an anonymous namespace as well, and you actually have disparate declarations of the enum which are not equivalent to the compiler.
Note that if this kind of setup would be the goal, you should be able to leverage the CompatibleKey overload by using a transparent comparator instead of the default (e.g. less<void> instead of less<FooClass::enum>).
Listing
Anti-bitrot:
#include <boost/bimap.hpp>
struct FooClass{
enum class Enum { Enum1, Enum2, Enum3, Enum4 };
};
namespace {
using results_bimap = boost::bimap<FooClass::Enum, std::string>;
using position = results_bimap::value_type;
auto const EnumsAsStrings = []() {
results_bimap result;
result.insert(position(FooClass::Enum::Enum1, "Enum1"));
result.insert(position(FooClass::Enum::Enum2, "Enum2"));
result.insert(position(FooClass::Enum::Enum3, "Enum3"));
result.insert(position(FooClass::Enum::Enum4, "Enum4"));
return result;
};
} // namespace
int main() {
std::string enumString =
EnumsAsStrings().left.at(FooClass::Enum::Enum1);
assert(enumString == "Enum1");
}
The following MCVE works, so it looks like you're not providing all the relevant information as to what your problem is:
Live Coliru Demo
#include <boost/bimap.hpp>
#include <string>
struct FooClass
{
enum Enum{Enum1,Enum2,Enum3,Enum4};
};
using results_bimap = boost::bimap<FooClass::Enum, std::string>;
using position = results_bimap::value_type;
const auto EnumsAsStrings = []() {
results_bimap result;
result.insert(position(FooClass::Enum1, "Enum1"));
result.insert(position(FooClass::Enum2, "Enum2"));
result.insert(position(FooClass::Enum3, "Enum3"));
result.insert(position(FooClass::Enum4, "Enum4"));
return result;
};
int main()
{
std::string enumString = EnumsAsStrings().left.at(FooClass::Enum1);
}
Okay, so at the end it wasn't anything related to this code (directly). I was calling the lambda like this EnumsAsStrings().left.at(FooClass::Enum1) and it couldn't implicitly convert from FooClass to FooClass::Enum and that was creating the errors. Thank you to everyone who tried to answer my question!
Does there exist a Boost Hana method for compile-time converting the types of members of a Struct concept to a STL container of std::string's of the typenames?
For example,
MyType t();
std::array<std::string, 3> ls = boost::hana::typesToString(t);
for(std::string x : ls){
std::cout << x << std::endl;
}
Yields "int string bool" to STDOUT,
With
class MyType{
int x;
std::string y;
bool z;
}
The documentation clearly provides methods for getting the members and their values of an instance of a Struct concept, but I haven't found anything there that does this for the types of the members. A simpler task would be to do:
int x;
std::string tName = boost::hana::typeId(x); //tName has value "int"
I've read this post but I'd like to know if there's a clean way out-of-the-box in Hana. Even better would be a way to iterate through the members of the Struct without having to know them by name.
If you are using Clang, Hana has an experimental feature hana::experimental::type_name. This can be used to get the type-names of the members of the struct:
#include <boost/hana.hpp>
#include <boost/hana/experimental/type_name.hpp>
namespace hana = boost::hana;
template <typename Struct>
auto member_type_names() {
constexpr auto accessors = hana::accessors<Struct>();
return hana::transform(
accessors,
hana::compose(
[](auto get) {
using member_type
= std::decay_t<decltype(get(std::declval<Struct>()))>;
return hana::experimental::type_name<member_type>();
},
hana::second
)
);
}
Demo (live on Wandbox):
#include <iostream>
#include <string>
struct MyType {
int a;
std::string b;
float c;
};
BOOST_HANA_ADAPT_STRUCT(MyType, a, b, c);
int main() {
hana::for_each(member_type_names<MyType>(), [](auto name) {
// Note that the type of `name` is a hana::string, not a std::string
std::cout << name.c_str() << '\n';
});
}
Outputs:
int
std::__1::basic_string<char>
float
I have a map defined and used like this
// def.h
struct X{};
struct Y{};
struct myStruct
{
X x;
Y y;
};
typedef std::unordered_map<std::pair<std::string, std::string>, myStruct> myMap;
namespace std
{
template<> struct pair<std::string, std::string>
{
std::string s1,s2;
pair(const std::string& a, const std::string& b):s1(a),s2(b){}
bool operator < (const pair<std::string,std::string>& r)
{
return (0 < r.s1.compare(s1) && (0 < r.s2.compare(s2)));
}
};
}
//use.cpp
class CUse
{
myMap m;
public:
CUse():m(0){}
};
Some errors emitted by the compiler are extracted as below
At the constructor CUse initialization,
note: see reference to function template instantiation
'std::unordered_map,myStruct,std::hash<_Kty>,std::equal_to<_Kty>,std::allocator>>::unordered_map(unsigned __int64)' being compiled
At the declaration of m in CUse
note: see reference to class template instantiation
'std::unordered_map,myStruct,std::hash<_Kty>,std::equal_to<_Kty>,std::allocator>>' being compiled
As #Bo Persson and #Sean Cline mentioned in the comments, you will need to use a custom hash function/functor to do that.
LIVE DEMO
#include <unordered_map>
#include <string>
#include <tuple>
#include <functional>
#include <cstddef>
#include <iostream>
struct myStruct { int x, y; };
using Key = std::pair<std::string, std::string>;
namespace something
{
struct Compare //custom hash function/functor
{
std::size_t operator()(const Key& string_pair) const
{
// just to demonstrate the comparison.
return std::hash<std::string>{}(string_pair.first) ^
std::hash<std::string>{}(string_pair.second);
}
};
}
using myMap = std::unordered_map<Key, myStruct, something::Compare>;
int main()
{
myMap mp =
{
{ { "name1", "name2" },{ 3,4 } },
{ { "aame1", "name2" },{ 8,4 } },
{ std::make_pair("fame1", "name2"),{ 2,4 } }, // or make pair
{ std::make_pair("fame1", "bame2"),{ 1,2 } }
};
for(const auto& it: mp)
{
std::cout << it.first.first << " " << it.first.second << " "
<< it.second.x << " " << it.second.y << std::endl;
}
return 0;
}
However, every symmetric pair will make almost same hashes and that can cause,
hash collisions and thereby less performance. Nevertheless, additional specializations for std::pair to compose hashes are available in boost.hash
An alternative solution, could be using std::map<>. There you can also specify the custom function/functor for the std::pair, in order to achieve the same map structure. Even though there you will not have to face hash-collisions, that would be well sorted which you might not want.
LIVE DEMO
#include <map>
#include <string>
#include <tuple>
#include <iostream>
struct myStruct { int x, y; };
using Key = std::pair<std::string, std::string>;
namespace something
{
struct Compare
{
bool operator()(const Key& lhs, const Key& rhs) const
{
// do the required comparison here
return std::tie(lhs.first, lhs.second) < std::tie(rhs.first, rhs.second);
}
};
}
using myMap = std::map<Key, myStruct, something::Compare>;
could you tell me why it is not good to use my data type in std ? My
type is defined in my own program anyway.
You shouldn't make it under the namespace of std, because it can cause a UB. A well defined situations/exceptions where you can extend std namespace are given here: https://en.cppreference.com/w/cpp/language/extending_std
Answer to the secondary question:
Thank you but could you tell me why it is not good to use my data type in std ? My type is defined in my own program anyway
You feel that you can define whatever you want in your program, right? (That is the impression you gave, at least.) Well, C++ implementations feel the same way about namespace std -- it is their namespace and they can define whatever they want in it (subject to the C++ standard, of course).
If an implementation needs to define a (possibly undocumented) helper function/class/whatever, the expectation is that it can be placed in namespace std without conflicting with your program. Case in point: what would happen to your program if your C++ library decided that it needed to define a specialization of the std::pair template for std::pair<std::string, std::string>? To my knowledge, the standard neither requires nor prohibits such a specialization, so the existence of it is left to the implementor's discretion.
Namespaces exist to prevent naming conflicts. In particular, namespace std exists to isolate C++ implementation details from user programs. Adding your code to namespace std destroys that isolation, hence the standard declares it undefined behavior. Don't do it.
(That being said, there is nothing stopping you from writing a wrapper class around std::pair<std::string, std::string> to get the functionality you need. Just do it in your own namespace.)
You need to define a specialization of std::hash for your key type, like so:
#include <unordered_map>
#include <string>
using KeyType = std::pair<std::string, std::string>;
namespace std
{
template<>
struct hash<KeyType>
{
size_t operator()(KeyType const& kt) const
{
size_t hash = 0;
hash_combine(hash, kt.first);
hash_combine(hash, kt.second);
return hash;
}
// taken from boost::hash_combine:
// https://www.boost.org/doc/libs/1_55_0/doc/html/hash/reference.html#boost.hash_combine
template <class T>
inline static void hash_combine(std::size_t& seed, const T& v)
{
std::hash<T> hasher;
seed ^= hasher(v) + 0x9e3779b9 + (seed<<6) + (seed>>2);
}
};
}
int main()
{
std::unordered_map<KeyType, int> us;
return 0;
}
I was reviewing some older code of mine and I saw the code using pointers to implement a tree of Variant objects. It is a tree because each Variant can contain an unordered_map of Variant*.
I looked at the code and wondered why isn't it just using values, a std::vector<Variant>, and std::unordered_map<std::string, Variant>, instead of Variant*.
So I went ahead and changed it. It seemed okay except one thing, I got errors:
/usr/local/include/c++/6.1.0/bits/stl_pair.h:153:11: error: 'std::pair<_T1, _T2>::second' has incomplete type
_T2 second; /// #c second is a copy of the second object
^~~~~~ main.cpp:11:8: note: forward declaration of 'struct Variant'
struct Variant
^~~~~~~
So I figured I could trick the compiler into delaying the need to know that type, which didn't work either.
Working Not Working! (MCVE)
I thought this worked earlier but it actually doesn't, I forgot ::type on the using HideMap...
#include <vector>
#include <unordered_map>
#include <iostream>
template<typename K, typename V>
struct HideMap
{
using type = std::unordered_map<K, V>;
};
struct Variant
{
using array_container = std::vector<Variant>;
// Does not work either
using object_container = typename HideMap<std::string, Variant>::type;
// Fails
//using object_container = std::unordered_map<std::string, Variant>;
private:
union Union
{
std::int64_t vint;
array_container varr;
object_container vobj;
// These are required when there are union
// members that need construct/destruct
Union() {}
~Union() {}
};
Union data;
bool weak;
};
int main()
{
Variant v;
std::cout << "Works" << std::endl;
}
So, my question is, why does it work okay for vector and not unordered_map?
If the problem is the inability to use incomplete types, is there a way to delay the instantiation of the unordered_map? I really don't want every object property to be a separate new allocation.
This uses placement new to defer the initialization of the Union to the constructor where Variant is a complete type. You need to reinterpret_cast everywhere you need to use the Union. I made an effort to not have any strict-alignment violations.
#include <algorithm>
#include <iostream>
#include <unordered_map>
#include <vector>
struct Variant {
Variant();
~Variant();
private:
std::aligned_union<0, std::vector<Variant>,
std::unordered_map<std::string, void *>,
std::int64_t>::type data;
};
namespace Variant_detail {
using array_container = std::vector<Variant>;
using object_container = std::unordered_map<std::string, Variant>;
union Union {
std::int64_t vint;
array_container varr;
object_container vobj;
// These are required when there are union
// members that need construct/destruct
Union() {}
~Union() {}
};
}
Variant::Variant() {
//make sure that std::unordered_map<std::string, Variant> is not too large
static_assert(sizeof(std::unordered_map<std::string, Variant>) <=
sizeof data, "Variant map too big");
static_assert(alignof(std::unordered_map<std::string, Variant>) <=
alignof(decltype(data)), "Variant map has too high alignment");
auto &my_union = *new (&data) Variant_detail::Union;
my_union.vint = 42;
}
Variant::~Variant() {
reinterpret_cast<Variant_detail::Union &>(data).~Union();
}
int main() {
Variant v;
std::cout << "Works" << std::endl;
}
I am using std::transform with an std::back_inserter to append elements to an std::deque. Now the transformation may fail and will return a invalid object (say an uninitialized boost::optional or a null pointer) in some cases. I would like to filter out the invalid objects from getting appended.
I thought about using boost::filter_iterator, but not sure how to present the end() parameter of the filtered range.
The documentation of boost::filter_iterator suggests that output filtering is possible. Should I just specialize operator == for std::back_insert_iterator in this case to always return false?
In addition to this, if I want to append values of initialized boost::optional or pointers, can I chain boost::filter_iterator and boost::indirect_iterator?
I am trying to avoid rolling out my own transform_valid function that takes an optional extractor function.
Is it even possible to use filter_iterator as an output iterator?
I suggest using boost range (algorithms & adaptors) for ease of use, you'd write:
boost::copy(
data | transformed(makeT) | filtered(validate) /* | indirected */,
std::back_inserter(queue));
Here is a complete working example of that:
#include <boost/range.hpp>
#include <boost/range/adaptors.hpp>
#include <boost/range/algorithm.hpp>
#include <boost/optional.hpp>
#include <vector>
#include <deque>
typedef boost::optional<int> T;
typedef std::deque<T> Q;
static T makeT(int i)
{
if (i%2) return T();
else return i;
}
static bool validate(const T& optional)
{
return (bool) optional; // select the optional that had a value set
}
int main()
{
static const int data[] = { 1,2,3,4,5,6,7,8,9 };
Q q;
using boost::adaptors::filtered;
using boost::adaptors::transformed;
// note how Boost Range elegantly supports an int[] as an input range
boost::copy(data | transformed(makeT) | filtered(validate), std::back_inserter(q));
// demo output: 2, 4, 6, 8 printed
for (Q::const_iterator it=q.begin(); it!=q.end(); ++it)
{
std::cout << (*it? "set" : "unset") << "\t" << it->get_value_or(0) << std::endl;
}
return 0;
}
Update
With a little help from this answer: Use boost::optional together with boost::adaptors::indirected
I now include an elegant demonstration of using the indirected range adaptor as well for immediate output of the queue (dereferencing the optionals):
Note that for (smart) pointer types there would obviously be no need to provide the pointee<> specialisation. I reckon this is by design: optional<> is not, and does not model, a pointer
#include <boost/range.hpp>
#include <boost/range/adaptors.hpp>
#include <boost/range/algorithm.hpp>
#include <boost/optional.hpp>
namespace boost {
template<typename P> struct pointee<optional<P> > {
typedef typename optional<P>::value_type type;
};
}
typedef boost::optional<int> T;
static T makeT(int i) { return i%2? T() : i; }
static bool validate(const T& optional) { return (bool) optional; }
int main() {
using namespace boost::adaptors;
static int data[] = { 1,2,3,4,5,6,7,8,9 };
boost::copy(data | transformed(makeT)
| filtered(validate)
| indirected,
std::ostream_iterator<int>(std::cout, ", "));
}