I'm trying to load parameters from a JSON file into my own parameter class using cereal (v1.12) on VS2013.
Compilation and loading into a POD works fine, but loading into my own class throws cereal::RapidJSONException.
Here is my code:
#include <iostream>
#include "cereal/archives/json.hpp"
struct CMyStruct {
CMyStruct(int f_value) : m_value(f_value) {}
int m_value;
};
template<class Archive> void load(Archive& ar, CMyStruct& parameter) {
int input;
ar(input);
parameter.m_value = input;
};
void test_cereal() {
int l_scalar_i(42);
CMyStruct l_scalar(42);
std::ifstream l_jsonFile("example_minimal.json");
cereal::JSONInputArchive l_ar(l_jsonFile);
l_ar( cereal::make_nvp("scalar", l_scalar_i) ); // WORKS
l_ar( cereal::make_nvp("scalar", l_scalar) ); // throws cereal::RapidJSONException
return;
}
The JSON file:
{
"bool": false,
"scalar": 3
}
Following the call stack shows that document.h:244 inside RapidJSON throws the exception:
ConstMemberIterator MemberEnd() const
{ RAPIDJSON_ASSERT(IsObject()); return data_.o.members + data_.o.size; }
However, I have no idea what to make of it. How can I make my code work?
Thanks a lot!
UPDATE WITH SOLUTION:
As written by ChrisBFX, it is necessary to implement load_minimal/save_minimal in order to deserialize an C++ object to a JSON scalar.
load/save on the other hand serializes a C++ object from/to a JSON object.
For my code, the load() function has to be removed and replaced by this:
template<class Archive>
int save_minimal(const Archive& ar,
const CMyStruct& parameter)
{
return parameter.m_value;
};
template<class Archive>
void load_minimal(const Archive& ar,
CMyStruct& parameter,
const int & value_i)
{
parameter.m_value = value_i;
};
Now it works like a charm! :-)
Your CMStruct is not a scalar but an object. So cereal expects
{
"bool": false,
"scalar": {
"value": 3
}
}
If you want to serialize your struct as a scalar you have to provide a minimal_load function for it. See http://uscilab.github.io/cereal/serialization_functions.html "minimal split serialization".
Related
I've got the following code (simplified):
namespace nlohmann {
// https://github.com/nlohmann/json/issues/1749#issuecomment-772996219
template <class T>
void to_json(nlohmann::json &j, const std::optional<T> &v) {
if (v.has_value())
j = *v;
else
j = nullptr;
}
template <class T>
void from_json(const nlohmann::json &j, std::optional<T> &v) {
if (j.is_null())
v = std::nullopt;
else
v = j.get<T>(); /* ERROR LOCATION */
}
} // namespace nlohmann
namespace discordpp{
class Component {
public:
Component(const std::optional<std::vector<Component>> &components)
: components(components) {}
std::optional<std::vector<Component>> components;
// (Resolved) NLOHMANN_DEFINE_TYPE_INTRUSIVE(Component, components)
friend void to_json(nlohmann::json &nlohmann_json_j,
const Component &nlohmann_json_t) {
nlohmann_json_j["components"] = nlohmann_json_t.components;
}
friend void from_json(const nlohmann::json &nlohmann_json_j,
Component &nlohmann_json_t) {
nlohmann_json_j.at("components").get_to(nlohmann_json_t.components); /* ERROR LOCATION */
}
};
}// namespace discordpp
Compiling returns the error error: no matching function for call to ‘nlohmann::basic_json<>::get<std::vector<discordpp::Component, std::allocator<discordpp::Component> > >() const’
Is there maybe something I could predeclare to solve this?
Tried setting up an adl_serializer like this, same issue
namespace nlohmann {
template<> struct adl_serializer<discordpp::Component> {
static void to_json(json &nlohmann_json_j, const discordpp::Component &nlohmann_json_t) {
NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(
NLOHMANN_JSON_TO, type, custom_id, disabled, style, label, emoji,
url, options, placeholder, min_values, max_values, components))
}
static void from_json(const json &nlohmann_json_j, discordpp::Component &nlohmann_json_t) {
NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(
NLOHMANN_JSON_FROM, type, custom_id, disabled, style, label, emoji,
url, options, placeholder, min_values, max_values, components))
}
};
} // namespace nlohmann
I changed it to a vector of shared pointers and now I get error: no matching function for call to ‘nlohmann::basic_json<>::get<discordpp::Component>() const’ in the adl_serializer I wrote to handle it
template <typename T> struct adl_serializer<std::vector<std::shared_ptr<T>>> {
static void to_json(json &j, const std::vector<std::shared_ptr<T>> &v) {
j = json::array();
for (auto t : v) {
j.push_back(*t);
}
}
static void from_json(const json &j, std::vector<std::shared_ptr<T>> &v) {
if (!j.is_array()) {
throw json::type_error::create(317, "j must be an array", j);
}
for (auto t : j) {
v.push_back(std::make_shared<T>(j.get<T>())); /* Error is here */
}
}
};
Mirrored at https://github.com/nlohmann/json/discussions/3047
Edit:
Full source (do a recursive clone): https://github.com/DiscordPP/echo-bot/tree/dev-objects
Failed build: https://github.com/DiscordPP/echo-bot/runs/3799394029?check_suite_focus=true
Edit 2: It's fine when I do std::vector<std::shared_ptr<Component>> components;, it just doesn't seem to be able handle 2 third party parsers, maybe?
I think the problem is that you try to have a member with type std::optional<std::vector<Component>> in class Component, and std::optional does not work with incomplete types. Replacing this with std::unique_ptr<std::vector<Component>> should work.
I think the problem is that you are calling get_to on a vector of Component, but you cannot convert json into Component.
A solution would be to add a constructor:
explicit Component(const nlohmann::json &nlohmann_json_j = nlohmann::json())
{
// do whatever you have to do
}
Note that I added = nlohmann::json() because std::vector<Component> needs the default constructor.
Note: I tested this approach and it works as expected.
Live demo:
https://replit.com/#ichramm/somerandomtest
Compile with:
clang++-7 -Iinclude -std=c++17 -o main main.cpp && ./main
Update: I am adding more details because it looks that the issue is not fully understood.
The problem has nothing to do with std::optional.
The problem is that there is a vector of Component with it is supposed to be created from a vector of json.
You provided no way read/write optionals from/to the json, but you didn't provide a way to read/write objects of type Component.
In the line j.get<T>();, T is std::vector<Component>, and j can be thought as std::vector<Json>.
Alternatively, you could create a function from_json for your type Component. I haven't tested it but it looks like it may work.
I was doing some thinking on #Niels's answer and I realized that friend void from_json is taking an already-constructed Component& but Component doesn't have a default constructor. I added Component() : components(std::nullopt) {} and we're off to the races!
I have a simple classe here :
template <typename T>
class clDaughter
{
public:
T* __pData;
uint16_t __u16Size;
clDaughter() [...]
~clDaughter() [...]
clDaughter(uint16_t const ku16Size)
{
this->__pData = (T*)malloc(ku16Size * sizeof(T));
this->__u16Size = ku16Size;
}
template<class Archive>
void save(Archive & ar) const
{
ar(this->__u16Size);
ar(cereal::binary_data(this->__pData, this->__u16Size * sizeof(T)));
}
template<class Archive>
void load(Archive & ar)
{
uint16_t u16LoadedSize;
ar(u16LoadedSize);
this->__pData = (T*)malloc(u16LoadedSize * sizeof(T));
this->__u16Size = u16LoadedSize;
ar(cereal::binary_data(this->__pData, this->__u16Size * sizeof(T)));
}
};
This is working fine, I mean, serialization in and out is tested ok.
Trouble starts when I want to use polymorphism here. This daughter class inherits from a pure virtual mother class, as well as other "daugter-like" classes.
class clDaugter_A : public clMother
{
[...]
}
class clDaugter_B : public clMother
{
[...]
}
And, when I want to register my clDaughter class using the CEREAL_REGISTER_TYPE(...) macro,
CEREAL_REGISTER_TYPE(clDaugter_A<int>)
CEREAL_REGISTER_TYPE(clDaugter_B<int>)
compiler crashes with
"cereal could not find any output serialization functions for the provided type and archive combination"
It really seems the problem comes from this binary_data(...) method because if I serialize the __pData array in a loop (ugly-style)
for (u16Idx = 0;..;..)
{
ar(this->__pData[u16Idx];
}
I have no error and it works fine. It is only when I use binary_data() and CEREAL_REGISTER_TYPE() together.
What did I miss ?
(to pre-empt the question, I do want to use binary_data() because it's something like 20-30 times faster than the loop and I need to be fast here)
Thank for your help
As far as I understand, there is no serialization (boost::serialization, actually) support for boost::any placeholder.
Does someone know if there is a way to serialize a custom boost::any entity?
The problem here is obvious: boost::any uses template-based placeholders to store objects and typeid to check if boost::any_cast is appropriate.
So, there is a custom abstract superclass placeholder and custom template-based derived classes, which are created the following way:
template <T> custom_placeholder : public placeholder {
virtual std::type_info type() const { return typeid(T); }
virtual ...
};
Obviously, this brings some troubles when even thinking about serializing this stuff. Maybe someone knows some trick to make such kind of serialization (and of course, proper deserialization)?
Thank you
If you want to stick with boost::any i am not sure but you can write your own "boost::any". I'm using this code for proxy methods to pass the parameters.
#include <iostream>
#include <boost\smart_ptr\scoped_ptr.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/serialization/access.hpp>
#include <boost/serialization/shared_ptr.hpp>
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/export.hpp>
#include <sstream>
class my_placeholder
{
public:
virtual ~my_placeholder(){}
my_placeholder(){}
private:
friend class boost::serialization::access;
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
// serialize base class information
//ar & boost::serialization::base_object<bus_stop>(*this);
//ar & m_placeholder;
}
};
template<typename T>
class my_derivedplaceholder:
public my_placeholder
{
public:
my_derivedplaceholder()
{
}
my_derivedplaceholder(T &value)
{
m_value=value;
}
T m_value;
private:
friend class boost::serialization::access;
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
// serialize base class information
ar & boost::serialization::base_object<my_placeholder>(*this);
ar & m_value;
}
};
BOOST_CLASS_EXPORT_GUID(my_derivedplaceholder<int>, "p<int>");
class my_any
{
public:
my_any()
{
}
template<typename T>
my_any(const T &value)
{
m_placeholder.reset(new my_derivedplaceholder<T>(const_cast<T&>(value)));
}
template<typename T>
void operator=(const T &value)
{
m_placeholder.reset(new my_derivedplaceholder<T>(const_cast<T&>(value)));
}
protected:
friend class boost::serialization::access;
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
// serialize base class information
//ar & boost::serialization::base_object<bus_stop>(*this);
ar & m_placeholder;
}
template<typename T>
friend T my_anycast(my_any &val);
boost::shared_ptr<my_placeholder> m_placeholder;
};
template<typename T>
T my_anycast(my_any &val)
{
boost::shared_ptr<my_derivedplaceholder<T>> concrete=boost::dynamic_pointer_cast<my_derivedplaceholder<T>>(val.m_placeholder);
if (concrete.get()==NULL)
throw std::invalid_argument("Not convertible");
return concrete->m_value;
}
void main()
{
my_any m=10;
int a=my_anycast<int>(m);
std::cout << a << std::endl;
std::stringstream ss,ss2;
boost::archive::text_oarchive oa(ss);
oa << m;
boost::archive::text_iarchive ia(ss);
my_any m2;
ia >> m2;
std::cout << my_anycast<int>(m2) << std::endl;
}
It is not possible at all, at least for arbitrary types. Note that maybe you could serialize using some tricky code (like finding the size of the elements contained in the any), but the any code relies on the compiler statically putting the any type_code and the proper types inside the placeholder. You surely cannot do that in deserialization in C++, as the type that you'd get from the deserialization is not known at compile time (as required by the newly formed boost::any).
The best solution is to build some kind of specialized any type for the exact types of elements you're going to serialize. Then, you can have special cases for the actual type of element being deserialized, but note that each element type serialization/deserialization has to be phisically written as static C++ code.
PD. Some others suggested using boost::variant as a representation of this specialized type holding the exact types you're going to serialize. You need a way of discerning the exact type on deserialization, though (maybe assigning identifiers to types in the variant).
Assuming you have to use boost::any and you cannot switch to variant, a map<type_info const*, string(*)(any)> based solution could get you done.
You have to initialize at runtime such a map with all the types you plan to use. Of course, you can use something along the lines of
template <typename T>
struct any_serializer
{
static string perform(any a)
{
T const& x = any_cast<T const&>(a);
stringstream out;
out << x;
return out.str();
}
};
and populate the map with addresses of any_serializer<T>::perform under the key &typeid(T). You can specialize the class any_serializer and use some (ugly) macros to populate the map.
More difficult is of course the deserialization. I haven't had a look at boost::lexical_cast for a while, perhaps it can provide some help. I am afraid that this is totally problem-dependant. However, you only need one function, which takes a string and returns one any. You may also want to prepend your output string with a custom type identifier.
There is no need to create new class. Try to use xany https://sourceforge.net/projects/extendableany/?source=directory
xany class allows to add new methods to any's existing functionality. By the way there is a example in documentation which does exactly what you want.
I have a "generator" class that basically constructs its subclass. To use this thing I simply subclass it and pass it the correct parameters to build the object I want built. I want to serialize these things and there's no good reason to do it for each subclass since all the data is in the base. Here's what I've got as example:
#include <boost/serialization/serialization.hpp>
template < typename T >
struct test_base
{
// works...
//template < typename Archive >
//void serialize(Archive &, unsigned int const)
// {
//}
};
template < typename T >
void f(test_base<T> const&) {}
struct test_derived : test_base<int>
{
};
namespace boost { namespace serialization {
template < typename Archive, typename T >
void serialize(Archive &, test_base<T> &, unsigned int const)
{
}
}}
#include <boost/archive/binary_oarchive.hpp>
#include <sstream>
int main()
{
int x = 5;
test_derived d;
//boost::serialization::serialize(x, d, 54); // <- works.
std::ostringstream str;
boost::archive::binary_oarchive out(str);
out & d; // no worky.
}
I want the free version to work if possible. Is it?
Version above pukes up error about serialize not being a member of test_derived.
Clarification why the problem happens:
boost::serialization has to ways of implementing the serialize function. As class method or (in your case) the non-intrusive way of defining a function in the boost::serialization namespace.
So the compiler has to somehow decide which implementation to choose. For that reason boost has a 'default' implementation of the boost::serialization::serialize template function.
Signature:
template<class Archive, class T>
inline void serialize(Archive & ar, T & t, const BOOST_PFTO unsigned int file_version)
Within that function there is a call to T::serialize(...). So when you don't want the intusive version you have to override the boost::serialization::serialize function with something more explicit than the default function-template.
Now the problem: In your case the compiler has to decide if it
a) chooses the version where a parameter has to be casted implicit (test_derived& to test_base&)
b) use the generic function without casting (T is test_derived&)
You want the compiler to use variant a) but the compiler prefers b)
Solution:
I don't know a really good solution. I think i would go with a macro which generates implementations of serialize(...) with the explicit type.
If that isn't a possible solution for you, you could also tell the compiler more explicit what to call:
out & *((test_base<int>*)&d);
and wrap it in some helper function (because no one wants to look at such code all the day)
I hope that is a clear description and helps
In case my explanation was not clear, here is an example:
#include <iostream>
class Base
{
public:
virtual ~Base()
{
}
};
class Derived : public Base
{
public:
virtual ~Derived()
{
}
};
void foo(Base& bar)
{
std::cout << "special" << std::endl;
}
template<typename T>
void foo(T& bar)
{
std::cout << "generic" << std::endl;
}
int main()
{
Derived derived;
foo(derived); // => call to generic implementation
foo(*((Base*) &bla)); // => call to special
return 0;
}
Using boost::serialization, what's the "best" way to serialize an object that contains cached, derived values in mutable members?
class Example
{
public:
Example(float n) :
num(n),
sqrt_num(-1.0)
{}
// compute and cache sqrt on first read
float get_sqrt() const
{
if(sqrt_num < 0)
sqrt_num = sqrt(num);
return sqrt_num;
}
template <class Archive>
void serialize(Archive& ar, unsigned int version)
{ ... }
private:
float num;
mutable float sqrt_num;
};
I'd like to avoid splitting serialize() into separate save() and load() methods, for maintenance reasons.
One suboptimal implementation of serialize:
template <class Archive>
void serialize(Archive& ar, unsigned int version)
{
ar & num;
sqrt_num = -1.0;
}
This handles the deserialization case, but in the serialization case, the cached value is killed and must be recomputed.
What is the best practice in this case?
Splitting your saving and loading methods doesn't mean you have to maintain two copies of your serialization code. You can split them and then join them back again with a common function.
private:
friend class boost::serialization::access;
BOOST_SERIALIZATION_SPLIT_MEMBER()
template <class Archive>
void save(Archive& ar, const unsigned int version) const {
const_cast<Example*>(this)->common_serialize(ar, version);
}
template <class Archive>
void load(Archive& ar, const unsigned int version) {
common_serialize(ar, version);
sqrt_num = -1;
}
template <class Archive>
void common_serialize(Archive& ar, const unsigned int version) {
ar & num;
}
You probably noticed the const_cast. That's an unfortunate caveat to this idea. Although the serialize member function is non-const for saving operations, the save member function needs to be const. As long as the object you're serializing wasn't originally declared const, though, it's safe to cast it away as shown above. The documentation briefly mentions the need to cast for const members; this is similar.
With the changes above, your code will correctly print "2" for both ex1 and ex2, and you only have to maintain one copy of the serialization code. The load code only contains code specific to re-initializing the object's internal cache; the save function doesn't touch the cache.
You can check the Archive::is_loading field, and load cached values if it's true.
template <class Archive>
void serialize(Archive& ar, unsigned int version)
{
ar & num;
if(Archive::is_loading::value == true)
sqrt_num = -1.0;
}