boost:serialization reconstruction (loading) - c++
I'm using boost:serialization to save data structures into a file. The actual data is a pointer vector of classes and subclasses.
However the constructor of the class which is serialized takes as parameter another instantiated class Agent which is an object that controls communication with a simulation API (webots).
I see that in boost::serialization examples, the serializable objects need an empty constructor class() {}; to be used for the reconstruction. However this is impractical in my case. How can I use reconstruction but include the object which communicates with the API ?
One of the serializable classes has this constructor:
State(Agent &A, ACTION_MODE const& m);
and I've seen from examples in boost docs that I need something like this:
State() {};
Yet Agent &A must be passed as parameter.
Should I find a way around this (using an extern, a singleton, a global object) or is there a way to modify this behavior when reconstructing ? I'm sure I'm missing something here.
Thank you
EDIT:
Maybe I didn't explain this clearly enough. I get an error message when trying to "load" by reconstructing the serialized data.
error: no matching function to call State::State()
Which is what made me look into boost::serialize code, and think that it is calling a constructor or copy operator.
How do I make it use a specific constructor to serialize the data and take as an argument the Agent reference &a ?
EDIT#2:
template <class S, class P, class A> void Task<S,P,A>::save(const char* file)
{
std::ofstream ofs(file);
assert(ofs.good());
boost::archive::text_oarchive oa(ofs);
oa << states;
ofs.close();
}
template <class S, class P, class A> void Task<S,P,A>::load(const char* file)
{
std::ifstream ifs(file);
boost::archive::text_iarchive ia(ifs);
ia >> states;
ifs.close();
}
States is friend to boost::serialization::access and has a function serialize.
Saving works fine, loading is the problem.
states is: boost::ptr_vector<S> states; where S is a type of State polymorphic class.
State is the base class and has "serialize"
template <class Archive>
void State::serialize(Archive& ar, const unsigned int version)
{
ar & accel.Xaxis & accel.Yaxis & accel.Zaxis;
ar & gyro.Xaxis & gyro.Yaxis & gyro.Zaxis;
ar & gps.Yaxis;
ar & positions;
ar & reward & value & hash_value;
}
guState inherits from State.
template <class Archive>
void guState::serialize(Archive& ar, const unsigned int version)
{
ar & boost::serialization::base_object<State>(*this);
ar & accel.Xaxis & accel.Yaxis & accel.Zaxis;
ar & gyro.Xaxis & gyro.Yaxis & gyro.Zaxis;
ar & gps.Yaxis;
ar & positions;
ar & reward & value & hash_value;
}
accel, gyro, gps are simple structures with 3 double variables. They get serialized above^^.
Positions is an std::map<std::string,float> positions;
Looking at the serialized text file, everything appears Ok.
I cannot understand why it calls a constructor when trying to load the file.
EDIT#3:
Base Constructor is:
State(Agent &A, ACTION_MODE const& m);
Derived Constuctor is:
guState::guState(Agent& A, ACTION_MODE const& m) :
State(A, m)
{
...
}
Agent reference &A, kept in each State (or derived State) refers to an object obtained from the simulation API. It controls a robot. I cannot serialize it, and it doesn't make sense serializing it.
When I use:
namespace boost { namespace serialization {
template <class Archive>
void save_construct_data(Archive & ar,const guState* d,const unsigned int file_version)
{
ar << guState::caller;
ar << guState::mode;
}
template <class Archive>
void load_construct_data(Archive & ar, guState* d,const unsigned int file_version)
{
Agent &a;
ACTION_MODE &m;
ar >> a;
ar >> m;
::new(d) guState(a,m);
}
}
}
I get the following errors:
invalid use of non-static data member State::caller
invalid use of non-static data member State::mode
referring to the references used from the constructor.
And:
error: 'a' declared as reference but not initialized
error: 'm' declared as reference but not initialized
As you can see, it makes no sense to try and save the reference to Agent, because that reference (even if it could be saved or serialized) will probably be different every time the application is started.
And in loading the construct data, apart from me probably using the wrong syntax, it makes no sense to construct from a serialized reference to agent.
What I belive i need, is a way to tell the load_construct_data how to obtain a reference to an Agent (after initializing an agent object) and use that reference to construct the data.
Does that make any sense ? Do you think this is doable ?
EDIT#4
namespace boost { namespace serialization {
template <class Archive>
void save_construct_data(Archive & ar,const guState* d,const unsigned int file_version)
{
ar << guState::caller;
}
template <class Archive>
void load_construct_data(Archive & ar, guState* d,const unsigned int file_version)
{
Agent * a;
ACTION_MODE mode = RAND_SING;
ar >> a;
::new(d) guState(*a,mode);
}
}
}
It will not allow to serialize guState::caller
I have also made class Agent serializable, and overloaded the load_construct_data and save_construct_data of Agent in order to request from the simulation app a new instance of Agent to control the API.
EDIT:
There is a part of the manual that I think we both missed: The section non-default constructors here. To make it work you need to a save_construct_data and a load_construct_data function. There is a slight technicallity in terms of where these are friended explored here.
Also, you said you had this problem when trying to load only, but that you could save fine. This makes me think that you might have omitted
BOOST_CLASS_EXPORT_GUID(state, "state")
This omission might lead to segmentation faults once you get the load compiling (see the export section of the manual)
To make sure I was not mistaken, I made a compiling example, which I add in case its useful.
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/export.hpp>
#include <iostream>
#include <fstream>
//base class
struct base
{
base(double d) : m_d(d) {}
virtual double run() = 0;
private:
friend class boost::serialization::access;
double m_d;
template <class Archive>
void serialize(Archive& ar, const unsigned int version)
{
ar & m_d;
}
};
//forward declare the save construct data before friending it
// (something about friend being in a different namespace)
class derived;
namespace boost { namespace serialization {
template<class Archive>
inline void save_construct_data(Archive & ar, const derived * t, const unsigned int file_version);
}}
//derived class with non-default constructor
struct derived : public base
{
derived(double a , double b) :
base(a+b),
m_a(a),m_b(b),m_c(a*b)
{}
//some checks
double get_a() const {return m_a;}
double get_b() const {return m_b;}
double get_c() const {return m_c;}
double run(){return 1.0;}
private:
friend class boost::serialization::access;
template<class Archive>
friend void boost::serialization::save_construct_data(Archive & ar, const derived * t, const unsigned int file_version);
template <class Archive>
void serialize(Archive& ar, const unsigned int version)
{
ar & boost::serialization::base_object<base>(*this);
//only need to return c, a and b already done for constructor
ar & m_c;
}
double m_a, m_b, m_c;
};
//Save and load the data required for the constructor.
namespace boost { namespace serialization {
template <class Archive>
inline void save_construct_data(
Archive & ar,const derived* d,const unsigned int file_version
)
{
// save data required to construct instance
ar << d->m_a;
ar << d->m_b;
}
template <class Archive>
inline void load_construct_data(
Archive & ar, derived* d,const unsigned int file_version
)
{
double a,b;
ar >> a;
ar >> b;
// invoke inplace constructor to initialize instance of my_class
::new(d) derived(a,b);
}
}
}
//register the derived class with boost.
BOOST_CLASS_EXPORT_GUID(derived, "derived")
int
main (int ac, char **av)
{
std::ofstream ofs("filename");
base* p = new derived(2,3);
// save data to archive
{
boost::archive::text_oarchive oa(ofs);
oa << p;
}
// ... some time later restore the class instance to its orginal state
base* p2;
{
std::ifstream ifs("filename");
boost::archive::text_iarchive ia(ifs);
ia >> p2;
}
derived* d = static_cast<derived*>(p2);
std::cout<<"p2 vals are: "<<d->get_a()<<" "<<d->get_b()<<" "<<d->get_c()<<std::endl;
}
OLD RESPONSE:
Not sure I entirely understood your problem (a fuller example would help me) -
The constructor doesn't usually come into it when you serialize an object: you serialize the raw data?
Do you mean that you don't want to serialize all the raw data of the object, but just want to reconstruct it again when you deserialize the object (using the constructor)? If so then you can do this by seriazing the data that you need for the reconstruction and splitting the save and load operations.
struct my_class
{
my_class(Agent& A, ACTION_MODE const & m)
: m_state(A,M)
{}
private:
State m_state;
friend class boost::serialization::access;
void save(Archive & ar, const unsigned int version) const
{
// note, version is always the latest when saving
Agent tmp_A = m_state.get_A();
ACTION_MODE tmp_m = m_state.get_m();
ar & tmp_A;
ar & tmp_m;
}
template<class Archive>
void load(Archive & ar, const unsigned int version)
{
Agent tmp_A;
ACTION_MODE tmp_m
ar & tmp_A;
ar & tmp_m;
m_state = State(tmp_A,tmp_m);
}
BOOST_SERIALIZATION_SPLIT_MEMBER()
}
Does that help, or have I missed the point?
Related
binary reading and writing complicated stucts in c++
I need the ability to save/read my data structures in my project, but all the data is in the form of quite complicated and distinct structures themselves which I typically implement through other structs and vectors. I have wrapped all of them up into a single struct so that I have something like struct master{ std::vector<apprentice_type1> a; std::vector<apprentice_type2> b; //etc. std::string label; }; with other ones defined like struct apprentice_type1{ vec location; int point_label; std::vector<int> relational_data; }; struct vec{ double x,y,z; }; So it gets pretty complicated! I was desperately hoping something nice, quick and naive like master obj; //write to obj.... std::ofstream ofs("data.dat", std::ios::binary); ofs.write((char *)&obj, sizeof(obj)); would work, but at present it doesn't seem to. Before I get lost in the debugging rabbit hole is this actually possible the way I'm approaching it or do I need to rethink? If so, how? Thanks.
If you want an alternative to Boost serialization and have access to a C++11 compiler, you can also check out cereal. It works in a near identical fashion to Boost serialize but is a header only library so there is nothing to link against.
[...] or do I need to rethink? If so, how? You will probably need to provide a full implementation (i.e. explore the "rabbit-hole"). This is a known problem (stream serialization) and there is no single best-approach solution to it, because most implementations need to solve different needs. You can do one of the following: implement std::i/ostream serialization; This means you will go over your classes and implement operator>>(std::istream*, your_type&) and it's inverse operator<<(std::ostream*, your_type&). implement serialization based on a stream library (like boost.archive). use a JSON or XML library. use google protocol buffers (or something else) roll your own implementation, depending on your needs.
If you go for boost::serialization, here is a little sample: #include <fstream> #include <vector> #include <boost/archive/binary_iarchive.hpp> #include <boost/archive/binary_oarchive.hpp> #include <boost/serialization/vector.hpp> template <typename T> inline const boost::serialization::nvp<T> archive_value(const char* name, T& value) { return boost::serialization::make_nvp(name, value); } const unsigned Version = 0; class Point{ public: double x,y,z; private: template <typename P, typename Archive> static void serialize(P& p, Archive& ar, const unsigned int version) { std::cout << "Point\n"; ar & archive_value("X", p.x); ar & archive_value("Y", p.y); ar & archive_value("Z", p.z); } public: template <typename Archive> void serialize(Archive& ar, const unsigned int version) const { serialize(*this, ar, version); } template <typename Archive> void serialize(Archive& ar, const unsigned int version) { serialize(*this, ar, version); } }; BOOST_CLASS_VERSION(Point, Version) struct Scene{ std::vector<Point> points; private: template <typename S, typename Archive> static void serialize(S& s, Archive& ar, const unsigned int version) { std::cout << "Scene\n"; ar & archive_value("Points", s.points); } public: template <typename Archive> void serialize(Archive& ar, const unsigned int version) const { serialize(*this, ar, version); } template <typename Archive> void serialize(Archive& ar, const unsigned int version) { serialize(*this, ar, version); } }; BOOST_CLASS_VERSION(Scene, Version) template <typename Archive> void register_types(Archive& ar) { ar.template register_type<Point>(); ar.template register_type<Scene>(); } int main() { Scene scene; scene.points = { { 0, 0, 0 }, { 1, 1, 1 }, { 2, 2, 2 } }; // Output { std::ofstream out("test.dat", std::ios_base::binary); boost::archive::binary_oarchive output(out); // First the version! output & archive_value("Version", Version); // Next the types! register_types(output); // Finally the data output & archive_value("Scene", scene); } scene.points = {}; // Input { int version; std::ifstream in("test.dat", std::ios_base::binary); boost::archive::binary_iarchive input(in); // First the version! input & archive_value("Version", Version); // Next the types! register_types(input); // Finally the data input & archive_value("Scene", scene); } for(const auto& p : scene.points) std::cout << p.x << '\n'; } Note: The file format may evolve and serialization functions (input and/or output) may get adjustment depending on the file version.
serializing classes using boost serialization without changing the class
this piece of code has to be written every time we make a class i.e.from template<class archive> to ar & BOOST_SERIALIZATION_NVP(b) . How can we make it short? and how can we serialize the stl containers ? class Employee { private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & BOOST_SERIALIZATION_NVP(a); ar & BOOST_SERIALIZATION_NVP(b); } int a; int b; public: Employee(int a, int b) { this->a = a; this->b = b; } };
I suggest you start with the documentation :) http://www.boost.org/doc/libs/1_55_0/libs/serialization/doc/index.html (start under Serializable Concept) STL containers are serializable when you include the relevant header: #include <boost/serialization/map.hpp> #include <boost/serialization/string.hpp> A host of other stuff is supported out of the box. You could make generic wrappers for types that have already been made "reflectible" by other means (e.g. Fusion Sequences, Qt QObjects etc.)
How do I serialize a class containing pointers to primitives?
I am trying to use boost's functionality for serializing pointers to primitives (so that I don't have to de-reference and do a deep store myself). However, I get a pile of errors when I try to do it. Here is a simple example of a class that is supposed to contain save and load methods which write and read the class content from a file. This program does not compile: #include <boost/archive/text_iarchive.hpp> #include <boost/archive/text_oarchive.hpp> #include <boost/serialization/shared_ptr.hpp> #include <boost/shared_ptr.hpp> #include <fstream> class A { public: boost::shared_ptr<int> sp; int const * p; int const& get() {return *p;} void A::Save(char * const filename); static A * const Load(char * const filename); ////////////////////////////////// // Boost Serialization: // private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar,const unsigned int file_version) { ar & p & v; } }; // save the world to a file: void A::Save(char * const filename) { // create and open a character archive for output std::ofstream ofs(filename); // save data to archive { boost::archive::text_oarchive oa(ofs); // write the pointer to file oa << this; } } // load world from file A * const A::Load(char * const filename) { A * a; // create and open an archive for input std::ifstream ifs(filename); boost::archive::text_iarchive ia(ifs); // read class pointer from archive ia >> a; return a; } int main() { } Note that I am not interested in a solution that dereferences the pointer; I want boost to take care of that for me (many of these classes might be pointing to the same underlying object).
In http://www.boost.org/doc/libs/1_54_0/libs/serialization/doc/index.html: By default, data types designated primitive by Implementation Level class serialization trait are never tracked. If it is desired to track a shared primitive object through a pointer (e.g. a long used as a reference count), It should be wrapped in a class/struct so that it is an identifiable type. The alternative of changing the implementation level of a long would affect all longs serialized in the whole program - probably not what one would intend. Hence: struct Wrapped { int value; private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar,const unsigned int file_version) { ar & value; } }; boost::shared_ptr<Wrapped> sp; Wrapped const * p;
Deserialize object with references and no default constructor (boost::serialization)
Is it possible in boost::serialization library to deserialize (polymorphic) objects with references and no default constructor? class Example { int& value; public: Example(int _value): value(_value) {} virtual ~Example() {} friend class boost::serialization::access; template<typename Archive> void serialize(Archive & ar, const unsigned int file_version) { ar & value; } }; class Usage { Example* example; public: Usage(): example(new Example(123)) {} ~Usage() { delete example; } friend class boost::serialization::access; template<typename Archive> void serialize(Archive & ar, const unsigned int file_version) { ar & example; } }; ... // serialize and deserialize object with reference and no default constructor { Usage source; std::ostringstream oss; boost::archive::text_oarchive oa(oss); oa & source; Usage target; std::istringstream iss(oss.str()); boost::archive::text_iarchive ia(iss); ia & target; // does not compile }
As for non-default constructible object, I'd recommend to look the item Non-Default Constructors here. Your class can be serialized by writing your own function template load_construct_data and save_construct_data.
Boost Serializing of Object containing Map (with object values) and Multimap (with std::string values): what is needed?
See below a main() and two very simple classes. Then per Boost serialization (and what is shown) my questions are: 1) Does class B need the normal overloaded stream insertion operators '<<' and '>>' to be defined? Currently in my real code it doesn't have these. 2) Does class A in the store() and load() methods need to iterate through the map and multimap containers explicitely, storing/loading their key:value pairs explicitely? e.g. something like: void A::store(const char* filename){ std::ofstream ofs(filename); boost::archive::text_oarchive oa(ofs); std::map< std::string, B >::iterator it; BMap.size(); oa << BMap.size(); for( it = BMap.begin(); it != BMap.end(); it++ ){ oa << it->first; oa << it->second; } //similar for strMultimap } I assume that I don't need to do this, but am not certain. 3) Assuming class B has only the two data members shown, does it need a default contructor included explicitely? (as opposed to the implicit default constructor) 4) Does B need to have an overide for the comparison operator '>'? I assume that it doesn't since this is a very simple class. Finally, any other comments per anything that I've failed to cover is appreciated! Example code for my above questions: //includes ommitted int main() { std::string file("test.dat"); A * pA = new A; pA->store(file.c_str()); pA->fillMaps(); //release data pA->load(file.c_str()); return 0; } //includes ommitted class A { friend class boost::serialization::access; public: std::map< std::string, B > BMap; std::multimap< std::string, std::string > strMultimap; template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & BMap; ar & strMultimap; } void store(const char* filename){ std::ofstream ofs(filename); boost::archive::text_oarchive oa(ofs); oa << this; } void load(const char* filename){ std::ifstream ifs(filename); boost::archive::text_iarchive ia(ifs); ia >> this; } void fillMaps(){ //code to allocate B objects and put them in BMap and fill strMultimap with whatever number of key:value pairs } class B { friend class boost::serialization::access; public: std::string str; unsigned int num; B::B(void) : str("a string") , num(7) { } template<class Archive> void serialize(Archive & ar, const unsigned int version) { ar & str; ar & num; } }
1) You don't need stream operators for class B, but it does need a serialize() method. I had to wrap the serialization with the BOOST_SERIALIZATION_NVP (name value pair) macro: ar & BOOST_SERIALIZATION_NVP(someNamedValue); // use this macro for everything you want to name There might be a way to avoid naming your map, but I don't know how that's done. 2) No, class A doesn't need map-specific serialization code. Just make sure you include <boost/serialization/map.hpp>. 3) The implicit default constructor should be fine. You only ever need an explicit default constructor if a) you've already provided a non-default constructor or b) you want to change the behavior of the default constructor. 4) No operator < is needed :) Here's some sample code which compiled, but I haven't run: #include <boost/serialization/map.hpp> struct A { struct B { template<class Archive> void serialize(Archive &ar, const unsigned int version) { } }; typedef std::map<int, SomeClass> MyMap; MyMap myMap; template<class Archive> void serialize(Archive &ar, const unsigned int version) { ar & BOOST_SERIALIZATION_NVP(myMap); } };