Serialization tree structure using boost::serialization - c++

I have to serialize libkdtree++ in my program, the tree structures are briefly described as following:
struct _Node_base {
_Node_base * _M_parent, *_M_left, * _M_right;
template<Archive>
serialize(Archive &ar, const unsigned int version) {
ar & _M_left & _M_right;
}
}
template<typename V>
struct _Node : public _Node_base {
typedef V value_type;
value_type value;
template<Archive>
serialize(Archive &ar, const unsigned int version) {
ar.register_type(static_cast<_Node*>(NULL));
ar & boost::serialization::base_object<_Node_base>(*this);
ar & value;
}
}
struct Tree {
_Node * root;
template<Archive>
serialize(Archive &ar, const unsigned int version) {
ar & root;
}
}
This program reports "stream error".
But from the "serailzed file", it lacks the value fields for the children nodes of roots. Thus I think it is possible that BaseNode serialized _M_left and _M_right pointer. However since _Node_base have no idea about the value type of _Node, so it looks hard to add "ar.register_type" to _Node_base.serialize().

The pointer_conflict exception documentation states (sic):
pointer_conflict, // an attempt has been made to directly
// serialization::detail an object
// after having already serialzed the same
// object through a pointer. Were this permited,
// it the archive load would result in the
// creation of an extra copy of the obect.
I think the conflict occurs where each is serialised by a ptr in BaseNode::serialize and via the direct object, the *Node expression, in Node::serialize. However since the base_object function takes a reference and not a ptr I'm not sure how you would avoid this.
One possibility is to not serialize the parent ptr. Instead, after deserialization, do a walk of the tree and fix up the parent ptrs to point to the node parent. E.g. add the following method to BaseNode :
void fix (BaseNode* parent = 0)
{
this->parent = parent;
if (left != 0)
left->fix (this);
if (right != 0)
right->fix (this);
}
Then just call root->fix ()

The following solution for libkdtree++ & boost::serialization seems work:
// KDTree::_Node
friend class boost::serialization::access;
template<class Archive>
//void serialize(Archive & ar, const unsigned int version)
void save(Archive & ar, const unsigned int version) const
{
ar.register_type(static_cast< _Link_type>(NULL));
ar & boost::serialization::base_object<_Node_base>(*this);
_Link_type left = static_cast<_Link_type>(_M_left);
_Link_type right = static_cast<_Link_type>(_M_right);
ar & left & right;
ar & _M_value;
}
template<class Archive>
void load(Archive & ar, const unsigned int version)
{
ar.register_type(static_cast< _Link_type>(NULL));
ar & boost::serialization::base_object<_Node_base>(*this);
_Link_type left, right;
ar & left & right;
ar & _M_value;
if (left) {
left->_M_parent = this;
}
if (right) {
right->_M_parent = this;
}
_M_left = left;
_M_right = right;
}
BOOST_SERIALIZATION_SPLIT_MEMBER()

Related

binary reading and writing complicated stucts in c++

I need the ability to save/read my data structures in my project, but all the data is in the form of quite complicated and distinct structures themselves which I typically implement through other structs and vectors. I have wrapped all of them up into a single struct so that I have something like
struct master{
std::vector<apprentice_type1> a;
std::vector<apprentice_type2> b; //etc.
std::string label;
};
with other ones defined like
struct apprentice_type1{
vec location;
int point_label;
std::vector<int> relational_data;
};
struct vec{
double x,y,z;
};
So it gets pretty complicated! I was desperately hoping something nice, quick and naive like
master obj;
//write to obj....
std::ofstream ofs("data.dat", std::ios::binary);
ofs.write((char *)&obj, sizeof(obj));
would work, but at present it doesn't seem to. Before I get lost in the debugging rabbit hole is this actually possible the way I'm approaching it or do I need to rethink? If so, how?
Thanks.
If you want an alternative to Boost serialization and have access to a C++11 compiler, you can also check out cereal. It works in a near identical fashion to Boost serialize but is a header only library so there is nothing to link against.
[...] or do I need to rethink? If so, how?
You will probably need to provide a full implementation (i.e. explore the "rabbit-hole").
This is a known problem (stream serialization) and there is no single best-approach solution to it, because most implementations need to solve different needs.
You can do one of the following:
implement std::i/ostream serialization; This means you will go over your classes and implement operator>>(std::istream*, your_type&) and it's inverse operator<<(std::ostream*, your_type&).
implement serialization based on a stream library (like boost.archive).
use a JSON or XML library.
use google protocol buffers (or something else)
roll your own implementation, depending on your needs.
If you go for boost::serialization, here is a little sample:
#include <fstream>
#include <vector>
#include <boost/archive/binary_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/serialization/vector.hpp>
template <typename T>
inline const boost::serialization::nvp<T> archive_value(const char* name, T& value) {
return boost::serialization::make_nvp(name, value);
}
const unsigned Version = 0;
class Point{
public:
double x,y,z;
private:
template <typename P, typename Archive>
static void serialize(P& p, Archive& ar, const unsigned int version) {
std::cout << "Point\n";
ar & archive_value("X", p.x);
ar & archive_value("Y", p.y);
ar & archive_value("Z", p.z);
}
public:
template <typename Archive>
void serialize(Archive& ar, const unsigned int version) const {
serialize(*this, ar, version);
}
template <typename Archive>
void serialize(Archive& ar, const unsigned int version) {
serialize(*this, ar, version);
}
};
BOOST_CLASS_VERSION(Point, Version)
struct Scene{
std::vector<Point> points;
private:
template <typename S, typename Archive>
static void serialize(S& s, Archive& ar, const unsigned int version) {
std::cout << "Scene\n";
ar & archive_value("Points", s.points);
}
public:
template <typename Archive>
void serialize(Archive& ar, const unsigned int version) const {
serialize(*this, ar, version);
}
template <typename Archive>
void serialize(Archive& ar, const unsigned int version) {
serialize(*this, ar, version);
}
};
BOOST_CLASS_VERSION(Scene, Version)
template <typename Archive>
void register_types(Archive& ar) {
ar.template register_type<Point>();
ar.template register_type<Scene>();
}
int main() {
Scene scene;
scene.points = { { 0, 0, 0 }, { 1, 1, 1 }, { 2, 2, 2 } };
// Output
{
std::ofstream out("test.dat", std::ios_base::binary);
boost::archive::binary_oarchive output(out);
// First the version!
output & archive_value("Version", Version);
// Next the types!
register_types(output);
// Finally the data
output & archive_value("Scene", scene);
}
scene.points = {};
// Input
{
int version;
std::ifstream in("test.dat", std::ios_base::binary);
boost::archive::binary_iarchive input(in);
// First the version!
input & archive_value("Version", Version);
// Next the types!
register_types(input);
// Finally the data
input & archive_value("Scene", scene);
}
for(const auto& p : scene.points)
std::cout << p.x << '\n';
}
Note: The file format may evolve and serialization functions (input and/or output) may get adjustment depending on the file version.

Able to serialize with boost but unable to deserialize std::shared_ptr

boost serialization
namespace boost {
namespace serialization {
template <class Archive, class T>
inline void save
(Archive &archive,
const std::shared_ptr<T> subtree,
const unsigned int file_version)
{
// only the raw pointer has to be saved
const T *const subtree_x = subtree.get();
archive << subtree_x;
}
template <class Archive, class T>
inline void load
(Archive &archive,
std::shared_ptr<T> subtree,
const unsigned int file_version)
{
T *p_subtree;
archive >> p_subtree;
#if BOOST_WORKAROUND(BOOST_DINKUMWARE_STDLIB, == 1)
subtree.release();
subtree = std::shared_ptr< T >(p_subtree);
#else
subtree.reset(p_subtree);
#endif
}
template <class Archive, class T>
inline void serialize
(Archive &archive,
std::shared_ptr<T> subtree, // no const or else get compile-time error
const unsigned int file_version)
{
boost::serialization::split_free(archive, subtree, file_version);
}
} // namespace serialization
} // namespace boost
tree class
class Tree{
private:
class TreeNode{
public:
std::shared_ptr<TreeNode> node_factory(const T &new_key, const long &new_index)
{
return std::shared_ptr<TreeNode>(new TreeNode(new_key, new_index));
}
friend class Tree;
private:
friend class boost::serialization::access;
template<class Archive>
void serialize(Archive &archive, const unsigned int /* file_version */){
archive & key;
archive & index;
archive & left;
archive & right;
}
T key;
long index;
std::shared_ptr<TreeNode> left;
std::shared_ptr<TreeNode> right;
}; // End Tree Node Class Definition
friend class boost::serialization::access;
template <class Archive>
void serialize(Archive &archive, const unsigned int version){
archive & root;
}
};
writer
bool save(std::shared_ptr<Tree> &tree, const std::string &search_tree_file_name)
{
// create and open a binary archive for output
std::ofstream writer(search_tree_file_name, std::ofstream::out | std::ofstream::binary);
if(writer){
boost::archive::binary_oarchive serial_writer(writer);
//set_flags(0, true);
// write class instance to archive
serial_writer << *tree;
// archive and stream closed when destructors are called
}else if(writer.fail()){
writer.clear();
}
return true;
}
reader
enter code here bool load(std::shared_ptr<Tree> &tree, const std::string &search_tree_file_name)
{
// create and open a binary archive for output
std::ifstream reader(search_tree_file_name, std::ifstream::in | std::ifstream::binary);
if(reader){
boost::archive::binary_iarchive serial_reader(reader);
// read class state from archive
serial_reader >> *tree;
// archive and stream closed when destructors are called
}else if(reader.fail()){
reader.clear();
}
return true;
}
I have written to and verified the successful serialization to a file but fail to deserialize from and into a usable object.
Whether I am writing in text or binary, I can verify the serialized output is correct but, for some reason, the serialize output does not deserialize and I am left with an empty object when loading.
Have a look at these links, might provide you some clue.
http://www.boost.org/doc/libs/1_49_0/libs/serialization/doc/shared_ptr.html
& http://www.boost.org/doc/libs/1_49_0/libs/serialization/doc/shared_ptr2.html
Although #Arun provided great documentation references useful for utilizing std::shared_ptr, I instead chose to employ boost::shared_ptr with boost::serialization and it has cured my de-serialization problem.

Boost serialization throws "input stream error" after a certain file size

I am using Boost::serialization to read/write to a file which contains 3d models.
After a certain size of the file (around 8-12kb+), serialization throws an "Input Stream Error" exception every time I try to read the file. It works fine every single time untill then - for example I can read/write a 1kb model 12 times successfully, then on the 13th time and onwards it will throw exception!
Here's the code to serialize/deserialize:
JonsPackagePtr ReadJonsPkg(const std::string& jonsPkgName)
{
std::ifstream jonsPkgStream(jonsPkgName.c_str(), std::fstream::binary || std::fstream::in);
JonsPackagePtr pkg(new JonsPackage()); // "JonsPackagePtr" is a boost_shared_ptr typedef
if (jonsPkgStream && jonsPkgStream.good() && jonsPkgStream.is_open())
{
boost::archive::binary_iarchive iar(jonsPkgStream);
iar >> (*pkg.get());
}
return pkg;
}
bool WriteJonsPkg(const std::string& jonsPkgName, const JonsPackagePtr pkg)
{
std::ofstream outStream(jonsPkgName.c_str(), std::fstream::out | std::fstream::binary);
bool ret = false;
if (outStream.is_open())
{
boost::archive::binary_oarchive oar(outStream);
oar << (*pkg.get());
ret = true;
}
return ret;
}
Here's the contents that I archive:
/* PackageHeader definition */
struct PackageHeader
{
std::string mSignature;
uint8_t mMajorVersion;
uint8_t mMinorVersion;
PackageHeader();
};
/* PackageMesh definition */
struct PackageMesh
{
std::vector<float> mVertexData;
std::vector<uint32_t> mIndiceData;
PackageMesh();
};
/* PackageModel definition */
struct PackageModel
{
std::string mName;
std::vector<PackageModel> mChildren;
std::vector<PackageMesh> mMeshes;
Mat4 mTransform;
PackageModel();
};
/* JonsPackage definition */
struct JonsPackage
{
PackageHeader mHeader;
std::vector<PackageModel> mModels;
JonsPackage();
};
typedef boost::shared_ptr<JonsPackage> JonsPackagePtr;
JonsPackagePtr ReadJonsPkg(const std::string& jonsPkgName);
bool WriteJonsPkg(const std::string& jonsPkgName, const JonsPackagePtr pkg);
/* PackageHeader inlines */
inline PackageHeader::PackageHeader() : mSignature("jons"), mMajorVersion(LatestMajorVersion), mMinorVersion(LatestMinorVersion)
{
}
/* PackageModel inlines */
inline PackageModel::PackageModel() : mName(""), mTransform(1.0f)
{
}
/* PackageMesh inlines */
inline PackageMesh::PackageMesh()
{
}
/* JonsPackage inlines */
inline JonsPackage::JonsPackage()
{
}
Finally here's my non-intrusive serialization definitions:
namespace boost
{
namespace serialization
{
template<class Archive>
void serialize(Archive & ar, JonsEngine::PackageHeader& header, const unsigned int version)
{
ar & header.mMajorVersion;
ar & header.mMinorVersion;
ar & header.mSignature;
}
template<class Archive>
void serialize(Archive & ar, JonsEngine::PackageModel& model, const unsigned int version)
{
ar & model.mName;
ar & model.mChildren;
ar & model.mMeshes;
ar & model.mTransform;
}
template<class Archive>
void serialize(Archive & ar, JonsEngine::PackageMesh& mesh, const unsigned int version)
{
ar & mesh.mVertexData;
ar & mesh.mIndiceData;
}
template<class Archive>
void serialize(Archive & ar, JonsEngine::JonsPackage& pkg, const unsigned int version)
{
ar & pkg.mHeader;
ar & pkg.mModels;
}
template<class Archive>
void serialize(Archive & ar, glm::detail::tmat4x4<glm::mediump_float> transform, const unsigned int version)
{
ar & transform[0];
ar & transform[1];
ar & transform[2];
ar & transform[3];
}
template<class Archive>
void serialize(Archive & ar, glm::detail::tvec4<glm::mediump_float> vec, const unsigned int version)
{
ar & vec.x;
ar & vec.y;
ar & vec.z;
ar & vec.w;
}
} // namespace serialization
} // namespace boost
As I mentioned above, it is only after a certain file size (8-12kb+) that it starts throwing the exceptions when I try to read it. Why on earth is this and what could possibly cause it? It goes fine up untill this point...
Thanks
in ReadJonsPkg you should use bitwise or: std::fstream::binary | std::fstream::in

Boost.Serialization: Error on Calling Class Method during serialization process

I have the following problem when I try to call method in "TestSerialize" class during serialization process.
Here is my code:
class TestSerialize
{
public:
std::string GetVal() { return Val + "abc"; }
void SetVal(std::string tVal) { Val = tVal.substr(0, 2); }
protected:
std::string Val;
friend class boost::serialization::access;
template<class Archive> void save(Archive & ar, const unsigned int version) const
{
using boost::serialization::make_nvp;
std::string tVal = GetVal(); // Error here
ar & make_nvp("SC", tVal);
}
template<class Archive> void load(Archive & ar, const unsigned int version)
{
using boost::serialization::make_nvp;
std::string tVal;
ar & make_nvp("SC", tVal);
SetVal(tVal);
}
BOOST_SERIALIZATION_SPLIT_MEMBER();
};
int main()
{
TestSerialize tS;
std::ofstream ofs("test.xml");
boost::archive::xml_oarchive oa(ofs, boost::archive::no_header);
oa << BOOST_SERIALIZATION_NVP(tS);
ofs.close();
return 0;
}
The error that I encountered is:
'TestSerialize::GetVal' : cannot convert 'this' pointer from 'const TestSerialize' to 'TestSerialize &'
This error only happens on "save" but not "load"
I wonder why I get this error. I would like to know what Boost.Serialization do such that we have these two different behaviors.
I use Boost Library 1.47.0
save is a const function and can only call other const functions. GetVal isn't. Change it:
std::string GetVal() const { ... }

boost:serialization reconstruction (loading)

I'm using boost:serialization to save data structures into a file. The actual data is a pointer vector of classes and subclasses.
However the constructor of the class which is serialized takes as parameter another instantiated class Agent which is an object that controls communication with a simulation API (webots).
I see that in boost::serialization examples, the serializable objects need an empty constructor class() {}; to be used for the reconstruction. However this is impractical in my case. How can I use reconstruction but include the object which communicates with the API ?
One of the serializable classes has this constructor:
State(Agent &A, ACTION_MODE const& m);
and I've seen from examples in boost docs that I need something like this:
State() {};
Yet Agent &A must be passed as parameter.
Should I find a way around this (using an extern, a singleton, a global object) or is there a way to modify this behavior when reconstructing ? I'm sure I'm missing something here.
Thank you
EDIT:
Maybe I didn't explain this clearly enough. I get an error message when trying to "load" by reconstructing the serialized data.
error: no matching function to call State::State()
Which is what made me look into boost::serialize code, and think that it is calling a constructor or copy operator.
How do I make it use a specific constructor to serialize the data and take as an argument the Agent reference &a ?
EDIT#2:
template <class S, class P, class A> void Task<S,P,A>::save(const char* file)
{
std::ofstream ofs(file);
assert(ofs.good());
boost::archive::text_oarchive oa(ofs);
oa << states;
ofs.close();
}
template <class S, class P, class A> void Task<S,P,A>::load(const char* file)
{
std::ifstream ifs(file);
boost::archive::text_iarchive ia(ifs);
ia >> states;
ifs.close();
}
States is friend to boost::serialization::access and has a function serialize.
Saving works fine, loading is the problem.
states is: boost::ptr_vector<S> states; where S is a type of State polymorphic class.
State is the base class and has "serialize"
template <class Archive>
void State::serialize(Archive& ar, const unsigned int version)
{
ar & accel.Xaxis & accel.Yaxis & accel.Zaxis;
ar & gyro.Xaxis & gyro.Yaxis & gyro.Zaxis;
ar & gps.Yaxis;
ar & positions;
ar & reward & value & hash_value;
}
guState inherits from State.
template <class Archive>
void guState::serialize(Archive& ar, const unsigned int version)
{
ar & boost::serialization::base_object<State>(*this);
ar & accel.Xaxis & accel.Yaxis & accel.Zaxis;
ar & gyro.Xaxis & gyro.Yaxis & gyro.Zaxis;
ar & gps.Yaxis;
ar & positions;
ar & reward & value & hash_value;
}
accel, gyro, gps are simple structures with 3 double variables. They get serialized above^^.
Positions is an std::map<std::string,float> positions;
Looking at the serialized text file, everything appears Ok.
I cannot understand why it calls a constructor when trying to load the file.
EDIT#3:
Base Constructor is:
State(Agent &A, ACTION_MODE const& m);
Derived Constuctor is:
guState::guState(Agent& A, ACTION_MODE const& m) :
State(A, m)
{
...
}
Agent reference &A, kept in each State (or derived State) refers to an object obtained from the simulation API. It controls a robot. I cannot serialize it, and it doesn't make sense serializing it.
When I use:
namespace boost { namespace serialization {
template <class Archive>
void save_construct_data(Archive & ar,const guState* d,const unsigned int file_version)
{
ar << guState::caller;
ar << guState::mode;
}
template <class Archive>
void load_construct_data(Archive & ar, guState* d,const unsigned int file_version)
{
Agent &a;
ACTION_MODE &m;
ar >> a;
ar >> m;
::new(d) guState(a,m);
}
}
}
I get the following errors:
invalid use of non-static data member State::caller
invalid use of non-static data member State::mode
referring to the references used from the constructor.
And:
error: 'a' declared as reference but not initialized
error: 'm' declared as reference but not initialized
As you can see, it makes no sense to try and save the reference to Agent, because that reference (even if it could be saved or serialized) will probably be different every time the application is started.
And in loading the construct data, apart from me probably using the wrong syntax, it makes no sense to construct from a serialized reference to agent.
What I belive i need, is a way to tell the load_construct_data how to obtain a reference to an Agent (after initializing an agent object) and use that reference to construct the data.
Does that make any sense ? Do you think this is doable ?
EDIT#4
namespace boost { namespace serialization {
template <class Archive>
void save_construct_data(Archive & ar,const guState* d,const unsigned int file_version)
{
ar << guState::caller;
}
template <class Archive>
void load_construct_data(Archive & ar, guState* d,const unsigned int file_version)
{
Agent * a;
ACTION_MODE mode = RAND_SING;
ar >> a;
::new(d) guState(*a,mode);
}
}
}
It will not allow to serialize guState::caller
I have also made class Agent serializable, and overloaded the load_construct_data and save_construct_data of Agent in order to request from the simulation app a new instance of Agent to control the API.
EDIT:
There is a part of the manual that I think we both missed: The section non-default constructors here. To make it work you need to a save_construct_data and a load_construct_data function. There is a slight technicallity in terms of where these are friended explored here.
Also, you said you had this problem when trying to load only, but that you could save fine. This makes me think that you might have omitted
BOOST_CLASS_EXPORT_GUID(state, "state")
This omission might lead to segmentation faults once you get the load compiling (see the export section of the manual)
To make sure I was not mistaken, I made a compiling example, which I add in case its useful.
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/export.hpp>
#include <iostream>
#include <fstream>
//base class
struct base
{
base(double d) : m_d(d) {}
virtual double run() = 0;
private:
friend class boost::serialization::access;
double m_d;
template <class Archive>
void serialize(Archive& ar, const unsigned int version)
{
ar & m_d;
}
};
//forward declare the save construct data before friending it
// (something about friend being in a different namespace)
class derived;
namespace boost { namespace serialization {
template<class Archive>
inline void save_construct_data(Archive & ar, const derived * t, const unsigned int file_version);
}}
//derived class with non-default constructor
struct derived : public base
{
derived(double a , double b) :
base(a+b),
m_a(a),m_b(b),m_c(a*b)
{}
//some checks
double get_a() const {return m_a;}
double get_b() const {return m_b;}
double get_c() const {return m_c;}
double run(){return 1.0;}
private:
friend class boost::serialization::access;
template<class Archive>
friend void boost::serialization::save_construct_data(Archive & ar, const derived * t, const unsigned int file_version);
template <class Archive>
void serialize(Archive& ar, const unsigned int version)
{
ar & boost::serialization::base_object<base>(*this);
//only need to return c, a and b already done for constructor
ar & m_c;
}
double m_a, m_b, m_c;
};
//Save and load the data required for the constructor.
namespace boost { namespace serialization {
template <class Archive>
inline void save_construct_data(
Archive & ar,const derived* d,const unsigned int file_version
)
{
// save data required to construct instance
ar << d->m_a;
ar << d->m_b;
}
template <class Archive>
inline void load_construct_data(
Archive & ar, derived* d,const unsigned int file_version
)
{
double a,b;
ar >> a;
ar >> b;
// invoke inplace constructor to initialize instance of my_class
::new(d) derived(a,b);
}
}
}
//register the derived class with boost.
BOOST_CLASS_EXPORT_GUID(derived, "derived")
int
main (int ac, char **av)
{
std::ofstream ofs("filename");
base* p = new derived(2,3);
// save data to archive
{
boost::archive::text_oarchive oa(ofs);
oa << p;
}
// ... some time later restore the class instance to its orginal state
base* p2;
{
std::ifstream ifs("filename");
boost::archive::text_iarchive ia(ifs);
ia >> p2;
}
derived* d = static_cast<derived*>(p2);
std::cout<<"p2 vals are: "<<d->get_a()<<" "<<d->get_b()<<" "<<d->get_c()<<std::endl;
}
OLD RESPONSE:
Not sure I entirely understood your problem (a fuller example would help me) -
The constructor doesn't usually come into it when you serialize an object: you serialize the raw data?
Do you mean that you don't want to serialize all the raw data of the object, but just want to reconstruct it again when you deserialize the object (using the constructor)? If so then you can do this by seriazing the data that you need for the reconstruction and splitting the save and load operations.
struct my_class
{
my_class(Agent& A, ACTION_MODE const & m)
: m_state(A,M)
{}
private:
State m_state;
friend class boost::serialization::access;
void save(Archive & ar, const unsigned int version) const
{
// note, version is always the latest when saving
Agent tmp_A = m_state.get_A();
ACTION_MODE tmp_m = m_state.get_m();
ar & tmp_A;
ar & tmp_m;
}
template<class Archive>
void load(Archive & ar, const unsigned int version)
{
Agent tmp_A;
ACTION_MODE tmp_m
ar & tmp_A;
ar & tmp_m;
m_state = State(tmp_A,tmp_m);
}
BOOST_SERIALIZATION_SPLIT_MEMBER()
}
Does that help, or have I missed the point?