I want to give a shot to cereal serialization library . I am at beginner level c++ and retired level at java :)
I try to serialize vector with it but cant.
Any suggestion ?
I have below codes and gives an error. What am I missing here ?
struct personv1 {
matrix<float, 0, 1 > _descriptor;
string person_name;
string notes;
matrix<rgb_pixel> s_chips;
template<class Archive>
void serialize(Archive & archive)
{
// serialize things by passing them to the archive
archive( _descriptor, person_name,notes , s_chips );
}
};
std::vector<personv1> person;
and:
std::stringstream ss;
{
cereal::BinaryOutputArchive oarchive(ss); // Create an output archive
oarchive(person); // Write the data to the archive
} // archive goes out of scope, ensuring all contents are flushed
{
cereal::BinaryInputArchive iarchive(ss); // Create an input archive
iarchive(person); // Read the data from the archive
}
But it gives error:
---GUI-master-MAC/cereal/cereal.hpp:462: error: static_assert failed "cereal could not find any output serialization functions for the provided type and archive combination.
Types must either have a serialize function, load/save pair, or load_minimal/save_minimal pair (you may not mix these).
Serialize functions generally have the following signature:
template<class Archive>
void serialize(Archive & ar)
{
ar( member1, member2, member3 );
}
static_assert(traits::detail::count_output_serializers<T, ArchiveType>::value != 0,
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Related
I'm trying to load parameters from a JSON file into my own parameter class using cereal (v1.12) on VS2013.
Compilation and loading into a POD works fine, but loading into my own class throws cereal::RapidJSONException.
Here is my code:
#include <iostream>
#include "cereal/archives/json.hpp"
struct CMyStruct {
CMyStruct(int f_value) : m_value(f_value) {}
int m_value;
};
template<class Archive> void load(Archive& ar, CMyStruct& parameter) {
int input;
ar(input);
parameter.m_value = input;
};
void test_cereal() {
int l_scalar_i(42);
CMyStruct l_scalar(42);
std::ifstream l_jsonFile("example_minimal.json");
cereal::JSONInputArchive l_ar(l_jsonFile);
l_ar( cereal::make_nvp("scalar", l_scalar_i) ); // WORKS
l_ar( cereal::make_nvp("scalar", l_scalar) ); // throws cereal::RapidJSONException
return;
}
The JSON file:
{
"bool": false,
"scalar": 3
}
Following the call stack shows that document.h:244 inside RapidJSON throws the exception:
ConstMemberIterator MemberEnd() const
{ RAPIDJSON_ASSERT(IsObject()); return data_.o.members + data_.o.size; }
However, I have no idea what to make of it. How can I make my code work?
Thanks a lot!
UPDATE WITH SOLUTION:
As written by ChrisBFX, it is necessary to implement load_minimal/save_minimal in order to deserialize an C++ object to a JSON scalar.
load/save on the other hand serializes a C++ object from/to a JSON object.
For my code, the load() function has to be removed and replaced by this:
template<class Archive>
int save_minimal(const Archive& ar,
const CMyStruct& parameter)
{
return parameter.m_value;
};
template<class Archive>
void load_minimal(const Archive& ar,
CMyStruct& parameter,
const int & value_i)
{
parameter.m_value = value_i;
};
Now it works like a charm! :-)
Your CMStruct is not a scalar but an object. So cereal expects
{
"bool": false,
"scalar": {
"value": 3
}
}
If you want to serialize your struct as a scalar you have to provide a minimal_load function for it. See http://uscilab.github.io/cereal/serialization_functions.html "minimal split serialization".
TLDR: I would like to transition a class serialisation from implementation level object_class_info to object_serializable, keeping compatibility with old data files while always writing files in the new format.
Rationale: When I started using the Boost Serialization library, I didn’t realise that it already came with a header (complex.hpp) to serialise std::complex<double>. Instead, I wrote my own, free-standing serialisation function:
namespace boost { namespace serialization {
template<class Archive, typename T>
void serialize(Archive& ar, std::complex<T>& comp, const unsigned int version) {
ar & reinterpret_cast<T(&)[2]>(comp)[0];
ar & reinterpret_cast<T(&)[2]>(comp)[1];
}
}
This by default enables version and class info tracking, which slows code down quite a bit. The serialization function which comes with Boost is a fair bit faster.
I would now like to transition to using the Boost version always when writing out new data files, but still be able to read in old data files. Reading in new files with an old binary is not an issue.
The problem is that the new serialisation is not versioned (obviously). Further, I don’t even see how I could attempt to read in an archive using the old version of the code and immediately write it out again using the new version, as the deserialisation/serialisation traits are global properties.
What would be the best way to either a) transparently read in old and new files while always writing new files or b) reading in an old file and immediately writing it out in the new format?
You can keep the old implementation and use it if the file version is "old".
Use the boost version of complex serialization only when saving or when the file version is "new".
This should be trivial if you have an object containing the complex data to serialize, because can bump the version on the containing object to achieve this
UPDATE
Sample, using a simple wrapper to invoke the old style of serialization:
Live On Coliru
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/complex.hpp>
template <typename T> struct old_format_wrapper {
T& wrapped;
old_format_wrapper(T& w) : wrapped(w) {}
};
template <typename T>
old_format_wrapper<T> old_format(T& w) { return {w}; }
namespace boost { namespace serialization {
template<class Archive, typename T>
void serialize(Archive& ar, old_format_wrapper<std::complex<T> >& comp, unsigned) {
ar & reinterpret_cast<T(&)[2]>(comp.wrapped)[0];
ar & reinterpret_cast<T(&)[2]>(comp.wrapped)[1];
}
} }
struct IHaveComplexData {
std::complex<double> data;
template <typename Ar> void serialize(Ar& ar, unsigned version) {
switch(version) {
case 0: { // old
auto wrap = old_format(data);
ar & wrap;
}
break;
case 1: // new
default:
ar & data; // uses boost serialization
break;
}
}
};
int main() {
{
boost::archive::text_oarchive oa(std::cout);
IHaveComplexData o { { 2, 33 } };
oa << o;
}
{
std::istringstream iss("22 serialization::archive 13 0 0 0 0 2.00000000000000000e+00 3.30000000000000000e+01");
boost::archive::text_iarchive ia(iss);
IHaveComplexData o;
ia >> o;
std::cout << o.data;
}
}
Prints (depending on your boost version):
22 serialization::archive 13 0 0 0 0 2.00000000000000000e+00 3.30000000000000000e+01
(2,33)
Of course, you can now set BOOST_CLASS_VERSION(IHaveComplexData, 1)
I loaded multiple data into boost::archive::text_oarchive, now I need to extract the data.
But because the archive contains multiple records, I would need an iterator.
something like
//input archive
boost::archive::text_iarchive iarch(ifs);
//read until the end of file
while (!iarch.eof()){
//read current value
iarch >> temp;
...//do something with temp
}
is there any standard way to iterate over elements of the archive?
I found only iarchive.iterator_type, but is it what I need and how do I use it?
The iterator type you are looking at actually comes from
class shared_ptr_helper {
...
typedef std::set<
boost::shared_ptr<const void>,
collection_type_compare
> collection_type;
typedef collection_type::const_iterator iterator_type;
which is used during the load of the archive rather than being an iterator for external use I think.
If you look at the link http://www.boost.org/doc/libs/1_53_0/libs/serialization/doc/index.html under tutorial -> STL Collection you will see the following example:
#include <boost/serialization/list.hpp>
class bus_route
{
friend class boost::serialization::access;
std::list<bus_stop *> stops;
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
ar & stops;
}
public:
bus_route(){}
};
If that isn't quite what you need then you would probably need to look at overriding load and save as per http://www.boost.org/doc/libs/1_53_0/libs/serialization/doc/tutorial.html#splitting
and adding handling as required.
I have been trying to make this work for a long time now.
In my project there are 6 classes that are being serialized using the exact tutorial from boost, by implementing the template function serialize.
Those classes are: State, guState, Policy, Action, Param, Vec3D.
When I serialize and save them, it works fine. I get indeed a text file, with various numbers and strings in it.
No complains, no warnings, no exceptions thrown. The only case is that if I try to serialize a pointer member of a class, the hole process becomes a zombie. So I do not try doing so, and saving works.
When however I try loading, I get:
terminate called after throwing an instance of
'boost::archive::archive_exception' what(): stream error
Now the interesting part is that I serialize two boost::ptr_vectors, one which consists of State pointers and one which consists of Policy pointers.
The state vector, I have saved and loaded without a problem.
The policy vector, I can save, but when I try loading i get the exception.
Furthermore, after reading the boost tutorials, I was under the impression that In order to load I did not need anything else other than the serialize function.
However when I tried loading, boost serialization would complain for not finding a default constructor such as State(), Policy(), etc (I implement my own constructors in each class ).
After reading this tutorial here I implemented a default constructor which does nothing, so that boost-serialization would work. Indeed it did compile, and I got the results mentioned above.
I have tried going down a much complicated road seen in my old question here where I tried seperating and implementing save_construct_data and load_construct_data, but I found this too instrusive, again I got the exact error as above.
Can someone please help me, explain how the loading works, what is the deal with the default constructors ? Or at least point me to a link that might be helpfull. I have gone through the manuals in boost, and they do not explain much about reconstruction.
Thank you.
Edit (Added a few snippets)
class State
{
friend class boost::serialization::access;
template<class Archive>
void serialize(Archive & ar, const unsigned int version);
protected:
std::map<std::string,float> positions;
float reward;
std::size_t hash_value;
bool exists(const Action* A);
bool operator== (State const& S);
std::size_t hash();
void clean_noise();
State(){}; // NOTE: This is used only by serializer, without it, code won't compile
public:
enum position { standing, on_chest, on_back, on_left, on_right, getting_up };
position current_position;
Policy *my_policy;
Vec3D gps;
boost::ptr_vector<Action> actions;
State(ACTION_MODE &m);
~State();
bool operator== (State const* S);
bool operator< (State const* S) const ;
const float& getR() const;
bool addAction(Action *A);
Action* findAction(const Action *A);
boost::ptr_vector<Action>& getAllActions();
void printState();
virtual bool isTerm();
};
template <class Archive>
void State::serialize(Archive& ar, const unsigned int version)
{
ar & BOOST_SERIALIZATION_NVP(positions);
ar & BOOST_SERIALIZATION_NVP(gps);
ar & BOOST_SERIALIZATION_NVP(current_position);
ar & BOOST_SERIALIZATION_NVP(reward);
ar & BOOST_SERIALIZATION_NVP(hash_value);
ar & BOOST_SERIALIZATION_NVP(actions);
ar & BOOST_SERIALIZATION_NVP(my_policy);
}
Other Classes inheriting from State, have also their serialize functions, using:
ar & boost::serialization::base_object<State>(*this);
Class Policy:
class Policy
{
friend class boost::serialization::access;
template<class Archive>
void serialize(Archive & ar, const unsigned int version);
Policy() {}; // NOTE: Again same as with state, used only by serialize load
protected:
float QValue;
State *state;
public:
//! Base class constructor
Policy(State *s);
...
};
template <class Archive>
void Policy::serialize(Archive& ar, const unsigned int version)
{
ar & BOOST_SERIALIZATION_NVP(action);
ar & BOOST_SERIALIZATION_NVP(state);
ar & BOOST_SERIALIZATION_NVP(QValue);
ar & BOOST_SERIALIZATION_NVP(r);
}
As you can see those are the two main classes, Other classes are also serialized because of reconstruction dependencies ( class Action, class Param, etc )
The master Class:
template <class S, class P> class Task
{
protected:
...
//! Container of states of type S (template parameter)
boost::ptr_vector<S> states;
//! Container of policies of type P (template parameter)
boost::ptr_vector<P> policies;
...
public:
Task(Agent &a, ACTION_MODE &m);
...
void save_to_file();
void load_from_file(std::string filename);
};
template <class S, class P>
void Task<S,P>::save_to_file()
{
std::string output = ramdisk+"serialized";
char *file = (char*)output.c_str();
std::ofstream ofs(file);
assert(ofs.good());
boost::archive::text_oarchive oa(ofs);
oa << states;
oa << policies;
ofs.close();
}
template <class S, class P>
void Task<S,P>::load_from_file(std::string filename)
{
char *file = (char*)output.c_str();
std::cout << file << std::endl;
std::ifstream ifs(file);
boost::archive::text_iarchive ia(ifs);
ia >> states;
ia >> policies;
ifs.close();
}
Effectively contains the two boost::ptr_vectors which hold the States and Policies.
States are saved and loaded without a problem.
The problem arises when loading policies. Saving them does not seem to create a problem (but then again I might be wrong).
Having tested save/load without policies, and with, The issue appears to be with policy reconstruction.
Note the default constructors used only by serialization, without which the code will not compile.
EDIT#2: After running application using valgrind and memcheck, it reports that there is a pointer memory leak. However since I am not good at debugging with valgrind I can't really tell where the leak is occuring or if it is relevant to my serialization (I think it is).
The problem is that you are serializing states and policies while Policy also holds references to the same instances of State. You can only serialize classes which don't have such cross-references. Boost should throw a pointer-conflict exception when writing to the file. In my tests it dependet on the order of writes if the exception was thrown or not - which is unfortunate because loading fails even if writing succeeded.
A workaround would be to delete the line oa << states when saving and loading and fix up the pointers by hand in a post-loading step.
About the constructors: it's mostly something the boost-api needs to do it's template magic. However when using versioning it's important to specify default-values for your member-variables so they are not left uninitialized when loading files with an older version-number.
I want to write a serialize function for a class that can optionally compress the data. I would like to use the compression facilities provided in boost::iostreams. Does anyone know how to do this?
struct X
{
X() {}
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
ar & compression;
if(compression == 0)
{
ar & data;
}
else if(compression == 1)
{
// use boost::iostream compression
// facilities to serialize data
}
}
int compression;
std::vector<int> data;
};
The only way I can see to do that is compress the data first and then use ar.load_binary and ar.save_binary. To compress the data, you could use a filtering_stream with std::ostringstream as sink and an appropriate compression filter.
Any reason you don't want to push the compression down the stack (that is, build your archive over a compressing stream)?