I am referring this : https://theboostcpplibraries.com/boost.serialization-class-hierarchies
I want to serialize object of my class, to std::vector<unsigned char>
class Person {
int id;
std::string name;
}
And then deserialize back to that class.
But I do not completely understand the implementation in the above URL. Can anybody help?
In such case it is best to see original documentation.
Blogs are often written by people who are in learning stage so in most cases they are not trustworthy (I don't say this is the case, but you should have limited trust to blogs written by someone you do not known).
The whole trick here is template method void serialize(Archive & ar, const unsigned int version).
Since it is a template once it is used to serialize and once to deserialize data.
Since code is doing writing and reading at the same time, stream operators << >> can't be used. Authors of library decided to use bit wise and operator & to express that it can do a reading and writing.
It looks strange like definition of a reference, but note on left side you have argument of method and or right side you have a field name.
Related
This might seem like an opinion question but I'm really looking for some good ways to go about doing this. so what's this? I basically want to have an abstract class named Repo let's say. This class is going to define the abstraction for what a Repo should be capable of doing. In this case, I just want to be able to save something, say you provide a name and data and it's supposed to store it for you. Then we can have a FileRepository that would save them on disc, S3Repository for example to store them in the AWS S3, or even MemoryRepository where we just save them in the memory.
Great, but how do I abstract this out? obviously I could get the bytes and each derived class would use their own stream to save it but what if the data is large and we don't want to load it up in the memory? let's say we want to save a 5GB file, we don't want to load that up in the memory.
I looked at the AWS SDK for C++ and it seems like they take a lambda with an ostream in it so you can write content. I tried to mimic something like that here so you can either just pass your istream, or give a lambda that takes an ostream and does whatever its heart desires.
Just wondering if there is a better way for this? it's often difficult to find good practices in c++ since there are a billion ways to do the same thing and many people do things very differently. I'd just love some insight here. I'm fairly new to C++ so a good explanation would be highly appreciated.
class Repo {
public:
virtual void add_with_ostream(const string& name, const std::function<void (ostream&)>& f) = 0;
template<typename T>
void add(const string& name, const T& data) {
this->add_with_ostream(name, [&data](ostream& output_stream) {
output_stream << data;
});
}
virtual void add_with_istream(const string&name, const istream& input_stream) {
this->add_with_ostream(name, [&input_stream](ostream& output_stream) {
output_stream << input_stream.rdbuf();
});
}
};
Edit: The question title was based on a deep missunderstanding of the compiler error I got. I was (sillyly) assuming the error was, that I tried to deserialize to an object I declared inside of the function. This was utterly wrong. I didn't do enough debug efforts myself or I could have found out what was wrong. So the title was quite missleading and I changed it. Thanks to Андрей Беньковский for helping.
I'm writing Serialization functions for 3D Models in my engine using cereal, which proves to be really efficient and easy to use.
So far everything worked great when I tested (de-)serializing a simple Mesh.
But now I'm trying to deserialize another class but ran in a problem I don't get.
void loadFile(std::string filepath)
{
DescriptionFile file;
{
ifstream stream = ifstream(filepath, std::ifstream::binary);
PortableBinaryInputArchive archive(stream);
archive(file);
stream.close();
}
}
This is my class, that should be deserialized:
struct DescriptionFile
{
public:
DescriptionFile(){}
map<string, MeshDescription*> meshDescriptions;
map<string, ModelDescription*> modelDescriptions;
public:
template<class Archive>
void serialize(Archive & archive)
{
archive(meshDescriptions, modelDescriptions);
}
};
It gives me compiler error:
Cereal does not support serializing raw pointers - please use a smart pointer
Even though it's not a pointer. In another part of the code something similar works just fine. I'd be glad if somebody could help me solve this.
I never used Cereal, but it looks like it expects you to use something like this:
map<string, unique_ptr<MeshDescription> >
To get std::unique_ptr I usually #include <memory>
From cereal documentation:
cereal supports serializing smart pointers but not dumb pointers (that is to say raw pointers, such as int *) or references. Pointer support can be found by including <cereal/types/memory.hpp>.
May be it means you have to include their header instead of the standard.
P.S. When your object owns resources (e.g. dynamically allocated MeshDescription) always delegate resource management (allocation, deallocation, copying, etc) to a separate class (smart pointer, collection, wrapper, etc). See rule of 0.
I'm working on a body of code which deals with a custom String implementation rather than the std::string (long story but for various reasons this has to be used) which I will refer to as "String" from here on.
I was able to easily pack the String without issue using the "raw" type to pack the raw char bytes and the size but I'm having problems unpacking it.
I was able to manually unpack it as shown below.
// before this I've unpacked to the point where the following object has the string
msgpack::object_kv& kv = obj.via.map.ptr[0];
// kv.key == the String I want
String key = String(key.via.raw.ptr, key.via.raw.size); // this works
But I want to use the built in >> operator or the .as template function and haven't been able to manage it. I don't have access to modify the String class to add the msgpack_unpack function nor to add MSGPACK_DEFINE
I tried creating a struct and giving it a msgpack_unpack function, but apparently it calls it with msgpack::object::implicit_type which my compiler replies with
error: 'struct msgpack::object::implicit_type' is private
And then I can't figure out any way of getting the msgpack::object out of the "implicit_type" object.
Any ideas?
So I figured it out! Thanks to hetepeperfan for the idea
After taking another crack at using operator>> overloading, the problem that prevented it from working before the strange way the operator>> being overloaded is called from the msgpack code.
namespace msgpack {
String& operator>>(msgpack::object o, String& v) {
v = String(o.via.raw.ptr, o.via.raw.size);
return v;
}
}
I needed to both use the msgpack namespace, and match the signature by returning the String by reference and take the object in NOT by reference. This seems a little ridiculous but it works and I can now use the built in "as<>" and convert functionality. awesome!
I have tried to read carefully all the advice given in the C++FAQ on this subject. I have implemented my system according to item 36.8 and now after few months (with a lot of data serialized), I want to make changes in both public interface of some of the classes and the inheritance structure itself.
class Base
{
public:
Vector field1() const;
Vector field2() const;
Vector field3() const;
std::string name() const {return "Base";}
};
class Derived : public Base
{
public:
std::string name() const {return "Derived";}
};
I would like to know how to make changes such as:
Split Derived into Derived1 and Derived2, while mapping the original Derived into Derived1 for existing data.
Split Base::field1() into Base::field1a() and Base::field1b() while mapping field1 to field1a and having field1b empty for existing data.
I will have to
deserialize all the gigabytes of my old data
convert them to the new inheritance structure
reserialize them in a new and more flexible way.
I would like to know how to make the serialization more flexible, so that when I decide to make some change in the future, I would not be facing conversion hell like now.
I thought of making a system that would use numbers instead of names to serialize my objects. That is for example Base = 1, Derived1 = 2, ... and a separate number-to-name system that would convert numbers to names, so that when I want to change the name of some class, I would do it only in this separate number-to-name system, without changing the data.
The problems with this approach are:
The system would be brittle. That is changing anything in the number-to-name system would possibly change the meaning of gigabytes of data.
The serialized data would lose some of its human readability, since in the serialized data, there would be numbers instead of names.
I am sorry for putting so many issues into one question, but I am inexperienced at programming and the problem I am facing seems so overwhelming that I just do not know where to start.
Any general materials, tutorials, idioms or literature on flexible serialization is most welcomed!
It's probably a bit late for that now, but whenever designing
a serialization format, you should provide for versionning.
This can be mangled into the type information in the stream, or
treated as a separate (integer) field. When writing the class
out, you always write the latest version. When reading, you
have to read both the type and the version before you can
construct; if you're using the static map suggested in the FAQ,
then the key would be:
struct DeserializeKey
{
std::string type;
int version;
};
Given the situation you are in now, the solution is probably to
mangle the version into the type name in a clearly recognizable
way, say something along the lines of
type_name__version; if the
type_name isn't followed by two underscore,
then use 0. This isn't the most efficient method, but it's
usually acceptable, and will solve the problem with backwards
compatibility, while providing for evolution in the future.
For your precise questions:
In this case, Derived is just a previous version of
Derived1. You can insert the necessary factory function into
the map under the appropriate key.
This is just classical versionning. Version 0 of Base has
a field1 attribute, and when you deserialize, you use it to
initialize field1a, and you initialize field1b empty.
Version 2 of Base has both.
If you mangle the version into the type name, as I suggest
above, you shouldn't have to convert any existing data. Long
term, of course, either some of the older versions simply
disappear from your data sets, so that you can remove the
support for them, or your program keeps getting bigger, with
support for lots of older versions. In practice, I've usually
seen the latter.
maybe Thrift Can help you do it.
I would like to implement a Serialization Class which takes in an object and converts it into a binary stream and stored in a file. Later, the object should be reconstructed from a file.
Though this functionality is provided by BinaryFormatter in C#, I would like to design my
own Serialization class from scratch.
Can someone point to some resources ?
Thanks in advance
I would like to give you a negative answer. It is less useful but it still may be.
I have been using boost serialization for several years and it was one of the greatest strategic mistakes of my company. It produces very large output, it is very slow, it propagates a whole bunch of dependencies making everything impossibly slow to compile, and then it is hard to get out because you have existing serialized formats. Further, it behaves differently on different compilers, thus upgrade from VS2005 to 2010 actually caused us to write a compatibility layer, which is also hard coz the code is very hard to understand.
Here are 2 solutions for C++ serialization:
Stephan Beal's s11n serialization library
boost serialization library
I personally only have experience with the 1st one and actually only used text based serializers, but i know that it's easy to define binary serializers for use with s11n.
I have been using boost::serialization library for a while now and I think it is very good. You just need to create the serialization code like this:
class X {
private:
std::string value_;
public:
template void serialize(Archive &ar, const unsigned int version) {
ar & value_;
};
}
No need to create the de-serialization code ( that's why they used the & operator ). But if you prefer you can still use the << and >> operators.
Also it's possible to write the serialization method for a class with no modification ( ex.: if you need to serialize an object that comes from a library ). In this case, you should do something like:
namespace boost { namespace serialization {
template
void serialize(Archive &ar, X &x const unsigned int version) {
ar & x.getValue();
};
}}
The C++ Middleware Writer may be of interest. It has performance advantages over the serialization library in Boost. It also automates the creation of serialization functions.