I have a core data structure that I am loading values into: it is a hashmap of vectors. The vectors contain a struct, however. Further, the struct uses a template type.
I need to serialize this data structure and save to the disk periodically. Then, later--in a different program--I need to load the serialized data structure.
Here is a streamlined version of the structs. I minimally define them, but there are other data items (members), in addition to this bare bones version.
#include<vector>
#include<string>
#include<map>
#include<fstream>
#include<stdlib.h>
#include <boost/serialization/vector.hpp>
#include <boost/serialization/map.hpp>
#include <boost/archive/text_oarchive.hpp>
using namespace std;
template<typename T>
struct DataUnit{
size_t time;
string transaction_string;
T transaction;
}
template<typename T>
struct DataStructure{
map<string transaction_hash, vector<DataUnit<T>> > hashmap;
int max_transactions;
// I have a method to add stuff, but omitted for readability
}
I started with the first struct, DataUnit and modified it as follows:
template<typename T>
struct DataUnit{
size_t time;
string transaction_string;
T transaction;
template<class Archive>
void serialize(Archive & ar, const unsigned int version){
ar & time;
ar & transaction;
ar & transaction_string;
}
};
Eventually, I need to serialize the data structure. However, when I run just this with the following code:
int main(){
DataUnit<int> hi;
hi.time = time(NULL);
hi.transaction = 1;
hi.transaction_string = "world";
return 0;
}
The world blows up with errors from boost. As far as I can tell, I followed the tutorial example exactly. How do I boost-serialize these objects?
Some of the errors (but there are so many I can't believe it isn't something fundamental...):
In function `boost::archive::text_oarchive::text_oarchive(std::ostream&, unsigned int)
undefined reference to `boost::archive::text_oarchive_impl::text_oarchive_impl(std::ostream&, unsigned int)'
last error:
undefined reference to `boost::archive::archive_exception::~archive_exception()'
and it goes on from there...but I don't see where I have lacked any includes...(boost was installed via Cygwin)...
(running the code as an administrator...the text file I am outputting is present, with read write permissions...the ofs object is being created successfully)...
Currently, totally out of ideas... (tried linking lboost_serialization, reinstalling boost) No idea if I am missing something from the code ^^^
The problem is the order of your dependencies on the build command like. You need to list dependencies after the modules that use them. Also you don't compile .h files. They should be included in the .cpp files that use them. Try this command:
g++ -std=c++11 main.cpp hashmap_transaction.cpp -o run.exe -lboost_serialization
Related
I'm trying to serialize my abstract class according to those questions:
Get private data members for non intrusive boost serialization C++
Error serializing an abstract class with boost
Error serializing an abstract class with boost
My neuron.h looks like this:
class Neuron {
public:
struct access;
API virtual ~Neuron();
API virtual double activate( double x, double b ) = 0;
};
I have to keep all the Boost related members in neuron.cpp to prevent including Boost headers when using neuron.h in some other codes.
My neuron.cpp looks like this:
#include "Neuron.h"
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
BOOST_SERIALIZATION_ASSUME_ABSTRACT(Neuron);
struct Neuron :: access {
template <class Archive>
static void serialize(Archive &ar, Neuron& n, const unsigned int version) {}
};
namespace boost {
namespace serialization {
template<class Archive>
void serialize(Archive & ar, Neuron& n, const unsigned int version)
{
Neuron::access::serialize(ar, n, version);
}
} // namespace serialization
} // namespace boost
Neuron::~Neuron() {
}
The problem is, that when I'm using its inherited classes elsewhere, I'm getting the error
***/boost/boost/serialization/access.hpp:116:11: error: ‘class Neuron’ has no member named ‘serialize’
What am I doing wrong here?
I think the key here is "when I'm using its inherited classes elsewhere". Correct me (and your question, please) if I'm wrong, but this suggests that you are getting the compile error while compiling a source file other than neuron.cpp.
This makes sense, given what the compiler has to work with. You might have noticed that changes to one source file tend to not require re-compiling other source files. So adding something -- like an overload of serialize() -- to neuron.cpp does not change how other translation units are compiled. (It can change how everything is linked together in the end, but we're not there yet.) If another translation unit tries to serialize Neuron, the stuff in neuron.cpp does not matter. The compiler is not aware of an appropriate overload of serialize(), so serializing Neuron in another source file results in intrusive serialization. That is, the compiler will look for a member function of Neuron called serialize().
In order for your overload of serialize() to affect how other translation units are compiled, it needs to be declared in a header file.
Since you cannot put Boost stuff in neuron.h, you might have to create a new header file, say neuron_boost.h. This file would #include "neuron.h" then provide the declarations needed for Boost serialization. Source files that serialize descendants of Neuron would include neuron_boost.h while other source files could continue to include the original neuron.h.
Related to this. I'd like to avoid using global variables so I resorted to using structs with enum and std::string[] (see link) in order to build menus for a small application. I would also like to have these enums in a separate header file. The selected answer in the link implies using --std=c++17, which I'd like to avoid, at least for now, and decided to use a static const std::string[] -- no need to include extra array or vector since this is initialized once, never modified, only called, ALL is always known.
As other answers on this have made it clear, I need to either initialize A::names outside the struct, or use a static const std::string& setter (see this, for example). But all the answers so far dealt with a std::string, not an array, std::string[].
This is a simple example of what I tried. It simply tries to print the contents of A::names using a for() loop iterating through the enum in struct A:
a.h:
#ifndef A_H_INCLUDED
#define A_H_INCLUDED
#include <string>
struct A
{
enum E { ONE, TWO, ALL };
static const std::string names[ALL];
};
#endif // A_H_INCLUDED
a.cpp:
#include "a.h"
static const std::string A::names[A::ALL] { "one", "two" };
main.cpp:
#include "a.h"
#include <iostream>
int main()
{
for(int i=A::ONE; i<A::ALL; ++i)
std::cout << A::names[i] << '\n';
return 0;
}
The error after g++ main.cpp is:
main.cpp:(.text+0x24): undefined reference to `A::names[abi:cxx11]'
collect2: error: ld returned 1 exit status
Seeing the cxx11, I thought g++ --std=c++11 main.cpp would solve it, but it doesn't.
So, what am I doing wrong, or, how could I adapt the version with the setter to return an array, std::string[]? My goal is to have an alternative to a global variable, that has only one instance in memory no matter how many calls.
Here's an adapted code, from a small program, on how I would build a menu using struct with enum and string (menu_design = new QMenu... and menuDesignAction() is the function that updates):
for(unsigned char i=0; i<A::ALL; ++i) // needs initializing
{
QAction *tmpAction {new QAction(tr(A::names[i].c_str()))};
tmpAction->setObjectName(QString("%1").arg(i));
connect(tmpAction, SIGNAL(triggered(bool)), this, SLOT(menuDesignAction()));
menu_design->addAction(tmpAction);
}
As a side-note, in the snippet above, I have to use .c_str(), but I am using a std::string in the enum. If I could make it *char[] instead of std::string[], would I avoid extra calls? If I am not wrong, how could the answers to my problem (assuming there are) be adapted so as to be able to fit somehow in the Qt snippet?
Is it possible to use a polymorphic_binary_iarchive to deserialize an object serialized with binary_oarchive?
My library has been written using binary archives for all the serialisation methods (as well as EOS portable archives). This results in overloaded serialize methods everywhere, significant code bloat and link time cost. I'd like to switch to polymorphic archive methods to reduce the number of exposed serialize methods and facilitate easy use of other archives (in particular XML). However, I would also like to maintain backwards compatibility with already serialized data.
polymorphic_binary_iarchive seems to work usually, but there is at least one case where it fails: vector.hpp changes the method of serialization depending on use_array_optimization, which for binary archives is set for any type that is_bitwise_serializable. So types that contain vector<size_t> fail to deserialize.
I'm tempted to try to specialize use_array_optimization for polymorphic_binary_iarchive (and the oarchive similarly). Is there any reason this would not work, or any other reasons why this is just A Bad Idea(TM)?
Edit
I tried that specialization:
struct use_array_optimization_in_polymorphic_archive {
template <class T>
struct apply : public boost::serialization::is_bitwise_serializable< T > {};
};
namespace boost {
namespace serialization {
template <>
struct use_array_optimization<boost::archive::polymorphic_iarchive> {
template <class ValueType>
struct apply : boost::mpl::apply1<
use_array_optimization_in_polymorphic_archive,
BOOST_DEDUCED_TYPENAME boost::remove_const<ValueType>::type>::type {};
};
}}
But it doesn't work because array.hpp then expects polymorphic_iarchive to implement load_array, which is only implemented for a basic_binary_iprimitive.
I'm also concerned that this will change behaviour for all polymorphic_iarchive implementations, not just the polymorphic_binary_iarchive. More thought required...
Edit2
Herefollows some code by way of demonstration. Toggle POLY_ON to use the polymorphic archive to deserialize; this works for doubles. Toggle VEC_ON to use vectors, demonstrating the problem. NB: I haven't yet double checked that this is exactly the same problem, but I'm reasonably sure it is. NNB: This is using Boost 1.59.
#include <fstream>
#include <boost/archive/binary_oarchive.hpp>
// #define POLY_ON
#ifdef POLY_ON
#include <boost/archive/polymorphic_binary_iarchive.hpp>
#else
#include <boost/archive/binary_iarchive.hpp>
#endif
// #define VEC_ON
#ifdef VEC_ON
#include <vector>
#include <boost/serialization/vector.hpp>
#endif
class bank_balance {
private:
friend class boost::serialization::access;
template <class archive>
void serialize(archive& ar, const unsigned int version) {
ar & date_;
ar & rate_;
}
#ifdef VEC_ON
std::vector<double> date_;
std::vector<double> rate_;
#else
double date_;
double rate_;
#endif
public:
bank_balance() : date_(0) {}
bank_balance(
#ifdef VEC_ON
std::vector<double> date, std::vector<double> rate
#else
double date, double rate
#endif
)
: date_(date), rate_(rate)
{}
bool operator==(const bank_balance& other) const {
return date_ == other.date_ && rate_ == other.rate_;
}
};
int main() {
std::ofstream ofs("bank_balance.ser");
#ifdef VEC_ON
const bank_balance balance({45367, 45369}, {5.6, 2.43});
#else
const bank_balance balance(45367, 5.6);
#endif
{
boost::archive::binary_oarchive oa(ofs);
oa << balance;
}
bank_balance balance2;
{
std::ifstream ifs("bank_balance.ser");
#ifdef POLY_ON
boost::archive::polymorphic_binary_iarchive ia(ifs);
#else
boost::archive::binary_iarchive ia(ifs);
#endif
ia >> balance2;
}
if (balance == balance2) std::cout << "ok\n";
else std::cout << "dammit\n";
return 0;
}
Is it possible to use a polymorphic_binary_iarchive to deserialize an object serialized with binary_oarchive?
Short answer: yes.
The only difference is the call-site interface here.
EDIT
Perhaps, unintentionally, this "promise" was broken when they introduced optimized serialization for POD containers.
Here's my analysis, with yours.cpp from your question, and mine.cpp as edited below:
{
std::ofstream ofs("bank_balance.ser");
#ifdef POLY_ON
boost::archive::polymorphic_binary_oarchive oa(ofs);
#else
boost::archive::binary_oarchive oa(ofs);
#endif
oa << balance;
}
I compile all flavours with the following command line:
for src in yours mine; do for a in {,-DPOLY_ON}\ {,-DVEC_ON}; do time g++ $a -O2 -std=c++11 $src.cpp -lboost_{system,serialization} -o "$src${a//[D _]/}.exe"; done; done
Which results in mine.exe, mine-POLYON.exe, mine-POLYON-VECON.exe, mine-VECON.exe, yours.exe, yours-POLYON.exe, yours-POLYON-VECON.exe and yours-VECON.exe. Running them:
(set -x; for a in ./*.exe; do $a; done)
Results in
+ ./mine.exe
ok: true
+ ./mine-POLYON.exe
ok: true
+ ./mine-POLYON-VECON.exe
ok: true
+ ./mine-VECON.exe
ok: true
+ ./yours.exe
ok
+ ./yours-POLYON.exe
ok
+ ./yours-POLYON-VECON.exe
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append
+ ./yours-VECON.exe
ok
Note that all combinations are fine if you write using the same archive implementation as while reading. You're also right that sadly ./yours-POLYON-VECON.exe is the only one to break. I think this is unintentional but your hunch could be spot on:
doc
Note that the concept of polymophic archives is fundamentally incompatible with the serialization of new types that are marked "primitive" by the user with:
BOOST_CLASS_IMPLEMENTATION(my_primitive_type, boost::serialization::primitive_type)
Code to implement serialization for these types is instantiated "on the fly" in the user's program. But this conflicts with the whole purpose of the polymorphic archive. An attempt to serialize such a primitive type will result in a compilation error since the common polymorhic interface is static and cannot instantiate code for a new type.
It looks like the vector optimization path might be sharing one of these code paths.
RECOMMENDATION
I'd recommend making a conversion tool to convert old format files to the new format. You can read using the non-polymorphic iarchive and write using the polymorphic archive. That, of course, means you'll have to compile both approaches for this release, but
it doesn't have to be baked into the main executable(s) - only the conversion tools does need the "old" method
the conversion tool will not have to be versioned, it stays the same for all future releases; this means you can stop building new versions and drop the required extra code.
I know this is an old topic, but I've been fighting with this precise problem, and think I have found a good solution.
The core of the problem is that polymorphic archives never get dispatched to use the optimized load/save versions in boost/serialization/vector.hpp. How I fixed this was to overload those dispatch functions for the polymorphic archive types like this:
namespace boost::serialization
{
template<class U, class Allocator>
inline void load(boost::archive::polymorphic_iarchive& ar, std::vector<U, Allocator>& t, const unsigned int file_version)
{
using use_optimized = boost::has_trivial_constructor<U>::type;
load(ar, t, file_version, use_optimized());
}
template<class U, class Allocator>
inline void save(boost::archive::polymorphic_oarchive& ar, const std::vector<U, Allocator>& t, const unsigned int file_version)
{
using use_optimized = boost::has_trivial_constructor<U>::type;
save(ar, t, file_version, use_optimized());
}
}
I cannot seem to get boost::serialization to work well. I have sprinkled the class .cpp files with BOOST_CLASS_EXPORT_GUID macros, after including the class and archive headers, but I still get unregistered_class exceptions.
I have looked around and it seems that whatever I can find is 1. either outdated, dealing with old versions of the library, or 2. works only for a simple one-file solution in which all the serializable classes are defined one after the other. Nothing I've found helps.
My solution at present consists of a project, compiled into a static library, that contains the core functionality with the basic archivable classes, and another test project that will eventually be fleshed out into a more concrete logic layer. Getting everything to work with boost::serialization is proving a nightmare. I'm almost tempted to write it myself.
Anyway, the class in question which raises the exception is defined in a header, which looks something like this:
#include <boost/serialization/assume_abstract.hpp>
#include <boost/serialization/serialization.hpp>
// Other includes...
namespace GameCore { class Component; }
// Forward declare some boost::serialization functions that appear at the bottom.
// ...
BOOST_SERIALIZATION_ASSUME_ABSTRACT(GameCore::Component);
namespace GameCore
{
// Some forward declares..
//////////////////////////////////////////////////////////////////////////
// Base component type.
//////////////////////////////////////////////////////////////////////////
class Component : public Updatable, public Object
{
friend class boost::serialization::access;
protected:
template <typename Archive>
friend void boost::serialization::serialize(Archive& archive, Component& object, const unsigned int version);
template <typename Archive> friend void boost::serialization::load_construct_data(Archive& archive, Component* t, const unsigned int version);
template <typename Archive> friend void boost::serialization::save_construct_data(Archive& archive, const Component* t, const unsigned int version);
public:
Component(GameObject& owner);
virtual ~Component() = 0;
// Irrelevant stuff..
GameObject& gameObject;
Transform* transform;
};
}
// The component includes have to be placed here because it would otherwise create a cyclic inclusion when trying to compile the
// individual component classes, say, Transform, which would end up including itself.
#include "Transform.h"
namespace boost
{
namespace serialization
{
template<class Archive>
inline void save_construct_data(Archive& archive, const GameCore::Component* t, const unsigned int version)
{
archive << t->gameObject;
}
template<class Archive>
inline void load_construct_data(Archive& archive, GameCore::Component* t, const unsigned int version)
{
// Retrieve data from archive required to construct new instance.
GameCore::GameObject owner;
archive >> owner;
// Invoke inplace constructor to initialize instance of class.
::new(t)GameCore::Component(owner);
}
//////////////////////////////////////////////////////////////////////////
// Serialization function for save/load.
//////////////////////////////////////////////////////////////////////////
template <typename Archive>
void serialize(Archive& archive, GameCore::Component& t, const unsigned int version)
{
archive & boost::serialization::base_object<GameCore::Object>(t);
archive & boost::serialization::base_object<GameCore::Updatable>(t);
archive & t.gameObject;
archive & t.transform;
}
}
}
That is one header file. Sorry for the verbosity. Its .cpp file starts like this:
#include "Component.h"
#include <boost/serialization/export.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>
// Other includes...
BOOST_CLASS_EXPORT_GUID(GameCore::Component, "GameCore::Component");
// Class method definitions here.
The exception is raised when I try to archive an instance object that derives from Component, which itself is an abstract class. I'm archiving through a simple method defined in a different manager class:
std::ofstream outputFile(fileName);
boost::archive::text_oarchive outputArchive(outputFile);
outputArchive << objects;
where objects is a std::list of Objects. Object is the base class from which all things, including Components, derive.
I apologize if this sounds convoluted, but there's only three layers of inheritance, and I believe I had a neat and effective architecture before thoughts of serialization crept in.
If you could help me get rid of the irrational unregistered_class exceptions I'll light a candle for your souls!
Update: Funny thing is, the exception isn't raised for all derived classes of Component.
After frying my neurones looking for an answer, I stumbled upon this line in the documentation:
Static Libraries and Serialization
Code for serialization of data types can be saved in libraries just as
it can for the rest of the type implementation. This works well, and
can save huge amount of compilation time. Only compile serialization
definitions in the library. Explicitly instantiate serialization code
for ALL archive classes you intend to use in the library. For exported
types, only use BOOST_CLASS_EXPORT_KEY in headers. For exported types,
only use BOOST_CLASS_EXPORT_IMPLEMENT in definitions compiled in the
library. For any particular type, there should be only one file which
contains BOOST_CLASS_EXPORT_IMPLEMENT for that type. This ensures that
only one copy of serialization code will exist within the program. It
avoids wasted space and the possibility of having different versions
of the serialization code in the same program. Including
BOOST_CLASS_EXPORT_IMPLEMENT in multiple files could result in a
failure to link due to duplicated symbols or the throwing of a runtime
exception.
Splitting BOOST_CLASS_EXPORT into BOOST_CLASS_EXPORT_KEY and BOOST_CLASS_EXPORT_IMPLEMENT seems to work.
I am trying to serialize a simple class with Plain-old-Data types using boost serialization. However, my only requirement is that I cannot use RTTI. Thus I am compiling with -fno-rtti using gcc 4.4.1 for ARM Linux with latest Boost 1.47 library.
So here is my class:
#include <boost/archive/xml_oarchive.hpp>
#include <boost/archive/xml_iarchive.hpp>
class libNemoMemento
{
friend class boost::serialization::access;
private:
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
ar & BOOST_SERIALIZATION_NVP(temperature);
ar & BOOST_SERIALIZATION_NVP(voltage);
ar & BOOST_SERIALIZATION_NVP(bandwidth);
ar & BOOST_SERIALIZATION_NVP(power);
}
int temperature;
unsigned int voltage;
unsigned int bandwidth;
unsigned char power;
public:
libNemoMemento(void) {}
virtual ~libNemoMemento(void) {}
};
I haven't even implemented the actual save and load functions yet (they seem to be pretty straightforward from looking at the Boost documentation) but I am already getting the following compiler errors:
In file included from /home/me/prebuild/third-party/boost/include/boost/serialization/detail/shared_ptr_132.hpp:29,
from /home/me/third-party/boost/include/boost/serialization/shared_ptr_132.hpp:35,
from /home/me/third-party/boost/include/boost/archive/shared_ptr_helper.hpp:29,
from /home/me/third-party/boost/include/boost/archive/xml_iarchive.hpp:133,
from serialize_test.h:21,
from serialize_test.cpp:15:
/home/me/third-party/boost/include/boost/serialization/detail/shared_count_132.hpp: In member function 'virtual void* boost_132::detail::sp_counted_base_impl<P, D>::get_deleter(const std::type_info&)':
/home/me/third-party/boost/include/boost/serialization/detail/shared_count_132.hpp:274: error: cannot use typeid with -fno-rtti
In file included from /home/me/prebuild/third-party/boost/include/boost/serialization/shared_ptr_132.hpp:35,
from /home/me/prebuild/third-party/boost/include/boost/archive/shared_ptr_helper.hpp:29,
from /home/me/prebuild/third-party/boost/include/boost/archive/xml_iarchive.hpp:133,
from serialize_test.h:21,
from serialize_test.cpp:15:
/home/me/prebuild/third-party/boost/include/boost/serialization/detail/shared_ptr_132.hpp: In function 'D* boost_132::get_deleter(const boost_132::shared_ptr<U>&)':
/home/me/prebuild/third-party/boost/include/boost/serialization/detail/shared_ptr_132.hpp:465: error: cannot use typeid with -fno-rtti
make: *** [all] Error 1
So.. the question is, is it possible to serialize this simple class using boost serialization without using RTTI?? I have looked around and it seems like it may be possible using some boost macros and machinery (#include < boost/serialization/extended_type_info_no_rtti.hpp > alludes to this) but I am a new Boost user and pretty clueless as to how to proceed.
PS: My code compiles fine if I remove -fno-rtti.
Take a look at this:
http://boost.2283326.n4.nabble.com/boost-serialization-and-RTTI-td2566883.html