I am currently saving my object graph in xml file with boost serialization. The library is great, it automatically restores the objects and all member pointers and references.
Problem :
I need to have random access to the data ( read just one object, or delete just one object ). With boost serialization the scenario is like this :
--load all objects from file
-- use and modify objects
-- write all objects to archive again
The drawback with this is if only one object changed , I want to write to the archive only that object, and not the entire object graph which is huge.
So I am looking for library which support storing all objects to database, and which also supports boost and stl types. Currently I am using these boost types:
boost multi-index
Smart Ptr ( shared_ptr,weak_ptr,scoped_ptr)
optional,variant
Anyone knows such library or something close to it, or have idea how to go for resolving this problem?
Have you tried ODB? http://www.codesynthesis.com/products/odb/
It is very complicate to do such a thing. Since C++ (the runtime environment) does not know anything about itself you have to code it on your own. My advice is to split the graph onto different files and to update only the files that are needed (this means that you have to control serialization on your own, not using boost lib).
Maybe Wt::Dbo? By adding a single template function to your classes, Dbo maps them to an SQL database.
Related
I'm asking here because i have already tried to search but i have no idea if this things even exist and what their names are.
I start explaining that with custom pattern i mean this: suppose that i need to serialize objects or data of type foo, bar and boo, usually the library handle this for the user in a very simple way, what comes first goes in first in the serialization process, so if i serialize all the foo first they are written "at the top" of the file and all the bar and boo are after the foo.
Now I would like to keep order in my file and organize things based on a custom pattern, it's this possible with Boost ? What section provides this feature ?
Second thing, that is strictly related to the first one, I also would like to access my serialized binary files in a way that I'm not forced to parse and read all the previous values to extract only the one that I'm interested in, kinda like the RAM that works based on memory address and offers a random access without forcing you to parse all the others addresses.
Thanks.
On the first issue: the Boost serialization library is agnostic as to what happens after it turns an object into its serialized form. It does this by using input and output streams. Files are just that - fostream/fistream. For other types of streams however, the order/pattern that you speak of doesn't make sense. Imagine you're sending serialized objects over the network - the library can't know that it'll have to rearrange the order of objects and, in fact, it can't do that once they've been sent. For this reason, it does not support what you're looking for.
What you can do is create a wrapper that either just caches serialized versions of the objects and arranges them in memory before you tell it to write them out to a file, or that knows that since you're working with files, it can later tellg to the appropriate place in the file and append (this approach would require you to store the locations of the objects you wrote to the file).
As for the second thing - random access file reading. You will have to know exactly where the object is in memory. If you know that the structure of your file won't change, you can seekg on the file stream before handing it to boost for deserialization. If the file structure will change however, you still need to know the location of objects in the file. If you don't want to parse the file to find it, you'll have to store it somewhere during serialization. For example - you can maintain a sort of registry of objects at the top of the file. You will still have to parse it, but it should be just a simple [Object identifier]-[location in file] sort of thing.
I'm creating a C++ object serialization library. This is more towards self-learning and enhancements & I don't want to use off-the-shelf library like boost or google protocol buf.
Please share your experience or comments on good ways to go about it (like creating some encoding with tag-value etc).
I would like to start by supporting PODs followed by support to non-linear DSs.
Thanks
PS: HNY2012
If you need serialization for inter process communication, then I suggest to use some interface language (IDL or ASN.1) for defining interfaces.
So it will be easier to make support for other languages (than C++) too. And also, it will be easier to implement code/stub generator.
I have been working on something similar for the last few months. I couldn't use Boost because the task was to serialize a bunch of existing classes (huge existing codebase) and it was inappropriate to have the classes inherit from the interface which had the serialize() virtual function (we did not want multiple inheritance).
The approach taken had the following salient features:
Create a helper class for each existing class, designated with the task of serializing that particular class, and make the helper class a friend of the class being serialized. This avoids introduction of inheritance in the class being serialized, and also allows the helper class access to private variables.
Have each of the helper classes (let's call them 'serializers') register themselves into a global map. Each serializer class implements a clone() virtual function ('prototype' pattern), which allows one to retrieve a pointer to a serializer, given the name of the class, from this map. The name is obtained by using compiler-specific RTTI information. The registration into the global map is taken care of by instantiating static pointers and 'new'ing them, since static variables get created before the program starts.
A special stream object was created (derived from std::fstream), that contained template functions to serialize non-pointer, pointer, and STL data types. The stream object could only be opened in read-only or write-only modes (by design), so the same serialize() function could be used to either read from the file or write into the file, depending on the mode in which the stream was opened. Thus, there is no chance of any mismatch in the order of reading versus writing of the class members.
For every object being saved or restored, a unique tag (integer) was created based on the address of the variable and stored in a map. If the same address occurred again, only the tag was saved, not the deep-copied object itself. Thus, each object was deep copied only once into the file.
A page on the web captures some of these ideas shared above: http://www.cs.sjsu.edu/~pearce/modules/lectures/cpp/Serialization.htm. Hope that helps.
I wrote an article some years ago. Code and tools can be obsolete, but concepts can remain the same.
May be this can help you.
I have a class named object that have some instance vars. I want to save it to file. How can I cast it to type(e.g. int) that can be written to file?
If I understand correcty you want to save an object to hard drive? That is called serialization. You might want to check the boost library for that.
If it has several data members - you cannot do this (without using some external (3rd party) lib - that's an addition). Instead, write your own function for storing your object into a file, and one for reading from it.
Is there something like a class that might be used to store Files and directories in, just like the way Zip files might be used?
Since I haven't found any "real" class to write Zip files (real class as in real class),
It would be nice to be able to store Files and Directories in a container-like file.
A perfect API would probably look like this:
int main()
{
ContainerFile cntf("myContainer.cnt", ContainerFile::CREATE);
cntf.addFile("data/some-interesting-stuff.txt");
cntf.addDirectory("data/foo/");
cntf.addDirectory("data/bar/", ContainerFile::RECURSIVE);
cntf.close();
}
... I hope you get the Idea.
Important Requirements are:
The Library must be crossplatform
anything *GPL is not acceptable in this case (MIT and BSD License are)
I already played with the thought of creating an Implentation based on SQLite (and its ability to store binary blobs). Unfortunately, it seems impossible to store Directory structures in a SQLite Database, which makes it pretty much useless in this case.
Is it useless to hope for such a class library?
In an SQLite db you can store directory-like structures... you just have to have a "Directories" table, with one entry for each directory, having at least an index and a "parent" field (which holds the index of another directory, or 0 if it has no parent). Then you can have a "Files" table, which contains file attributes, the index of the parent directory and the file content.
That's it, now you have your directory tree in a relational DB.
Someone pointed me to PhysicsFS, which has an API similar to what you describe, but it's a pure C API that does everything you need. A trivial object-oriented wrapper could be easily written.
You might like to check out http://www.cs.unc.edu/Research/compgeom/gzstream/
If you are making your own then redis may be a better choice than SQLite as I believe it handles binary data better.
I took the time to write a tiny, yet working wrapper around libarchive. I'm not exactly familiar with all features of Libarchive, but the result fits what I needed:
archive_wrapper.cpp # gist.github.com
It uses libmars for strings, etc. But I guess it wouldn't be too hard to replace the mars::mstring occurances with std::string.
And of course this wrapper is available under the MIT/X11 License (just as libmars), which means you can do whatever you want with it. ;-)
In my job, i need to parse different kind of data files from different data sources.Sometimes i parse them by writing directly c++ code (with the help of qt and boost:D), sometimes manually with a helper program.
I must note that data types are so different from each other it is so hard to create common a interface for all of them. But i want to do this job in a more generic way.I am planning to write a library to convert them and it should be easy to add new parser utility in future.I am also planning to use other helper programs inside my program, not manually.
My question is what kind of an architecture or pattern do you suggest, Basic condition is library must be extendable via new classes or dll's and also configurable.
By the way data can be in text, ascii or something like CSV(comma seperated values) and most of them are specific for a certain data.
Not to blow my own trumpet, but my small Open Source utility CSVfix has an extensible architecture based on deriving new C++ classes with a very simple interface. I did consider using a plugin-architecture with DLLs but it seemed like overkill for such a simple utility . If interested, you can get the binaries & sources here.
I'd suggest a 3-part model, where the common data-format is a String which should be able to contain every value:
Reader: In this layer the values are read from the source (ie. CSV-file) using some sort of file-format-descriptor. The values are then stored in some sort of intermediate data structure.
Connector/Converter: This layer is responsible for mapping the reader-data to the writer-fields.
Writer: This layer is responsible for writing a specific data structure to the target (ie. another file-format or a database).
This way you can write different Readers for different input files.
I think the hardest part would be creating the definition of the intermediate storage format/structure so that it is future-proof and flexible.
One method I used for defining data structure in my datafile read/write classes is to use std::map<std::string, std::vector<std::string>, string_compare> where the key is the variable name and the vector of strings is the data. While this is expensive in memory, it does not lock me down to only numeric data. And, this method allows for different lengths of data within the same file.
I had the base class implement this generic storage, while the derived classes implemented the reader/writer capability. I then used a factory to get to the desired handler, using another class that determined the file format.