I'm implementing a data buffer which receives audio data packages with procedure call (no network protocols just two applications running on same machine) from one application and puts it in a Struct and writes to a mapped file.
So the writer application may call my app's procedure, which would be smth like void writeData (DataItem data, Timestamp ts) for about 15 times a second with each data item size 2MB.
My app shall store the data into a struct like
Struct DataItem
{
long id;
... Data;
Time insertTime;
}
and write it to a file for future reading purposes.
So since its hard to save the struct to the file as it is, I think(?) I need to write it as binary. So I'm not sure that I need to use any kind of serialization like boost serialization or not?
And I don't know how to align this data for memory map files, and how to re-construct the data for reading purpose from the file as well.
I search internet but I couldn't find much code example. And sample code would be higly appriciated.
By the way I'm using Windows 7 x64 embedded and Visual Studio 2008.
Thanks...
A common C++ way to serialize would be:
struct myStruct
{
int IntData;
float FloatData;
std::string StringData;
};
std::ostream& operator<<(std::ostream &os, const myStruct &myThing)
{
os
<< myThing.IntData << " "
<< myThing.FloatData << " "
<< myThing.StringData << " "
;
return os;
}
std::istream& operator>>(std::istream &is, myStruct &myThing)
{
is
>> myThing.IntData
>> myThing.FloatData
>> myThing.StringData;
return is;
}
void WriteThing()
{
myStruct myThing;
myThing.IntData = 42;
myThing.FloatData = 0.123;
myThing.StringData = "My_String_Test";
std::ofstream outFile;
outFile.open("myFile.txt");
outFile << myThing;
}
void ReadThing()
{
myStruct myThing;
std::ifstream inFile;
inFile.open("myFile.txt");
inFile >> myThing;
}
Please Note:
std::string defines operators << and >>. Those will be called in the
code above.
streams will treat white space characters as delimiters. Storing Strings with blanks would require additional handling
If you plan to keep your data through updates of your
software, you must implement some sort of file versioning
refer to the docs of fstream to find out how to move the file pointer
using seek etc. on a single large file.
Use boost::serialization with text archive.
Is the most "standard" way of solving platform independence.
Optional, you can set a gzip compression on top of it.
Are you sure you are asking about C++ and not C#? Your code example looks like C#
In C++ If your struct format is not going to change, then you can just write the array out to disk.
here is an example as you requested, but this is really C 101 stuff
FILE* output=fopen ("myfile", "wb");
fwrite (array, sizeof (mystruct), number_of_elements_in_array, output);
fclose (output);
Related
I'm working on a C++ beginner level project (not absolute beginner like "what's a loop" but I wouldn't say it's intermediate level either).
In this project I need to save into a file some data stored in memory in struct variables (this is plain imperative programming, with no OOP involved).
I've read a bit about options like serialization, using some non-standard libraries and such. But I need to keep it as simple and clean as possible.
So far I have 2 structs, pretty much like these:
struct client {
string name;
string address;
double phone;
};
struct invoice {
string client_name;
double total;
};
I'm looking for something like this example provided at http://www.cplusplus.com/doc/tutorial/files:
#include <iostream>
#include <fstream>
using namespace std;
int main () {
ofstream myfile;
myfile.open ("example.txt");
myfile << "Writing this to a file.\n";
myfile.close();
return 0;
}
Is there a way to do something like that, but to write (and then be able to read) struct variables to a file, keeping it simple?
Some years ago I remember handling this in a very simple way in Pascal, when writing records to files. It was something like: open file, write record field 1, write field separador, write record field 2, write field separator, write record separator. Then when reading I would search for separators. Is this not recommended in C++?
Thanks.
You could check serialization for C++, eg. Boost::serialization in the Boost library.
Is it possible to serialize and deserialize a class in C++?
You have to associate 2 functions to this class : inputStruct and OutputStruct. Input should be able to read what Output generate. The easy way consist in writing each struct element in the same order that they are implemented.
The clear way to do that is implementing a serialize and deserialize function for every structure or class that you want to write to a file. You give the serialize function the reference of the output stream, and it writes each of the fields that you want to write. The deserialize do the opposite: it reads all the properties in the same order, and sets them in the current class or structure. If you use the stream operators for serialization, the output file will be a text file.
With c++ you can overload stream operator, so it will look pretty in your code, but in other languages you must use functions for that.
You can also use binary serialization, but it is more problematic, because you need to check the endianness of the platform that you use currently. If you will just use it on one platform, you can try write, and read functions. They need the pointer of the variable, and the size, and they copy them into/from the file. Use these for every property separate, never copy whole structures, because it can lead to errors easily.
UPDATE:
I made serialize and deserialize functions. I haven't tested them, so im not 100% sure they will work.
Static functions:
void serialize_client( ofstream& out, client& cl )
{
out << cl.name << endl;
out << cl.address << endl;
out << cl.phone << endl;
}
void deserialize_client( ifstream& in, client& cl )
{
getline( in, cl.name );
getline( in, cl.address );
in >> cl.phone;
}
Usage:
client client_instance;
deserialize_client( cin, client_instance );
serialize_client( cout, client_instance );
With operator overload:
ostream& operator<<( ostream& os, const client& cl )
{
os << cl.name << endl;
os << cl.address << endl;
os << cl.phone << endl;
return os;
}
istream& operator>>( istream& is, client& cl )
{
getlise( is, cl.name );
getlise( is, cl.address );
is >> cl.phone;
return is;
}
client client_instance;
cin >> client_instance;
cout << client_instance;
I've created an fstream object to write info to files.
I write strings to the new file like
fStreamObject << "New message.\n";
because I want each << to print a string to the next line.
I want to be able to set a property and make a call like
fstreamObject << "New message.";
which will write the string to the next line.
Are there flags/settings for fstream objects that allows this to be done?
I've seen the different file modes (i.e. ofstream::in, ofstream::out, etc.), but I couldn't find one that auto writes to a new line. Also, I'm not looking to write my own solution. I want to be able to use a built in feature.
No, there are no readily configurable capabilities of that sort within the standard streams.
You may have to subclass the stream type and fiddle with operator<< to get this to work the way you want, or do it with a helper function of some description:
fstreamObject << nl("New message.");
(but that's hardly easier than just having the \n in there (for a string, anyway).
It depends on what you mean by "setting the stream". If we consider this to be fairly broad then the answer happens to be "yes"!
Here is how:
Create a stream buffer which inserts a newline every time it is flushed, i.e., when sync() is called. Otherwise it just forwards characters.
Change the file stream's stream buffer to use this stream buffer filtering to the file stream's stream buffer.
Set the flag std::ios_base::unitbuf which causes a flush after every [properly written] output operation.
Here are is the example code to do just that:
#include <iostream>
class newlinebuf
: public std::streambuf {
std::ostream* stream;
std::streambuf* sbuf;
int overflow(int c) { return this->sbuf->sputc(c); }
int sync() {
return (this->sbuf->sputc('\n') == std::char_traits::eof()
|| this->sbuf->pubsync() == -1)? -1: 0;
}
public:
newlinebuf(std::ostream& stream)
: stream(&stream)
, sbuf(stream.rdbuf(this)) {
stream << std::unitbuf;
}
~newlinebuf() { this->stream->rdbuf(this->sbuf); }
};
int main() {
newlinebuf sbuf(std::cout);
std::cout << "hello" << "world";
}
Although this approach work, I would recommend against using it! On problem is that all composite output operators, i.e., those using multiple output operators to do their work, will cause multiple newlines. I'm not aware of anything which can be done to prevent this behavior. There isn't anything in the standard library which enables just configuring the stream to do this: you'll need to insert the newline somehow.
No, the C++ streams do not allow that.
There is no way to decide where one insertion stops and the next starts.
For example for custom types, their stream-inserters are often implemented as calls to other stream-inserters and member-functions.
The only things you can do, is write your own class, which delegates to a stream of your choosing, and does that.
That's of strictly limited utiliy though.
struct alwaysenter {
std::ostream& o;
template<class X> alwaysenter& operator<<(X&& x) {
o<<std::forward<X>(x);
return *this;
}
};
I found this question answered for Python, Java, Linux script, but not C++:
I'd like to write all outputs of my C++ program to both the terminal and an output file. Using something like this:
int main ()
{
freopen ("myfile.txt","w",stdout);
cout<< "Let's try this";
fclose (stdout);
return 0;
}
outputs it to only the output file named "myfile.txt", and prevents it from showing on the terminal. How can I make it output to both simultaneously? I use visual studio 2010 express (if that would make any difference).
Thanks in advance!
Possible solution: use a static stream cout-like object to write both to cout and a file.
Rough example:
struct LogStream
{
template<typename T> LogStream& operator<<(const T& mValue)
{
std::cout << mValue;
someLogStream << mValue;
}
};
inline LogStream& lo() { static LogStream l; return l; }
int main()
{
lo() << "hello!";
return 0;
}
You will probably need to explicitly handle stream manipulators, though.
Here is my library implementation.
There is no built in way to do this in one step. You have to write the data to a file and then write the data out on screen in two steps.
You can write a function that takes in the data and the filename and does this for you, to save you time, some sort of logging function.
I have a method to do this, and it is based on a subscriber model.
In this model all your logging goes to a "logging" manager and you then have "subscribers" that decide what to do with the messages. Messages have topics (for me a number) and loggers subscribe to one or more topic.
For your purpose, you create 2 subscribers, one that outputs to the file and one that outputs to the console.
In the logic of your code you simply output the message, and at this level not need to know what is going to be done with it. In my model though you can check first if there are any "listeners" as this is considered cheaper than constructing and outputting messages that will only end up in /dev/null (well you know what I mean).
One way to do this would be to write a small wrapper to do this, for example:
class DoubleOutput
{
public:
// Open the file in the constructor or any other method
DoubleOutput(const std::string &filename);
// ...
// Write to both the file and the stream here
template <typename T>
friend DoubleOutput & operator<<(const T& file);
// ...
private:
FILE *file;
}
To have a class instead of a function makes you use the RAII idiom (https://en.wikipedia.org/wiki/Resource_acquisition_is_initialization)
To use it:
DoubleOutput mystream("myfile");
mystream << "Hello World";
I'm trying to serialize objects to send over network through a socket using only STL. I'm not finding a way to keep objects' structure to be deserialized in the other host. I tried converting to string, to char* and I've spent a long time searching for tutorials on the internet and until now I have found nothing.
Is there a way to do it only with STL?
Are there any good tutorials?
I am almost trying boost, but if there is how to do it with STL I'd like to learn.
You can serialize with anything. All serialization means is that you are converting the object to bytes so that you can send it over a stream (like an std::ostream) and read it with another (like an std::istream). Just override operator <<(std::ostream&, const T&) and operator >>(std::istream&, T&) where T is each of your types. And all the types contained in your types.
However, you should probably just use an already-existing library (Boost is pretty nice). There are tons of things that a library like Boost does for you, like byte-ordering, taking care of common objects (like arrays and all the stuff from the standard library), providing a consistent means of performing serialization and tons of other stuff.
My first question will be: do you want serialization or messaging ?
It might seem stupid at first, since you asked for serialization, but then I have always distinguished the two terms.
Serialization is about taking a snapshot of your memory and restoring it later on. Each object is represented as a separate entity (though they might be composed)
Messaging is about sending information from one point to another. The message usually has its own grammar and may not reflect the organization of your Business Model.
Too often I've seen people using Serialization where Messaging should have been used. It does not mean that Serialization is useless, but it does mean that you should think ahead of times. It's quite difficult to alter the BOM once you have decided to serialize it, especially if you decide to relocate some part of information (move it from one object to another)... because how then are you going to decode the "old" serialized version ?
Now that that's been cleared up...
... I will recommend Google's Protocol Buffer.
You could perfectly rewrite your own using the STL, but you would end up doing work that has already been done, and unless you wish to learn from it, it's quite pointless.
One great thing about protobuf is that it's language agnostic in a way: ie you can generate the encoder/decoder of a given message for C++, Java or Python. The use of Python is nice for message injection (testing) or message decoding (to check the output of a logged message). It's not something that would come easy were you to use the STL.
Serializing C++ Objects over a Network Socket
This is 6 years late but I just recently had this problem and this was one of the threads that I came across in my search on how to serialize object through a network socket in C++. This solution uses just 2 or 3 lines of code. There are a lot of answers that I found work but the easiest that I found was to use reinterpret_cast<obj*>(target) to convert the class or structure into an array of characters and feed it through the socket. Here's an example.
Class to be serialized:
/* myclass.h */
#ifndef MYCLASS_H
#define MYCLASS_H
class MyClass
{
public:
int A;
int B;
MyClass(){A=1;B=2;}
~MyClass(){}
};
#endif
Server Program:
/* server.cpp */
#include "myclass.h"
int main (int argc, char** argv)
{
// Open socket connection.
// ...
// Loop continuously until terminated.
while(1)
{
// Read serialized data from socket.
char buf[sizeof(MyClass)];
read(newsockfd,buf, sizeof(MyClass));
MyClass *msg = reinterpret_cast<MyClass*>(buf);
std::cout << "A = " << std::to_string(msg->A) << std::endl;
std::cout << "B = " << std::to_string(msg->B) << std::endl;
}
// Close socket connection.
// ...
return 0;
}
Client Program:
/* client.cpp */
#include "myClass.h"
int main(int argc, char *argv[])
{
// Open socket connection.
// ...
while(1)
{
printf("Please enter the message: ");
bzero(buffer,256);
fgets(buffer,255,stdin);
MyClass msg;
msg.A = 1;
msg.B = 2;
// Write serialized data to socket.
char* tmp = reinterpret_cast<char*>(&msg);
write(sockfd,tmp, sizeof(MyClass));
}
// Close socket connection.
// ...
return 0;
}
Compile both server.cpp and client.cpp using g++ with -std=c++11 as an option. You can then open two terminals and run both programs, however, start the server program before the client so that it has something to connect to.
Hope this helps.
I got it!
I used strinstream to serialize objects and I sent it as a message using the stringstream's method str() and so string's c_str().
Look.
class Object {
public:
int a;
string b;
void methodSample1 ();
void methosSample2 ();
friend ostream& operator<< (ostream& out, Object& object) {
out << object.a << " " << object.b; //The space (" ") is necessari for separete elements
return out;
}
friend istream& operator>> (istream& in, Object& object) {
in >> object.a;
in >> object.b;
return in;
}
};
/* Server side */
int main () {
Object o;
stringstream ss;
o.a = 1;
o.b = 2;
ss << o; //serialize
write (socket, ss.str().c_str(), 20); //send - the buffer size must be adjusted, it's a sample
}
/* Client side */
int main () {
Object o2;
stringstream ss2;
char buffer[20];
string temp;
read (socket, buffer, 20); //receive
temp.assign(buffer);
ss << temp;
ss >> o2; //unserialize
}
I'm not sure if is necessary convert to string before to serialize (ss << o), maybe is possible directly from char.
I think you should use google Protocol Buffers in your project.In network transport Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:
are simpler
are 3 to 10 times smaller
are 20 to 100 times faster
are less ambiguous
generate data access classes that are easier to use programmaticall
and so on. I think you need read https://developers.google.com/protocol-buffers/docs/overview about protobuf
I try to write and read object of class into and from binary file in C++. I want to not write the data member individually but write the whole object at one time. For a simple example:
class MyClass {
public:
int i;
MyClass(int n) : i(n) {}
MyClass() {}
void read(ifstream *in) { in->read((char *) this, sizeof(MyClass)); }
void write(ofstream *out){ out->write((char *) this, sizeof(MyClass));}
};
int main(int argc, char * argv[]) {
ofstream out("/tmp/output");
ifstream in("/tmp/output");
MyClass mm(3);
cout<< mm.i << endl;
mm.write(&out);
MyClass mm2(2);
cout<< mm2.i << endl;
mm2.read(&in);
cout<< mm2.i << endl;
return 0;
}
However the running output show that the value of mm.i supposedly written to the binary file is not read and assigned to mm2.i correctly
$ ./main
3
2
2
So what's wrong with it?
What shall I be aware of when generally writing or reading an object of a class into or from a binary file?
The data is being buffered so it hasn't actually reached the file when you go to read it. Since you using two different objects to reference the in/out file, the OS has not clue how they are related.
You need to either flush the file:
mm.write(&out);
out.flush()
or close the file (which does an implicit flush):
mm.write(&out);
out.close()
You can also close the file by having the object go out of scope:
int main()
{
myc mm(3);
{
ofstream out("/tmp/output");
mm.write(&out);
}
...
}
Dumping raw data is a terrible idea, from multiple angles. This will break even worse once you add pointer data.
One suggestion would be to use Boost.Serialization which allows for far more robust data dumping.
Your main problem is the file does not contain the contents yet due to fstream buffering. Close or flush the file.
I'll echo "you shouldn't be doing this". If you print out sizeof(myc) in the code above it's probably 4, as you'd expect... BUT try changing read and write to be virtual. When I did so, it prints out the size as 16. Those 12 bytes are internal guts with sensitive values—and saving them out and then reading them back in would be like expecting a pointer value to be still good if you wrote it and loaded it again.
If you want to circumvent serialization and map C++ object memory directly to disk, there are ways to hack that. But rules are involved and it's not for the faint of heart. See POST++ (Persistent Object Storage for C++) as an example.
I'll add that you did not check the fail() or eof() status. If you had you'd have known you were misusing the fstream API. Try it again with:
void read(ifstream *in) {
in->read((char *) this, sizeof(myc));
if (in->fail())
cout << "read failed" << endl;
}
void write(ofstream *out){
out->write((char *) this, sizeof(myc));
if (out->fail())
cout << "write failed" << endl;
}
...and see what happens.
My C++ is pretty rust and highly under-tested, but you may want to take a look at Serialization and Unserialization. FAQ
I've done something similar using output.write((char*)&obj, sizeof(obj)), obj being an instance of your class. You may want to loop this if you want to write the data inside the object instead, which would generally be the case as you need members to be readable, right ?
Same thing for reading with read function. But if you have dynamic allocation to do then with this data, you need to handle it.