I'm trying to serialize objects to send over network through a socket using only STL. I'm not finding a way to keep objects' structure to be deserialized in the other host. I tried converting to string, to char* and I've spent a long time searching for tutorials on the internet and until now I have found nothing.
Is there a way to do it only with STL?
Are there any good tutorials?
I am almost trying boost, but if there is how to do it with STL I'd like to learn.
You can serialize with anything. All serialization means is that you are converting the object to bytes so that you can send it over a stream (like an std::ostream) and read it with another (like an std::istream). Just override operator <<(std::ostream&, const T&) and operator >>(std::istream&, T&) where T is each of your types. And all the types contained in your types.
However, you should probably just use an already-existing library (Boost is pretty nice). There are tons of things that a library like Boost does for you, like byte-ordering, taking care of common objects (like arrays and all the stuff from the standard library), providing a consistent means of performing serialization and tons of other stuff.
My first question will be: do you want serialization or messaging ?
It might seem stupid at first, since you asked for serialization, but then I have always distinguished the two terms.
Serialization is about taking a snapshot of your memory and restoring it later on. Each object is represented as a separate entity (though they might be composed)
Messaging is about sending information from one point to another. The message usually has its own grammar and may not reflect the organization of your Business Model.
Too often I've seen people using Serialization where Messaging should have been used. It does not mean that Serialization is useless, but it does mean that you should think ahead of times. It's quite difficult to alter the BOM once you have decided to serialize it, especially if you decide to relocate some part of information (move it from one object to another)... because how then are you going to decode the "old" serialized version ?
Now that that's been cleared up...
... I will recommend Google's Protocol Buffer.
You could perfectly rewrite your own using the STL, but you would end up doing work that has already been done, and unless you wish to learn from it, it's quite pointless.
One great thing about protobuf is that it's language agnostic in a way: ie you can generate the encoder/decoder of a given message for C++, Java or Python. The use of Python is nice for message injection (testing) or message decoding (to check the output of a logged message). It's not something that would come easy were you to use the STL.
Serializing C++ Objects over a Network Socket
This is 6 years late but I just recently had this problem and this was one of the threads that I came across in my search on how to serialize object through a network socket in C++. This solution uses just 2 or 3 lines of code. There are a lot of answers that I found work but the easiest that I found was to use reinterpret_cast<obj*>(target) to convert the class or structure into an array of characters and feed it through the socket. Here's an example.
Class to be serialized:
/* myclass.h */
#ifndef MYCLASS_H
#define MYCLASS_H
class MyClass
{
public:
int A;
int B;
MyClass(){A=1;B=2;}
~MyClass(){}
};
#endif
Server Program:
/* server.cpp */
#include "myclass.h"
int main (int argc, char** argv)
{
// Open socket connection.
// ...
// Loop continuously until terminated.
while(1)
{
// Read serialized data from socket.
char buf[sizeof(MyClass)];
read(newsockfd,buf, sizeof(MyClass));
MyClass *msg = reinterpret_cast<MyClass*>(buf);
std::cout << "A = " << std::to_string(msg->A) << std::endl;
std::cout << "B = " << std::to_string(msg->B) << std::endl;
}
// Close socket connection.
// ...
return 0;
}
Client Program:
/* client.cpp */
#include "myClass.h"
int main(int argc, char *argv[])
{
// Open socket connection.
// ...
while(1)
{
printf("Please enter the message: ");
bzero(buffer,256);
fgets(buffer,255,stdin);
MyClass msg;
msg.A = 1;
msg.B = 2;
// Write serialized data to socket.
char* tmp = reinterpret_cast<char*>(&msg);
write(sockfd,tmp, sizeof(MyClass));
}
// Close socket connection.
// ...
return 0;
}
Compile both server.cpp and client.cpp using g++ with -std=c++11 as an option. You can then open two terminals and run both programs, however, start the server program before the client so that it has something to connect to.
Hope this helps.
I got it!
I used strinstream to serialize objects and I sent it as a message using the stringstream's method str() and so string's c_str().
Look.
class Object {
public:
int a;
string b;
void methodSample1 ();
void methosSample2 ();
friend ostream& operator<< (ostream& out, Object& object) {
out << object.a << " " << object.b; //The space (" ") is necessari for separete elements
return out;
}
friend istream& operator>> (istream& in, Object& object) {
in >> object.a;
in >> object.b;
return in;
}
};
/* Server side */
int main () {
Object o;
stringstream ss;
o.a = 1;
o.b = 2;
ss << o; //serialize
write (socket, ss.str().c_str(), 20); //send - the buffer size must be adjusted, it's a sample
}
/* Client side */
int main () {
Object o2;
stringstream ss2;
char buffer[20];
string temp;
read (socket, buffer, 20); //receive
temp.assign(buffer);
ss << temp;
ss >> o2; //unserialize
}
I'm not sure if is necessary convert to string before to serialize (ss << o), maybe is possible directly from char.
I think you should use google Protocol Buffers in your project.In network transport Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:
are simpler
are 3 to 10 times smaller
are 20 to 100 times faster
are less ambiguous
generate data access classes that are easier to use programmaticall
and so on. I think you need read https://developers.google.com/protocol-buffers/docs/overview about protobuf
Related
This question already has answers here:
How to use boost serialization for binary output?
(2 answers)
Closed 1 year ago.
I have succeeded to serialize my Employee object into file format using Boost library, and I want to send it through socket using TCP/IP communication protocol, but I guess that this protocol does not work for files.
I am thinking of serializing and deserializing the object into binary format but I did not know how to do it, I did not find examples on the internet, so would you please help me ?
I will share with you my code :
#include <fstream>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>
using namespace std;
class Employee {
private:
friend class boost::serialization::access;
int id;
string name;
float salary;
template<class Archive>
void serialize(Archive &a, const unsigned version){
a & id & name & salary;
}
public:
Employee(){}
Employee(int i, string n, float s):id(i),name(n),salary(s)
{}
};
int main()
{
const string filename = "emp.dat";
Employee e1(11,"Harry",4500.00f);
Employee e2(22,"Ravi",8800.00f);
Employee e3(33,"Tim",6800.00f);
Employee e4(44,"Rajiv",3400.00f);
// Serialize and persist the object
{
std::ofstream outfile(filename);
boost::archive::text_oarchive archive(outfile);
archive << e1 << e2 << e3 << e4;
}
// Deserialize and restore the object
Employee restored_e1;
Employee restored_e2;
Employee restored_e3;
Employee restored_e4;
{
std::ifstream infile(filename);
boost::archive::text_iarchive archive(infile);
archive >> restored_e1 >> restored_e2
>> restored_e3 >> restored_e4;
}
return 0;
}
[Summary]
a) look for a std::stream compatible class that accepts a socket. (I think boost asio has one).
b) create a file handle and assign the socket handle to it.
c) caveat: The boost serialize Archive concept does not lend itself easily to what you are trying to do. See below.
[Detail]
Firstoff, what you are asking here has nothing to do with text or binary file format. Either would work in principle with tcp/ip. text serializes your data in human readable (text) form whereas binary would simply replace the readable (text) with a binary representation. This has nothing to do with where the data is going.
That said: What you are trying to do is serialize to a tcp/ip socket instead of a file. This is possible, although it is not immediately obvious how. boost serialize uses an Archive as a wrapper around a std::stream. If you have a stream that writes data to a tcp/ip socket rather than a file, than that would be what you wanted.
[My recommendation]
I'll start with c): caveat. The boost Archive concept does not lend itself easily to serialization via sockets. The reason is that usually you have exactly ONE Archive PER FILE. Translated to socket: You can serialize exactly ONE Archive PER Connection. You will have to close the connection and reopen it, if you need to serialize more than one "archive". closing and reopening takes time.
Given the fact that you also don't have a std::otcpstream ready, my own approach was to serialize my Archives to std::o/istringstream and send the entire string buffer via TCP/IP. Like this:
std::ostringstream out;
{
boost::text_archive outA(out);
outA << do some serialization;
}
send_via_tcp(out.str());
Note: The brackets here are important. The Archive destructor will write an 'end' tag to the stream, so unless the destructor gets called, the out.str() call will yield an incomplete string.
[serializing directly to TCP/IP]
That said, you could try to serialize directly to TCP/IP. Problem is, there is no std::stream class that serializes to a socket. You'd expect a std::otcpstream for example, but there is none (in std).
a) I think boost asio has a stream that uses TCP/IP so you can look there.
b) The second option is to open a socket and then create a file handle for this socket. The function _dup() accepts a socket as input (I think). That way you have a file handle (rather than a socket) and can create an ofstream/ifstream for it. Or perhaps create the ofstream/ifstream first and reassign the file handle with _dup.
I think this will cause all read/write calls to go to the TCP/IP stack instead of the file system so in theory you should be able to write/read an Archive (exactly one). But like I said above, I'd simply use the stringstream method, if your archives aren't too big.
In my project (Unreal Engine 4) I don't have an output stream - instead of this I can communicate via UE_LOG function, which works pretty much similar to printf(). The problem is that I just made a .dll library (without Unreal includes) which I want to communicate through the iostream. My idea is - inside .dll library I use standard cout to write messages into ostream, I use all of it in Unreal Engine functions, where I grab ostream in form of string and output it into UE_LOG function.
Problem is I always treated std::cout as a part of magic, without thinking what is really inside (I am pretty sure most of us did). How I can handle this? Easy ways won't work (like grabbing stringstream and outputing it into UE_LOG).
My idea is - inside .dll library I use standard cout to write messages into ostream
You actually can replace the output buffer used with std::cout with your own implementation. Use the std::ostream::rdbuf() function to do so (example from the reference docs):
#include <iostream>
#include <sstream>
int main()
{
std::ostringstream local;
auto cout_buff = std::cout.rdbuf(); // save pointer to std::cout buffer
std::cout.rdbuf(local.rdbuf()); // substitute internal std::cout buffer with
// buffer of 'local' object
// now std::cout work with 'local' buffer
// you don't see this message
std::cout << "some message";
// go back to old buffer
std::cout.rdbuf(cout_buff);
// you will see this message
std::cout << "back to default buffer\n";
// print 'local' content
std::cout << "local content: " << local.str() << "\n";
}
(in case my edit won't be positively reviewed)
From OP: Thanks to your hints I finally found how to solve my problem. Suppose I want to get stream from cout and send it to printf (because I think stdio library is superior to iostream). Here how I can do this:
#include <iostream>
#include <sstream>
#include <cstdio>
using namespace std;
class ssbuf : public stringbuf{
protected:
int sync(){
printf("My buffer: %s",this->str().c_str());
str("");
return this->stringbuf::sync();
}
};
int main(){
ssbuf *buf = new ssbuf();
cout.rdbuf(buf);
cout<<"This is out stream "<<"and you cant do anything about it"<<endl;
cout<<"(don't) "<<"Vote Trump"<<endl;
}
Code is very raw, but it does it's job. I made child class of buffer which has method sync() downcasting original virtual method sync(). Except this it works like usual buffer, just grabs all console-out stream - exactly what we wanted. The str("") inside is to clean the buffer - probably not outputted stream doesn't clean itself.
Great thanks for help! Big GRIN for you! :D
i am working on a network project of mine in order to learn more about networking and right now i have designed a simple protocol/structure that i fill and send to the server, the problem is that all vectors and probably arrays aswell are invalid on the server side.
im gonna try to explain it with code, its alot easier that way.
My protocol:
typedef struct NETWORK_PROTOCOL {
int packet_size;
int number_of_data_files;
std::vector<std::string> data_files;
}
so its a pretty simple protocol, and what i did is that i fill it with data and its completely valid on the client side, however as soon as i send it to the server and try to convert it back it the vector is invalid but the integers are still valid.
this is how i create and send the data from the client:
NETWORK_PROTOCOL Protocol;
//Fills protocol with data
int sendt = send(ClientSocket, (const char*)&Protocol, Protocol.packet_size, 0);
and when it hits the server i still get the full size of the data, but as i said earlier it does not convert back properly :/
Code on the server side that tries to cast it back:
NETWORK_PROTOCOL* Protocol;
iResult = recv(ClientSocket, buffer, BUFFLEN, 0);
//then i have some validation code to check if the whole packet arrived since its TCP
Protocol = reinterpret_cast<NETWORK_PROTOCOL*>(buffer);
//And now the vector is invalid :/
im not really sure how to fix this problem, i thought it would be easy to convert it back since it is the exact same data on both sides. Any help to fix this issue is greatly appreciated.
std::vector can't be transferred this way: internally it uses pointers, so you send only a pointer, without any actual information, and that pointer is not valid on the receiving side.
In order to send the contents of vector, you need to somehow serialize it (convert it to the representation in which it can be easily transferred). For example, you can use is Boost.Serialization
#include <sstream>
// include headers that implement a archive in simple text format
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/vector.hpp>
struct NETWORK_PROTOCOL
{
private:
friend class boost::serialization::access;
// When the class Archive corresponds to an output archive, the
// & operator is defined similar to <<. Likewise, when the class Archive
// is a type of input archive the & operator is defined similar to >>.
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
ar & packet_size;
ar & number_of_data_files; // you don't actually need it
ar & data_files;
}
public:
int packet_size;
int number_of_data_files;
std::vector<std::string> data_files;
};
Now you can serialize it like this:
std::ostringstream ofs;
boost::archive::text_oarchive oa(ofs);
oa << protocol; // protocol is your instance of NETWORK_PROTOCOL, which you want to send
// and then you'll be able to get a buffer from ofs using ofs.str()
Deserialize it like this:
NETWORK_PROTOCOL protocol;
std::istringstream ifs(buf);
boost::archive::text_iarchive ia(ifs);
ia >> protocol;
For practical usages you may want to use binary archives instead. If you decide to go with boost.serialization, I recommend starting looking here.
You may also like Google Protocol Buffers: https://developers.google.com/protocol-buffers/docs/cpptutorial
This comment is longer than allowed. So I put it as an answer; although I think it answers partially.
To send all the data in one shipment wasted space and bandwidth, because you'd have to take a maximum for the number of names and their sizes. So I suggest you divide your transmission in phases.
In the first phase you send the number of filenames that you are transmitting. In this way you prepare the server for receiving n file names. Then in the second phase you make a loop divided in two transmissions. The first transmission you send the file name size, then you prepare a buffer for receiving the filename.
For these modes you only use basic types (size_t and char *).
On the server side you can build your vector<string>, if you want to give that illusion
I hope it helps you
I'm implementing a data buffer which receives audio data packages with procedure call (no network protocols just two applications running on same machine) from one application and puts it in a Struct and writes to a mapped file.
So the writer application may call my app's procedure, which would be smth like void writeData (DataItem data, Timestamp ts) for about 15 times a second with each data item size 2MB.
My app shall store the data into a struct like
Struct DataItem
{
long id;
... Data;
Time insertTime;
}
and write it to a file for future reading purposes.
So since its hard to save the struct to the file as it is, I think(?) I need to write it as binary. So I'm not sure that I need to use any kind of serialization like boost serialization or not?
And I don't know how to align this data for memory map files, and how to re-construct the data for reading purpose from the file as well.
I search internet but I couldn't find much code example. And sample code would be higly appriciated.
By the way I'm using Windows 7 x64 embedded and Visual Studio 2008.
Thanks...
A common C++ way to serialize would be:
struct myStruct
{
int IntData;
float FloatData;
std::string StringData;
};
std::ostream& operator<<(std::ostream &os, const myStruct &myThing)
{
os
<< myThing.IntData << " "
<< myThing.FloatData << " "
<< myThing.StringData << " "
;
return os;
}
std::istream& operator>>(std::istream &is, myStruct &myThing)
{
is
>> myThing.IntData
>> myThing.FloatData
>> myThing.StringData;
return is;
}
void WriteThing()
{
myStruct myThing;
myThing.IntData = 42;
myThing.FloatData = 0.123;
myThing.StringData = "My_String_Test";
std::ofstream outFile;
outFile.open("myFile.txt");
outFile << myThing;
}
void ReadThing()
{
myStruct myThing;
std::ifstream inFile;
inFile.open("myFile.txt");
inFile >> myThing;
}
Please Note:
std::string defines operators << and >>. Those will be called in the
code above.
streams will treat white space characters as delimiters. Storing Strings with blanks would require additional handling
If you plan to keep your data through updates of your
software, you must implement some sort of file versioning
refer to the docs of fstream to find out how to move the file pointer
using seek etc. on a single large file.
Use boost::serialization with text archive.
Is the most "standard" way of solving platform independence.
Optional, you can set a gzip compression on top of it.
Are you sure you are asking about C++ and not C#? Your code example looks like C#
In C++ If your struct format is not going to change, then you can just write the array out to disk.
here is an example as you requested, but this is really C 101 stuff
FILE* output=fopen ("myfile", "wb");
fwrite (array, sizeof (mystruct), number_of_elements_in_array, output);
fclose (output);
In my application, I want to redirect the output that would normally go to the stdout stream to a function I define. I read that you can redirect stdio to a file, so why not to a function?
For example:
void MyHandler( const char* data );
//<<Magical redirection code>>
printf( "test" );
std::cout << "test" << std::endl;
//MyHandler should have been called with "test" twice, at this point
How can I achieve this / similar behaviour?
#Konrad Rudolph is right, you can totally do this, easily, at least for cout/cerr/clog. You don't even need your own streambuf implementation, just use an ostringstream.
// Redirect cout.
streambuf* oldCoutStreamBuf = cout.rdbuf();
ostringstream strCout;
cout.rdbuf( strCout.rdbuf() );
// This goes to the string stream.
cout << "Hello, World!" << endl;
// Restore old cout.
cout.rdbuf( oldCoutStreamBuf );
// Will output our Hello World! from above.
cout << strCout.str();
Same thing works for cerr and clog, but in my experience that will NOT work for stdout/stderr in general, so printf won't output there. cout goes to stdout, but redirecting cout will not redirect all stdout. At least, that was my experience.
If the amount of data is expected to be small, the freopen/setbuf thing works fine. I ended up doing the fancier dup/dup2 thing redirecting to a pipe.
Update: I wrote a blog post showing the dup2 method I ended up using, which you can read here. It's written for OS X, but might work in other Unix flavors. I seriously doubt it would work in Windows. Cocoa version of the same thing here.
Invoking a callback function whenever something’s written to the underlying standard output stream is hard: it requires overriding low-level, system-specific functions (on POSIX systems, it would require at least overriding write, and the way this is called might depend on the standard library implementation, so it might be different between e.g. glibc and musl).
But depending on what exactly you’re after you can solve this in C++ without resorting to low-level OS specific functions, by manipulating the C++ stream buffers directly.
For this you need to create your own std::streambuf implementation, i.e. your own stream buffer.
Once you have that, you can redirect the std::cout stream by switching the buffer:
auto buf = callback_streambuf(MyHandler);
auto pold_buffer = std::cout.rdbuf(&buf);
std::cout << "Hello" << std::cout;
// Restore original buffer:
std::cout.rdbuf(pold_buffer);
However, what this implementation won’t do is call your callback function exactly twice. Instead, the number of calls will depend on several factors, but it generally will not depend on the number of stream insertions (<<), and there is no way around that!
For the specific example above, the callback is called once, with the data "Hello\n".
The following minimal implementation illustrates how to get a streambuf to call your handler:
class callback_streambuf : public std::streambuf {
public:
callback_streambuf(std::function<void(char const*, std::streamsize)> callback) : callback(callback) {}
protected:
std::streamsize xsputn(char_type const* s, std::streamsize count) {
callback(s, count);
return count;
}
private:
std::function<void(char const*, std::streamsize)> callback;
};
This implementation has several caveats. For instance, it does the wrong thing when attempts are made to use it as an input stream. It doesn’t override overflow (since I don’t think this is ever called, although I find conflicting information about this on the internet; at any rate, adding overflow would be trivial). I didn’t implement synchronisation, so the callback will be called concurrently from multiple threads. Furthermore, there is no error handling, since the callback does not return a success status. I also had to change the signature of the callback to
void MyHandler(char const* data, std::streamsize count);
The second parameter is required since data isn’t a string, it’s a raw char buffer, so there is no way to determine its length intrinsically, and MyHandler can do nothing useful with the data without knowing its length.
Answer: Yes you can, via a dup. freopen will only reopen stdout to a file, as you talked about.
Check out How to buffer stdout in memory and write it from a dedicated thread
It's possible to disable stdin/stdout by dereferencing its pointer:
FILE fp_old = *stdout; // preserve the original stdout
*stdout = *fopen("/dev/null","w"); // redirect stdout to null
HObject m_ObjPOS = NewLibraryObject(); // call some library which prints unwanted stdout
*stdout=fp_old; // restore stdout
The std::cout object has a fixed meaning, and that is to output to the standard out stream. The user of your program gets to control where standard out is connected to, not you. What you can do is decide whether you wish to write to a file, to standard out or to any other output stream. So in your code you switch what stream you write to.
Again, the point of writing to the standard out stream is to give the user a flexibility in choosing where the output goes to. You're not supposed to redirect standard out; this is something the user is supposed to have the freedom to do.
Another thing is that you shouldn't mix C IO and C++ IO in a C++ program. Choose which IO library you wish to work with and stick to it.
That said, you can in C++ quite elegantly switch streams for a function to take input from by templating the handler function on the template parameters of std::basic_istream<>. Then the function will read its input from the input stream independently of the real kind of stream it's working with. Here's an example:
#include<iostream>
#include<fstream>
#include<string>
template<class Ch, class Tr>
void dodge_this(std::basic_istream<Ch, Tr>& in)
{
// in is an input stream. read from it as you read from std::cin.
}
int main(int argc, char* argv[])
{
if( std::string(argv[1]) == "cin" ) {
dodge_this(std::cin);
} else if( std::string(argv[1]) == "file" ) {
std::ifstream file("input.txt");
dodge_this(file);
} else {
dodge_this(dev_null_stream); // i just made that up. you get the idea.
}
}
Another option is to place your handler class calls into the inherited streambuf class. I had a requirement to redirect cout to a Win GUI edit control in a dialog box that may be of some use. Here is the class code:
//-------------------------------- DlgStringbuf Definition -----------------------
class DlgStringbuf : public std::stringbuf
{
public:
DlgStringbuf(void) : _hwndDlg(NULL), _editControlID(0), _accum(""), _lineNum(0) {}
void SetDlg(HWND dlg, int editControlID)
{ _hwndDlg = dlg; _editControlID = editControlID; }
void Clear(void)
{ _accum.clear(); _lineNum = 0; }
protected:
virtual std::streamsize xsputn(const char* s, std::streamsize num)
{
std::mutex m;
std::lock_guard<std::mutex> lg(m);
// Prepend with the line number
std::string str(s, (const uint32_t)num);
str = std::to_string(_lineNum) + ": " + str + "\r\n";
// Accumulate the latest text to the front
_accum = str + _accum;
// Write to the Win32 dialog edit control.
if(_hwndDlg != NULL)
SetDlgItemTextW(_hwndDlg, _editControlID, (LPCWSTR)(std::wstring(_accum.begin(), _accum.end())).c_str());
_lineNum++;
return(num);
}//end xsputn.
private:
std::string _accum;
HWND _hwndDlg;
int _editControlID;
uint32_t _lineNum;
};//end DlgStringbuf.
//-------------------------------- DlgStream Definition ------------------------------
class DlgStream : public std::ostream
{
public:
DlgStream(void) : std::ostream(&_sbuff) {}
void SetDlg(HWND dlg, int editControlID)
{ _sbuff.SetDlg(dlg, editControlID); }
void Clear(void)
{ _sbuff.Clear(); }
private:
DlgStringbuf _sbuff;
};
...and in the WinMain, someplace after the dialog box and its edit control is created:
// Redirect all cout usage to the activity dlg box.
// Save output buffer of the stream - use unique pointer with deleter that ensures to restore
// the original output buffer at the end of the program.
auto del = [&](streambuf* p) { cout.rdbuf(p); };
unique_ptr<streambuf, decltype(del)> origBuffer(cout.rdbuf(), del);
// Redirect the output to the dlg stream.
_dlgStream.SetDlg(hwndActivityDlg, IDC_EDIT_ACTIVITY);
_dlgStream.copyfmt(cout);
cout.rdbuf(_dlgStream.rdbuf());
cout << "this is from cout";
You can use sprintf to write to a character array and then read the value:
char buf[1024];
sprintf(buf, "test");
MyHandler(buf);
there are also snprintf and a few others depending on platform