I'm currently working on a small project: there's a protocol for sending some strings via UDP implemented with standard C interface.
Although it works pretty fine, I'd like to rewrite it with some more sophisticated C++ (consider it exercise).
Currently it's something like that: A client wants that string so it sends the following struct:
struct request {
uint8_t msg_type;// == 1
uint64_t key; // generated randomly to identify each request
}
In new implementation, I want to use boost::asio so in server I have a following piece of code:
boost::asio::io_service io_service;
boost::asio::ip::udp::endpoint client_endpoint;
boost::asio::ip::udp::socket socket(io_service,
boost::asio::ip::udp::endpoint(boost::asio::ip::udp::v4(),
m_serverPort));
boost::asio::streambuf sb;
boost::asio::streambuf::mutable_buffers_type mutableBuf =
sb.prepare(sizeof(request));
size_t received_bytes = socket.receive_from(mutableBuf, client_endpoint);
sb.commit(received_bytes);
request r;
std::istream is(&sb);
is >> msg_type;
is >> key;
key = __bswap64(key); // I'm using network byteorder for numbers sent with this protocol
// and there's no ntohll function on Snow Leopard (at least I can't
// find one)
sb.consume(received_bytes);
And here's my problem: the "key" value which I try to receive this way is wrong - I mean I get something that I did not send.
Here are my suspicions:
__bswap64 does not convert network to host (little-endian) byteorder
I misunderstood how to use boost::asio::streambuf with streams
There's some incompatibility between old C interface and boost (but I don't think so
cause I've found out that boost functions are just wrappers for it)
EDIT:
hmm they say "don't praise a ford till you get over". Now I have a very similar issue in another place of my code. I have a following struct which is sent as a reply for request metioned above:
struct __attribute__ ((packed)) CITE_MSG_T
{
uint8_t msg_id;
uint64_t key; // must be the same as in request
uint16_t index; // part number
uint16_t parts; // number of all parts
CITE_PART_T text; // message being sent
};
//where CITE_PART_T is:
struct __attribute__ ((packed)) CITE_PART_T
{
uint16_t data_length;
char* data;
};
and following piece of code: http://pastebin.com/eTzq6AWQ.
Unfortunately there's another bug in it and again I read something I haven't sent - replyMsg.parts and replyMsg.index is always 0 although old implementation says they're for example 3 and 10. What's wrong this time? As you can see I take care of padding and I use read instead of operator>>. If you wonder why I read that struct field by field here's an answer: A server sends two different structures, both beginning with msg_id, one if it succeceeds and another if it fails. Right now, I simply have no idea how to do it other way.
You're using formatted input, as though the data being sent were textual -- you need unformatted input. Read about the std::istream::read member function, as it's what you should be using rather than operator>>.
Note that this would have been immediately obvious if you had been checking the stream state after each extraction, as one always should in non-throw-away code.
You forgot about padding. Your request structure probably has at least three bytes inserted by the compiler between the first and the second member, as in:
struct request {
uint8_t msg_type;
char __pad__[3]; // or 7 on 64-bit machine.
uint64_t key;
};
You can fix that, say in GCC, with attributes (see the GCC manual):
struct __attribute__ ((__packed__)) request { ...
And yes, I did miss the fact that you are trying to read text instead of binary. Fix that first, get bitten by alignment/padding later :)
Related
I would like to serialize/deserialize some structured data in order to send it over the network via a char* buffer.
More precisely, suppose I have a message of type struct Message.
struct Message {
Header header;
Address address;
size_t size; // size of data part
char* data;
} message
In C, I would use something such as:
size = sizeof(Header) + sizeof(Address) + sizeof(size_t) + message.size;
memcpy(buffer, (char *) message, size);
to serialize, and
Message m = (Message) buffer;
to deserialize.
What would be the "right" way to do it in C++. Is it better to define a class rather than a struct. Should I overload some operators? are there alignment issues to consider?
EDIT: thanks for pointing the "char *" problem. The provided C version is incorrect. The data section pointed to by the data field should be copied separately.
Actually there are many flavors:
You can boost let it do for you: http://www.boost.org/doc/libs/1_52_0/libs/serialization/doc/tutorial.html
Overloading the stream operators << for serialization and >> for deserialization works well with file and string streams
You could specify a constructor Message (const char*) for constructing from a char*.
I am a fan of static methods for deserialization like:
Message {
...
static bool desirialize (Message& dest, char* source);
}
since you could catch errors directly when deserializing.
And the version you proposed is ok, when applying the modifications in the comments are respected.
Why not insert a virtual 'NetworkSerializable' Class into your inheritance tree? A 'void NetSend(fd socket)' method would send stuff, (without exposing any private data), and 'int(bufferClass buffer)' could return -1 if no complete, valid message was deserilalized, or, if a valid message has been assembled, the number of unused chars in 'buffer'.
That encapsulates all the assembly/disassembly protocol state vars and other gunge inside the class, where it belongs. It also allows message/s to be assembled from multiple stream input buffers.
I'm not a fan of static methods. Protocol state data associated with deserialization should be per-instance, (thread-safety).
i am working on a network project of mine in order to learn more about networking and right now i have designed a simple protocol/structure that i fill and send to the server, the problem is that all vectors and probably arrays aswell are invalid on the server side.
im gonna try to explain it with code, its alot easier that way.
My protocol:
typedef struct NETWORK_PROTOCOL {
int packet_size;
int number_of_data_files;
std::vector<std::string> data_files;
}
so its a pretty simple protocol, and what i did is that i fill it with data and its completely valid on the client side, however as soon as i send it to the server and try to convert it back it the vector is invalid but the integers are still valid.
this is how i create and send the data from the client:
NETWORK_PROTOCOL Protocol;
//Fills protocol with data
int sendt = send(ClientSocket, (const char*)&Protocol, Protocol.packet_size, 0);
and when it hits the server i still get the full size of the data, but as i said earlier it does not convert back properly :/
Code on the server side that tries to cast it back:
NETWORK_PROTOCOL* Protocol;
iResult = recv(ClientSocket, buffer, BUFFLEN, 0);
//then i have some validation code to check if the whole packet arrived since its TCP
Protocol = reinterpret_cast<NETWORK_PROTOCOL*>(buffer);
//And now the vector is invalid :/
im not really sure how to fix this problem, i thought it would be easy to convert it back since it is the exact same data on both sides. Any help to fix this issue is greatly appreciated.
std::vector can't be transferred this way: internally it uses pointers, so you send only a pointer, without any actual information, and that pointer is not valid on the receiving side.
In order to send the contents of vector, you need to somehow serialize it (convert it to the representation in which it can be easily transferred). For example, you can use is Boost.Serialization
#include <sstream>
// include headers that implement a archive in simple text format
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/serialization/vector.hpp>
struct NETWORK_PROTOCOL
{
private:
friend class boost::serialization::access;
// When the class Archive corresponds to an output archive, the
// & operator is defined similar to <<. Likewise, when the class Archive
// is a type of input archive the & operator is defined similar to >>.
template<class Archive>
void serialize(Archive & ar, const unsigned int version)
{
ar & packet_size;
ar & number_of_data_files; // you don't actually need it
ar & data_files;
}
public:
int packet_size;
int number_of_data_files;
std::vector<std::string> data_files;
};
Now you can serialize it like this:
std::ostringstream ofs;
boost::archive::text_oarchive oa(ofs);
oa << protocol; // protocol is your instance of NETWORK_PROTOCOL, which you want to send
// and then you'll be able to get a buffer from ofs using ofs.str()
Deserialize it like this:
NETWORK_PROTOCOL protocol;
std::istringstream ifs(buf);
boost::archive::text_iarchive ia(ifs);
ia >> protocol;
For practical usages you may want to use binary archives instead. If you decide to go with boost.serialization, I recommend starting looking here.
You may also like Google Protocol Buffers: https://developers.google.com/protocol-buffers/docs/cpptutorial
This comment is longer than allowed. So I put it as an answer; although I think it answers partially.
To send all the data in one shipment wasted space and bandwidth, because you'd have to take a maximum for the number of names and their sizes. So I suggest you divide your transmission in phases.
In the first phase you send the number of filenames that you are transmitting. In this way you prepare the server for receiving n file names. Then in the second phase you make a loop divided in two transmissions. The first transmission you send the file name size, then you prepare a buffer for receiving the filename.
For these modes you only use basic types (size_t and char *).
On the server side you can build your vector<string>, if you want to give that illusion
I hope it helps you
I am trying to pass data from an x64 app to a x86 app using named pipes and overlapped I/O like what is defined here:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365603(v=vs.85).aspx
My server application's call to WriteFileEx succeeds and the structure I am sending through the pipe seems ok, however when I call ReadFile on the client side the data structure I retrieve is corrupted or different to the data that I sent, but it also successfully reads.
My client application has a unicode character set and the server's character set is 'not set', which I assume defaults to multibyte. I'm not in a position to change the server's character set to unicode.
Would this data corruption just be because I need to convert from multibyte to wide char on the client after I retrieve / read the data structure? If so is there built in helper functions that I can call on do to that?
Data structure being sent (defined identically on the server and client):
typedef struct
{
int id;
float vertices[VERTICES_COUNT][VERTICES_COMPONENTS];
unsigned short indices[INDICES_COUNT];
float texCoords[TEXTURE_COORD_COUNT][TEXTURE_COORD_COMPONENT];
unsigned char texData[TEXTURE_SIZE];
} MESHINST, *LPMESHINST;
typedef struct
{
OVERLAPPED oOverlap;
HANDLE pipeInst;
int addedCount;
MESHINST meshes[MESH_GROUP_BUFFER];
int removedCount;
int removed[MESH_REMOVE_BUFFER];
} MESHGROUPINST, *LPMESHGROUPINST;
WriteFileEx call on the server:
LPMESHGROUPINST meshes = (LPMESHGROUPINST)lpOverLap;
fWrite = WriteFileEx(
meshes->pipeInst,
(wchar_t*)meshes,
sizeof(MESHGROUPINST),
(LPOVERLAPPED)meshes,
(LPOVERLAPPED_COMPLETION_ROUTINE)CompletedWriteRoutine);
ReadFile call on the client:
(in header)
MESHGROUPINST _meshes;
(in cpp)
do
{
_success = ReadFile(
_pipe,
(wchar_t*)&_meshes,
sizeof(MESHGROUPINST),
&_numOfBytesRead,
NULL);
} while (!_success);
What is the type of _meshes in the ReadFile call? If it's a pointer, you'll be reading into the pointer, not the data being pointed to:
&_meshes
Should be:
_meshes
Also, it looks like you're writing process-specific HANDLE and OVERLAPPED info. Did you mean to write those?
You'll need to add more code for better help.
You need to ensure the structure is sent and received with 1-byte packing. Use #pragma pack(1) around the struct you wish to send/receive:
#pragma pack(1)
typedef struct
{
int id;
float vertices[VERTICES_COUNT][VERTICES_COMPONENTS];
unsigned short indices[INDICES_COUNT];
float texCoords[TEXTURE_COORD_COUNT][TEXTURE_COORD_COMPONENT];
unsigned char texData[TEXTURE_SIZE];
} MESHINST, *LPMESHINST;
#pragma pack()
I am trying to create C-style structs in Clojure, so I can call a poorly documented C++ API from Clojure.
The API is designed to send and receive serialized protobuf messages (the good) preceded by a C Header struct (the bad). The initial handshake is an RPCHandshakeHeader struct and the process is roughly described in the code below:
struct RPCHandshakeHeader {
char magic[8];
int version;
static const char REQUEST_MAGIC[9];
static const char RESPONSE_MAGIC[9];
};
[...snip...]
const char RPCHandshakeHeader::REQUEST_MAGIC[9] = "FooBar?\n";
[...snip...]
RPCHandshakeHeader header;
memcpy(header.magic, RPCHandshakeHeader::REQUEST_MAGIC, sizeof(header.magic));
header.version = 1;
socket = new CActiveSocket();
socket->Initialize();
socket->Open((const uint8 *)"localhost", 5000);
socket->Send((uint8*)&header, sizeof(header));
[...code to read response...]
How can I do this in clojure? Do I need to use JNA/JNI?
Is there a way to create a C struct, turn it into binary and send it over a socket? (I think this is what I need to do)
Sounds like a job for gloss! I don't know the details of this part of the API, but you want to look particularly at compile-frame, and repeated for the character strings.
I'm trying to serialize objects to send over network through a socket using only STL. I'm not finding a way to keep objects' structure to be deserialized in the other host. I tried converting to string, to char* and I've spent a long time searching for tutorials on the internet and until now I have found nothing.
Is there a way to do it only with STL?
Are there any good tutorials?
I am almost trying boost, but if there is how to do it with STL I'd like to learn.
You can serialize with anything. All serialization means is that you are converting the object to bytes so that you can send it over a stream (like an std::ostream) and read it with another (like an std::istream). Just override operator <<(std::ostream&, const T&) and operator >>(std::istream&, T&) where T is each of your types. And all the types contained in your types.
However, you should probably just use an already-existing library (Boost is pretty nice). There are tons of things that a library like Boost does for you, like byte-ordering, taking care of common objects (like arrays and all the stuff from the standard library), providing a consistent means of performing serialization and tons of other stuff.
My first question will be: do you want serialization or messaging ?
It might seem stupid at first, since you asked for serialization, but then I have always distinguished the two terms.
Serialization is about taking a snapshot of your memory and restoring it later on. Each object is represented as a separate entity (though they might be composed)
Messaging is about sending information from one point to another. The message usually has its own grammar and may not reflect the organization of your Business Model.
Too often I've seen people using Serialization where Messaging should have been used. It does not mean that Serialization is useless, but it does mean that you should think ahead of times. It's quite difficult to alter the BOM once you have decided to serialize it, especially if you decide to relocate some part of information (move it from one object to another)... because how then are you going to decode the "old" serialized version ?
Now that that's been cleared up...
... I will recommend Google's Protocol Buffer.
You could perfectly rewrite your own using the STL, but you would end up doing work that has already been done, and unless you wish to learn from it, it's quite pointless.
One great thing about protobuf is that it's language agnostic in a way: ie you can generate the encoder/decoder of a given message for C++, Java or Python. The use of Python is nice for message injection (testing) or message decoding (to check the output of a logged message). It's not something that would come easy were you to use the STL.
Serializing C++ Objects over a Network Socket
This is 6 years late but I just recently had this problem and this was one of the threads that I came across in my search on how to serialize object through a network socket in C++. This solution uses just 2 or 3 lines of code. There are a lot of answers that I found work but the easiest that I found was to use reinterpret_cast<obj*>(target) to convert the class or structure into an array of characters and feed it through the socket. Here's an example.
Class to be serialized:
/* myclass.h */
#ifndef MYCLASS_H
#define MYCLASS_H
class MyClass
{
public:
int A;
int B;
MyClass(){A=1;B=2;}
~MyClass(){}
};
#endif
Server Program:
/* server.cpp */
#include "myclass.h"
int main (int argc, char** argv)
{
// Open socket connection.
// ...
// Loop continuously until terminated.
while(1)
{
// Read serialized data from socket.
char buf[sizeof(MyClass)];
read(newsockfd,buf, sizeof(MyClass));
MyClass *msg = reinterpret_cast<MyClass*>(buf);
std::cout << "A = " << std::to_string(msg->A) << std::endl;
std::cout << "B = " << std::to_string(msg->B) << std::endl;
}
// Close socket connection.
// ...
return 0;
}
Client Program:
/* client.cpp */
#include "myClass.h"
int main(int argc, char *argv[])
{
// Open socket connection.
// ...
while(1)
{
printf("Please enter the message: ");
bzero(buffer,256);
fgets(buffer,255,stdin);
MyClass msg;
msg.A = 1;
msg.B = 2;
// Write serialized data to socket.
char* tmp = reinterpret_cast<char*>(&msg);
write(sockfd,tmp, sizeof(MyClass));
}
// Close socket connection.
// ...
return 0;
}
Compile both server.cpp and client.cpp using g++ with -std=c++11 as an option. You can then open two terminals and run both programs, however, start the server program before the client so that it has something to connect to.
Hope this helps.
I got it!
I used strinstream to serialize objects and I sent it as a message using the stringstream's method str() and so string's c_str().
Look.
class Object {
public:
int a;
string b;
void methodSample1 ();
void methosSample2 ();
friend ostream& operator<< (ostream& out, Object& object) {
out << object.a << " " << object.b; //The space (" ") is necessari for separete elements
return out;
}
friend istream& operator>> (istream& in, Object& object) {
in >> object.a;
in >> object.b;
return in;
}
};
/* Server side */
int main () {
Object o;
stringstream ss;
o.a = 1;
o.b = 2;
ss << o; //serialize
write (socket, ss.str().c_str(), 20); //send - the buffer size must be adjusted, it's a sample
}
/* Client side */
int main () {
Object o2;
stringstream ss2;
char buffer[20];
string temp;
read (socket, buffer, 20); //receive
temp.assign(buffer);
ss << temp;
ss >> o2; //unserialize
}
I'm not sure if is necessary convert to string before to serialize (ss << o), maybe is possible directly from char.
I think you should use google Protocol Buffers in your project.In network transport Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:
are simpler
are 3 to 10 times smaller
are 20 to 100 times faster
are less ambiguous
generate data access classes that are easier to use programmaticall
and so on. I think you need read https://developers.google.com/protocol-buffers/docs/overview about protobuf