JSON or protocol buffer to replace the custom TCP message - c++

Originally, We have two applications communication with TCP/IP, and both of them are implemented by C++. The messages between them are custom message type.
Now the client program will be changed to web application based on nodejs and the communication between them will be changed to message bus such as rabbitmq
So the message type between them should be changed.
JSON comes to my mind first, however the custom messages are too complicated, which are defined by template and inheritance. It seems that convert the custom message to JSON is not a good option. Am I right?
class Address {
int network;
int addressType;
//...
};
class MsgType{
unsigned char msgSeq;
unsigned int msgLen;
//...
};
class Message{
Address destination;
Address source;
MsgType msgType;
//...
};
template <typename T, int RESPONSE_TYPE>
class ResponseMessage : public Message{
//...
}
typedef struct{
int number;
int type;
}ConfigResp;
class CfgResp : public ResponseMessage<ConfigResp, CONFIG_REQUEST>
{
//...
}
Protocol Buffers is another option for me to do that. What should I do?
redefine the custom message into protocol buffer? no no
Here is my solution: Just wrap the whole original custom message (binary type) into protocol buffer as one message in the server side, then decode the custom message(binary type) in client side. Is that possible?

It looks like you are structuring your application to become more extensible. Not using a nice message format is completely counter to that aim.
Don't embed your binary format inside a protocol buffer chunk. You'll gain nothing - you'll need to rewrite parsing and writing code for each component that wants to use the message bus. Thats wasted time and effort.
There is pain in mapping your C++ structures to JSON or protocol buffers. But it will make hooking into those messages using node.js or other things peeking into the message bus much easier later.
Personally I'd use protocol buffers - since they're more type safe. And there are differences between handling of various types in JSON libraries, because the JSON format is (intentionally) lax. In particular I've found handling of long integers problematic.
Typically I write a helper template struct for each class I need to convert - then conversion becomes a lot of boilerplate. Something like
template<typename T> class ProtocolBufferHelper {
}
template<> class ProtocolBufferHelper<Address> {
typedef address_buffer protocol_buffer_type;
void writeToProtocolBuffer( const Address &a, address_buffer & buffer) {
buffer.setNetwork(a.network);
...
}
...
}
template<> class ProtocolBufferHelper<Message> {
void writeToProtocolBuffer( const Message &m, address_buffer & buffer) {
::writeToProtocolBuffer(buffer.getDestination(), m.destination);
::writeToProtocolBuffer(buffer.getSource(), m.source);
...
}
}
template<typename T> void writeToProtocolBuffer( const T &value, ProtocolBufferHelper<T>::protocol_buffer_type & buffer ) {
ProtocolBufferHelper<T>::writeToProtocolBuffer(value, buffer);
}
You'll have to forgive me for not remembering exactly what the protocol buffer syntax is in C++ (its been a while...). but hopefully its enough to get you started.

Related

Packet wrangling from C to C++ or... how Neither Variant nor Any is a Union

The problem in a nutshell: I'm considering a rewrite of some packet handling code for a UDP protocol ... from C to C++. Obviously, I can just use the structures that I had with C to model the data, but then what would be the point? So, for the sake of argument, we have:
struct { uint8:2 type; uint8:6 con_id; uint16 seq_num } packet_header;
struct { packet_header ph; uint16 command; } control_packet;
struct { packet_header ph; uchar8 data[DATASIZE]; } data_packet;
In this case, "type" is 0 for data, 1 for control, 2 for encrypted data and 3 for encrypted control. There would also be structs reprenting each of the control packets, for argument, lets say "open" and "close".
Now... I would normally have a type that the function that fetches the packet returns and the function that sends accepts:
struct {
size_t pkt_len;
union {
uchar8 raw_packet[DATASIZE];
packet_header ph;
control_packet cp;
control_open_packet cop;
control_close_packet ccp;
data_packet dp;
} p;
} packet;
... this is all simplified for argument... but the point is: when sending a packet, we want to create a packet of a certain type and send a generic packet. Conversely, when receiving a packet, you want to receive a generic packet and and cast it to more specific types as you classify it.
So where's the question? My question is: if Variant and Any are not appropriate ways to express this, what are? Am I stuck with union in this case? Certainly I don't violate the stipulation of POD types (so far) for the union. I've been scouring both C++11 and Boost (and to some extent looking at C++14 ... although I'm stuck with C++11) and I don't see offered solutions.
You can use both union as it is in your example or class hierarchy with packet_header a the top and specific derived classes for each type of the packet:
struct packet_header { uint8:2 type; uint8:6 con_id; uint16 seq_num };
struct control_packet : packet_header { uint16 command; };
struct data_packet: packet_header { uchar8 data[DATASIZE]; };
struct packet
{
size_t pkt_len;
union
{
uchar8 raw_packet[DATASIZE];
packet_header ph;
control_packet cp;
control_open_packet cop;
control_close_packet ccp;
data_packet dp;
};
};
Maybe you need to seek for elegance and flexibility in the code that handles your packets. I doubt that defining data in some other way (using boost or using C++17 features) will lead to a considerable gain.
Note the anonymous union in the struct packet. It allows accessing fields of specific packet types directly. C++ style expects the name of the struct directly after the struct keyword.

generic message dispatching library?

Is there a standard way to get rid of the switch/case block in a read loop?
i.e.
enum msg_type
{
message_type_1,
//msg types
}
struct header
{
msg_type _msg_type;
uint64_t _length;
}
struct message1
{
header _header;
//fields
}
struct message2
{
header _header;
//fields
}
//socket read loop
void read(//blah)
{
//suppose we have full message here
char* buffer; //the buffer that holds data
header* h = (header*)buffer;
msg_type type = h->_msg_type;
switch(type)
{
case msg_type_1:
message1* msg1 = (message1*)buffer;
//Call handler function for this type
//rest
}
}
this means that I have to inherit from a handler container base class which is of the form:
class handler_container_base
{
public:
virtual void handle(message1* msg){}
virtual void handle(message2* msg){}
//etc
}
and pass an object of that type to where the message loop can see and ask him to call those back.
One problem is, even when I want to implement and register only one handler for a single type I have to inherit from this class.
Another is this just looks ugly.
I was wondering if there are existing libraries which handle this problem (should be free). Or is there no better way of doing this rather than like this?
Other approaches that avoid inheritance are:
For a closed set of types:
Use a variant:
variant<message1_t, message2_t> my_message;
With a visitor you can do the rest. I recommend boost.variant.
You can also use a boost::any, for an open set of types, and copy the messages around at runtime.At some point you will have to cast back to the original type, though.
Another solution goes along the lines of Poco.DynamicAny, which will try to convert, to the type on the left in an assignment, similar to a dynamic language. But you need to register converters yourself for your types.

Storing a message meant for serial communication as a member variable in c++ viable solution?

I am creating some C++ libraries from some simple C functions that primarily have to do with USB serial communication. I am writing it with visual studio 2010 with the intent to port to mac and linux.
I currently have a SerialCommunication class and a PumpComm subclass. Currently, there is a C function for CreateMessage(uint8 msg[]), SendMessage(uint8 msg[]), ReadSerial(uint8 readmsg[]) etc.
My question is: are there any disadvantages from changing msg[] to a member variable rather than a function parameter? (pseudo code):
int main()
{
SerialPort sp;
sp.OpenSerial("COM19");
sp.CreateMessage("Messagetosend1234");
sp.SendMsg();
}
class SerialPort
{
private:
uint8 msg[];
public:
int32 OpenSerial();
int32 CreateMessage(uint8 msg[]);
}
The advantages I see include: 1) the msg is clearly owned by the serial port it will be sent to, 2) the scope of the msg is the scope of the serial port, 3) the implementation of the msg can be hidden (when I add getters/setters).. So basically advantages of OOP. I'm fairly new to c++ and OOP so I may be missing something, especially when it comes to serial communication. Thanks for your time.
Are messages part of the COM channel? I'd say you send messages over the channel. You design is tightly coupling the "how" (channel) with "what" (message). Also, you implicitly add a lot of state dependencies to your code - for example you have to call CreateMessage before calling SendMsg.
To sum up, there are some things that might backfire at you later:
If you need to access the channel from multiple threads
If your message becomes more than a byte buffer
If you refactor your code and mistakenly swap some calls
Therefore I'd keep message and channel two separate classes:
class Message {
private:
uint8 *buffer;
size_t length;
public:
Message(uint8 *buf, size_t len) : buffer(buf), length(len) { }
size_t getLength() const { return length; }
const uint8 *getData() const { return buffer; }
};
class SerialPort
{
public:
int32 OpenSerial();
int32 SendMsg(const Message& msg);
}

How Protocol buffer interact with legacy protocol code

every one, I have a question about how protocol buffer interact with existed protocol mechanism,Say code below:
class PacketBase
{
public:
PacketBase();
private:
int msgType;
int msgLen;
private:
MessageBuilder* m_pMsgBuilder; /// do Write and Read From msg stream
};
class LoginRequest : public PacketBase
{
/// here we can use proto replace the msg field defined here
/// invoke write or read method in packetbase to serialze or de-serialize
};
Can protocol buffer do the job while maintain the class hierarchy unchanged?
Well, since your msgType and msgLen fields are both private, I think your question boils down to "Can I replace LoginRequest with a protocol buffer that has a no-args constructor?" and the answer here is an unqualified "yes".
Protocol buffers take care of (de-)serializing fields while maintaining type information; there's really no reason for you to write that code yourself. If you were to create a LoginRequest protobuf, for example, you could just construct it, call setters to set its fields, and then serialize it to an ostringstream. That's probably sufficient for what you want, right?

C++ design - Network packets and serialization

I have, for my game, a Packet class, which represents network packet and consists basically of an array of data, and some pure virtual functions
I would then like to have classes deriving from Packet, for example: StatePacket, PauseRequestPacket, etc. Each one of these sub-classes would implement the virtual functions, Handle(), which would be called by the networking engine when one of these packets is received so that it can do it's job, several get/set functions which would read and set fields in the array of data.
So I have two problems:
The (abstract) Packet class would need to be copyable and assignable, but without slicing, keeping all the fields of the derived class. It may even be possible that the derived class will have no extra fields, only function, which would work with the array on the base class. How can I achieve that?
When serializing, I would give each sub-class an unique numeric ID, and then write it to the stream before the sub-class' own serialization. But for unserialization, how would I map the read ID to the appropriate sub-class to instanciate it?
If anyone want's any clarifications, just ask.
-- Thank you
Edit: I'm not quite happy with it, but that's what I managed:
Packet.h: http://pastebin.com/f512e52f1
Packet.cpp: http://pastebin.com/f5d535d19
PacketFactory.h: http://pastebin.com/f29b7d637
PacketFactory.cpp: http://pastebin.com/f689edd9b
PacketAcknowledge.h: http://pastebin.com/f50f13d6f
PacketAcknowledge.cpp: http://pastebin.com/f62d34eef
If someone has the time to look at it and suggest any improvements, I'd be thankful.
Yes, I'm aware of the factory pattern, but how would I code it to construct each class? A giant switch statement? That would also duplicade the ID for each class (once in the factory and one in the serializator), which I'd like to avoid.
For copying you need to write a clone function, since a constructor cannot be virtual:
virtual Packet * clone() const = 0;
Which each Packet implementation implement like this:
virtual Packet * clone() const {
return new StatePacket(*this);
}
for example for StatePacket. Packet classes should be immutable. Once a packet is received, its data can either be copied out, or thrown away. So a assignment operator is not required. Make the assignment operator private and don't define it, which will effectively forbid assigning packages.
For de-serialization, you use the factory pattern: create a class which creates the right message type given the message id. For this, you can either use a switch statement over the known message IDs, or a map like this:
struct MessageFactory {
std::map<Packet::IdType, Packet (*)()> map;
MessageFactory() {
map[StatePacket::Id] = &StatePacket::createInstance;
// ... all other
}
Packet * createInstance(Packet::IdType id) {
return map[id]();
}
} globalMessageFactory;
Indeed, you should add check like whether the id is really known and such stuff. That's only the rough idea.
You need to look up the Factory Pattern.
The factory looks at the incomming data and created an object of the correct class for you.
To have a Factory class that does not know about all the types ahead of time you need to provide a singleton where each class registers itself. I always get the syntax for defining static members of a template class wrong, so do not just cut&paste this:
class Packet { ... };
typedef Packet* (*packet_creator)();
class Factory {
public:
bool add_type(int id, packet_creator) {
map_[id] = packet_creator; return true;
}
};
template<typename T>
class register_with_factory {
public:
static Packet * create() { return new T; }
static bool registered;
};
template<typename T>
bool register_with_factory<T>::registered = Factory::add_type(T::id(), create);
class MyPacket : private register_with_factory<MyPacket>, public Packet {
//... your stuff here...
static int id() { return /* some number that you decide */; }
};
Why do we, myself included, always make such simple problems so complicated?
Perhaps I'm off base here. But I have to wonder: Is this really the best design for your needs?
By and large, function-only inheritance can be better achieved through function/method pointers, or aggregation/delegation and the passing around of data objects, than through polymorphism.
Polymorphism is a very powerful and useful tool. But it's only one of many tools available to us.
It looks like each subclass of Packet will need its own Marshalling and Unmarshalling code. Perhaps inheriting Packet's Marshalling/Unmarshalling code? Perhaps extending it? All on top of handle() and whatever else is required.
That's a lot of code.
While substantially more kludgey, it might be shorter & faster to implement Packet's data as a struct/union attribute of the Packet class.
Marshalling and Unmarshalling would then be centralized.
Depending on your architecture, it could be as simple as write(&data). Assuming there are no big/little-endian issues between your client/server systems, and no padding issues. (E.g. sizeof(data) is the same on both systems.)
Write(&data)/read(&data) is a bug-prone technique. But it's often a very fast way to write the first draft. Later on, when time permits, you can replace it with individual per-attribute type-based Marshalling/Unmarshalling code.
Also: I've taken to storing data that's being sent/received as a struct. You can bitwise copy a struct with operator=(), which at times has been VERY helpful! Though perhaps not so much in this case.
Ultimately, you are going to have a switch statement somewhere on that subclass-id type. The factory technique (which is quite powerful and useful in its own right) does this switch for you, looking up the necessary clone() or copy() method/object.
OR you could do it yourself in Packet. You could just use something as simple as:
( getHandlerPointer( id ) ) ( this )
Another advantage to an approach this kludgey (function pointers), aside from the rapid development time, is that you don't need to constantly allocate and delete a new object for each packet. You can re-use a single packet object over and over again. Or a vector of packets if you wanted to queue them. (Mind you, I'd clear the Packet object before invoking read() again! Just to be safe...)
Depending on your game's network traffic density, allocation/deallocation could get expensive. Then again, premature optimization is the root of all evil. And you could always just roll your own new/delete operators. (Yet more coding overhead...)
What you lose (with function pointers) is the clean segregation of each packet type. Specifically the ability to add new packet types without altering pre-existing code/files.
Example code:
class Packet
{
public:
enum PACKET_TYPES
{
STATE_PACKET = 0,
PAUSE_REQUEST_PACKET,
MAXIMUM_PACKET_TYPES,
FIRST_PACKET_TYPE = STATE_PACKET
};
typedef bool ( * HandlerType ) ( const Packet & );
protected:
/* Note: Initialize handlers to NULL when declared! */
static HandlerType handlers [ MAXIMUM_PACKET_TYPES ];
static HandlerType getHandler( int thePacketType )
{ // My own assert macro...
UASSERT( thePacketType, >=, FIRST_PACKET_TYPE );
UASSERT( thePacketType, <, MAXIMUM_PACKET_TYPES );
UASSERT( handlers [ thePacketType ], !=, HandlerType(NULL) );
return handlers [ thePacketType ];
}
protected:
struct Data
{
// Common data to all packets.
int number;
int type;
union
{
struct
{
int foo;
} statePacket;
struct
{
int bar;
} pauseRequestPacket;
} u;
} data;
public:
//...
bool readFromSocket() { /*read(&data); */ } // Unmarshal
bool writeToSocket() { /*write(&data);*/ } // Marshal
bool handle() { return ( getHandler( data.type ) ) ( * this ); }
}; /* class Packet */
PS: You might dig around with google and grab down cdecl/c++decl. They are very useful programs. Especially when playing around with function pointers.
E.g.:
c++decl> declare foo as function(int) returning pointer to function returning void
void (*foo(int ))()
c++decl> explain void (* getHandler( int ))( const int & );
declare getHandler as function (int) returning pointer to function (reference to const int) returning void