Could someone please help and tell me how to use protocol buffers. Actually I want to exchange data through sockets between a program running on unix and anoother running on windows in order to run simulation studies.
The programs that use sockets to exchange data, are written in C/C++ and I would be glad if somneone could help me to use protocol buffers in order to exchange data in the form of :
struct snd_data{
char *var="temp";
int var1=1;
float var2;
double var2;
}
I tried several ways, but still data are not exchanged correctly. Any help would be very appreciated
Thanks for your help,
You start by defining your message in a .proto file:
package foo;
message snd_data {
required string var= 1;
required int32 var1 = 2;
optional float var2 = 3;
optional double var3 = 4;
}
(I guess the float and double actually are different variables...)
Then you compile it using protoc and then you have code implementing your buffer.
For further information see: http://code.google.com/apis/protocolbuffers/docs/cpptutorial.html
How are you writing your messages to the socket? Protobufs is not endian-sensitive itself, but neither does protobufs define a transport mechanism -- protobuf defines a mapping between a message and its serialized form (which is a sequence of (8-bit) bytes) and it is your responsibility to transfer this sequence of bytes to the remote host.
In our case, we define a very simple transport protocol; first we write the message size as an 32-bit integer (big endian), then comes the message itself. (Also remember that protobuf messages are not self-identifying, which means that you need to know which message you are sending. This is typically managed by having a wrapper message containing optional fields for all messages you want to send. See the protobuf website and mailing list archives for more info about this technique.)
Endianess is handled within protobuf.
See:
https://groups.google.com/forum/?fromgroups#!topic/protobuf/XbzBwCj4yL8
How cross-platform is Google's Protocol Buffer's handling of floating-point types in practice?
Are both machines x86? Otherwise you need to watch for big endian and little endian differences. Its also worth paying attention to struct packing. Passing pointer can be problematic too due to the fact pointer are different sizes on different platforms. All in there is far too little information in your post to say, for certain, what is going wrong ...
The answer lies in the endianess of the data being transmitted, this is something you need to consider very carefully and check. Look here to show what endianness can do and cause data to get messed up on both the receiver and sender. There is no such perfect measure of transferring data smoothly, just because data sent from a unix box guarantees the data on the windows box will be in the same order in terms of memory structure for the data. Also the padding of the structure on the unix box will be different to the padding on the windows box, it boils down to how the command line switches are used, think structure alignment.
Related
I know that this might be a silly question, but I am a newbie C++ developer and I need some clarifications about the endianness.
I have to implement a communication interface that relies on SCTP protocol in order to communicate between two different machines (one ARM based, and the other Intel based).
The aim is to:
encode messages into a stream of bytes to be sent on the socket (I used a vector of uint8_t, and positioned each byte of the different fields -taking care of splitting uint16/32/64 to single bytes- following big-endian convention)
send the bytestream via socket to the receiver (using stcp)
retrieve the stream and parse it in order to fill the message object with the correct elements (represented by header + TV information elements)
I am confused on where I could have problem with the endianness of the underlying architecture of the 2 machines in where the interface will be used.
I think that taking care of splitting objects into single bytes and positioning them using big-endian can preclude that, at the arrival, the stream is represented differently, right? or am I missing something?
Also, I am in doubt about the role of C++ representation of multiple-byte variables, for example:
uint16_t var=0x0123;
//low byte 0x23
uint8_t low = (uint8_t)var;
//hi byte 0x01
uint8_t hi = (uint8_t)(var >> 8);
This piece of code is endianness dependent or not? i.e. if I work on a big-endian machine I suppose that the above code is ok, but if it is little-endian, will I pick up the bytes in different order?
I've searched already for such questions but no one gave me a clear reply, so I have still doubts on this.
Thank you all in advance guys, have a nice day!
This piece of code is endianness dependent or not?
No the code doesn't depend on endianess of the target machine. Bitwise operations work the same way as e.g. mathematical operators do.
They are independent of the internal representation of the numbers.
Though if you're exchanging data over the wire, you need to have a defined byte order known at both sides. Usually that's network byte ordering (i.e. big endian).
The functions of the htonx() ntohx() family will help you do en-/decode the (multibyte) numbers correctly and transparently.
The code you presented is endian-independent, and likely the correct approach for your use case.
What won't work, and is not portable, is code that depends on the memory layout of objects:
// Don't do this!
uint16_t var=0x0123;
auto p = reinterpret_cast<char*>(&var);
uint8_t hi = p[0]; // 0x01 or 0x23 (probably!)
uint8_t lo = p[1]; // 0x23 or 0x01 (probably!)
(I've written probably in the comments to show that these are the likely real-world values, rather than anything specified by Standard C++)
I have a struct that has 5 unsigned 8 bit integers that mimics a frame with 5 packets. After researching, I know need to serialize the data, field by field, especially since I am sending from a Windows machine to a Linux machine and back.
here is my struct:
typedef struct pressure{
UINT8 a;
UINT8 b;
UINT8 c;
UINT8 d;
UINT8 e;
}pressure;
The issue is I cant use htons() since my members must be 8 bits. How do I manually serialize this? It would be greatly appreciated if you could provide a short code sample that shows how to serialize and what to pass to send().
You can either write each individual byte using ostream::put, or - if you've ensured they're contiguous in memory in pressure (which will be done on every compiler I've ever used without you doing anything actively) - write the lot of them using ostream::write, as in:
my_ostream.write(static_cast<const char*>(&my_pressure.a), 5);
That said, consider keeping the values in an array so you're guaranteed they're contiguous in memory.
You don't need htonX / ntohX etc. - they're for normalising/denormalising multi-byte integer representations, which you don't have here.
Just write it if you are sending from an Intel x86 based machine to another Intel x86 machine (which most linuxes are).
If however you plan to send it to a machine based on another processor the safest thing is to just send ASCI characters so something like:
char[] buffer = sprintf("|%03d|%03d|%03d|%03d|%03d|",a,b,c,d,e);
Would give you a fixed length string readable by any machine. Its a good idea to have some sort of field separator (the "|" in this case) to help the receiver verify the string has not been garbled by an unreliable network.
Getting into socket programming with C++, and I'd like to find a way to marshal a series of data types into a string of bytes to be sent over a network via a socket using UDP.
Here's the method header
char * Message::marshal( int &n)
and the series of data types I need to marshal are:
int int int int float //need to be all marshaled into the same byte string
The length is saved to n, and the string of bytes is returned in a character array. I'd also like to know how to unmarshal that string of bytes as well.
The server I'm working on most likely doesn't have boost library, and is probably not updated to C++ 11. The last server I was in wasn't, so I'm going to assume they didn't update them yet.
One method of serializing I saw was using streams, but I saw something about non portability (I'm writing the source code in windows, and then I'll change a few things for unix and compile it on the unix server).
I need to use standard C++ library stuff as again I have no control over the server.
I eventually found this post that led me to a solution. Didn't bother converting double to network byte order since the client and server are on the same machine. And for UDP, make sure to cast the structure as a char*.
I need to send a packet of data over a UDP connection in C++. The first message I need to send is built up of two 32 bit integers and a 64 bit integer. What is the best way to combine multiple variable types into one block of data ready for sending over a UDP connection?
It depends on the requirements for your network. Do you care about endianness? If you do, you should use not just any serialisatioin, but a safe one in regards to endianness.
Generally, each class/struct sendable through the network should have special methods or overloaded operators to stream them in and out. Ultimately you'll have to use macros/functions like hton/ntoh for streaming primitive types eg int, int64, float, double etc.
Upd: also if your network endpoint applications run on different platforms/compilers, you may have different sizes of int, long, short etc. So when serialising, you'll have to convert your integers to some predefined types with sizes guaranteed to be the same on all supported platforms.
I wrote a DNS resolver by hand in C, and structs is indeed the way I did it. Use bit fields to specify sizes of each piece. More about bit fields: http://msdn.microsoft.com/en-us/library/ewwyfdbe.aspx
Make sure to use hton/ntoh to take care of byte order. More information here; http://www.beej.us/guide/bgnet/output/html/multipage/htonsman.html
In fact, peruse beej's guide -- mucho useful information there!
In C++ we send data using socket on the network. I am aware that we need to use htons() , ntohs() function to maintain byte order big endian and little endian.
support we have following data to be sent
int roll;
int id;
char name[100];
This can also be wrapped into struct.
My confusion here is, for roll and id, we can use htons() function. But for the string name, what should and how should we do it? Do we need to use any such function? will it work on every machine like mac, intel and other network?
I want to send all three fields in one packet.
You'd use htonl for int, not htons.
The name doesn't need to be reordered, since the bytes of the array correspond directly to bytes on the network.
The issue of byte-order only arises with words larger than a byte, since different architectures choose different ends at which to place the least-significant byte.
For char arrays this conversion is not necessary since they do not have a network byte order but are sequentially transmitted. The reason that ntohs and htons exist, is that some data types consist of lesser and more significant bits, which are interpreted differently on different architectures. This is not the case in strings.
To add to helpful comments here - if your structs get much more complex you could be better off considering a serialization library like Boost.Serialization or Google Protocol Buffers, which handle endianness for you under the covers.
When encoding the string, make sure you send a length (probably a short handled using htons) before the string itself, don't just send 100 chars every time.