In C++ I have data structure something like this:
struct Data
{
int N;
double R;
char Name[20];
};
This Data I have to send over from a client to server on a different system (I have to send an array of Data structs, but i could send it just one by one). I would like to send it over as binary data, so that I could extract the data on the other end put it inside the same struct type.
If both (client and sever) are compiled with the same compiler the sizeof(Data) and all bit paddings within structure would be the same. But as server is 64bit running Linux and client could be even 32bit windows, the ordering of data within Data could be different.
Am I Right? What would be the best way of dealing with this the problem?
Assuming that all clients and servers will always be built with the same compiler for same architecture and OS is almost always a bad idea. It is better to write code that explicitly packs and unpacks the member of the structure as a stream of bytes with a specified ordering, or converts the data to non-binary formats (e.g. JSON or XML) that can be parsed on the other side.
You could use Boost.Serialization to marshal/unmarshal (i.e. transform your data from your source computer format to one that is suitable for transmission, and from that format to the one used by your destination computer) your data and Boost.Asio to handle communication.
Proper protocol definition would help you to solve this issue. Have a fixed length header followed by variable length data.
Something like this
Header***
[DataSize] - 2 bytes
[MSG ID] - 1 byte
Packet**
[N] - 4 bytes
[R] - 8 bytes
[Len] - 2 bytes
[Name] - Len bytes
read the bytes and fill the structure
Related
I have a u_char* dynamic array having binary data of some network packet. I want to change the destination port number in the packet with some integer value. Suppose that the port number offset within the packet is ofs, with length of 4 bytes.
I tried the following 2 methods:
u_char* packet = new u_char[packet_size]; // Packet still empty
// Read packet from network ...
int new_port = 1234;
Method #1:
std::copy((u_char*)&new_port, (u_char*)&new_port+4, packet+ofs);
Method #2:
std::string new_port_str = std::to_string(new_port);
auto new_port_bytes = new_port_str.c_str();
std::copy(new_port_bytes, new_port_bytes+4, packet+ofs);
Both methods give garbage value for port number (but the rest of the packet is OK). Could anyone help me ?
You have to convert the integer from whatever internal representation your platform happens to use to the format the particular network protocol you're using requires them to be in when sent over the network.
This depends on the particular network protocol you're trying to use -- check its documentation for precisely the format it requires ports to be expressed in. My bet will be it's network byte order. You probably have functions like htons to convert shorts to network byte order.
Another problem -- how many bytes is int on your platform? How many bytes does the network protocol use to express ports? I'll bet the numbers are 4 and 2 respectively. So that's another issue. (Or maybe it isn't. I don't know for sure how many bytes an int is on your platform nor do I know what protocol you're trying to work with, so I have to guess.)
You can't just write code randomly and expect it to work. You have to think about what you're trying to do and understand the requirements.
My recommendation would be to look at the specification for the network protocol you're working with and figure out exactly which bytes in the data have to change and what they have to change to. Then write code to change each byte to the correct value according to the network protocol specification. This will ensure your code works correctly on any platform.
I want to send a payload that has this structure:
uint8_t num_of_products;
//1 product
uint8_t action;
time_t unix_time;//uint64_t
uint32_t name_len;//up to 256
char* name;
uint8_t num_of_ips;
//1 ip out if num_of_ips
uint8_t ip_ver; //0 error , 1 ipv4 , 2 ipv6
char* IP;// ipv4 : 4 , ipv6 : 16
before sending the packet I aggregate products using memcpy into jumbo size mbuf
from tests I did name_len must go through hton in order to not look "inverted" in wireshark.
my question is ,what logic can I apply in order to get the byte order right for a custom structure with inner variables with unknown size
i.e what should go through hton what should be left as is
If you are aiming to have your message in network byte order (big endian), only your integer fields that take up more than one byte will need the htonl treamnet. In your case, that would be time_t unix_time and uint32_t name_len. Your strings and single-byte fields (such as num_of_products) won't need any specific conversion.
As some of the commenters in your question suggested - it's really up to you if you want to use a strict network byte order. Serializing your message to have a strict byte ordering is useful if you intend your code to run across different platforms.
Writing efficient byte packing code is annoyingly hard. You wind up writing a lot of code to just to save a few bytes of network bandwidth.
User jxh mentioned JSON as a possible encoding for your message. Not sure why he deleted his answer because it was on point. But in any case, a standard messaging format of either JSON or XML (or any ascii text schema) is 100x easier to observe in wireshark and when debugging.
I have a server sending a multi-dimensional character array
char buff1[][3] = { {0xff,0xfd,0x18} , {0xff,0xfd,0x1e} , {0xff,0xfd,21} }
In this case the buff1 carries 3 messages (each having 3 characters). There could be multiple instances of buffers on server side with messages of variable length (Note : each message will always have 3 characters). viz
char buff2[][3] = { {0xff,0xfd,0x20},{0xff,0xfd,0x27}}
How should I store the size of these buffers on client side while compiling the code.
The server should send information about the length (and any other structure) of the message with the message as part of the message.
An easy way to do that is to send the number of bytes in the message first, then the bytes in the message. Often you also want to send the version of the protocol (so you can detect mismatches) and maybe even a message id header (so you can send more than one kind of message).
If blazing fast performance isn't the goal (and you are talking over a network interface, which tends to be slower than computers: parsing may be cheap enough that you don't care), using a higher level protocol or format is sometimes a good idea (json, xml, whatever). This also helps with debugging problems, because instead of debugging your custom protocol, you get to debug the higher level format.
Alternatively, you can send some sign that the sequence has terminated. If there is a value that is never a valid sequence element (such as 0,0,0), you could send that to say "no more data". Or you could send each element with a header saying if it is the last element, or the header could say that this element doesn't exist and the last element was the previous one.
struct ss
{ int data ;string name;}*o;
this is my structure in the client application
i want to send that to the server (in TCP Server under windows);
How to do that..
i Know the serilization is the solution.
but i don't know how to do that.. please help me if you can.
http://www.parashift.com/c++-faq-lite/serialization.html
If you use Boost for serialization then read this
Serialize and send objects by TCP using boost
I normally write my own data format for transferring this data.
I will create a character buffer.
Put the size of the packet.
Convert integer to bytes and copy it.
Append the string length
Copy the name
send it across.
First one must know whether client and server are known to be always the same architecture or not. This decides whether you can just send data as it is or whether you must care about endianness and the size of an integer.
In any case, ntohl and htonl will take care of byte ordering and allow you to transfer the int in a simple, defined way (no-op on machines that are already network-byte-order).
About the string, you can send both the size and the contents over TCP just fine (converting the size with htonl), assuming that the string data is either in the same encoding on both sides, or a "general, agnostic" encoding is always used, such as UTF-8.
If you don't know what encodings the machines on both ends are using, you are in trouble. In that case, you must include a message that defines this and convert accordingly (similar to as for example webservers do).
Having TCP operate in "normal mode" means that Nagle's algorithm will be enabled, so you can just use 3 calls to send and the network layer will coerce that into as few packets as it believes is reasonable (instead of sending an individual packet just for an integer).
That all for the simple case in your example, or you can do some proper serialization, which is much more work, of course.
This is a very good guide: http://beej.us/guide/bgnet/output/html/multipage/index.html. it is not strictly for windows but the changes are very slight.
You basically need to serialize your data to a buffer and then use send function. pass your socket identifier and buffer etc...
int send(
__in SOCKET s,
__in const char *buf,
__in int len,
__in int flags
);
from http://msdn.microsoft.com/en-us/library/ms740149(v=vs.85).aspx
// send some data to a socket
send(
socket, // the open socket
o, // pointer to the data
sizeof( ss ), // number of bytes
0 ); // no special flags
The difficulty is at the other end! The recipient needs to know how many bytes to read, and what structure to store them in.
You either need to write your own code to deal with these problems, or use a protocol that works on top of sockets. The code isn't hard, but some experience with some of the many protocols available would help with the gotchas!
Our server is seemingly packet based. It is an adaptation from an old serial based system. It has been added, modified, re-built, etc over the years. Since TCP is a stream protocol and not a packet protocol, sometimes the packets get broken up. The ServerSocket is designed in such a way that when the Client sends data, part of the data contains the size of our message such as 55. Sometimes these packets are split into multiple pieces. They arrive in order but since we do not know how the messages will be split, our server sometimes does not know how to identify the split message.
So, having given you the background information. What is the best method to rebuild the packets as they come in if they are split? We are using C++ Builder 5 (yes I know, old IDE but this is all we can work with at the moment. ALOT of work to re-design in .NET or newer technology).
TCP guarantees that the data will arrive in the same order it was sent.
That beeing said, you can just append all the incoming data to a buffer. Then check if your buffer contains one or more packets, and remove them from the buffer, keeping all the remaining data into the buffer for future check.
This, of course, suppose that your packets have some header that indicates the size of the following data.
Lets consider packets have the following structure:
[LEN] X X X...
Where LEN is the size of the data and each X is an byte.
If you receive:
4 X X X
[--1--]
The packet is not complete, you can leave it in the buffer. Then, other data arrives, you just append it to the buffer:
4 X X X X 3 X X X
[---2---]
You then have 2 complete messages that you can easily parse.
If you do it, don't forget to send any length in a host-independant form (ntohs and ntohl can help).
This is often accomplished by prefixing messages with a one or two-byte length value which, like you said, gives the length of the remaining data. If I've understood you correctly, you're sending this as plain text (i.e., '5', '5') and this might get split up. Since you don't know the length of a decimal number, it's somewhat ambiguous. If you absolutely need to go with plain text, perhaps you could encode the length as a 16-bit hex value, i.e.:
00ff <255 bytes data>
000a <10 bytes data>
This way, the length of the size header is fixed to 4 bytes and can be used as a minimum read length when receiving on the socket.
Edit: Perhaps I misunderstood -- if reading the length value isn't a problem, deal with splits by concatenating incoming data to a string, byte buffer, or whatever until its length is equal to the value you read in the beginning. TCP will take care of the rest.
Take extra precautions to make sure that you can't get stuck in a blocking read state should the client not send a complete message. For example, say you receive the length header, and start a loop that keeps reading through blocking recv() calls until the buffer is filled. If a malicious client intentionally stops sending data, your server might be locked until the client either disconnects, or starts sending.
I would have a function called readBytes or something that takes a buffer and a length parameter and reads until that many bytes have been read. You'll need to capture the number of bytes actually read and if it's less than the number you're expecting, advance your buffer pointer and read the rest. Keep looping until you've read them all.
Then call this function once for the header (containing the length), assuming that the header is a fixed length. Once you have the length of the actual data, call this function again.