I am trying to send C++ ojbects through a tcp connection:
My objects are all serializable, using boost serialization.
The TCP server/client is made with boost asio.
Basically I would like to send message like that would contain the message type (the type of the object being sent) and the data itself (the serialized object) and the size of the data so I can process the buffer (the size can vary for objects of the same type, as it is not POD).
I am a bit stuck, because I don't know how I can send this. I don't understand what are the steps to convert the data to a char buffer, and adding the extra information (message type & size) at the beginning of the buffer, and then giving this buffer to the send function of the tcp connection, all that with doing as few copies as possible.
Thanks.
-
Here you can find a good example on how to use boost::serialization together with boost::asio.
Something like this is the core of what you need:
std::ostringstream archive_stream;
boost::archive::text_oarchive archive(archive_stream);
archive << YOUR_DATA;
outbound_data_ = archive_stream.str();
boost::asio::async_write(socket_, boost::asio::buffer(outbound_data_), handler);
Related
Take the following:
struct Header{
std::size_t body_size;
};
struct Body{
std::string data;
};
struct Packet{
Header header;
Body body;
};
Suppose now, that I want to send a Packet object over a tcp socket.
To do this, I want to serialise the Header into a std::string, which contains information about how large the body is, and send that string over through the socket.
But this serialised header string itself has a variable size since the body_size is not fixed, so how would I know how many bytes to read (of the serialised header string)
So what kind of protocols are used in sending data like this?
The most commonly used protocol for sending data over a TCP socket like you propose is HTTP.
In HTTP, the body size is sent as a string in the relevant header [request, response or chunk] with special characters (\r\n for a request or response, ; for a chunk) to identify the end of the size string.
You'll need to decide on your own special character; NULL would be the easiest for null terminated strings.
One of the key issues of sending data as you propose, is that TCP can split the data up into a number of (what are commonly called) packets, so the receiver may not receive the whole of your Packet at once.
BTW Packet is a poor name in this context.
You don't need to know the size of your complete Packet before you send it. You can call boost:asio async_write with a collection of const_buffers to send both the Header and Body of your Packet.
On the receive side, I recommend using async_read_some with a string buffer large enough for your whole Packet including the Header string.
You should be able to read the Body size from the characters before the NULL and calculate your Packet size from the Body size plus the number of characters before the NULL + 1 (for the NULL). If this is less that the size reported by your ReadHandler then your Body has been split across multiple TCP packets, so you'll need to receive the other packets to reconstruct your Packet Body.
Or, you could save yourself the trouble and simply use a tried and tested HTTP library like via-httplib or boost::beast.
I want to implement a client for a sensor that sends data over tcp and uses the following protocol:
the message-header starts with the byte-sequence 0xAFFEC0CC2 of type uint32
the header in total is 24 Bytes long (including the start sequence) and contains the size in bytes of the message-body as a uint32
the message-body is sent directly after the header and not terminated by a demimiter
Currently, I got the following code (assume a connected socket exists)
typedef unsigned char byte;
boost::system::error_code error;
boost::asio::streambuf buf;
std::string magic_word_s = {static_cast<char>(0xAF), static_cast<char>(0xFE),
static_cast<char>(0xC0), static_cast<char>(0xC2)};
ssize_t n = boost::asio::read_until(socket_, buf, magic_word_s, error);
if(error)
std::cerr << boost::system::system_error(error).what() << std::endl;
buf.consume(n);
n = boost::asio::read(socket_, buf, boost::asio::transfer_exactly(20);
const byte * p = boost::asio::buffer_cast<const byte>(buf.data());
uint32_t size_of_body = *((byte*)p);
unfortunately the documentation for read_until remarks:
After a successful read_until operation, the streambuf may contain additional data beyond the delimiter. An application will typically leave that data in the streambuf for a subsequent read_until operation to examine.
which means that I loose synchronization with the described protocol.
Is there an elegant way to solve this?
Well... as it says... you just "leave" it in the object, or temporary store it in another, and handle the whole message (below called 'packet') if it is complete.
I have a similar approach in one of my projects. I'll explain a little how I did it, that should give you a rough idea how you can handle the packets correctly.
In my Read-Handler (-callback) I keep checking if the packet is complete. The meta-data information (header for you) is temporary stored in a map associated with the remote-partner (map<RemoteAddress, InfoStructure>).
For example it can look like this:
4 byte identifier
4 byte message-length
n byte message
Handle incoming data, check if identifier + message-length are received already, continue to check if message-data is completed with received data.
Leave rest of the packet in the temporary buffer, erase old data.
Continue with handling when next packet arrives or check if received data completes next packet already...
This approach may sound a little slow, but I get even with SSL 10MB/s+ on a slow machine.
Without SSL much higher transfer-rates are possible.
With this approach, you may also take a look into read_some or its asynchronous version.
I'm using boost 1.53 and serialization to transfer an array of 520 floats over TCP/IP. I put a debug code printout to see the amount of data to be send : it's about 5 K. No problem for me here, but this value somehow depends on the actual data to be serialized. It could be 5400, 5500 and so on.
The question is : what is the right way to receive such data block? For the moment I use read_some() call. But as I've figured out it doesn't guarantee that the whole serialized block of data will be read out. Am I wrong?
How to ensure that there will be a complete archive at RX side? Is there any exception to be thrown when it is not possible to deserialize a chunk of data?
as far as tcpip packet can be received to a number of smaller packets so I'd recommend to add some additional data to tcpip
something like this:
serialize you data to stream
get size of stream
send to tcpip buffer starting with size of stream and then data from the stream
receiver reads size and then reads the rest of the packet.
after you received the full packet - call deserialization
Yes. read_some is potentially a no-op on conforming implementations[1].
Instead do a loop using read() and gcount(), like:
std::istream& is = gotten_from_somewhere_or_a_parameter();
std::vector<byte> v(256);
std::streamsize bytes_read;
do
{
is.read(v.data(),v.size());
bytes_read = stream.gcount ();
// do something with the bytes read
} while(bytes_read);
[1] Notably, gcc's standard library implementation seems to always return something for std::filebuf but on MSVC, the first call will simply always return 0 bytes read :)
struct ss
{ int data ;string name;}*o;
this is my structure in the client application
i want to send that to the server (in TCP Server under windows);
How to do that..
i Know the serilization is the solution.
but i don't know how to do that.. please help me if you can.
http://www.parashift.com/c++-faq-lite/serialization.html
If you use Boost for serialization then read this
Serialize and send objects by TCP using boost
I normally write my own data format for transferring this data.
I will create a character buffer.
Put the size of the packet.
Convert integer to bytes and copy it.
Append the string length
Copy the name
send it across.
First one must know whether client and server are known to be always the same architecture or not. This decides whether you can just send data as it is or whether you must care about endianness and the size of an integer.
In any case, ntohl and htonl will take care of byte ordering and allow you to transfer the int in a simple, defined way (no-op on machines that are already network-byte-order).
About the string, you can send both the size and the contents over TCP just fine (converting the size with htonl), assuming that the string data is either in the same encoding on both sides, or a "general, agnostic" encoding is always used, such as UTF-8.
If you don't know what encodings the machines on both ends are using, you are in trouble. In that case, you must include a message that defines this and convert accordingly (similar to as for example webservers do).
Having TCP operate in "normal mode" means that Nagle's algorithm will be enabled, so you can just use 3 calls to send and the network layer will coerce that into as few packets as it believes is reasonable (instead of sending an individual packet just for an integer).
That all for the simple case in your example, or you can do some proper serialization, which is much more work, of course.
This is a very good guide: http://beej.us/guide/bgnet/output/html/multipage/index.html. it is not strictly for windows but the changes are very slight.
You basically need to serialize your data to a buffer and then use send function. pass your socket identifier and buffer etc...
int send(
__in SOCKET s,
__in const char *buf,
__in int len,
__in int flags
);
from http://msdn.microsoft.com/en-us/library/ms740149(v=vs.85).aspx
// send some data to a socket
send(
socket, // the open socket
o, // pointer to the data
sizeof( ss ), // number of bytes
0 ); // no special flags
The difficulty is at the other end! The recipient needs to know how many bytes to read, and what structure to store them in.
You either need to write your own code to deal with these problems, or use a protocol that works on top of sockets. The code isn't hard, but some experience with some of the many protocols available would help with the gotchas!
I have a problem concerning boost asio libraries. I successfully tried to create a socket between a client and a server, this involves creation of resolvers in order to specify ip and port to the server (the server only requires port) and other objects, but, most importantly, it is necessary to use write and read_some as functions to read and write from/in the socket.
I would really appreciate to use a stream, and this is possible in boost asio, but that's strange...
In almost all examples using streams, to create a server it is necessary to provide port, ok, let's talk about the client... client side, it is necessary to use the iostream constructor to specify coordinates for connecting the stream, here's the code:
tcp::iostream() s(argv[1], "daytime");
Well, I don't really understand what is passed in the first parameter and really don't know what daytime might ever represent...
Basically, here, I'm telling: "Hey stream, you must connect to this server..." but how can I specify ip and port of that server?
Note that, on the opposite, everything is almost clear server side:
boost::asio::io_service io_s;
tcp::acceptor acc(io_s, tcp::endpoint(tcp::v4(), 1950));
for (;;) {
tcp::iostream stream;
acc.accept(*stream.rdbuf());
stream << "Message" << std::endl;
}
Using this model, I would like to use
stream << mymessage_to_send << std::endl;
stream >> a_string_containing_my_message;
in order to send and receive.
How can I do this?
Thank you very much.
The boost asio sample code you quoted:
tcp::iostream s(argv[1], "daytime");
uses "daytime" as a lookup into the services table (usually in /etc/services on a linux system), which would identify that the port for the daytime service is 13.
If you want to connect to a port that is not one of the well known services, you can do so with something like:
tcp::iostream s("localhost", "57002");
Note that the port number is supplied as a string, not as an unsigned short integer as one might be tempted to try.
Of course, "localhost" can be replaced with an IP address "127.0.0.1"
Let's solve all 3 issues here:
Creating the iostream around the socket client side.
This is really simple:
boost::asio::ip::tcp::iostream socketStream;
socketStream.connect( hostname, std::to_string( port ) );
You have to check the state of the stream to see if it connected successfully.
Creating the iostream around the socket server side.
Assuming you have your acceptor object and it is bound and listening..
boost::asio::ip::tcp::iostream connectionSocketStream; // from the connection object
acceptor.accept( *connectionSocketStream.rdbuf() );
// or
acceptor.async_accept( *connectionSocketStream.rdbuf(), callback );
where callback is a function that takes an error code.
Streaming the objects
Now for the streaming itself and here your issue is that when you stream out the string "Message" the client side will need to know where this message begins and ends, and the regular iostream won't write anything to specify this. This is a flaw in iostream itself really.
The answer therefore is to use a boost archive, and you can use a text or binary archive as long as you use the same both ends. It even doesn't matter if one side is using 64-bit big-endian and the other side 32-bit little endian or any other mix.
Using binary archive you would send a message this way:
boost::archive::binary_oarchive oarch( socketStream, boost::archive::no_header );
oarch << "Message";
Remember to flush the stream (socketStream, not oarch) when you have completed sending all you wish to send at this point.
and receive a message
boost::archive::binary_iarchive iarch( socketStream, boost::archive::no_header );
iarch >> message;
You would potentially create one archive and use it throughout, especially for outbound. For inbound you may have issues if you get a streaming error as it will break your archive.
You can use a text archive instead of a binary one.
The boost archive will automatically put in header information so it knows when an object is complete and will only return to you once it has a complete object or something has broken.
Note: primitive types, eg std::string and even vector< int > etc. are automatically handled by an archive. Your own classes will need special overloads as to how to stream them. You should read boost::archive documentation.
Note: You can connect the archive object to the stream before the stream has been opened. The archive works around the streambuf object which does not change dependent on the stream opening successfully.
Creating without no_header would be an issue though as the archives immediately try to use the stream on construction (to read or write their header)
I've written a client/server system using Boost.Asio. The source is available on GitHub: Client.cpp and Server.cpp. Using Boost.Serialization together with Boost.Asio allows me to send arbitrary datastructures over the wire. I must say it is quite impressive!