I know it can be used to send/receive structured object from file,
but can it be used to send/receive sequences of structured object from a socket?
http://code.google.com/p/protobuf/
Protocol Buffers is a structured data serialization (and de-serialization) framework. It is only concerned with encoding a selection of pre-defined data types into a data stream. What you do with that stream is up to you. To quote the wiki:
If you want to write multiple messages
to a single file or stream, it is up
to you to keep track of where one
message ends and the next begins. The
Protocol Buffer wire format is not
self-delimiting, so protocol buffer
parsers cannot determine where a
message ends on their own. The easiest
way to solve this problem is to write
the size of each message before you
write the message itself. When you
read the messages back in, you read
the size, then read the bytes into a
separate buffer, then parse from that
buffer.
So yes, you could use it to send/receive multiple objects via a socket but you have to do some extra work to differentiate each object stream.
I'm not familiar with protobuf, but the documentation says you can create a FileInputStream (which can then be used to create a CodedInputStream) using a file descriptor. If you're on a system that supports BSD sockets, you should presumably be able to give it a socket file descriptor rather than an ordinary one.
Protocol Buffers does not handle any surrounding network/file I/O operations. You might want to consider using Thrift, which includes socket communication libraries and server libraries with the serialization/deserialization.
Related
I use custom code to create SSL connection over native Berkeley sockets interface. I need to wrap the resulted socket with iostream to use existing algorithms written in C++ with these sockets data.
Is there any easy way to do it without need to implement stream and streambuf from scratch?
I learned boost::iostreams and boost::asio.
I didn't find any way to wrap existing OpenSSL session with boost::asio. Or may be anyone knows how to do that?
After boost:asio I concentrated my research on boost:iostreams.
boost::iostreams looks like good idea, however, its problem is that it uses read buffering. So, if we need to read just 1 byte from SSL session, it asks the TCP device to read 4 kilobytes and results in timeout. From the other hand, when I set buffer size to 0, boost::iostreams start to call write method for each byte, so when I try to write 10 bytes to stream, it calls SSL_write 10 times. TCP device itself can not use write buffering, because there are no way to forward flush method to device, so application level protocol may expect that data is sent to another peer while the data remains in output buffer.
So, we need unbuffered read and buffered flushable write; is that possible with boost::iostreams?
I found solution myself.
First of all, it is required to mark the device as flushable. Because there are not ready-made template for such device, you have to inherit device<dual_use, Ch> and override its category with multiple inheritance:
struct category : device<dual_use, Ch>::category, flushable_tag
Now when you will call flush on stream, it will forward the call to your device.
Next step is to disable stream own buffering (i. e. call open with 2nd and 3rd parameters equal to 0).
In such configuration boost will write to device each byte of data separatelly. However, you can implement buffering on device level, and flush the buffer on flush call.
I am using Libevent library 2.0 for socket communication.
In order to add data to evbuffer, I am using evbuffer_add. The bufferevent stores the data in its internal buffer and transfers the data via socket using some predefined timeout and watermark settings.
My question is, is there any way to control the data transfer? Can we transfer the data explicitly any time and after any random number of bytes being written?
The idea behind this function is fire-and-forget. However, you can add a callback so that when the send finally happens, you can do some things:
evbuffer_add_cb
This doesn't allow you much control, but you can use it for some behaviors like appending the buffer.
SO helpful folks:
This is another NEWBIE question. I am new at C++, Google Protocol Buffers, and serializing messages over HTTP without SOAP (even with SOAP for that matter). I am trying to send image data over http. I was told that Google Protocol Buffers was the way to go. So I eagerly started going through all of the documentation and used CocoaPods to install and include Google Protocol Buffer classes to my project. I created my .protoc files and generated the classes. Now I'm populating the data using the generated classes. Now what???
I cannot find any information on how to send the data. I have found a few other questions on SO that contained what might be meaningful information if I had a clue. This is what I have so far:
void message::myMessage::transmit(const uint32_t ipaddress, uint32_t port, message::MatMessage* rawImage)
{
message::HostMessage *transmitter;
transmitter->set_ipaddress(ipaddress);
transmitter->set_port(port);
//open socket with ip and port
//send unsing socket
}
HostMessage (*transmitter) is generated C++ class and header from a .protoc file that contains only the ipAddress and port number. OK, so I have this transmitter with an ipaddress and a port. Now what? How do I open a socket with it?
Maybe once I open a socket the other answers will make more sense to me. Can someone please help unconfuse me?
Thank you.
Protocol Buffers is just a serialization tool. It will convert your message object into bytes, and will convert bytes back into a message object. The library does not directly implement a way to transport those bytes.
If you have a raw socket, you could wrap it in FileInputStream/FileOutputStream to read/write protobufs from/to it*. Since protobufs are not self-delimiting, you need to write the size as a prefix, followed by the data, and interpret these correctly on the receiving end. See my answer to this other question for code to do that: Are there C++ equivalents for the Protocol Buffers delimited I/O functions in Java?
Another alternative is to use a higher-level transport library like ZeroMQ, which implements sending and receiving of byte-blob messages. Use protobufs to encode/decode the byte blobs, then hand them off to ZeroMQ for transport.
* This works on Unix, where sockets are file descriptors. On Windows, they aren't, so you'll need to implement the ZeroCopyInputStream/ZeroCopyOutputStream directly in terms of send()/recv(), which is not too hard if you use CopyingInputStreamAdapter/CopyingOutputStreamAdaptor.
I'm using Boost asio to send and receive framed TCP messages. Does anyone know the correct way to use a streambuf?
Should I receive into one streambuf per connection, and parse the messages out to pass around in my application?
Or can I read into a single streambuf per message, and pass the streambufs themselves around?
Ideally I'd like an easy way of parsing individual data fields into/out of a message, such as using an istream/ostream.
Any advice appreciated!
Typically you don't want to pass the buffers around, you would want to deserialize the content into some form of internal message which is easier to pass around. You may want to investigate something like Google's protocol buffers or boost serialization etc. to allow you to send real messages over the wire.
Typically you'd want to use a buffer per connection, in the read handler deserialize the message, and reuse the buffer for the next read.
I am writing a C++ server side application called quote of the day. I am using the winsock2 library. I want to send the contents of a file back to the client, including newlines by using the send function. The way i tried it doesn't work. How would i go about doing this?
Reading the file and writing to the socket are 2 distinct operations. Winsock does not have an API for sending a file directly.
As for reading the file, simply make sure you open it in read binary mode if using fopen, or simply use the CreateFile, and ReadFile Win32 API and it will be binary mode by default.
Usually you will read the file in chunks (for example 10KB at a time) and then send each of those chunks over the socket by using send or WSASend. Once you are done, you can close the socket.
On the receiving side, read whatever's available on the socket until the socket is closed. As you read data into a buffer, write the amount read to a file.
Hmm... I think Win32 should have something similar to "sendfile" in Linux.
If it doesn't you still can use memory-mapping (but, don't forgot to handle files with size larger than available virtual address space). You probably will need to use blocking sockets to avoid returning to application until all data is consumed. And I think there was something with "overlapped" operation to implement async IO.
I recommend dropping winsock and instead using something more modern such as Boost.Asio:
http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio/tutorial.html
There is also an example on transmitting a file:
http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio/examples.html