I have been working on wrapping ENet into a set of easy to use functions for a few weeks now and seem to have a bit of an issue.
I have a std::stringstream and am attempting to send the contents to a remote machine using ENet then reconstruct the std::stringstream on the remote machine.
The reason I need to use a std::stringstream is due to the fact that I'm serializing my data with the Cereal Serialization Library which requires a stream.
With Azoth's help he has identified that I need to be using std::istringstream and std::ostringstream. Previously I was only using std::stringstream which was causing an exception.
However now an exception is being thrown within Cereal at portable_binary.hpp line 156:
throw Exception("Failed to read " + std::to_string(size) + " bytes from input stream! Read " + std::to_string(readSize));
Here's what I'm doing:
void Send(ENetHost* Host)
{
std::ostringstream SData;
{
cereal::PortableBinaryOutputArchive Archive(SData);
Archive(PacketData);
}
std::string Out = SData.str();
ENetPacket* Packet = enet_packet_create(Out.c_str(), Out.size(), ENET_PACKET_FLAG_RELIABLE);
enet_host_broadcast(Host, 0, Packet);
}
A Cereal Portable Binary Data Archive is constructed to hold a single vector.
The std::ostringstream is sent off to the host using ENet.
This part seems to work okay, I can print the information out before and after and it appears to be the same, albeit some weird symbols, but they print out the same on both ends.
Now a std::istringstream is created on the host with the data we received.
NetPacket(enet_uint8 const* Data)
{
std::istringstream SData(reinterpret_cast<char const*>(Data));
{
cereal::PortableBinaryInputArchive Archive(SData);
Archive(PacketData);
}
}
At this point I receive the exception at line:
Archive(PacketData)
I have a feeling the data is being changed somehow when it's sent through ENet and/or I'm not pulling the data out of the std::ostringstream correctly and/or not putting the data back into the std::istringstream correctly.
Thank you very much for your time I greatly appreciate it.
Disclaimer: I'm not familiar with enet.
You are getting this error because you aren't constructing the std::stringstream properly upon receiving the packet. A send/receive pair should look something like:
my_send_function()
{
std::ostringstream os;
{
cereal::PortableBinaryOutputArchive ar(os);
ar( whatever_needs_to_be_serialized );
} // the binary archives will flush their output
// immediately, but it's better to
// use cereal archives in an RAII matter all the time
std::string data = os.str();
create_packet(data.c_str(), data.size());
// send out
}
And then on the receiving end, something like this:
my_receive_function( uint8_t const * data ) // data came from some packet
{
MyDataType d;
std::istringstream is(reinterpet_cast<char const *>(data));
// this is safe to do since we generated the data using c_str(), which added
// a null terminator to the data
{
cereal::PortableBinaryInputArchive ar(is);
ar( d );
}
}
The basic idea here: use cereal and some ostringstream to generate a string (which is really just an array of bytes), send those raw bytes over the network, pull them into an istringstream, and then have cereal parse that.
Related
I know this has been asked before but I just can't find the answer so I am posting for some help.
I have a DLL, which once injected into a process creates a named pipe. The pipe will wait until a client is connected and will send data to the client until the client disconnects.
The client side, it will just connect to the pipe and receive data and do things with such data.
My question is, I want to be able to send more than 1 type of data, for example, float, int, strings, etc. How do I reconstruct the data into correct data (float, int strings and such)?
Here is my code for the client :
HANDLE hPipe;
DWORD dwWritten;
char Buffer[1024];
hPipe = CreateFile(TEXT("\\\\.\\pipe\\Pipe"),
GENERIC_READ | GENERIC_WRITE,
0,
NULL,
OPEN_EXISTING,
0,
NULL);
if (hPipe != INVALID_HANDLE_VALUE)
{
WriteFile(hPipe,
Buffer, //How do I put all the data into a buffer to send over to the client?
sizeof(Buffer), // = length of string + terminating '\0' !!!
&dwWritten,
NULL);
CloseHandle(hPipe);
}
The Server :
wcout << "Creating Pipe..." << endl;
HANDLE hPipe;
char buffer[1024];
DWORD dwRead;
hPipe = CreateNamedPipe(TEXT("\\\\.\\pipe\\Pipe"),
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT, // FILE_FLAG_FIRST_PIPE_INSTANCE is not needed but forces CreateNamedPipe(..) to fail if the pipe already exists...
1,
1024 * 16,
1024 * 16,
NMPWAIT_USE_DEFAULT_WAIT,
NULL);
while (hPipe != INVALID_HANDLE_VALUE)
{
if (ConnectNamedPipe(hPipe, NULL) != FALSE) // wait for someone to connect to the pipe
{
while (ReadFile(hPipe, buffer, sizeof(buffer) - 1, &dwRead, NULL) != FALSE)
{
/* add terminating zero */
buffer[dwRead] = '\0';
/* do something with data in buffer */
printf("%s", buffer);
}
}
DisconnectNamedPipe(hPipe);
}
My problem is, I have a bunch of data I would like to send to the client in 1 go, which could contain things like float, int, double, etc. Once I gather all the data from the server, I would like to send it to the client and have the client parse it by splitting the data like so :
void split(const string& s, char c,
vector<string>& v) {
string::size_type i = 0;
string::size_type j = s.find(c);
while (j != string::npos) {
v.push_back(s.substr(i, j - i));
i = ++j;
j = s.find(c, j);
if (j == string::npos)
v.push_back(s.substr(i, s.length()));
}
}
I'm sort of lost on how I can send all my data over to the client and correctly get the original value?
You have to use a library that converts your data to something which can be send over a socket. This is called a serializer! You can use a serializer also for serializing into data stream or maybe also in a GUI or whatever. Receiving the data simply needs "deserialize".
You can find a lot of seriaizer libs like:
http://www.boost.org/doc/libs/1_66_0/libs/serialization/doc/index.html
https://uscilab.github.io/cereal/
many many more!
The custom code can look like ( pseudo code! ):
class Check
{
int i;
float f;
template<SERIALIZER_TYPE>
void Serialize( SERIALIZER_TYPE& ser )
{
ser & i & f;
}
};
int main()
{
Check c;
std::string s;
Socket socket( ip, port );
WriteSerializer ser(socket);
ser & c & s;
}
As you can see, you have nothing to write your self for "knowing" how data types are serialized. Classes/structs have to provide a serializer method, so that they also can be split into there native data types.
Edit:
Added question from comments:
Is it possible to send this data from my DLL to the EXE through named pipe instead of saving the file?
For cereal taken from the docs:
cereal comes with excellent standard library support as well as binary, XML, and JSON serializers. If you need something else, cereal was written to be easily extensible for adding custom serialization archives or types.
So one option is simply to write the new interface for your needs.
But take a look into the example code:
void x()
{
std::ofstream file( "out.xml" );
cereal::XMLOutputArchive archive( file ); // depending on the archive type, data may be
// output to the stream as it is serialized, or
// only on destruction
archive( some_data, more_data, data_galore );
}
As you can see there is used a std::ofstream as output. So you can simply use that ofstream which is opened for a socket.
How you can connect a std::ostream with a socket is answered e.g. here:
How can I create an 'ostream' from a socket?
But it is quite simply to this job also by hand. You simply have to write your own buffer class for the socket and connect it to a ofstream. Not more then 10 lines of code I believe!
As a result you can stream now your variables and objects as xml, json or whatever over a socket.
Edit: From the comments: Yes, using a pipe instead a socket will also be adaptable to iostream and the technique is exactly the same and based on implementing something around streambuf.
c++ connect output stream to input stream
I am in hope that on windows it will work the same way and istream is also as ostream adaptable.
To go more in details about a complete solution will not longer fit here in Q&A style I believe. So if there are further questions about connecting something to iostream, please start new question!
If you use WriteFile API to write to pipe, then you can send Buffer of bytes. I like #Klaus idea about serialization to pack all your data and if I need to send objects over pipe from client to server it would be my implementation of choice.
However I consider it overkill if you need to send just some pair of data (like "abc 1.12345"). I would simply put them in the buffer with known delimiter and send from server to client, and then just parse the string on the client.
Answering on your question "//How do I put all the data into a buffer to send over to the client?"
Here's some code snippets:
std::string strToSend;
// ... some initialization
std::wstring wstr = std::wstring(strToSend.begin(), strToSend.end());
LPTSTR pchStr = wstr.c_str();
LPCTSTR pchSend;
StringCchCopy( pchSend, 1024, pchStr );
Use pchSend in WriteFile(hPipe, pchSend, ... call.
Please also check the following example for some code ideas: Named Pipe Client
First of all we need to understand what is a pipe here. Pipe and Queue are medium for communication between multiple process. Every message you submit/drop in a queue or pipe are unique as one set of message and while you are reading it, will read only one message at a time. If you put 10, 20.45 and “Hellow” into a pipe and read them from a client it will read 10 as first message 20.45 as second message and “Hellow” as third message. Normally they will be coming a string value which you need to convert into appropriate type. You should have your own logic to check whether a value is int or float or a normal string. First just read the data and print them all and think how to cope up with them.
I got a client and a server application which will send each other data by using the Asio (Standalone) library. Both applications consists of two (logical) parts:
A high level part: dealing with complex objects e.g. users, permissions,...
A low level part: sending data over network between client and server
Let's assume the complex objects are already serialized by using Protocoll Buffers and the low level part of the application receives the data as std::string from the high level part. I would like to use this function from Protocoll Buffers for this job:
bool SerializeToString(string* output) const;: serializes the message
and stores the bytes in the given string. Note that the bytes are
binary, not text; we only use the string class as a convenient
container.
And say I transfer this data with async_write on the client side:
size_t dataLength = strlen(data);
//writes a certain number of bytes of data to a stream.
asio::async_write(mSocket,
asio::buffer(data, dataLength),
std::bind(&Client::writeCallback, this,
std::placeholders::_1,
std::placeholders::_2));
How can I read this data on the server side? I don't know how much data I will have to read. Therefore this will not work (length is unknown):
asio::async_read(mSocket,
asio::buffer(mResponse, length),
std::bind(&Server::readCallback, this,
std::placeholders::_1,
std::placeholders::_2));
What is the best way to solve this problem? I could think of two solutions:
Append a 'special' character at the end of data and read until I reach this 'end of data signal'. The problem is, what if this character appears in data somehow? I don't know how Protocoll Buffers serializes my data.
Send a binary string with size_of_data + data instead of data. But I don't know how to serialize the size in an platform independent way, add it to the binary data and extract it again.
Edit: Maybe I could use this:
uint64_t length = strlen(data);
uint64_t nwlength = htonl(length);
uint8_t len[8];
len[0] = nwlength >> 56;
len[1] = nwlength >> 48;
len[2] = nwlength >> 40;
len[3] = nwlength >> 32;
len[4] = nwlength >> 24;
len[5] = nwlength >> 16;
len[6] = nwlength >> 8;
len[7] = nwlength >> 0;
std::string test(len);
mRequest = data;
mRequest.insert(0, test);
and send mRequest to the server? Any traps or caveats with this code?
How could I read the length on server side and the content afterwards?
Maybe like this:
void Server::readHeader(){
asio::async_read(mSocket,
asio::buffer(header, HEADER_LENGTH),
std::bind(&Server::readHeaderCallback, this,
std::placeholders::_1,
std::placeholders::_2),
asio::transfer_exactly(HEADER_LENGTH));
}
void Server::readHeaderCallback(const asio::error_code& error,
size_t bytes_transferred){
if(!error && decodeHeader(header, mResponseLength)){
//reading header finished, now read the content
readContent();
}
else{
if(error) std::cout << "Read failed: " << error.message() << "\n";
else std::cout << "decodeHeader failed \n";
}
}
void Server::readContent(){
asio::async_read(mSocket,
asio::buffer(mResponse, mResponseLength),
std::bind(&Server::readContentCallback, this,
std::placeholders::_1,
std::placeholders::_2),
asio::transfer_exactly(mResponseLength));
}
void Server::readContentCallback(const asio::error_code& error,
size_t bytes_transferred){
if (!error){
//handle content
}
else{
//#todo remove this cout
std::cout << "Read failed: " << error.message() << "\n";
}
}
Please note that I try to use transfer_exactly. Will this work?
When sending variable length messages over a stream-based protocol, there are generally three solutions to indicate message boundaries:
Use a delimiter to specify message boundaries. The async_read_until() operations provide a convenient way to read variable length delimited messages. When using a delimiter, one needs to consider the potential of a delimiter collision, where the delimiter appears within the contents of a message, but does not indicate a boundary. There are various techniques to handle delimiter collisions, such as escape characters or escape sequences.
Use a fixed-length header with a variable-length body protocol. The header will provide meta-information about the message, such as the length of the body. The official Asio chat example demonstrates one way to handle fixed-length header and variable-length body protocols.
If binary data is being sent, then one will need to consider handling byte-ordering. The hton() and ntoh() family of functions can help with byte-ordering. For example, consider a protocol that defines the field as two bytes in network-byte-order (big-endian) and a client reads the field as a uint16_t. If the value 10 is sent, and a little-endian machine reads it without converting from network-order to local-order, then the client will read the value as 2560. The Asio chat example avoids handling endianness by encoding the body length to string instead of a binary form.
Use the connection's end-of-file to indicate the end of a message. While this makes sending and receiving messages easy, it limits the sender to only one message per connection. To send an additional message, one would need to established another connection.
A few observations about the code:
The Protocol Buffers' SerializeToString() function serializes a message to a binary form. One should avoid using text based functions, such as strlen(), on the serialized string. For instance, strlen() may incorrectly determine the length, as it will treat the first byte with a value of 0 as the terminating null byte, even if that byte is part of the encoded value.
When providing an explicitly sized buffer to an operation via asio::buffer(buffer, n), the default completion condition of transfer_all will function the same as transfer_exactly(n). As such, the duplicate use of variables can be removed:
asio::async_read(mSocket,
asio::buffer(header, HEADER_LENGTH),
std::bind(&Server::readHeaderCallback, this,
std::placeholders::_1,
std::placeholders::_2));
The htonl() overloads support uint16_t and uint_32t, not uint64_t.
Asio supports scatter/gather operations, allowing a receive operation to scatter-read into multiple buffers, and transmit operations can gather-write from multiple buffers. As such, one does not necessarily need to have both the fixed-length header and message-body contained with a single buffer.
std::string body_buffer;
body.SerializeToString(&body_buffer);
std::string header_buffer = encode_header(body_buffer.size());
// Use "gather-write" to send both the header and data in a
// single write operation.
std::vector<boost::asio::const_buffer> buffers;
buffers.push_back(boost::asio::buffer(header_buffer));
buffers.push_back(boost::asio::buffer(body_buffer));
boost::asio::write(socket_, buffers);
client must call
socket.shutdown(asio::ip::tcp::socket::shutdown_both);
socket.close();
on the server size read until EOF detected
std::string reveive_complete_message(tcp::socket& sock)
{
std::string json_msg;
asio::error_code error;
char buf[255];
while (1)
{
//read some data up to buffer size
size_t len = sock.read_some(asio::buffer(buf), error);
//store the received buffer and increment the total return message
std::string str(buf, len);
json_msg += str;
if (error == asio::error::eof)
{
//EOF received, the connection was closed by client
break;
}
else if (error)
{
throw asio::system_error(error);
}
}
return json_msg;
}
I am currently implementing a network protocol with Boost Asio. The domain classes already exist and I am able to
write packets to a std::istream
and read packets from a std::ostream.
A Network Packet contains a Network Packet Header. The header starts with the Packet Length field, which has a size of two bytes (std::uint16_t).
I am using TCP/IPv4 as the transport layer, therefore I try to implement the following:
Read the length of the packet to know its total length. This means reading exactly two bytes from the socket.
Read the rest of the packet. This means reading exactly kActualPacketLength - sizeof(PacketLengthFieldType) bytes from the socket.
Concat both read binary data.
Therefore I need at least two calls to boost::asio::read (I am starting synchronously!).
I am able to read a packet with one call to boost::asio::read if I hard-code the expected length:
Packet const ReadPacketFromSocket() {
boost::asio::streambuf stream_buffer;
boost::asio::streambuf::mutable_buffers_type buffer{
stream_buffer.prepare(Packet::KRecommendedMaximumSize)};
std::size_t const kBytesTransferred{boost::asio::read(
this->socket_,
buffer,
// TODO: Remove hard-coded value.
boost::asio::transfer_exactly(21))};
stream_buffer.commit(kBytesTransferred);
std::istream input_stream(&stream_buffer);
PacketReader const kPacketReader{MessageReader::GetInstance()};
return kPacketReader.Read(input_stream);
}
This reads the complete packet data at once and returns a Packet instance. This works, so the concept is working.
So far so good. Now my problem:
If I make two consecutive calls to boost::asio::read with the same boost::asio::streambuf I can't get it to work.
Here is the code:
Packet const ReadPacketFromSocket() {
std::uint16_t constexpr kPacketLengthFieldSize{2};
boost::asio::streambuf stream_buffer;
boost::asio::streambuf::mutable_buffers_type buffer{
stream_buffer.prepare(Packet::KRecommendedMaximumSize)};
std::size_t const kBytesTransferred{boost::asio::read(
// The stream from which the data is to be read.
this->socket_,
// One or more buffers into which the data will be read.
buffer,
// The function object to be called to determine whether the read
// operation is complete.
boost::asio::transfer_exactly(kPacketLengthFieldSize))};
// The received data is "committed" (moved) from the output sequence to the
// input sequence.
stream_buffer.commit(kBytesTransferred);
BOOST_LOG_TRIVIAL(debug) << "bytes transferred: " << kBytesTransferred;
BOOST_LOG_TRIVIAL(debug) << "size of stream_buffer: " << stream_buffer.size();
std::uint16_t packet_size;
// This does seem to modify the streambuf!
std::istream istream(&stream_buffer);
istream.read(reinterpret_cast<char *>(&packet_size), sizeof(packet_size));
BOOST_LOG_TRIVIAL(debug) << "size of stream_buffer: " << stream_buffer.size();
BOOST_LOG_TRIVIAL(debug) << "data of stream_buffer: " << std::to_string(packet_size);
std::size_t const kBytesTransferred2{
boost::asio::read(
this->socket_,
buffer,
boost::asio::transfer_exactly(packet_size - kPacketLengthFieldSize))};
stream_buffer.commit(kBytesTransferred2);
BOOST_LOG_TRIVIAL(debug) << "bytes transferred: " << kBytesTransferred2;
BOOST_LOG_TRIVIAL(debug) << "size of stream_buffer: " << stream_buffer.size();
// Create an input stream with the data from the stream buffer.
std::istream input_stream(&stream_buffer);
PacketReader const kPacketReader{MessageReader::GetInstance()};
return kPacketReader.Read(input_stream);
}
I have the following problems:
Reading the packet length from the boost::asio::streambuf after the first socket read seems to remove the data from the boost::asio::streambuf.
If I use two distinct boost::asio::streambuf instances I do not know how to "concat" / "append" them.
At the end of the day I need a std::istream with the correct data obtained from the socket.
Can someone please guide me into the correct direction? I've tried to make this work for several hours now...
Maybe this approach isn't the best, so I am open to suggestions to improve my design.
Thanks!
I believe the behaviour is by design.
To concatenate the buffers, you can use BUfferSequences (using make_buffers) and use buffer iterators, or you can stream the second into the first:
boost::asio::streambuf a, b;
std::ostream as(&a);
as << &b;
Now you can throw away b as it's pending data have been appended to a
See it Live on Coliru
Before I forget, I want to summarize my current solution, which doesn't use a boost::asio::streambuf, since it seems to be impossible to read from it without modifying it. Instead I use a std::vector<std::uint8_t> (ByteVector) as the data holder for the buffers.
The following source code contains my current solution:
Packet const ReadPacketFromSocket() {
ByteVector const kPacketLengthData{this->ReadPacketLengthFromSocket()};
PacketHeader::PacketLengthType kPacketLength{
static_cast<PacketHeader::PacketLengthType>(
(kPacketLengthData[1] << 8) | kPacketLengthData[0])};
ByteVector rest_packet_data(Packet::KRecommendedMaximumSize);
boost::asio::read(
this->socket_,
boost::asio::buffer(rest_packet_data),
boost::asio::transfer_exactly(
kPacketLength - sizeof(PacketHeader::PacketLengthType)));
ByteVector data{
VectorUtils::GetInstance().Concatenate(
kPacketLengthData,
rest_packet_data)};
// Create an input stream from the vector.
std::stringstream input_stream;
input_stream.rdbuf()->pubsetbuf(
reinterpret_cast<char *>(&data[0]), data.size());
PacketReader const kPacketReader{MessageReader::GetInstance()};
return kPacketReader.Read(input_stream);
}
ByteVector ReadPacketLengthFromSocket() {
ByteVector data_holder(sizeof(PacketHeader::PacketLengthType));
boost::asio::read(
this->socket_,
boost::asio::buffer(data_holder),
boost::asio::transfer_exactly(sizeof(PacketHeader::PacketLengthType)));
return data_holder;
}
This works like a charm, I have successfully exchanged packets with messages from my domain model between two processes using this approach.
But: This solution feels wrong, since I have to do lots of conversions. Maybe someone else can provide me with a cleaner approach? What do you think about my solution?
I am writing a program in c++ that uses c sockets. I need a function to receive data that I would like to return a string. I know this will not work:
std::string Communication::recv(int bytes) {
std::string output;
if (read(this->sock, output, bytes)<0) {
std::cerr << "Failed to read data from socket.\n";
}
return output;
}
Because the read()* function takes a char array pointer for an argument. What is the best way to return a string here? I know I could theoretically read the data into a char array then convert that to a string but that seems wasteful to me. Is there a better way?
*I don't actually mind using something other that read() if there is a more fitting alternative
Here is all of the code on pastebin which should expire in a week. If I don't have an answer by then I will re-post it: http://pastebin.com/HkTDzmSt
[UPDATE]
I also tried using &output[0] but got the output contained the following:
jello!
[insert a billion bell characters here]
"jello!" was the data sent back to the socket.
Here are some functions that should help you accomplish what you want. It assumes you'll only receive ascii character from the other end of the socket.
std::string Communication::recv(int bytes) {
std::string output(bytes, 0);
if (read(this->sock, &output[0], bytes-1)<0) {
std::cerr << "Failed to read data from socket.\n";
}
return output;
}
or
std::string Communication::recv(int bytes) {
std::string output;
output.resize(bytes);
int bytes_received = read(this->sock, &output[0], bytes-1);
if (bytes_received<0) {
std::cerr << "Failed to read data from socket.\n";
return "";
}
output[bytes_received] = 0;
return output;
}
When printing the string, be sure to use cout << output.c_str() since string overwrite operator<< and skip unprintable character until it reaches size. Ultimately, you could also resize at the end of the function to the size received and be able to use normal cout.
As pointed out in comments, sending the size first would also be a great idea to avoid possible unnecessary memory allocation by the string class.
I receive a binary file via POST in a C++ CGI script and I'm using the Cgicc library to get its contents like so:
std::ofstream myfile;
myfile.open ("file.out", std::ios::in | std::ios::binary);
try
{
cgicc::Cgicc cgi;
cgicc::const_file_iterator file = cgi.getFile("bitmap");
if(file != cgi.getFiles().end())
{
file->writeToStream(myfile);
}
}
catch(std::exception &exception)
{
std::cout << exception.what();
}
The result is a binary file containing the bytes.
Now, because each byte should represent one pixel of an 8 bit bitmap file, I want to construct the entire bitmap file. In order to achieve this, I think I can use the easyBMP library, but since I need to create the image pixel by pixel, I need to somehow iterate over the received bytes. Does anyone know how this can be achieved? Can I get an iterator somehow to an std::ostream / std::ostrstream / std::ostringstream?
std::ostringstream stream;
file->writeToStream(stream);
//foreach byte in stream do { ... }
If you use std::ostringstream you can get std::string from it, using std::ostringstream::str function http://cplusplus.com/reference/iostream/ostringstream/str/ . Also, you can open your file and read it...