How can I convert serialized data in boost::beast to a string so that I could process it in a FIFO manner? - c++

I have an application of a client where I need to receive http "long running requests" from a server. I send a command, and after getting the header of the response, I have to just receive json data separated by \r\n until the connection is terminated.
I managed to adapt boost beast client example to send the message and receive the header and parse it and receive responses from the server. However, I failed at finding a way to serialize the data so that I could process the json messages.
The closest demonstration of the problem can be found in this relay example. In that example (p is a parser, sr is a serializer, input is a socket input stream and output is an socket output stream), after reading the http header, we have a loop that reads continuously from the server:
do
{
if(! p.is_done())
{
// Set up the body for writing into our small buffer
p.get().body().data = buf;
p.get().body().size = sizeof(buf);
// Read as much as we can
read(input, buffer, p, ec);
// This error is returned when buffer_body uses up the buffer
if(ec == error::need_buffer)
ec = {};
if(ec)
return;
// Set up the body for reading.
// This is how much was parsed:
p.get().body().size = sizeof(buf) - p.get().body().size;
p.get().body().data = buf;
p.get().body().more = ! p.is_done();
}
else
{
p.get().body().data = nullptr;
p.get().body().size = 0;
}
// Write everything in the buffer (which might be empty)
write(output, sr, ec);
// This error is returned when buffer_body uses up the buffer
if(ec == error::need_buffer)
ec = {};
if(ec)
return;
}
while(! p.is_done() && ! sr.is_done());
A few things I don't understand here:
We're done reading the header. Why do we need boost beast and not boost asio to read a raw tcp message? When I tried to do that (with both async_read/async_read_some) I got an infinite reads of zero size.
The documentation of parser says (at the end of the page) that a new instance is needed for every message, but I don't see that in the example.
Since tcp message reading is not working, is there a way to convert the parser/serializer data to some kind of string? Even write it to a text file in a FIFO manner, so that I could process it with some json library? I don't want to use another socket like the example.
The function boost::beast::buffers() failed to compile for the parser and the serializer, and for the parser there's no consume function, and the serializer's consume seems to be for particular http parts of the message, which fires an assert if I do it for body().
Besides that, I also failed at getting consistent chunks of data from the parser and the buffer with old-school std::copy. I don't seem to understand how to combine the data together to get the stream of data. Consuming the buffer with .consume() at any point while receiving data leads to need buffer error.
I would really appreciate someone explaining the logic of how all this should work together.

Where is buf? You could read directly into the std::string instead. Call string.resize(N), and set the pointer and size in the buffer_body::value_type to string.data() and string.size().

Related

boost asio find beginning of message in tcp based protocol

I want to implement a client for a sensor that sends data over tcp and uses the following protocol:
the message-header starts with the byte-sequence 0xAFFEC0CC2 of type uint32
the header in total is 24 Bytes long (including the start sequence) and contains the size in bytes of the message-body as a uint32
the message-body is sent directly after the header and not terminated by a demimiter
Currently, I got the following code (assume a connected socket exists)
typedef unsigned char byte;
boost::system::error_code error;
boost::asio::streambuf buf;
std::string magic_word_s = {static_cast<char>(0xAF), static_cast<char>(0xFE),
static_cast<char>(0xC0), static_cast<char>(0xC2)};
ssize_t n = boost::asio::read_until(socket_, buf, magic_word_s, error);
if(error)
std::cerr << boost::system::system_error(error).what() << std::endl;
buf.consume(n);
n = boost::asio::read(socket_, buf, boost::asio::transfer_exactly(20);
const byte * p = boost::asio::buffer_cast<const byte>(buf.data());
uint32_t size_of_body = *((byte*)p);
unfortunately the documentation for read_until remarks:
After a successful read_until operation, the streambuf may contain additional data beyond the delimiter. An application will typically leave that data in the streambuf for a subsequent read_until operation to examine.
which means that I loose synchronization with the described protocol.
Is there an elegant way to solve this?
Well... as it says... you just "leave" it in the object, or temporary store it in another, and handle the whole message (below called 'packet') if it is complete.
I have a similar approach in one of my projects. I'll explain a little how I did it, that should give you a rough idea how you can handle the packets correctly.
In my Read-Handler (-callback) I keep checking if the packet is complete. The meta-data information (header for you) is temporary stored in a map associated with the remote-partner (map<RemoteAddress, InfoStructure>).
For example it can look like this:
4 byte identifier
4 byte message-length
n byte message
Handle incoming data, check if identifier + message-length are received already, continue to check if message-data is completed with received data.
Leave rest of the packet in the temporary buffer, erase old data.
Continue with handling when next packet arrives or check if received data completes next packet already...
This approach may sound a little slow, but I get even with SSL 10MB/s+ on a slow machine.
Without SSL much higher transfer-rates are possible.
With this approach, you may also take a look into read_some or its asynchronous version.

Windows XP socket error with recv()

I'm having a strange behaviour with the recv() function.
My C++ (MFC) application with WinSock implements a simple HTTP client (non-blocking socket) for accessing HTML pages on a web server. Some of these pages are taking a few seconds for loading. On Windows 7 this is not a problem, because recv() also returns partial data. But on Windows XP the recv() function always returns SOCKET_ERROR and the error code is WSAEWOULDBLOCK. Only when the connection is finished the data is returned in one access.
Does anyone know this problem? How can I force Windows XP to also receive partial data?
I setted the buffer size (SO_RCVBUF) to 1000 Bytes. On Windows 7 this is also reflected to the TCP Window Size - on XP not.
The real problem which I have with this issue is, that I don't know how to check if the connection is still alive or not. How can I check if a connection is still alive? Or how can I specify a timeout (max time between two received packets from the server)?
By default, a socket operates in blocking mode, so the only way you can get a WSAEWOULDBLOCK error at all is if you explicitly put the socket into non-blocking mode instead. Doing so, you agree to handle WSAEWOULDBLOCK (otherwise, don't use non-blocking mode).
WSAEWOULDBLOCK is not a real error, it is just an indication that the operation you attempted to perform cannot be completed at that moment because it would block the calling thread. You need to detect this "error" and simply retry the same operation again at a later time, preferably after a socket state change is detected.
For recv(), WSAEWOULDBLOCK simply means there is no data available on the socket to be read at that moment. In non-blocking mode, you should be using select() (or WSAEventSelect(), or WSAAsyncSelect(), or Overlapped I/O, or an I/O Completion Port) to detect inbound data before you then read it.
That being said, you are implementing an HTTP client, so you must follow the HTTP protocol properly, regardless of the socket I/O mode you are using, regardless of your socket buffer sizes. You must follow the pseudo code logic I outlined in this answer on another question:
You must follow the rules outlined in RFC 2616. Namely:
Read until the "\r\n\r\n" sequence is encountered. Do not read any more bytes past that yet.
Analyze the received headers, per the rules in RFC 2616 Section 4.4. They tell you the actual format of the remaining response data.
Read the data per the format discovered in #2.
Check the received headers for the presence of a Connection: close header if the response is using HTTP 1.1, or the lack of a Connection: keep-alive header if the response is using HTTP 0.9 or 1.0. If detected, close your end of the socket connection because the server is closing its end. Otherwise, keep the connection open and re-use it for subsequent requests (unless you are done using the connection, in which case do close it).
Process the received data as needed.
In short, you need to do something more like this instead (pseudo code):
string headers[];
byte data[];
string statusLine = read a CRLF-delimited line;
int statusCode = extract from status line;
string responseVersion = extract from status line;
do
{
string header = read a CRLF-delimited line;
if (header == "") break;
add header to headers list;
}
while (true);
if ( !((statusCode in [1xx, 204, 304]) || (request was "HEAD")) )
{
if (headers["Transfer-Encoding"] ends with "chunked")
{
do
{
string chunk = read a CRLF delimited line;
int chunkSize = extract from chunk line;
if (chunkSize == 0) break;
read exactly chunkSize number of bytes into data storage;
read and discard until a CRLF has been read;
}
while (true);
do
{
string header = read a CRLF-delimited line;
if (header == "") break;
add header to headers list;
}
while (true);
}
else if (headers["Content-Length"] is present)
{
read exactly Content-Length number of bytes into data storage;
}
else if (headers["Content-Type"] == "multipart/byteranges")
{
string boundary = extract from Content-Type header;
read into data storage until terminating boundary has been read;
}
else
{
read bytes into data storage until disconnected;
}
}
if (!disconnected)
{
if (responseVersion == "HTTP/1.1")
{
if (headers["Connection"] == "close")
close connection;
}
else
{
if (headers["Connection"] != "keep-alive")
close connection;
}
}
check statusCode for errors;
process data contents, per info in headers list;
As you can see, HTTP requires reading CRLF-delimited lines of text, or fixed lengths of raw bytes. To do that, you must call recv() in a loop until you encounter the terminating CRLF, or have received the expected number of bytes, whichever the case may be. Whether you use a synchronous loop that just ignores WSAEWOULDBLOCK errors while looping, or you use a state machine driven by asynchronous events/callbacks, that is up to you to decide. That doesn't change how you must process the HTTP protocol.
This applies to all versions of Windows (even all platforms that use BSD-style socket APIs). What you are encountering is not a Windows bug at all. It is an underlying flaw in your understanding of how to use socket I/O correctly and effectively.
As for checking if the connection is alive, recv() will return 0 if the server closed the connection gracefully, or will report an error otherwise (usually WSAECONNABORTED or WSAECONNRESET, though there can be others). But an abnormal disconnect may take a long time to detect, so you should implement timeouts in your code instead. In synchronous mode, you can use setsockopt(SO_RCVTIMEO). In non-blocking mode, you can use select(). In asynchronous (overlapped) mode, you can use WaitForSingleObject() on whatever event/object you use to drive your state machine.
You can't expect recv to give you any data on a non-blocking socket. If there's no data available it returns WOULDBLOCK. You just need to call recv again (normally after select notifies you some data is available). Whether you get data on the first (or any) call is going to depend on how fast the server is sending it.
When the socket is closed you'll get a different error from recv, like WSAECONNRESET or WSAENOTCONN. select will also notify you when the socket is closed.
It's very strange.
Today I have changed my software to use blocking sockets. But it still doesn't work on Windows XP. Windows 7 is no problem.
So I thought: Let's try another PC. On this PC (also Windows XP) it does work. Now I tried a 3rd PC with Windows XP and here it also works.
I still don't know what the problem is but I think there must be a bug with the PC.

Sending data via socket aborts unexpected

i am trying to send data via tcp socket to a server. The idea behind that is a really simple chat programm.
The string I am trying to send looks like the following:
1:2:e9e633097ab9ceb3e48ec3f70ee2beba41d05d5420efee5da85f97d97005727587fda33ef4ff2322088f4c79e8133cc9cd9f3512f4d3a303cbdb5bc585415a00:2:xc_[z kxc_[z kxc_[z kxc_[==
As you can see there a few unprintable characters which I don't think are a problem here.
To send this data I am using the following code snippet.
bool tcp_client::send_data(string data)
{
if( send(sock , data.c_str(), strlen(data.c_str()) , 0) < 0)
{
perror("Send failed : ");
return false;
}
return true;
}
After a few minutes of trying things out I came up, that data.c_str() cuts my string of.
The result is:
1:2:e9e633097ab9ceb3e48ec3f70ee2beba41d05d5420efee5da85f97d97005727587fda33ef4ff2322088f4c79e8133cc9cd9f3512f4d3a303cbdb5bc585415a00:2:xc_[z
I think that there is some kind of null sequence inside my string which is a problem for the c_str() function.
Is there a way to send the whole string as I mentioned aboved without cutting it off?
Thanks.
Is there a way to send the whole string as I mentioned aboved without cutting it off?
What about:
send(sock , data.c_str(), data.size() , 0);
There are only two sane ways to send arbitrary data (such as a array of characters) over stream sockets:
On the server: close the socket after data was sent (like in ftp, http 0.9, etc). On the client - read until socket is closed in a loop.
On the server: prefix the data with fixed-length size (nowadays people usualy use 64 bit integers for size, watch out for endiannes). On the client - read the size first (in a loop!), than read the data until size bytes are read (in a loop).
Everything else is going to backfire sooner or later.

Boost ASIO TCP separation of messages

I just started working with the Boost ASIO library, version 1.52.0. I am using TCP/SSL encryption with async sockets. From other questions asked here about ASIO, it seems that ASIO does not support receiving a variable length message and then passing the data for that message to a handler.
I'm guessing that ASIO puts the data into a cyclical buffer and loses all track of each separate message. If I have missed something and ASIO does provide a way to pass individual messages, then please advise as to how.
My question is that assuming I can't somehow obtain just the bytes associated with an individual message, can I use transfer_exactly in async_read to obtain just the first 4 bytes, which our protocol always places the length of the message. Then call either read or async_read (if read won't work with async sockets) to read in the rest of the message? Will this work? Any better ways to do it?
Typically I like to take the data I receive in an async_read and put it in a boost::circular_buffer and then let my message parser layer decide when a message is complete and pull the data out.
http://www.boost.org/doc/libs/1_52_0/libs/circular_buffer/doc/circular_buffer.html
Partial code snippets below
boost::circular_buffer TCPSessionThread::m_CircularBuffer(BUFFSIZE);
void TCPSessionThread::handle_read(const boost::system::error_code& e, std::size_t bytes_transferred)
{
// ignore aborts - they are a result of our actions like stopping
if (e == boost::asio::error::operation_aborted)
return;
if (e == boost::asio::error::eof)
{
m_SerialPort.close();
m_IoService.stop();
return;
}
// if there is not room in the circular buffer to hold the new data then warn of overflow error
if (m_CircularBuffer.reserve() < bytes)
{
ERROR_OCCURRED("Buffer Overflow");
m_CircularBuffer.clear();
}
// now place the new data in the circular buffer (overwrite old data if needed)
// note: that if data copying is too expensive you could read directly into
// the circular buffer with a little extra effor
m_CircularBuffer.insert(m_CircularBuffer.end(), pData, pData + bytes);
boost::shared_ptr<MessageParser> pParser = m_pParser.lock(); // lock the weak pointer
if ((pParser) && (bytes_transferred))
pParser->HandleInboundPacket(m_CircularBuffer); // takes a reference so that the parser can consume data from the circ buf
// start the next read
m_Socket.async_read_some(boost::asio::buffer(*m_pBuffer), boost::bind(&TCPSessionThread::handle_read, this, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}

C++ network programing in linux: Server Questions

I am learning how to network program using c/c++ and I have created a server(TCP) that is suppose to respond in specific ways to messages from a client in order to do this I created a class that the server class passes the message to and returns a string to report back to the client.
Here is my problem sometimes it reports the correct string back other times if just repeats what I sent to the message handler. Which no where in the code do I have it return what was passed in. So I am wondering am I handling getting the message correctly?
Secondly, I am unsure of how to keep a connection open in a while loop to continually pass messages back and forth. You can see how I did it in the code below but I am pretty sure this is incorrect, any help on this would be great. Thanks!
if (!fork())
{ // this is the child process
close(sockfd); // child doesn't need the listener
while ((numbytes = recv(new_fd, buf, MAXDATASIZE-1, 0)) > 0)
{
//numbytes = recv(new_fd, buf, MAXDATASIZE-1, 0);
buf[numbytes-1] = '\0';
const char* temp = ash.handleMessage(buf).c_str();
int size_of_temp = ash.handleMessage(buf).length();
send(new_fd, temp, size_of_temp, 0);
//send(new_fd, temp, size_of_temp+1, 0);
}
}//end if
Please excuse my ghetto code
Handles the message
Class Method handler uses
If your learning about sockets you should also know that you can't assume that what you send() as a "complete message", will be delivered as a complete message.
If you send() some big data from your client you might need to use multiple recv() on the server (or vice versa) to read it all. This is a big difference of how files usually work...
If you'r designing your own protocol you can opt to also send the messages length, like [LEN][message]. An easy example would be if the strings you send are limited to 256 bytes you can start with send()ing a short representing the strings length,
Or easier, decide that you use line-feeds (newline - \n) to terminate messages. The the protocol would look like
"msg1\nmsg2\n"
then you would have to recv(), and append the data, until you get a newline. This is all I can muster right now, there are a lot of great examples on the internet, but I would recommend getting the source of some "real" program and look at how it handles its network.
You are calling handleMessage twice. You didn't post the code, but it looks like you're returning a string. It might be better to do:
string temp = ash.handleMessage(buf);
int size_of_temp = temp.length();
This would avoid repeating any action that takes place in handleMessage.