I'm using cereal to serialize and deserialize data in c++, and I came upon a problem. I'm sending data using a socket to the client side, so i send a stringstream with the serialized files in a JSON format.
The problem is that I can't deserialize it with simple data types, nor with complex ones, on the client side as it fails a rapidjson check and it tells me it's not an object
What would be the proper way of deserializing data considering that i can't create an instance of a class from the server side ?
Here is a simple example in which i try to send the username of a user to the client side
The send function :
void DBUser::SendUser()
{
std::stringstream os;
{
cereal::JSONOutputArchive archive_out(os);
archive_out(CEREAL_NVP(m_user));
}
ServerSocket* instance= ServerSocket::GetInstance();
instance->SendData(os.str());
}
In case it is needed, here is the function used to send the stringstream to the client side
void ServerSocket::SendData(std::string message)
{
try {
asio::error_code ignored_error;
asio::write(*socket, asio::buffer(message), ignored_error);
}
catch (std::exception& e)
{
std::cerr << e.what() << std::endl;
}
}
And here is the code when i try to deserialize:
std::array<char, 5000> buf;
asio::error_code error;
size_t len = socket.read_some(asio::buffer(buf), error);
std::string testString;
std::stringstream is(buf.data());
{
cereal::JSONInputArchive archive_in(is);
archive_in(testString);
}
std::cout << "Message from server: ";
std::cout << testString;
std::cout << std::endl;
You need to null terminate the received socket data. Can you try null terminating it like buf[len] = '\0'; after this size_t len = socket.read_some(asio::buffer(buf), error); statement.
Goal and Instruction to the Protocol about Notchian Communication
I have a Server Application that uses boost::asio's asynchronous read/write Functions to communicate with connecting Notchian Clients. So far so good I read the Documented Website and only wrote a Status Handshake Packet. In Minecraft you can get those Packets at each Notchian Server. These Packets do use specific Data Types. My Server is just sending a String as a Json Response to the Client.
Code Section | How I wrote the ByteBuffer
typedef unsigned char byte; /* Sending unsigned bytes */
class LBuffer {
std::vector<byte> buf;
public:
std::vector<byte>& getBuf() {
return buf;
}
void write(byte data) {
buf.push_back(data);
}
void writeInt(int32_t data) {
buf.push_back(data >> 24);
buf.push_back((data << 8) >> 24);
buf.push_back((data << 16) >> 24);
buf.push_back((data << 24) >> 24);
}
void writeString(std::string data) {
std::copy(data.begin(), data.end(), std::back_inserter(buf));
}
};
Code Section | How I wrote the Packet to the Buffer
LBuffer createHandshakeStatusResponsePacket() {
LBuffer buffer;
buffer.write(0x00);
buffer.writeString("{{\"version\":{\"name\":\"1.8.7\",\"protocol\":47},\"players\":{\"max\":100,\"online\":5,\"sample\":[{\"name\":\"thinkofdeath\",\"id\":\"4566e69f-c907-48ee-8d71-d7ba5aa00d20\"}]},\"description\":{\"text\":\"Helloworld\"}}}");
return buffer;
}
Code Section | Writing Server with the ResponseBuf
int main() {
boost::asio::io_service svc;
tcp::acceptor a(svc);
a.open(tcp::v4());
a.set_option(tcp::acceptor::reuse_address(true));
a.bind({ {}, 6767 });
a.listen(5);
using session = std::shared_ptr<tcp::socket>;
std::function<void()> doAccept;
std::function<void(session)> doSession;
doSession = [&](session s) {
auto buf = std::make_shared<std::vector<byte>>(1024);
s->async_read_some(boost::asio::buffer(*buf), [&, s, buf](error_code ec, size_t bytes) {
if (ec)
std::cerr << "read failed: " << ec.message() << "\n";
else {
/*
As you see I dont read the Request from the Client..
But thats not relevant when I just want to send the Data
to receive it's Motd and so on..
*/
if (ec)
std::cerr << "endpoint failed: " << ec.message() << std::endl;
else {
std::vector<byte> responseBuf = createHandshakeStatusResponsePacket().getBuf();
async_write(*s, boost::asio::buffer(responseBuf), [&, s, buf](error_code ec, size_t) {
if (ec) std::cerr << "write failed: " << ec.message() << "\n";
});
}
doSession(s);
}
});
};
doAccept = [&] {
auto s = std::make_shared<session::element_type>(svc);
a.async_accept(*s, [&, s](error_code ec) {
if (ec)
std::cerr << "accept failed: " << ec.message() << "\n";
else {
doSession(s);
doAccept();
}
});
};
doAccept();
svc.run();
}
Results and Problems
When my Notchian Client reads the Packet that I sent as Response from the Server, it's giving me without any Delay this Result here:
Can't connect to Server
The Log from my Notchian Client said:
[04:42:54] [Client thread/ERROR]: Can't ping 127.0.0.1:6767: Internal Exception: io.netty.handler.codec.DecoderException:
java.io.IOException: Bad packet id 123
But how can it be the packetId 123 ? Because I'm sending the PacketId 0 at first.
Declaration
Notchian: Typically Software written from Notch ( so I grabbed it up )
ByteBuffer: Sending Bytes in a specific order.
I do hope for Tips and Solutions,
thanks
I have this requirement where my app have to connect to another app via sockets and will have to maintain persistent connection for quiet long time. My app will be a TCP client and the other is a TCP server. My app will send commands and the server will respond accordingly.
The problem am facing right now is how to read the whole data from server a string and return for app which will issue the next command. Reading synchronously (with asio::read) looked like a good option up until I observed socket hanging up until I terminate the server. Looking at the documentation I found that the library is correctly working.
his function is used to read a certain number of bytes of data from a stream. The call will block until one of the following conditions is true:
1. The supplied buffers are full. That is, the bytes transferred is equal to the sum of the buffer sizes.
2. An error occurred.
The problem is I don't know correct buffer size as the response from the server varies. So If I put a too small buffer it returns fine but missing some data. If I put too big it will hang forever until server quits.
So I thought I would do the async reading. It works only once and I don't know how to make it fetch data until whole data it read.
here is the relevant async code
#define ASIO_STANDALONE 1
#include <iostream>
#include <asio.hpp>
int main()
{
asio::io_context context;
size_t reply_length;
size_t length = 1024;
std::vector<char> buffer;
//create socket
asio::ip::tcp::socket socket(context);
socket.connect(asio::ip::tcp::endpoint(asio::ip::address::from_string("127.0.0.1"), 8088));
std::string dataOut = "list --files"; //some command to write
std::error_code error;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
buffer.resize(length);
asio::async_read(socket, asio::buffer(buffer), [&buffer, &context](const asio::error_code &ec, std::size_t bytes_transferred) {
std::copy(buffer.begin(), buffer.end(), std::ostream_iterator<char>(std::cout, ""));
std::cout << "\nRead total of:" << bytes_transferred << "\n";
context.run();
});
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
context.run();
}
Searching didn't help much solving my issue.
So my question is, how can I read all the data in a persistent socket with asio? Am not using boost.
You need to loop async_read calls. If you don't want your client to hang on read operation you can define the smallest possible buffer i.e. 1 byte.
Define function which takes socket, buffer and two additional parameters according to async_read's handler signature, and this function calls itself with async_read to make the loop of async_read calls - it reads until some error occures:
void onRead (
asio::ip::tcp::socket& socket,
std::array<char,1>& buf,
const system::error_code& ec,
std::size_t bytes)
{
if (ec)
{
if (ec == asio::error::eof && bytes == 1)
std::cout << buf[0];
return;
}
std::cout << buf[0];
asio::async_read(socket,asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1, // error code
std::placeholders::_2)); // transferred bytes
}
and the changes in main:
std::array<char,1> buf;
asio::write(socket, asio::buffer(dataOut), error);
if (!error)
{
std::cout << "Receiving...!" << std::endl;
asio::async_read(socket, asio::buffer(buf),
std::bind(onRead, std::ref(socket), std::ref(buf),
std::placeholders::_1,
std::placeholders::_2));
context.run();
}
else
{
std::cout << "send failed: " << error.message() << std::endl;
}
(I am using Boost, so you should replace system::error_code on asio::error_code).
Salutations fellow programmers,
I am trying to write a program that allows you input what you want and the program will send your input to the server.
At the moment, my goal is sending HTTP requests to a web page. It connects fine. But when the while loop runs in immediately sends something through the cin.getline procedure without me inputting anything. I thought this was weird but it seemed to be work anyway.
Every time I send something like: "GET / HTTP/1.1\r\n\r\n" it will return the correct thing, but anything else I input, like "OPTIONS" returns the source code + "application blocked" (I am at school so it makes sense).
So, I connected to hotspot shield VPN and tested the application, but to my horror when I input something to send it returns nothing.
I searched through stack overflow and google but I haven't been able to find anything so far; probably because I'm searching for the wrong solutions to the problem.
Anyway, if you have time, please scan through the code send some help. It could just be a VPN and school issue and I could try at home if the code seems to be working for you, so just let me know.
SPECIFIC OUTLINE OF PROBLEM:
When I use this outside the school network nothing is returned and the while loop doesn't seem to execute. I can connect but the program seems to be in an endless time-out or something.
cout << "Connected to " << hostName << endl;
while (true) {
cout << ">";
cin.getline(sendBuf, sizeof(sendBuf));
string s(sendBuf);
cout << s.c_str() << endl;
send(connectSocket, s.c_str(), sizeof(s.c_str()), 0);
int rec = recv(connectSocket, recvBuf, sizeof(recvBuf), 0);
if (rec > 0) {
cout << recvBuf << endl;
}
else if (rec <= 0) {
cout << "nothing" << endl;
}
}
system("pause");
}
system("pause");
}
my goal is sending HTTP requests to a web page
The code you showed does not attempt to implement any semblance of the HTTP protocol, not even close.
For one thing, if you look at your own example more carefully, you will see that the GET request (which BTW, is missing a required Host header, due to your use of HTTP 1.1) contains 2 line breaks, but cin.getline() (why not std::getline()?) reads only 1 line at a time. So, you read in one line, send it, and wait for a response that doesn't arrive since you didn't finish sending a complete request yet. That would explain why your while loop is hanging.
If you want the user to type in a complete HTTP request and then you send it as-is, you have to read in the ENTIRE request from the user, and then send it entirely to the server, before you can then attempt to receive the server's response. That means you have to handle line breaks between individual message headers, handle the terminating line break that separates the message headers from the message body, and detect the end of the body data.
I would suggest not relying on the user typing in a complete HTTP request as-is. I suggest you prompt the user for relevant pieces and let the user type normal text, and then your code can format that text into a proper HTTP request as needed.
When you are reading the server's response, you can't just blindly read arbitrary chunks of data. You have to process what you read, per the rules of the HTTP protocol. This is particularly important in order to determine when you have reached the end of the response and need to stop reading. The end of the response can be signaled in one of many different ways, as outlined in RFC 2616 Section 4.4 Message Length.
You are also making some common newbie mistakes in your TCP handling in general. TCP is a streaming transport, you are not taking into account that send() and recv() can sent/receive fewer bytes than requested. Or that recv() does not return null-terminated data.
With that said, try something like this:
void sendAll(SOCKET sckt, const void *buf, int buflen)
{
// send all bytes until buflen has been sent,
// or an error occurs...
const char *pbuf = static_cast<const char*>(buf);
while (buflen > 0)
{
int numSent = send(sckt, pbuf, buflen, 0);
if (numSent < 0) {
std::ostringstream errMsg;
errMsg << "Error sending to socket: " << WSAGetLastError();
throw std::runtime_error(errMsg.str());
}
pbuf += numSent;
buflen -= numSent;
}
}
int readSome(SOCKET sckt, void *buf, int buflen)
{
// read as many bytes as possible until buflen has been received,
// the socket is disconnected, or an error occurs...
char *pbuf = static_cast<char*>(buf);
int total = 0;
while (buflen > 0)
{
int numRecvd = recv(sckt, pbuf, buflen, 0);
if (numRecvd < 0) {
std::ostringstream errMsg;
errMsg << "Error receiving from socket: " << WSAGetLastError();
throw std::runtime_error(errMsg.str());
}
if (numRecvd == 0) break;
pbuf += numRecvd;
buflen -= numRecvd;
total += numRecvd;
}
return total;
}
void readAll(SOCKET sckt, void *buf, int buflen)
{
// read all bytes until buflen has been received,
// or an error occurs...
if (readSome(sckt, buf, buflen) != buflen)
throw std::runtime_error("Socket disconnected unexpectedly");
}
std::string readLine(SOCKET sckt)
{
// read a line of characters until a line break is received...
std::string line;
char c;
do
{
readAll(sckt, &c, 1);
if (c == '\r')
{
readAll(sckt, &c, 1);
if (c == '\n') break;
line.push_back('\r');
}
else if (c == '\n') {
break;
}
line.push_back(c);
}
while (true);
return line;
}
...
inline void ltrim(std::string &s) {
// erase whitespace on the left side...
s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](int ch) {
return !std::isspace(ch);
}));
}
inline void rtrim(std::string &s) {
// erase whitespace on the right side...
s.erase(std::find_if(s.rbegin(), s.rend(), [](int ch) {
return !std::isspace(ch);
}).base(), s.end());
}
inline void trim(std::string &s) {
// erase whitespace on both sides...
ltrim(s);
rtrim(s);
}
inline void upperCase(std::string &s)
{
// translate all characters to upper-case...
std::transform(s.begin(), s.end(), s.begin(), ::toupper);
}
...
std::string makeRequest(const std::string &host, const std::string &method, const std::string &resource, const std::vector<std::string> &extraHeaders, const void *body, int bodyLength)
{
std::ostringstream oss;
oss << method << " " << resource << " HTTP/1.1\r\n";
oss << "Host: " << host << "\r\n";
oss << "Content-Length: " << bodyLength << "\r\n";
for(auto &hdr : extraHeaders)
{
// TODO: ignore Host and Content-Length...
oss << hdr << "\r\n";
}
oss << "\r\n";
oss.write(static_cast<const char*>(body), bodyLength);
return oss.str();
}
bool getHeaderValue(const std::vector<std::string> &headers, const std::string &headerName, std::string &value)
{
value.clear();
std::string toFind = headerName;
upperCase(toFind);
// find the requested header by name...
for(auto &s : headers)
{
std::string::size_type pos = s.find(':');
if (pos != std::string::npos)
{
std::string name = s.substr(0, pos-1);
trim(name);
upperCase(name);
if (name == toFind)
{
// now return its value...
value = s.substr(pos+1);
trim(value);
return true;
}
}
}
// name not found
return false;
}
...
std::cout << "Connected to " << hostName << std::endl;
try
{
std::string method, resource, hdr, data;
std::string status, version, reason;
std::vector<std::string> headers;
int statusCode, rec;
do
{
headers.clear();
data.clear();
// get user input
std::cout << "Method > " << std::flush;
if (!std::getline(std::cin, method))
throw std::runtime_error("Error reading from stdin");
upperCase(method);
std::cout << "Resource > " << std::flush;
if (!std::getline(std::cin, resource))
throw std::runtime_error("Error reading from stdin");
std::cout << "Extra Headers > " << std::flush;
while (std::getline(std::cin, hdr) && !hdr.empty())
headers.push_back(hdr);
if (!std::cin)
throw std::runtime_error("Error reading from stdin");
std::cout << "Data > " << std::flush;
// use Ctrl-Z or Ctrl-D to end the data, depending on platform...
std::ios_base::fmtflags flags = std::cin.flags();
std::cin >> std::noskipws;
std::copy(std::istream_iterator<char>(std::cin), std::istream_iterator<char>(), std::back_inserter(data));
if (!std::cin)
throw std::runtime_error("Error reading from stdin");
std::cin.flags(flags);
std::cin.clear();
// send request
std::string request = makeRequest(hostName, method, resource, headers, data.c_str(), data.length());
std::cout << "Sending request: << std::endl << request << std::endl;
// TODO: reconnect to hostName if previous request disconnected...
sendAll(connectSocket, request.c_str(), request.length());
// receive response
headers.clear();
data.clear();
// read the status line and parse it...
status = readLine(connectSocket);
std::cout << status << std::endl;
std::getline(std::istringstream(status) >> version >> statusCode, reason);
upperCase(version);
// read the headers...
do
{
hdr = readLine(connectSocket);
std::cout << hdr << std::endl;
if (hdr.empty()) break;
headers.push_back(hdr);
}
while (true);
// The transfer-length of a message is the length of the message-body as
// it appears in the message; that is, after any transfer-codings have
// been applied. When a message-body is included with a message, the
// transfer-length of that body is determined by one of the following
// (in order of precedence):
// 1. Any response message which "MUST NOT" include a message-body (such
// as the 1xx, 204, and 304 responses and any response to a HEAD
// request) is always terminated by the first empty line after the
// header fields, regardless of the entity-header fields present in
// the message.
if (((statusCode / 100) != 1) &&
(statusCode != 204) &&
(statusCode != 304) &&
(method != "HEAD"))
{
// 2. If a Transfer-Encoding header field (section 14.41) is present and
// has any value other than "identity", then the transfer-length is
// defined by use of the "chunked" transfer-coding (section 3.6),
// unless the message is terminated by closing the connection.
if (getHeaderValue(headers, "Transfer-Encoding", hdr))
upperCase(hdr);
if (!hdr.empty() && (hdr != "IDENTITY"))
{
std::string chunk;
std::string::size_type oldSize, size;
do
{
chunk = readLine(connectSocket);
std::istringstream(chunk) >> std::hex >> size;
if (size == 0) break;
oldSize = data.size();
chunkData.resize(oldSize + size);
readAll(connectSocket, &data[oldSize], size);
std::cout.write(&data[oldSize], size);
readLine(connectSocket);
}
while (true);
std::cout << std::endl;
do
{
hdr = readLine(connectSocket);
std::cout << hdr << std::endl;
if (hdr.empty()) break;
headers.push_back(hdr);
}
while (true);
}
// 3. If a Content-Length header field (section 14.13) is present, its
// decimal value in OCTETs represents both the entity-length and the
// transfer-length. The Content-Length header field MUST NOT be sent
// if these two lengths are different (i.e., if a Transfer-Encoding
// header field is present). If a message is received with both a
// Transfer-Encoding header field and a Content-Length header field,
// the latter MUST be ignored.
else if (getHeaderValue(headers, "Content-Length", hdr))
{
std::string::size_type size;
if ((std::istringstream(hdr) >> size) && (size > 0))
{
data.resize(size);
readAll(connectSock, &data[0], size);
std::cout << data;
}
}
// 4. If the message uses the media type "multipart/byteranges", and the
// transfer-length is not otherwise specified, then this self-
// delimiting media type defines the transfer-length. This media type
// MUST NOT be used unless the sender knows that the recipient can parse
// it; the presence in a request of a Range header with multiple byte-
// range specifiers from a 1.1 client implies that the client can parse
// multipart/byteranges responses.
else if (getHeaderValue(headers, "Content-Type", hdr) &&
(hdr.compare(0, 10, "multipart/") == 0))
{
// TODO: extract 'boundary' attribute and read from
// socket until the terminating boundary is reached...
}
// 5. By the server closing the connection.
else
{
do
{
rec = readSome(connectSocket, recvBuf, sizeof(recvBuf));
if (rec == 0) break;
data.append(recvBuf, rec);
std::cout.write(recvBuf, rec);
}
while (rec == sizeof(recvBuf));
}
}
std::cout << std::endl;
// use status, headers, and data as needed ...
getHeaderValue(headers, "Connection", hdr);
upperCase(hdr);
if (version == "HTTP/1.0")
{
if (hdr != "KEEP-ALIVE")
break;
}
else
{
if (hdr == "CLOSE")
break;
}
}
while (true);
}
catch (const std::exception &e)
{
std::cerr << e.what() << std::endl;
}
closesocket(connectSocket);
std::cout << "Disconnected from " << hostName << std::endl;
std::system("pause");
Isn't HTTP fun? :-) This is, by far, not a complete HTTP implementation, but it should get you started. However, as you can see, HTTP can be quite complex to implement from scratch, and it has many rules and restrictions that you have to follow. You are better off not implementing HTTP manually at all. There are plenty of 3rd party HTTP libraries that are available for C++. Use one of them instead, and let them handle the hard work for you, so you can focus on your own business logic.
I am currently trying to transfer some JSON data over the network from a client to a server using the socket API of boost-asio. My client essentially does this:
int from = 1, to = 2;
boost::asio::streambuf buf;
ostream str(&buf);
str << "{"
<< "\"purpose\" : \"request\"" << "," << endl
<< "\"from\" : " << from << "," << endl
<< "\"to\" : " << to << "," << endl
<< "}" << endl;
// Start an asynchronous operation to send the message.
boost::asio::async_write(socket_, buf,
boost::bind(&client::handle_write, this, _1));
On the server side I have the choice between various boost::asio::async_read* functions.
I wanted to use JsonCpp to parse the received data. Studying the JsonCpp API (http://jsoncpp.sourceforge.net/class_json_1_1_reader.html) I found that the Reader operates on top of either a std::string, a char* array or a std::istream which I could operate from the boost::asio::streambuf passed to the functions.
The point is that as far as I know it is not necessarily the case that the entire content is transferred at once, so I would need some kind of confirmation that the buffer contains sufficient data to process the entire document using JsonCpp. How can I assure that the buffer contains enough data?
This is an area for application level protocol
Either
read until the stream end (the sender disconnects); this doesn't work with connections that are kept alive for more than a single message
supply a header like Content-Length: 12346\r\n to know in advance how much to read
supply a delimiter (a bit like mime boundaries, but you could use any sequence that is not allowed/supported as part of the JSON payload) (async_read_until)
Treat the payload as "binary-style" (BSON e.g.) and supply a (network-order) length field before the text transmission.
The ASIO Http server example contains a pretty nice pattern for parsing HTTP request/headers that you could use. This assumes that your parser can detect completeness and just 'soft-fails' until all information is present.
void connection::handle_read(const boost::system::error_code& e,
std::size_t bytes_transferred)
{
if (!e)
{
boost::tribool result;
boost::tie(result, boost::tuples::ignore) = request_parser_.parse(
request_, buffer_.data(), buffer_.data() + bytes_transferred);
if (result)
{
request_handler_.handle_request(request_, reply_);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else if (!result)
{
reply_ = reply::stock_reply(reply::bad_request);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else
{
socket_.async_read_some(boost::asio::buffer(buffer_),
boost::bind(&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
}
else if (e != boost::asio::error::operation_aborted)
{
connection_manager_.stop(shared_from_this());
}
}
I've provided an answer that parses JSON using Boost Spirit earlier Parse a substring as JSON using QJsonDocument; you could use this to detect the end of a proper JSON document (and if it's incomplete, the end will coincide with the start)
2 problems here : 1) tell the server how many bytes to read; 2) read the JSON
for 1) you can make your own simple protocol
300#my message here
sends a 300 byte sized message; # is the delimiter between size and message
int write_request(socket_t &socket, const char* buf_json)
{
std::string buf;
size_t size_json = strlen(buf_json);
buf = std::to_string(static_cast<long long unsigned int>(size_json));
buf += "#";
buf += std::string(buf_json);
return (socket.write_all(buf.data(), buf.size()));
}
to read on the server
//parse header, one character at a time and look for for separator #
//assume size header lenght less than 20 digits
for (size_t idx = 0; idx < 20; idx++)
{
char c;
if ((recv_size = ::recv(socket.m_sockfd, &c, 1, 0)) == -1)
{
std::cout << "recv error: " << strerror(errno) << std::endl;
return str;
}
if (c == '#')
{
break;
}
else
{
str_header += c;
}
}
to read JSON, you can use
https://github.com/nlohmann/json