I'm using cereal to serialize and deserialize data in c++, and I came upon a problem. I'm sending data using a socket to the client side, so i send a stringstream with the serialized files in a JSON format.
The problem is that I can't deserialize it with simple data types, nor with complex ones, on the client side as it fails a rapidjson check and it tells me it's not an object
What would be the proper way of deserializing data considering that i can't create an instance of a class from the server side ?
Here is a simple example in which i try to send the username of a user to the client side
The send function :
void DBUser::SendUser()
{
std::stringstream os;
{
cereal::JSONOutputArchive archive_out(os);
archive_out(CEREAL_NVP(m_user));
}
ServerSocket* instance= ServerSocket::GetInstance();
instance->SendData(os.str());
}
In case it is needed, here is the function used to send the stringstream to the client side
void ServerSocket::SendData(std::string message)
{
try {
asio::error_code ignored_error;
asio::write(*socket, asio::buffer(message), ignored_error);
}
catch (std::exception& e)
{
std::cerr << e.what() << std::endl;
}
}
And here is the code when i try to deserialize:
std::array<char, 5000> buf;
asio::error_code error;
size_t len = socket.read_some(asio::buffer(buf), error);
std::string testString;
std::stringstream is(buf.data());
{
cereal::JSONInputArchive archive_in(is);
archive_in(testString);
}
std::cout << "Message from server: ";
std::cout << testString;
std::cout << std::endl;
You need to null terminate the received socket data. Can you try null terminating it like buf[len] = '\0'; after this size_t len = socket.read_some(asio::buffer(buf), error); statement.
Related
I developed a c++ program for communication between client and server through socket. They are sending to each other string messages.
On the other hand, I developed a c++ program for serializing data into human readable form.
My problem is, how to modify those programs in order to send a structure from client to server and then the server serialize it and save it into a file ?
This is the server program :
#include <iostream>
#include <boost/asio.hpp>
using namespace boost::asio;
using ip::tcp;
using std::string;
using std::cout;
using std::endl;
string read_(tcp::socket & socket) {
boost::asio::streambuf buf;
boost::asio::read_until( socket, buf, "\n" );
string data = boost::asio::buffer_cast<const char*>(buf.data());
return data;
}
void send_(tcp::socket & socket, const string& message) {
const string msg = message + "\n";
boost::asio::write( socket, boost::asio::buffer(message) );
}
int main() {
boost::asio::io_service io_service;
//listen for new connection
tcp::acceptor acceptor_(io_service, tcp::endpoint(tcp::v4(), 1234 ));
//socket creation
tcp::socket socket_(io_service);
//waiting for connection
acceptor_.accept(socket_);
//read operation
string message = read_(socket_);
cout << message << endl;
//write operation
send_(socket_, "Hello From Server!");
cout << "Servent sent Hello message to Client!" << endl;
return 0;
}
This is the client program :
#include <iostream>
#include <boost/asio.hpp>
using namespace boost::asio;
using ip::tcp;
using std::string;
using std::cout;
using std::endl;
int main() {
boost::asio::io_service io_service;
//socket creation
tcp::socket socket(io_service);
//connection
socket.connect( tcp::endpoint( boost::asio::ip::address::from_string("127.0.0.1"), 1234 ));
// request/message from client
const string msg = "Hello from Client!\n";
boost::system::error_code error;
boost::asio::write( socket, boost::asio::buffer(msg), error );
if( !error ) {
cout << "Client sent hello message!" << endl;
}
else {
cout << "send failed: " << error.message() << endl;
}
// getting response from server
boost::asio::streambuf receive_buffer;
boost::asio::read(socket, receive_buffer, boost::asio::transfer_all(), error);
if( error && error != boost::asio::error::eof ) {
cout << "receive failed: " << error.message() << endl;
}
else {
const char* data = boost::asio::buffer_cast<const char*>(receive_buffer.data());
cout << data << endl;
}
return 0;
}
This is how serialize strcuture :
const char* PERSON_FORMAT_OUT = "(%s, %d, %c)\n";
//const char* PERSON_FORMAT_IN = "(%[^,], %d, %c)\n";
typedef struct Person {
char name[20];
int age;
char gender;
} Person;
int main (int argc, char* argv[]) {
Person p1 = {
.name = "Andrew",
.age = 22,
.gender = 'M'
};
//Person p2;
FILE* file;
fopen_s(&file, "people.dat", "w+");
fseek(file, 0, SEEK_SET);
fprintf_s(file, PERSON_FORMAT_OUT, p1.name, p1.age, p1.gender);
//fscanf_s(file, PERSON_FORMAT_IN, p2.name, 20, &p2.age, &p2.gender);
return 0;
}
If you intend to have common structures definitions so that both client and server can exchange information using the same format, I would recommend using Protocol Buffers (https://developers.google.com/protocol-buffers). You can generate a .proto file with the data definitions and share it between client/server code. Another option is to use gRPC (https://grpc.io/).
You can send any characters over the socket, so you could even define your own communication protocol with no standard structure as you did above. Another approach would be to define your data structure in JSON/XML and use a third-party library such as https://github.com/nlohmann/json to create and parse the structures. You could then convert these structures into STL strings or string streams and transport those over your socket connection.
The main drawback of going this way, and not the protobuf or grpc way, is that any time you need to change the structure definition you will need to update both your client and server code. You also get free goodies from using these, as they are more flexible, maintainable and scale better if the communication interface feature set grows.
Use Protobuf https://developers.google.com/protocol-buffers/docs/cpptutorial
Use ZeroMQ or any message queue rather than using plain socket.https://zeromq.org/
Best things for protocol is to use google protobufs
We solve this using:
FastBinaryEncoding - for defining data model and serialization
CppServer - for transport layer (tcp, ssl, http, https, WebSockets)
There is a relatively new kid on the block that does excellent job of defining multi platform/language structures and manage its serialization Flatbuffers. In contrast with protobuf, it does not have packing/unpacking step. Generally, it saves one memory copy and trade off is extra code needed on construction of object. If you care about performance, gRPC and flatbuffers is a probably a best choice.
Goal and Instruction to the Protocol about Notchian Communication
I have a Server Application that uses boost::asio's asynchronous read/write Functions to communicate with connecting Notchian Clients. So far so good I read the Documented Website and only wrote a Status Handshake Packet. In Minecraft you can get those Packets at each Notchian Server. These Packets do use specific Data Types. My Server is just sending a String as a Json Response to the Client.
Code Section | How I wrote the ByteBuffer
typedef unsigned char byte; /* Sending unsigned bytes */
class LBuffer {
std::vector<byte> buf;
public:
std::vector<byte>& getBuf() {
return buf;
}
void write(byte data) {
buf.push_back(data);
}
void writeInt(int32_t data) {
buf.push_back(data >> 24);
buf.push_back((data << 8) >> 24);
buf.push_back((data << 16) >> 24);
buf.push_back((data << 24) >> 24);
}
void writeString(std::string data) {
std::copy(data.begin(), data.end(), std::back_inserter(buf));
}
};
Code Section | How I wrote the Packet to the Buffer
LBuffer createHandshakeStatusResponsePacket() {
LBuffer buffer;
buffer.write(0x00);
buffer.writeString("{{\"version\":{\"name\":\"1.8.7\",\"protocol\":47},\"players\":{\"max\":100,\"online\":5,\"sample\":[{\"name\":\"thinkofdeath\",\"id\":\"4566e69f-c907-48ee-8d71-d7ba5aa00d20\"}]},\"description\":{\"text\":\"Helloworld\"}}}");
return buffer;
}
Code Section | Writing Server with the ResponseBuf
int main() {
boost::asio::io_service svc;
tcp::acceptor a(svc);
a.open(tcp::v4());
a.set_option(tcp::acceptor::reuse_address(true));
a.bind({ {}, 6767 });
a.listen(5);
using session = std::shared_ptr<tcp::socket>;
std::function<void()> doAccept;
std::function<void(session)> doSession;
doSession = [&](session s) {
auto buf = std::make_shared<std::vector<byte>>(1024);
s->async_read_some(boost::asio::buffer(*buf), [&, s, buf](error_code ec, size_t bytes) {
if (ec)
std::cerr << "read failed: " << ec.message() << "\n";
else {
/*
As you see I dont read the Request from the Client..
But thats not relevant when I just want to send the Data
to receive it's Motd and so on..
*/
if (ec)
std::cerr << "endpoint failed: " << ec.message() << std::endl;
else {
std::vector<byte> responseBuf = createHandshakeStatusResponsePacket().getBuf();
async_write(*s, boost::asio::buffer(responseBuf), [&, s, buf](error_code ec, size_t) {
if (ec) std::cerr << "write failed: " << ec.message() << "\n";
});
}
doSession(s);
}
});
};
doAccept = [&] {
auto s = std::make_shared<session::element_type>(svc);
a.async_accept(*s, [&, s](error_code ec) {
if (ec)
std::cerr << "accept failed: " << ec.message() << "\n";
else {
doSession(s);
doAccept();
}
});
};
doAccept();
svc.run();
}
Results and Problems
When my Notchian Client reads the Packet that I sent as Response from the Server, it's giving me without any Delay this Result here:
Can't connect to Server
The Log from my Notchian Client said:
[04:42:54] [Client thread/ERROR]: Can't ping 127.0.0.1:6767: Internal Exception: io.netty.handler.codec.DecoderException:
java.io.IOException: Bad packet id 123
But how can it be the packetId 123 ? Because I'm sending the PacketId 0 at first.
Declaration
Notchian: Typically Software written from Notch ( so I grabbed it up )
ByteBuffer: Sending Bytes in a specific order.
I do hope for Tips and Solutions,
thanks
I am learning some network programming and was recommended to use boost-asio. I did Daytime Tutorials 1&2 on: http://www.boost.org/doc/libs/1_64_0/doc/html/boost_asio/tutorial.html and wanted to modify it so that the server reacts to a client sending a serialized object then sends back results. I imagined using something like the following sequence with the intention that the client would sit in the handleRead loop waiting for the server to finish:
Server:
accept --> handleRead --> process_read --> perform action --> handleWrite
Client:
connect --> handleWrite --> handleRead --> process_read
However, when I do this, both the server and the client somehow get stuck in the read loop I have set up. This is expected on the client side, but the client should be writing and sending the data prior to getting to the read loop. When I break the connection on the client side or add a socket.close() to the end of the write function I wrote, all the rest of the server steps take place with the appropriate data having been sent from the client.
I originally thought this issue had to do with Nagle's algorithm being enabled, but adding
boost::asio::ip::tcp::no_delay option(true);
socket.set_option(option);
Didn't help at all. Am I missing something that wasn't in the tutorial as to how to get this data to send such as flushing the socket?
void myClient::handle_read()
{
boost::system::error_code e;
try
{
for (;;)
{
size_t len = boost::asio::read(socket, boost::asio::buffer(inBuffer), e);
std::cout << "Client Received: ";
if (e == boost::asio::error::eof)
{
break;
}
else if (e)
{
throw boost::system::system_error(e);
}
}
}
catch (std::exception& e)
{
std::cerr << e.what() << std::endl;
std::cout << e.what() << std::endl;
}
}
template <typename T>
void handleWrite(T& t)
{
std::ostringstream archive_stream;
std::string tempString;
boost::archive::text_oarchive archive(archive_stream);
archive << t;
tempString = archive_stream.str();
outBuffer.assign(tempString.begin(), tempString.end());
boost::system::error_code ignored_error;
size_t sent = boost::asio::write(socket, boost::asio::buffer(outBuffer), ignored_error);
std::cout << sent << std::endl;
}
I'm fairly new to c++ and network programming so any additional documentation is also appreciated. Thanks!
Of course you're stuck in your read loop. It doesn't exit until end of steam, and end of stream only happens when the peer closes the socket.
I am currently trying to transfer some JSON data over the network from a client to a server using the socket API of boost-asio. My client essentially does this:
int from = 1, to = 2;
boost::asio::streambuf buf;
ostream str(&buf);
str << "{"
<< "\"purpose\" : \"request\"" << "," << endl
<< "\"from\" : " << from << "," << endl
<< "\"to\" : " << to << "," << endl
<< "}" << endl;
// Start an asynchronous operation to send the message.
boost::asio::async_write(socket_, buf,
boost::bind(&client::handle_write, this, _1));
On the server side I have the choice between various boost::asio::async_read* functions.
I wanted to use JsonCpp to parse the received data. Studying the JsonCpp API (http://jsoncpp.sourceforge.net/class_json_1_1_reader.html) I found that the Reader operates on top of either a std::string, a char* array or a std::istream which I could operate from the boost::asio::streambuf passed to the functions.
The point is that as far as I know it is not necessarily the case that the entire content is transferred at once, so I would need some kind of confirmation that the buffer contains sufficient data to process the entire document using JsonCpp. How can I assure that the buffer contains enough data?
This is an area for application level protocol
Either
read until the stream end (the sender disconnects); this doesn't work with connections that are kept alive for more than a single message
supply a header like Content-Length: 12346\r\n to know in advance how much to read
supply a delimiter (a bit like mime boundaries, but you could use any sequence that is not allowed/supported as part of the JSON payload) (async_read_until)
Treat the payload as "binary-style" (BSON e.g.) and supply a (network-order) length field before the text transmission.
The ASIO Http server example contains a pretty nice pattern for parsing HTTP request/headers that you could use. This assumes that your parser can detect completeness and just 'soft-fails' until all information is present.
void connection::handle_read(const boost::system::error_code& e,
std::size_t bytes_transferred)
{
if (!e)
{
boost::tribool result;
boost::tie(result, boost::tuples::ignore) = request_parser_.parse(
request_, buffer_.data(), buffer_.data() + bytes_transferred);
if (result)
{
request_handler_.handle_request(request_, reply_);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else if (!result)
{
reply_ = reply::stock_reply(reply::bad_request);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else
{
socket_.async_read_some(boost::asio::buffer(buffer_),
boost::bind(&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
}
else if (e != boost::asio::error::operation_aborted)
{
connection_manager_.stop(shared_from_this());
}
}
I've provided an answer that parses JSON using Boost Spirit earlier Parse a substring as JSON using QJsonDocument; you could use this to detect the end of a proper JSON document (and if it's incomplete, the end will coincide with the start)
2 problems here : 1) tell the server how many bytes to read; 2) read the JSON
for 1) you can make your own simple protocol
300#my message here
sends a 300 byte sized message; # is the delimiter between size and message
int write_request(socket_t &socket, const char* buf_json)
{
std::string buf;
size_t size_json = strlen(buf_json);
buf = std::to_string(static_cast<long long unsigned int>(size_json));
buf += "#";
buf += std::string(buf_json);
return (socket.write_all(buf.data(), buf.size()));
}
to read on the server
//parse header, one character at a time and look for for separator #
//assume size header lenght less than 20 digits
for (size_t idx = 0; idx < 20; idx++)
{
char c;
if ((recv_size = ::recv(socket.m_sockfd, &c, 1, 0)) == -1)
{
std::cout << "recv error: " << strerror(errno) << std::endl;
return str;
}
if (c == '#')
{
break;
}
else
{
str_header += c;
}
}
to read JSON, you can use
https://github.com/nlohmann/json
I am currently trying to transfer some JSON data over the network from a client to a server using the socket API of boost-asio. My client essentially does this:
int from = 1, to = 2;
boost::asio::streambuf buf;
ostream str(&buf);
str << "{"
<< "\"purpose\" : \"request\"" << "," << endl
<< "\"from\" : " << from << "," << endl
<< "\"to\" : " << to << "," << endl
<< "}" << endl;
// Start an asynchronous operation to send the message.
boost::asio::async_write(socket_, buf,
boost::bind(&client::handle_write, this, _1));
On the server side I have the choice between various boost::asio::async_read* functions.
I wanted to use JsonCpp to parse the received data. Studying the JsonCpp API (http://jsoncpp.sourceforge.net/class_json_1_1_reader.html) I found that the Reader operates on top of either a std::string, a char* array or a std::istream which I could operate from the boost::asio::streambuf passed to the functions.
The point is that as far as I know it is not necessarily the case that the entire content is transferred at once, so I would need some kind of confirmation that the buffer contains sufficient data to process the entire document using JsonCpp. How can I assure that the buffer contains enough data?
This is an area for application level protocol
Either
read until the stream end (the sender disconnects); this doesn't work with connections that are kept alive for more than a single message
supply a header like Content-Length: 12346\r\n to know in advance how much to read
supply a delimiter (a bit like mime boundaries, but you could use any sequence that is not allowed/supported as part of the JSON payload) (async_read_until)
Treat the payload as "binary-style" (BSON e.g.) and supply a (network-order) length field before the text transmission.
The ASIO Http server example contains a pretty nice pattern for parsing HTTP request/headers that you could use. This assumes that your parser can detect completeness and just 'soft-fails' until all information is present.
void connection::handle_read(const boost::system::error_code& e,
std::size_t bytes_transferred)
{
if (!e)
{
boost::tribool result;
boost::tie(result, boost::tuples::ignore) = request_parser_.parse(
request_, buffer_.data(), buffer_.data() + bytes_transferred);
if (result)
{
request_handler_.handle_request(request_, reply_);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else if (!result)
{
reply_ = reply::stock_reply(reply::bad_request);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else
{
socket_.async_read_some(boost::asio::buffer(buffer_),
boost::bind(&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
}
else if (e != boost::asio::error::operation_aborted)
{
connection_manager_.stop(shared_from_this());
}
}
I've provided an answer that parses JSON using Boost Spirit earlier Parse a substring as JSON using QJsonDocument; you could use this to detect the end of a proper JSON document (and if it's incomplete, the end will coincide with the start)
2 problems here : 1) tell the server how many bytes to read; 2) read the JSON
for 1) you can make your own simple protocol
300#my message here
sends a 300 byte sized message; # is the delimiter between size and message
int write_request(socket_t &socket, const char* buf_json)
{
std::string buf;
size_t size_json = strlen(buf_json);
buf = std::to_string(static_cast<long long unsigned int>(size_json));
buf += "#";
buf += std::string(buf_json);
return (socket.write_all(buf.data(), buf.size()));
}
to read on the server
//parse header, one character at a time and look for for separator #
//assume size header lenght less than 20 digits
for (size_t idx = 0; idx < 20; idx++)
{
char c;
if ((recv_size = ::recv(socket.m_sockfd, &c, 1, 0)) == -1)
{
std::cout << "recv error: " << strerror(errno) << std::endl;
return str;
}
if (c == '#')
{
break;
}
else
{
str_header += c;
}
}
to read JSON, you can use
https://github.com/nlohmann/json