Reading Flatbuffers objects sent via ZMQ in C++ throws unhandled exception - c++

I am trying to send a reasonably big Flatbuffers object over the network via ZMQ and then read it using C++. When accessing the object, I get unhandled exceptions that I don't know how to solve. Even this minimal example fails:
The flatbuffers schema:
namespace flatbuffer;
table TestBuf {
testStatus:bool;
testNumber:double;
testInt:int;
}
root_type TestBuf;
The main.cpp using the REP socket:
int main() {
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REP);
socket.bind("tcp://*:5555");
std::cout << "Listening for requests." << std::endl;
std::cout << "-----" << std::endl;
double count = 0;
while (1) {
zmq::message_t request;
socket.recv(&request);
// Read incoming data
auto reqmsg = flatbuffer::GetTestBuf(&request);
std::cout << "Received: " << reqmsg << std::endl;
flatbuffers::FlatBufferBuilder fbb;
flatbuffer::TestBufBuilder builder(fbb);
count++;
builder.add_testNumber(count);
std::cout << "Sending " << count << std::endl;
auto response = builder.Finish();
fbb.Finish(response);
// Send the flatbuffer
int buffersize = fbb.GetSize();
zmq::message_t message(buffersize);
memcpy((void *)message.data(), fbb.GetBufferPointer(), buffersize);
socket.send(message);
}
return 0;
}
The main.cpp using the REQ socket:
int main() {
// Prepare ZMQ context and socket
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REQ);
std::cout << "Sending out data requests." << std::endl;
socket.connect("tcp://localhost:5555");
double count = 0;
while (1) {
// Formulate response
flatbuffers::FlatBufferBuilder fbb;
flatbuffer::TestBufBuilder builder(fbb);
count++;
builder.add_testNumber(count);
auto response = builder.Finish();
fbb.Finish(response);
// Send the flatbuffer
std::cout << "Sending. " << count << ". ";
int buffersize = fbb.GetSize();
zmq::message_t message(buffersize);
memcpy((void *)message.data(), fbb.GetBufferPointer(), buffersize);
socket.send(message);
std::cout << "Sent. ";
// Receive reply
zmq::message_t reply;
socket.recv(&reply);
// Read the data
auto inmsg = flatbuffer::GetTestBuf(&reply);
std::cout << " Received reply: " << inmsg << std::endl;
//auto num = inmsg->testNumber();
//std::cout << num << " test number.";
}
return 0;
}
This code runs fine and displays (I think) the raw buffer each program is receiving. Strangely, it is not changing, although the content of the message should be. If I uncomment the last two lines and try to access inmsg->testNumber(), I get this error message:
Unhandled exception at 0x000000013F373C53 in KUKAREQ.exe: 0xC0000005: Access violation reading location 0x00000000004B35D8.
I have sent Flatbuffers objects through ZMQ successfully before, but I have not read them in C++. I am fairly sure I followed the Flatbuffers tutorial closely, but something is obviously going wrong. Pointers? Buffer sizes? Either way I would appreciate help.
Edit: To clarify my comment on the accepted answer, the offending line was:
auto inmsg = flatbuffer::GetTestBuf(&reply);
It has to be changed to:
auto inmsg = flatbuffer::GetTestBuf(reply.data());
Whoever reads this question may also be interested to know that I later came across a bug which occurs when the FlatBufferBuilder functions are not called in the correct order. Apparently the order in which the Flatbuffers object is built is important. Finding that one took me a while - novices watch out.

Not familiar with ZeroMQ, but flatbuffer::GetTestBuf(&request) this looks problematic.. you need to pass the buffer, not the message structure. Likely request.data() or similar works better.
In general, if it crashes in FlatBuffers, you should use the verifier to verify the buffer you're passing to FlatBuffers. If that fails, it means you're not passing legal data to FlatBuffers, as is the case here.
Also, you may want to check if ZeroMQ can send buffers without copying, will be faster.

Related

zmq::message_t assign a string

I am trying to familiarize myself with ZeroMQ by creating a simple socket communication betwenn a publisher and a subscriber to send a test message. However, I can't find the information I want on how to put a string inside a zmq::message_t type message. Indications pointed to the use of "std::memcpy(message.data(), ms.data(), ms.size())" which I tried. However, by debuging the coding using Watchs, I see that the message is still empty after execution:
WATCH IMAGE and it is also empty when I print it out using a cout: Cmd
Is there another way to assign a string to a zmq::message_t message or is there something else wrong here?
My entire code is:
int main()
{
zmq::context_t context(1);
zmq::socket_t pub(context, ZMQ_PUB);
pub.bind("tcp://*:5555");
std::cout << "Pub Connected" << std::endl;
zmq::socket_t sub(context, ZMQ_SUB);
sub.connect("tcp://localhost:5555");
std::cout << "Sub Connected" << std::endl;
std::stringstream s;
s << "Hello World";
auto ms = s.str();
zmq::message_t message(ms.size());
memcpy(message.data(), ms.c_str(), ms.length());
pub.send(message, zmq::send_flags::none);
std::cout << "message: " << message << std::endl;
zmq_sleep(1);
sub.set(zmq::sockopt::subscribe, "Hello World");
zmq::message_t rx_msg;
sub.recv(rx_msg,zmq::recv_flags::none);
std::string rx_str;
rx_str.assign(static_cast<char*>(rx_msg.data()), rx_msg.size());
std::cout << "Message: " << rx_str << "received!" << std::endl;
}
There is a constructor for zmq::message_t that has the signature (docs)
message_t(const void *data_, size_t size)
so you could use this like
zmq::message_t message(static_cast<void*>(ms.data()), ms.size());

How to Send messages between server and client using C++ standard library networking TS

I have tried following the tutorial from boost, however the API is not identical so I have had to guess some parts.
My attempt so far is shown bellow:
#include <iostream>
#include <experimental/internet>
#include <experimental/socket>
#include <thread>
#include <chrono>
using namespace std::experimental;
int main(int argc, char* argv[])
{
std::thread server = std::thread([]()
{
std::cout << "Starting server" << std::endl;
net::io_context context;
net::ip::tcp::endpoint endpoint{net::ip::tcp::v4(), 1234};
net::ip::tcp::acceptor acceptor{context, endpoint};
acceptor.non_blocking(true);
std::cout << "opened server on " << endpoint << std::endl;
std::error_code error;
net::ip::tcp::socket socket(context);
while (true)
{
socket = acceptor.accept(error); //accept connections
if (!error) //if connected with a client
{
std::cout << "Connected to client!" << std::endl;
std::this_thread::sleep_for(std::chrono::seconds(2));
std::string data = "Hello World!";
net::const_buffer buf(&data, sizeof(data));
socket.send(buf);
std::cout << "Sent data!" << std::endl;
while(true) {}
}
}
});
std::thread client = std::thread([]()
{
net::io_context context;
net::ip::tcp::socket socket(context);
net::ip::tcp::endpoint server{net::ip::tcp::v4(), 1234};
std::error_code error;
while(true)
{
socket.connect(server, error); //attempt to connect
if (!error) //if connected
{
std::cout << "Connected to server!" << std::endl;
net::mutable_buffer buf;
while(buf.size() == 0)
{
socket.receive(buf);
}
std::cout << "Received data!" << std::endl;
std::cout << buf.data() << std::endl;
while(true) {}
}
}
});
server.join();
return 0;
}
The sever and client connect, but the message is not received by the client. The output from the program above is:
Starting server
opened server on 0.0.0.0:1234
Connected to server!
Connected to client!
Sent data!
And then it waits forever.
How do I get the socket to correctly receive the data?
This
std::string data = "Hello World!";
net::const_buffer buf(&data, sizeof(data));
is wrong. You want to send content of data string, not its internal bytes. &data gives you a pointer to underlying data of string instance, not its content. If you want to create buffer which represents content of data you can do:
const std::string data = "Hello World!";
net::const_buffer buf = net::buffer(data);
This
net::mutable_buffer buf;
while(buf.size() == 0)
{
socket.receive(buf);
}
gives you infinite loop because initial size of buf is 0, so receive reads 0 bytes and returns. Then while condition is checked, buf's size is still 0, and the loop goes on.
Before calling receive you need to specify the size of buffer - it indicates how many bytes must be read. You are sending Hello World! so
std::string msg;
msg.resize(12); // prepare space for incoming data
net::mutable_buffer buf = net::buffer(msg);
socket.receive(buf);
std::cout << "I got: " << msg << std::endl;

Sending data through ZeroMQ (zmqpp) using MsgPack gives 'msgpack::v1::insufficient_bytes' error

I made a PUB/SUB connection using zmqpp and now I want to send data from the publisher to the subscribers using the header-only, C++11 version of msgpack-c.
The publisher has to send 2 int64_t numbers -- header_1 and header_2 -- followed by a std::vector<T> -- data --, where T is determined by the (header_1, header_2) combination.
Sinse there aren't that many examples on how to combine msgpack and zmqpp, the idea I came up with is to send a 3-part message by using zmqpp::message::add/add_raw. Each part would be packed/unpacked using msgpack.
The publisher packs a single data part as follows:
zmqpp::message msg;
int64_t header_1 = 1234567;
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_1);
msg.add(buffer.data(), buffer.size());
And the receiver unpacks it like this:
zmqpp::message msg;
subscriberSock.receive(msg);
int64_t header_1;
msgpack::unpacked unpackedData;
// crash !
msgpack::unpack(unpackedData,
static_cast<const char*>(msg.raw_data(0)),
msg.size(0));
unpackedData.get().convert(&header_1);
When I run the code, I get the following error on the subscriber side:
terminate called after throwing an instance of 'msgpack::v1::insufficient_bytes'
what(): insufficient bytes
Aborted
Also, it seems that zmqpp has generated a 5-part message, even though I called add() only 3 times.
Q1: Am I packing/unpacking the data correctly ?
Q2: Is this the proper method for sending msgpack buffers using zmqpp ?
Here are the important parts of the code:
Publisher
zmqpp::socket publisherSock;
/* connection setup stuff ...*/
// forever send data to the subscribers
while(true)
{
zmqpp::message msg;
// meta info about the data
int64_t header_1 = 1234567;
int64_t header_2 = 89;
// sample data
std::vector<double> data;
data.push_back(1.2);
data.push_back(3.4);
data.push_back(5.6);
{
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_1);
msg.add(buffer.data(), buffer.size());
cout << "header_1:" << header_1 << endl; // header_1:1234567
}
{
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_2);
msg.add(buffer.data(), buffer.size());
cout << "header_2:" << header_2 << endl; // header_2:89
}
{
msgpack::sbuffer buffer;
msgpack::pack(buffer, data);
msg.add_raw(buffer.data(), buffer.size());
std::cout << "data: " << data << std::endl; // data:[1.2 3.4 5.6]
}
std::cout << msg.parts() << " parts" << std::endl; // prints "5 parts"... why ?
publisherSock.send(msg);
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
Subscriber
zmqpp::socket subscriberSock;
/* connection setup stuff ...*/
zmqpp::message msg;
subscriberSock.receive(msg);
int64_t header_1;
int64_t header_2;
std::vector<double> data;
std::cout << msg.parts() << " parts" << std::endl; // prints "5 parts"
{
// header 1
{
msgpack::unpacked unpackedData;
// crash !
msgpack::unpack(unpackedData,
static_cast<const char*>(msg.raw_data(0)),
msg.size(0));
unpackedData.get().convert(&header_1);
cout << "header_1:" << header_1 << endl;
}
// header 2
{
msgpack::unpacked unpackedData;
msgpack::unpack(unpackedData,
static_cast<const char*>(msg.raw_data(1)),
msg.size(1));
unpackedData.get().convert(&header_2);
cout << "header_2:" << header_2 << endl;
}
// data
{
msgpack::unpacked unpacked_data;
msgpack::unpack(unpacked_data,
static_cast<const char*>(msg.raw_data(2)),
msg.size(2));
unpacked_data.get().convert(&data);
std::cout << "data:" << data << std::endl;
}
}
EDIT: Problem solved: As pointed out by #Jens, the correct way of packing/sending data is by using zmqpp::message::add_raw()
zmqpp::message msg;
int64_t header_1 = 1234567;
msgpack::sbuffer buffer;
msgpack::pack(buffer, header_1);
msg.add_raw(buffer.data(), buffer.size());
I think the calls to msg.add(buffer.data(), buffer.size()do not add a array of buffer.size() bytes, but call message::add(Type const& part, Args &&...args), which
msg << buffer.data(), which probably calls message::operator<<(bool) since a pointer converts to bool
add(buffer.size()) which then calls msg << buffer.size(), which adds a size_t value as the next part.
Looking at the zmqpp::message class, using message::add_raw should do the trick.
PS: This is all without any guarantee because I have never used zmqpp or msgpack.

Boost sockets - client is not receiving all bytes from server

I am developing an application with C++ and having some difficulty with boost sockets. The server sends an image but not all the bytes are received by the client; the client always receives about 500 bytes less than the server sent. Provided below is the pertinent code and screenshots of the program running.
Server code:
int sent = boost::asio::write(*socket, response, boost::asio::transfer_all(), error);
std::cout << "Sent: " << sent << std ::endl;
Client code (I know that read_some will block if the total bytes sent by the server is divisible by 10000; this code is just for testing):
int len = 0;
int count = 0;
do {
len = socket->read_some( boost::asio::buffer( imageData, 10000 ) );
count += len;
std::cout << "len: " << len << std::endl;
std::cout << "count: " << count << std::endl;
} while(len == 10000);
std::cout << "Image Received of size: " << count << std::endl;
Screenshot of server:
Screenshot of client:
Thanks for your time; any help or suggestions would be greatly appreciated.
There're no guarantee you'll receive complete buffers of 10000 bytes.
I'd recommend following approach:
To send some binary data w/o any explicit terminator, first send its size and only then data itself. In this case client will know how many data in this chunk it should receive.

How can I receive multipart messages with ZeroMQ?

I can't get ZeroMQ C++ wrapper to receive multipart messages. The same code using C version works just fine, but it leads to an exception with no explanations at all with C++. The multipart handling code is as follows:
int _tmain(int argc, _TCHAR* argv[])
{
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REP);
socket.bind("tcp://*:5555");
while(true) {
// the following two lines lead to exception
zmq::message_t request;
socket.recv(&request);
//zmq_msg_t message;
//zmq_msg_init (&message);
//zmq_recv (socket, &message, 0);
}
return 0;
}
It is extremely simple; this version does not work. but if I comment out the first two lines in the while loop and uncomment the currently commented (C version) code, it works.
This is Windows XP sp3, Zeromq 2.1.1 and Visual Studio 2010 Express.
If I send single part messages, both versions work fine.
What am I doing wrong?
I'm also a newbie in ZMQ and I too had to struggle a lot in order to understand multipart messaging using REP/REQ in ZeroMQ. I had to go through multiple resources and stitch data in order to understand this. I think this answer will help many seekers in the near future that's why I am sharing the client and server code here. I have tested this code and it is working perfectly fine. However, being a newbie there are chances that I would have missed something vital. Please share your valuable inputs.
Server Code
void
serverMultipartREPREQ()
{
try
{
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REP);
socket.bind("tcp://*:5556");
std::cout << "Listening at port 5556..." << std::endl;
zmq::message_t reply;
socket.recv(reply, zmq::recv_flags::none);
auto rep = std::string(static_cast<char*> (reply.data()), reply.size());
std::cout << "Received: " << rep << std::endl;
while(1)
{
if (input == "exit")
break;
for (int j = 0; j < 3; ++j)
{
std::string s("Message no - " + std::to_string(j));
zmq::message_t message(s.length());
memcpy(message.data(), s.c_str(), s.length());
std::cout << "Sending: " << s << std::endl;
if (j != 2)
socket.send(message, zmq::send_flags::sndmore);
else
socket.send(message, zmq::send_flags::none);
}
}
}
catch (const zmq::error_t& ze)
{
std::cout << "Exception: " << ze.what() << std::endl;
}
Sleep(5000);
}
Client code
void
clientMultipartREQREP()
{
try
{
zmq::context_t context(1);
std::cout << "Connecting to socket at 5556" << std::endl;
zmq::socket_t socket(context, ZMQ_REQ);
socket.connect("tcp://localhost:5556");
std::cout << "Connected to socket at 5556" << std::endl;
std::string msg("Hii this is client...");
zmq::message_t message(msg.length());
memcpy(message.data(), msg.c_str(), msg.length());
socket.send(message, zmq::send_flags::none); // send to server (request message)
while (true)
{
__int64 more = 1;
if (more)
{
zmq::message_t message;
socket.recv(message, zmq::recv_flags::none);
auto rep = std::string(static_cast<char*> (message.data()), message.size());
std::cout << "Reading from client: " << rep << std::endl;
size_t size = sizeof(__int64);
socket.getsockopt(ZMQ_RCVMORE, &more, &size); // if msg is not the last one then more = 1 else more = 0
}
else
{
std::cout << "Done..." << std::endl;
break;
}
}
}
catch (const zmq::error_t& ze)
{
std::cout << "Exception: " << ze.what() << std::endl;
}
Sleep(5000);
}
Probably C version of code doesn't work either, but you don't check the return code of zmq_recv, so you don't notice it. Also, when receiving miltipart messages you should check if there are more message parts to be received through the socket, like this:
int64_t more = 0;
size_t more_size = sizeof(more);
socket.getsockopt(ZMQ_RCVMORE, &more, &more_size);
if (more != 0)
{
//has more parts
}
Also, take a look at ZmqMessage C++ library designed specifically for Sending and receiving ZeroMQ multipart messages.
I decided to use the C version of the code. In general all examples seem to be in C anyway.