How to receive binary data (protobuf) via REST SDK? - c++

I try to send and receive a protobuf object by using Microsoft REST SDK. If you are not familiar with protobuf, imagine that it can be any other raw data. On the client side the vector is empty. I think that I do not create correctly the streams on the server side.
The get method of the server looks like this:
void MyServer::handle_get(http_request message)
{
// a class created with protobuf
ns::user *user = new ns::user();
ns::user->set_id(-1);
ns::user->set_name("Dark");
ns::user->set_email("dark#foo.com");
// this schould be fine
int size = user->ByteSize();
void *buffer = malloc(size);
user->SerializeToArray(buffer, size);
// how to do?
concurrency::streams::container_buffer<std::vector<char>> inBuffer;
concurrency::streams::istream bodyStream = message.body();
inBuffer.putn((char*)buffer, size);
Concurrency::task<size_t> read = bodyStream.read_to_end(inBuffer);
message.reply(status_codes::OK, bodyStream, size, L"application/octet-stream");
delete user;
};
The client method like this:
utility::ostringstream_t buf;
buf << "Dark" << "/" << "Foo" << "?" << "id=" << "1";
http_response response = CheckResponse("server/user", server.request(methods::GET, buf.str()).get());
if (response.status_code() == status_codes::OK)
{
cout << "STATUS OK" << endl;
Concurrency::streams::istream iStream = response.body();
auto rslt0 = response.extract_vector();
auto rslt = rslt0.get();
cout << "Result has size: <" << rslt.size() << ">" << endl;
for (unsigned int i = 0; i < rslt.size(); i++)
{
cout << "Data" << i << ": <" << rslt.at(i) << ">" << endl;
}

Server side:
std::vector<uint8_t> data;
//TODO: fill data
//convert the vector into a async istream
auto instream = Concurrency::streams::bytestream::open_istream(data);
message.reply(status_codes::OK, instream);
Client side:
wchar_t urlbuf[256];
//TODO: fill the url
client->request(web::http::methods::GET,urlbuf).then([](web::http::http_response& resp){
if(resp.status_code() != status_codes::OK){
DLogError("error: status != ok");
return;
}
Concurrency::streams::container_buffer<std::vector<uint8_t> >inBuffer;
resp.body().read_to_end(inBuffer).then([](size_t bytesRead){
//TODO: use it
}
}

I answer my own question. There is no need for stream; I can use the binary data by this way:
Server:
void MyServer::handle_get(http_request message)
{
// a class created with protobuf
ns::user *user = new ns::user();
ns::user->set_id(-1);
ns::user->set_name("Dark");
ns::user->set_email("dark#foo.com");
int size = user->ByteSize();
unsigned char *buffer = new unsigned char(size);
user->SerializeToArray(buffer, size);
std::string out;
user->SerializeToString(&out);
std::vector<unsigned char> data(buffer, buffer + size);
string_t outT = utility::conversions::to_base64(data);
message.reply(status_codes::OK, outT);
delete user;
};
Client:
utility::ostringstream_t buf;
buf << "Dark" << "/" << "Foo" << "?" << "id=" << "1";
http_response response = CheckResponse("server/user", server.request(methods::GET, buf.str()).get());
if (response.status_code() == status_codes::OK)
{
cout << "STATUS OK" << endl;
geopp::gnUser *user = new geopp::gnUser();
auto rslt = response.extract_string();
auto rslt0 = utility::conversions::from_base64(rslt.get());
user->ParseFromArray(rslt0.data(), rslt0.size());
cout << "Result: name <" << user->name() << ">, id <" << user->id() << ">" << endl;
};

Related

Double free or corruption C++ Avro

I need to serialize a JSON string using Avro in C++.
I installed the libserdes library (https://github.com/confluentinc/libserdes) and I used the example code
./examples/kafka-serdes-avro-console-producer.cpp
to write a main,
which serializes a JSON string as follows.
{"s":"BTCUSDT","c":"spread","u":123456789,"a":20000.4,"b":21000.5,"A":1.25,"B":0.58,"p":-1,"P":-1,"st":-1,"rt":1675000000123000}
Here is the code
#include "fh_price_t.h"
#include "avro/Encoder.hh"
#include "avro/Decoder.hh"
#include <avro/Generic.hh>
#include <avro/Specific.hh>
#include <avro/Exception.hh>
#include "libserdes/serdescpp-avro.h"
#include <iostream>
#include <sstream>
#include <string>
#include <fstream>
#define FATAL(reason...) do { \
std::cerr << "% FATAL: " << reason << std::endl; \
exit(1); \
} while (0)
template<class T> std::string to_string_a (const T& x) {
std::ostringstream oss;
oss << x;
return oss.str();
}
/**
* Convert JSON to Avro Datum.
*
* Returns 0 on success or -1 on failure.
*/
int json2avro (Serdes::Schema *schema, const std::string &json,
avro::GenericDatum **datump) {
avro::ValidSchema *avro_schema = schema->object();
/* Input stream from json string */
std::istringstream iss(json);
auto json_is = avro::istreamInputStream(iss);
/* JSON decoder */
avro::DecoderPtr json_decoder = avro::jsonDecoder(*avro_schema);
avro::GenericDatum *datum = new avro::GenericDatum(*avro_schema);
try {
/* Decode JSON to Avro datum */
json_decoder->init(*json_is);
avro::decode(*json_decoder, *datum);
} catch (const avro::Exception &e) {
std::cerr << "% JSON to Avro transformation failed: "
<< e.what() << std::endl;
return -1;
}
*datump = datum;
}
int main (int argc, char* argv) {
izycoinstestavro::fh_price_t price{};
price.s = "BTCUSDT";
price.c = "spread";
price.u = 123456789;
price.a = 20000.4;
price.A = 1.25;
price.b = 21000.52;
price.B = 0.58;
price.p = -1;
price.P = -1;
price.st = -1;
price.rt = 1675000000123000;
std::ifstream file_ifstream("../src/avro/fh_price_t.json");
std::stringstream buffer;
buffer << file_ifstream.rdbuf();
std::string schema_json = buffer.str();
Serdes::Conf *sconf = Serdes::Conf::create();
Serdes::Schema *schema;
std::string errstr;
if (sconf->set("schema.registry.url", "http://localhost:8081", errstr))
FATAL("Conf failed: " << errstr);
if (sconf->set("serializer.framing", "cp1", errstr))
FATAL("Conf failed: " << errstr);
Serdes::Avro *serdes = Serdes::Avro::create(sconf, errstr);
if (!serdes)
FATAL("Failed to create Serdes handle: " << errstr);
schema = Serdes::Schema::add(serdes, "fh_price_t", schema_json, errstr);
if (!schema)
FATAL("Failed to register schema " << "fh_price_t" << ": " << errstr);
std::cout << "% Registered schema " << schema->name() << " with id " << schema->id() << std::endl;
avro::GenericDatum *datum = NULL;
std::vector<char> out;
/* Convert JSON to Avro object */
std::string line = to_string_a(price);
json2avro(schema, line, &datum);
//std::cout << to_string_a(price) << std::endl;
/*if (rr == -1) {
FATAL("Failed to convert JSON to Avro " << to_string_a(price) << ": " << errstr);
}
// Serialize Avro
if (serdes->serialize(schema, datum, out, errstr) == -1) {
std::cerr << "% Avro serialization failed: " << errstr << std::endl;
delete datum;
exit(1);
}
delete datum;
std::cout << "Data Size : " << out.size() << std::endl;*/
return 0;
}
When I run it, I get a double free or corruption (out) error.
The error occurs at the output of the function.
I have located the line that causes the error (because when I remove it, the error disappears)
avro::GenericDatum *datum = new avro::GenericDatum(*avro_schema);
I would like to know why and how to solve it.

How to parse a const char* from a double / long without the std::string library?

How can I parse a const char* from a double or long?
Mainly because my code is a lot faster when I use a const char*, so i decided to create a small base string class. But my code to parse a double has some bugs.
My code only works partially. Some help would be very appreciated.
I am using macos, g++ & c++17.
Code:
#include <iostream>
class bstring {
public:
const char* characters;
bstring(const char* c = "") { characters = c; }
static bstring parse(const double number, int precision = 100) {
// Convert.
int decimal, sign;
char *buffer;
buffer = ecvt(number, precision, &decimal, &sign);
int n = strlen(buffer);
// Add decimal.
char before[decimal];
strncpy(before, 0 + buffer, decimal);
char after[n - decimal - 1];
strncpy(after, decimal + buffer, n - decimal - 1);
// Remove zero padding.
int removed = 0;
while (true) {
size_t n = sizeof(after) - removed;
size_t index_to_remove = n - 1;
if (after[index_to_remove] == '0') {
for (size_t i = index_to_remove; i < n - 1; ++i) {
after[i] = after[i + 1];
}
removed += 1;
} else { break; }
}
bool is_zero = removed == sizeof(after);
int after_size = sizeof(after)-removed;
char* nafter = (char*)malloc(sizeof(char) * after_size);
// Concat.
char* new__{ new char[strlen(before) + 1 + after_size] };
new__ = strcpy(new__, before);
new__ = strcat(new__, ".");
if (is_zero) {
char a[] = "0";
new__ = strcat(new__, a);
} else {
new__ = strcat(new__, after);
}
// Assign.
bstring s = new__;
delete[] new__; new__ = NULL;
return s;
//
}
};
std::ostream& operator <<(std::ostream &s, bstring x) { return s << x.characters; }
int main() {
std::cout << "Should be " << "-1234.39950" << ": " << bstring::parse(-1234.39950) << std::endl;
std::cout << "Should be " << "-1.0" << ": " << bstring::parse(-1.0) << std::endl;
std::cout << "Should be " <<"0.0" << ": " << bstring::parse(0.0) << std::endl;
std::cout << "Should be " <<"0.3897495" << ": " << bstring::parse(0.3897495) << std::endl;
std::cout << "Should be " <<"1.0" << ": " << bstring::parse(1.0) << std::endl;
std::cout << "Should be " <<"100.00" << ": " << bstring::parse(1000.0) << std::endl;
std::cout << "Should be " <<"10000.000" << ": " << bstring::parse(1000000.0) << std::endl;
std::cout << "Should be " <<"1000000.0000" << ": " << bstring::parse(1000000000.0) << std::endl;
std::cout << "Should be " <<"1000000000.0000" << ": " << bstring::parse(1000000000000.0) << std::endl;
std::cout << "Should be " <<"1000000000000.0000" << ": " << bstring::parse(1000000000000000.0) << std::endl;
}
Edit:
Is this piece of code okay? Or am I doing something wrong by not deleting it / By where I assign the new__ to.
// Concat.
bstring concat(const char* c) {
int n = ::strlen(characters) + ::strlen(c);
if (n == 0) { return bstring(); }
if (::strlen(c) == 0) { return bstring(characters); }
char* new__{ new char[n + 1] };
new__ = strcpy(new__, characters);
new__ = strcat(new__, c);
// const char* n = new__;
// delete[] new__; new__ = NULL;
bstring s = new__;
return s;
}

Can the NETLINK/SOCK_DIAG interface be used to listen for `listen()` and `close()` events of said socket?

I've been gleaning information about the NETLINK socket which allows us to listen on what is happening in socket land. It seems to very partially work, but I'm wondering whether I missed something.
I'm using C++, although of course all the NETLINK is pure C, so here we have mainly C headers. The following code has three main parts:
Binding
First I create a socket() and bind() it.
The bind() is kind of magical. When using a bound NETLINK socket, you start receiving events without having to have a polling setup (which is why I'm trying to do this, to avoid polling for when a socket starts listening for connections).
I put -1 in the nl_groups so that way all events are sent to my socket. But, at this point, I seem to only receive two of them: TCP_ESTABLISHED and TCP_CLOSE. The one I really would like to receive is the TCP_LISTEN and "not listening" (which apparently is not going to be available...)
Explicit Request
I tried with an explicit request. I have it in the code below so you can see how I've done it. That request works as expected. I get an explicit reply if the socket exists or an error "No such file or directory" when the socket is closed. Great, except that mechanism means I'd be using a poll (i.e. I need my process to try over and over again on a timer until the socket is visible).
Note: the error when no one is listening is happening because the request is explicit, i.e. it includes the expected IP address and port that I'm interested in.
Response
The next part is a loop, which sits until a response is received. The recvmsg() call is blocking in this version, which is why it sits around in this test.
If I sent my explicit request (see point 2. above), then, as I mentioned, I get a reply if another process is listening, otherwise I get an error saying it's not listening. The state is clearly set to 10 (TCP_LISTEN), so everything works as expected.
When listening to all the events (-1 in the bind), the process will go on and receive more data as events happen. However, so far, the only events I've received are 1 and 7 (i.e. TCP_ESTABLISHED and TCP_CLOSE).
I used the following to compile my code:
g++ -Wall -o a test.cpp
Here is my test code with which I can reproduce my current results:
#include <iostream>
#include <linux/netlink.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <linux/sock_diag.h>
#include <linux/inet_diag.h>
int
main(int argc, char ** argv)
{
// socket / bind
int d = socket(AF_NETLINK, SOCK_RAW, NETLINK_SOCK_DIAG);
if(d < 0)
{
std::cerr << "error: could not create RAW socket.\n";
return 1;
}
struct sockaddr_nl addr = {};
addr.nl_family = AF_NETLINK;
addr.nl_pid = getpid();
addr.nl_groups = -1;
// You can find these flags in Linux source:
//
// "/usr/src/linux-headers-4.15.0-147/include/net/tcp_states.h
//
// (1 << 7) // TCPF_CLOSE
// | (1 << 8) // TCPF_CLOSE-WAIT
// | (1 << 10) // TCPF_LISTEN
// | (1 << 11) // TCPF_CLOSING
// ;
if(bind(d, reinterpret_cast<struct sockaddr *>(&addr), sizeof(addr)) != 0)
{
perror("bind failure\n");
return 1;
}
// request
struct sockaddr_nl nladdr = {};
nladdr.nl_family = AF_NETLINK;
struct nl_request
{
struct nlmsghdr f_nlh;
struct inet_diag_req_v2 f_inet;
};
nl_request req = {};
req.f_nlh.nlmsg_len = sizeof(req);
req.f_nlh.nlmsg_type = SOCK_DIAG_BY_FAMILY;
req.f_nlh.nlmsg_flags = NLM_F_REQUEST;
req.f_inet.sdiag_family = AF_INET;
req.f_inet.sdiag_protocol = IPPROTO_TCP;
req.f_inet.idiag_ext = 0;
req.f_inet.pad = 0;
req.f_inet.idiag_states = 0;
req.f_inet.id.idiag_sport = htons(4998);
req.f_inet.id.idiag_dport = 0;
req.f_inet.id.idiag_src[0] = htonl(0x0A00020A);
req.f_inet.id.idiag_dst[0] = 0;
req.f_inet.id.idiag_if = 0;
req.f_inet.id.idiag_cookie[0] = INET_DIAG_NOCOOKIE;
req.f_inet.id.idiag_cookie[1] = INET_DIAG_NOCOOKIE;
struct iovec vector = {};
vector.iov_base = &req;
vector.iov_len = sizeof(req);
struct msghdr msg = {};
msg.msg_name = &nladdr;
msg.msg_namelen = sizeof(nladdr);
msg.msg_iov = &vector;
msg.msg_iovlen = 1;
int const r(sendmsg(d, &msg, 0));
if(r < 0)
{
perror("sendmsg");
return 1;
}
// response
struct sockaddr_nl r_nladdr = {};
r_nladdr.nl_family = AF_NETLINK;
struct iovec r_vector = {};
long buf[8192 / sizeof(long)];
r_vector.iov_base = buf;
r_vector.iov_len = sizeof(buf);
for(int i(1);; ++i)
{
struct msghdr r_msg = {};
r_msg.msg_name = &r_nladdr;
r_msg.msg_namelen = sizeof(r_nladdr);
r_msg.msg_iov = &r_vector;
r_msg.msg_iovlen = 1;
//std::cout << "wait for message...\n";
ssize_t size(recvmsg(d, &r_msg, 0));
if(size < 0)
{
perror("recvmsg");
return 1;
}
if(size == 0)
{
std::cout << "end of message stream received." << std::endl;
break;
}
//std::cout << "got message #" << i << ": size = " << size << std::endl;
struct nlmsghdr const * h(reinterpret_cast<struct nlmsghdr *>(buf));
if(!NLMSG_OK(h, size))
{
std::cerr << "NLMSG_OK() says there is an error." << std::endl;
return 1;
}
do
{
if(h->nlmsg_type == NLMSG_DONE)
{
std::cout << "explicit end of message stream received (NLMSG_DONE)." << std::endl;
break;
}
if(h->nlmsg_type == NLMSG_ERROR)
{
struct nlmsgerr const * err(reinterpret_cast<struct nlmsgerr const *>(NLMSG_DATA(h)));
if(h->nlmsg_len < NLMSG_LENGTH(sizeof(*err)))
{
std::cerr << "unknown NLMSG_ERROR received." << std::endl;
}
else
{
// here is the location display an error when trying to get an
// event about the LISTEN and no one is listening on that port.
//
errno = -err->error;
perror("NLMSG_ERROR:");
}
return 1;
}
if(h->nlmsg_type != SOCK_DIAG_BY_FAMILY)
{
std::cerr << "unexpected message type (h->nlmsg_type) "
<< h->nlmsg_type
<< std::endl;
return 1;
}
//std::cout << "------- sock_diag info!\n";
struct inet_diag_msg const * k_msg(reinterpret_cast<struct inet_diag_msg const *>(NLMSG_DATA(h)));
if(h->nlmsg_len < NLMSG_LENGTH(sizeof(*k_msg)))
{
std::cerr << "unexpected message length (h->nlmsg_len) "
<< h->nlmsg_type
<< std::endl;
return 1;
}
switch(k_msg->idiag_state)
{
case 1:
case 7:
break;
default:
{
std::uint32_t const src_ip(ntohl(k_msg->id.idiag_src[0]));
std::uint32_t const dst_ip(ntohl(k_msg->id.idiag_dst[0]));
std::cout << "inet_diag_msg->idiag_family = " << static_cast<int>(k_msg->idiag_family) << "\n"
<< "inet_diag_msg->idiag_state = " << static_cast<int>(k_msg->idiag_state) << "\n"
<< "inet_diag_msg->idiag_timer = " << static_cast<int>(k_msg->idiag_timer) << "\n"
<< "inet_diag_msg->idiag_retrans = " << static_cast<int>(k_msg->idiag_retrans) << "\n"
<< "inet_diag_msg->id.idiag_sport = " << ntohs(k_msg->id.idiag_sport) << "\n"
<< "inet_diag_msg->id.idiag_dport = " << ntohs(k_msg->id.idiag_dport) << "\n"
<< "inet_diag_msg->id.idiag_src[0] = " << ((src_ip >> 24) & 255)
<< "." << ((src_ip >> 16) & 255) << "." << ((src_ip >> 8) & 255) << "." << (src_ip & 255) << "\n"
<< "inet_diag_msg->id.idiag_dst[0] = " << ((dst_ip >> 24) & 255)
<< "." << ((dst_ip >> 16) & 255) << "." << ((dst_ip >> 8) & 255) << "." << (dst_ip & 255) << "\n"
<< "inet_diag_msg->id.idiag_if = " << k_msg->id.idiag_if << "\n"
<< "inet_diag_msg->id.idiag_cookie[0] = " << k_msg->id.idiag_cookie[0] << "\n"
<< "inet_diag_msg->id.idiag_cookie[1] = " << k_msg->id.idiag_cookie[1] << "\n"
<< "inet_diag_msg->idiag_expires = " << k_msg->idiag_expires << "\n"
<< "inet_diag_msg->idiag_rqueue = " << k_msg->idiag_rqueue << "\n"
<< "inet_diag_msg->idiag_wqueue = " << k_msg->idiag_wqueue << "\n"
<< "inet_diag_msg->idiag_uid = " << k_msg->idiag_uid << "\n"
<< "inet_diag_msg->idiag_inode = " << k_msg->idiag_inode << "\n"
<< "\n";
}
break;
}
// next message
//
h = NLMSG_NEXT(h, size);
}
while(NLMSG_OK(h, size));
}
return 0;
}
To test that IP:port combo, I simply used the nc command like so:
nc -l 10.0.2.10 4998
You of course need the 10.0.2.10 IP on one of your interfaces for this to work.
My question is:
Did I do something wrong that I do not receive TCP_LISTEN events on that socket unless explicitly requested?
P.S. Just in case, I tried to run this test app as root. Same results.

C++ QueryDosDeviceA() return 0 but COMX is connected

so I'm doing a function to retrieve all the COM serial port names connected to the system using QueryDosDeviceA
I tried to use CreatFileA with "\\\\.\\COM6", and it doesn't throw me any errors.
But QueryDosDeviceA with "COM6" or "\\\\.\\COM6"
Function.cpp
std::vector<std::string> SerialPort::_SerialList()
{
std::vector<std::string> serialList;
std::string COMName("\\\\.\\COM"), queryName(COMName); //also tried with COMName("COM")
CHAR bufferTragetPath[5000];
std::string tmp;
DWORD path_size(0);
//test each COM name to get the one used by the system
for (int i(0); i < 255; i++)
{
queryName = COMName + std::to_string(i);
//Query the path of the COMName
path_size = QueryDosDeviceA((LPCSTR)&queryName, (LPSTR)&bufferTragetPath, 5000);
std::cout << std::endl << "Path for " << queryName << ":" << path_size;
if (path_size != 0) {
std::cout << "pushing..." << queryName << " on " << bufferTragetPath << std::endl;
serialList.push_back(queryName);
}
}
return serialList;
}
And here is the output :
Path for \\.\COM0:0
Path for \\.\COM1:0
Path for \\.\COM2:0
Path for \\.\COM3:0
Path for \\.\COM4:0
Path for \\.\COM5:0
Path for \\.\COM6:0
Path for \\.\COM7:0
Path for \\.\COM8:0
Path for \\.\COM9:0
...
Path for \\.\COM253:0
Path for \\.\COM254:0Port name: \\.\COM6
Hello World !
So it's finding nothing but the COM6 port can be used by CreateFileA
Thanks to #user253751, #JohnnyMopp, #dresscherjm
And #RemyLebeau
Here is the code that include the solution:
std::vector<std::string> SerialPort::_SerialList()
{
std::vector<std::string> serialList;
std::string COMName("COM"), queryName("");
CHAR bufferTragetPath[5000];
std::string tmp;
DWORD path_size(0);
//test each COM name to get the one used by the system and get his description name
for (int i(0); i < 255; i++)
{
queryName = COMName + std::to_string(i);
//Query the path of the COMName
path_size = QueryDosDeviceA(queryName.c_str(), bufferTragetPath, 5000);
std::cout << std::endl << "Path for " << queryName << ":" << path_size << " " << queryName;
if (path_size != 0) {
std::cout << "pushing..." << queryName << " on " << bufferTragetPath << std::endl;
serialList.push_back(tmp);
}
}
return serialList;
}
Basically i change the convertion of queryName from string to LPCSTR
//LPCSTR => Long Pointer Const String(= const char)
queryName.c_str() //return queryName as const char*
I've also removed a cast from bufferTragetPath that was useless
//from
(LPSTR)&bufferTragetPath
//to
bufferTragetPath //cause bufferTragetPath is char *

ZeroMQ: how to use multiple Publishers and a single Client, using C < C11

I am new to ZeroMQ.
I have multiple publishers and one client. Seeking suggestions to implement it in a best way.
Currently its making use of a reply - request pattern for a single client and a server; this has to be extended to multiple publishers and a single subscriber.
This application is going to run on a QNX-system that does not support C11, so zmq::multipart_t is not helping.
void TransportLayer::Init()
{
socket.bind( "tcp://*:5555" );
}
void TransportLayer::Receive()
{
while ( true ) {
zmq::message_t request;
string protoBuf;
socket.recv( &request );
uint16_t id = *( (uint16_t*)request.data() );
protoBuf = std::string( static_cast<char*>( request.data()
+ sizeof( uint16_t )
),
request.size() - sizeof( uint16_t )
);
InterfaceLayer::getInstance()->ParseProtoBufTable( protoBuf );
}
Send();
usleep( 1 );
}
void TransportLayer::Send()
{
zmq::message_t reply( 1 );
memcpy( reply.data(), "#", 1 );
socket.send( reply );
}
This is the code that I had written, this was initially designed to listen to only one client, now I have to extend it to listen to multiple clients.
I tried using zmq::multipart_t but this requires C11 support but the QNX-version we are using does not support C11.
I tried implementing the proposed solution.
I created 2 publishers connecting to same static location.
Observation :
I )
Execution Order :
1. Started Subscriber2. Started Publisher1 ( it published only one data value )
Subscriber missed to receive this data.
II )modified Publisher1 to send the same data in a while loop
Execution Order :
1. Started Subscriber2. Started Publisher13. Started Publsiher2.
Now I see that the Subscriber is receiving the data from both publishers.
This gives me an indication that there is a possibility for data loss.
How do I ensure there is absolutely no data loss?
Here is my source code :
Publisher 2 :
dummyFrontEnd::dummyFrontEnd():context(1),socket(context,ZMQ_PUB) {
}
void dummyFrontEnd::Init()
{
socket.connect("tcp://127.0.0.1:5555");
cout << "Connecting .... " << endl;
}
void dummyFrontEnd::SendData() {
while ( std::getline(file, line_str) ) {
std::stringstream ss(line_str);
std::string direction;
double tdiff;
int i, _1939, pgn, priority, source, length, data[8];
char J, p, _0, dash, d;
ss >> tdiff >> i >> J >> _1939 >> pgn >> p >> priority >> _0 >> source
>> dash >> direction >> d >> length >> data[0] >> data[1] >> data[2]
>> data[3] >> data[4] >> data[5] >> data[6] >> data[7];
timestamp += tdiff;
while ( gcl_get_time_ms() - start_time <
uint64_t(timestamp * 1000.0) - first_time ) { usleep(1); }
if (arguments.verbose) {
std::cout << timestamp << " " << i << " " << J << " " << _1939 << " "
<< pgn << " " << p << " " << priority << " " << _0 << " " << source
<< " " << dash << " " << direction << " " << d << " " << length
<< " " << data[0] << " " << data[1] << " " << data[2] << " "
<< data[3] << " " << data[4] << " " << data[5] << " " << data[6]
<< " " << data[7] << std::endl;
}
uint64_t timestamp_ms = (uint64_t)(timestamp * 1000.0);
protoTable.add_columnvalues(uint64ToString(timestamp_ms)); /* timestamp */
protoTable.add_columnvalues(intToString(pgn)); /* PGN */
protoTable.add_columnvalues(intToString(priority)); /* Priority */
protoTable.add_columnvalues(intToString(source)); /* Source */
protoTable.add_columnvalues(direction); /* Direction */
protoTable.add_columnvalues(intToString(length)); /* Length */
protoTable.add_columnvalues(intToString(data[0])); /* data1 */
protoTable.add_columnvalues(intToString(data[1])); /* data2 */
protoTable.add_columnvalues(intToString(data[2])); /* data3 */
protoTable.add_columnvalues(intToString(data[3])); /* data4 */
protoTable.add_columnvalues(intToString(data[4])); /* data5 */
protoTable.add_columnvalues(intToString(data[5])); /* data6 */
protoTable.add_columnvalues(intToString(data[6])); /* data7 */
protoTable.add_columnvalues(intToString(data[7])); /* data8 */
zmq::message_t create_values(protoTable.ByteSizeLong()+sizeof(uint16_t));
*((uint16_t*)create_values.data()) = TABLEMSG_ID; // ID
protoTable.SerializeToArray(create_values.data()+sizeof(uint16_t), protoTable.ByteSizeLong());
socket.send(create_values);
protoTable.clear_columnvalues();
usleep(1);
}
}
Publisher 1 :
dummyFrontEnd::dummyFrontEnd():context(1),socket(context,ZMQ_PUB) {
}
void dummyFrontEnd::Init()
{
socket.connect("tcp://127.0.0.1:5555");
cout << "Connecting .... " << endl;
}
void dummyFrontEnd::SendData()
{
cout << "In SendData" << endl;
while(1) {
canlogreq canLogObj = canlogreq::default_instance();
canLogObj.set_fromhours(11);
canLogObj.set_fromminutes(7);
canLogObj.set_fromseconds(2);
canLogObj.set_fromday(16);
canLogObj.set_frommonth(5);
canLogObj.set_fromyear(2020);
canLogObj.set_tohours(12);
canLogObj.set_tominutes(7);
canLogObj.set_toseconds(4);
canLogObj.set_today(17);
canLogObj.set_tomonth(5);
canLogObj.set_toyear(2020);
zmq::message_t logsnippetmsg(canLogObj.ByteSizeLong() + sizeof(uint16_t));
*((uint16_t*)logsnippetmsg.data()) = 20;
canLogObj.SerializeToArray(logsnippetmsg.data()+sizeof(uint16_t), canLogObj.ByteSizeLong());
socket.send(logsnippetmsg);
usleep(1);
canLogObj.clear_fromhours();
canLogObj.clear_fromminutes();
canLogObj.clear_fromseconds();
canLogObj.clear_fromday();
canLogObj.clear_frommonth();
canLogObj.clear_fromyear();
canLogObj.clear_tohours();
canLogObj.clear_tominutes();
canLogObj.clear_toseconds();
canLogObj.clear_today();
canLogObj.clear_tomonth();
canLogObj.clear_toyear();
}
}
Subscriber :
TransportLayer::TransportLayer():context(1),socket(context,ZMQ_SUB){ }
void TransportLayer::Init()
{
socket.bind("tcp://*:5555");
socket.setsockopt(ZMQ_SUBSCRIBE, "", 0);
}
void TransportLayer::Receive()
{
cout << "TransportLayer::Receive " << " I am in server " << endl;
static int count = 1;
// Producer thread.
while ( true ){
zmq::message_t request;
string protoBuf;
socket.recv(&request);
uint16_t id = *((uint16_t*)request.data());
cout << "TransportLayer : " << "request.data: " << request.data() << endl;
cout << "TransportLayer : count " << count << endl; count = count + 1;
cout << "TransportLayer : request.data.size " << request.size() << endl;
protoBuf = std::string(static_cast<char*>(request.data() + sizeof(uint16_t)), request.size() - sizeof(uint16_t));
cout << "ProtoBuf : " << protoBuf << endl;
InterfaceLayer *interfaceLayObj = InterfaceLayer::getInstance();
switch(id) {
case TABLEMSG_ID: cout << "Canlyser" << endl;
interfaceLayObj->ParseProtoBufTable(protoBuf);
break;
case LOGSNIPPET_ID: cout << "LogSnip" << endl;
interfaceLayObj->ParseProtoBufLogSnippet(protoBuf);
interfaceLayObj->logsnippetSignal(); // publish the signal
break;
default: break;
}
usleep(1);
}
}
Q : "how to use multiple Publishers and a single Client, using C < C11?"
So, the QNX-version was not explicitly stated, so let's work in general.
As noted in ZeroMQ Principles in less than Five Seconds, the single Client ( being of a SUB-Archetype ) may zmq_connect( ? ), however at a cost of managing some, unknown for me, way how all the other, current plus any future PUB-s were let to zmq_bind(), after which to let somehow let the SUB learn where to zmq_connect( ? ), so that to get some news from the newly bound PUB-peer.
So it would be a way smarter to make the single SUB-agent to perform a zmq_bind() and let any of the current or future PUB-s perform zmq_connect() as they come, directed to the single, static, known SUB's location ( this does not say, they cannot use any of the available transport-classes - one inproc://, another one tcp://, some ipc://, if QNX permits & system architecture requires to do so ( and, obviously, supposing the SUB-agent has exposed a properly configured AccessNode for receiving such connections ).
Next, your SUB-Client has to configure its subscription filtering topic-list: be it an order to "Do Receive EVERYTHING!" :
...
retCode = zmq_setsockopt( <aSubSocketINSTANCE>, ZMQ_SUBSCRIBE, "", 0 );
assert( retCode == 0 && "FAILED: at ZMQ_SUBSCRIBE order " );
...
Given this works, your next duty is to make the setup robust enough ( an explicit ZMQ_LINGER setting to 0, access-policies, security, scaled-resources, L2/L3-network protective measures, etc ).
And you are done to harness the ZeroMQ just fit right to your QNX-system design needs.