serialize complex C++ structures between client and server - c++

I am writing C++ ZeroMQ Client and server programs for the same platform. I need to trigger some functions with arguments on the server. The arguments are complex structures. I have just started to try this out. I am trying to fill a structure and fill it to a char* buffer to see if the bytes are filled out in sequence as per the structure.
But when i try to print the buffer, it prints garbage. Please advice what might be wrong. And is this the elegant way to do this? I cannot use gRPC or Protobuffs as the message contains complex structures.
struct employee {
uint8_t byt;
int arr[10] = {0};
int number;
uint32_t acct;
};
int main ()
{
// Prepare our context and socket
zmq::context_t context (1);
zmq::socket_t socket (context, ZMQ_PAIR);
struct employee *e = new employee;
e->byt = 0xff;
e->arr[0] = 15;
e->number = 25555;
e->acct = 45;
std::cout << "Connecting to hello world server…" << std::endl;
socket.connect ("tcp://localhost:5555");
char *temp = (char*)malloc(sizeof(employee));
memcpy(temp,e,sizeof(employee));
zmq::message_t request(sizeof(employee));
char *temp1 = temp;
for (int i = 0;i<sizeof(employee);i++) {
std::cout<<temp1[i]<<std::endl;
}
memcpy ((void *)request.data(),(void*)temp, sizeof(employee));
socket.send (request);
// Get the reply.
zmq::message_t reply;
socket.recv (&reply);
return 0;
}

Two points I would like to share here.
the buffer (temp) contains binary representation of your data structure. If you want to check if the content is meaningful, you can cast the pointer type back to its original pointer.ie:
struct employee * employeePtr = static_cast< struct employee *>(temp);
cout << employeePtr ->number;
...
The way you de-serialize is okay when the object you are trying to
serialize occupies continuous memory. When it is not the case, you will have to handle them some other way (using stream for example). Examples of such cases include:
when you have pointer, shared_ptr etc. to some allocated memory
container classes

Related

How to pass big amount of data(unknown size, minimum 10GB) using gRPC

From a source I am getting stream data which size will not be known before the final processing, but the minimum is 10 GB. I have to send this large amount of data using gRPC.
Need to mention here, this large amount data will be passed through the gRPC while the processing of the streaming is done. In this step, I have thought to store all the value in a vector.
Regarding sending large amount of data I have tried to get idea and found:
This where it is mentioned not to pass large data using gRPC. Here, mentioned to use any other message protocol where I have limitation to use something else rather than gRPC(at least till today).
From this post I have tried to know how chunk message can be sent but I am not sure is it related to my problem or not.
First post where I have found a blog to stream data using go language.
This one the presentation using python language of this post. But it is also incomplete.
gRPC example could be a good start bt cannot decode due to lack of C++ knowledge
From there, a huge Update I have done in the question. But the main theme of the question is not changed
What I have done so far and some points about my project. The github repo is available here.
A Unary rpc is present in the project
I know that my new Bi directional rpc will take some time. I want that the Unary rpc will not wait for the completion of the Bi directional rpc. Right now I am thinking in a synchronous way where Unary rpc is waiting to pass it's status for the streaming one completion.
I am avoiding the unnecessary lines in C++ code. But giving whole proto files
big_data.proto
syntax = "proto3";
package demo_grpc;
message Large_Data {
repeated int32 large_data_collection = 1 [packed=true];
int32 data_chunk_number = 2;
}
addressbook.proto
syntax = "proto3";
package demo_grpc;
import "myproto/big_data.proto";
message S_Response {
string name = 1;
string street = 2;
string zip = 3;
string city = 4;
string country = 5;
int32 double_init_val = 6;
}
message C_Request {
uint32 choose_area = 1;
string name = 2;
int32 init_val = 3;
}
service AddressBook {
rpc GetAddress(C_Request) returns (S_Response) {}
rpc Stream_Chunk_Service(stream Large_Data) returns (stream Large_Data) {}
}
client.cpp
#include <big_data.pb.h>
#include <addressbook.grpc.pb.h>
#include <grpcpp/grpcpp.h>
#include <grpcpp/create_channel.h>
#include <iostream>
#include <numeric>
using namespace std;
// This function prompts the user to set value for the required area
void Client_Request(demo_grpc::C_Request &request_)
{
// do processing for unary rpc. Intentionally avoided here
}
// According to Client Request this function display the value of protobuf message
void Server_Response(demo_grpc::C_Request &request_, const demo_grpc::S_Response &response_)
{
// do processing for unary rpc. Intentionally avoided here
}
// following function make large vector and then chunk to send via stream from client to server
void Stream_Data_Chunk_Request(demo_grpc::Large_Data &request_,
demo_grpc::Large_Data &response_,
uint64_t preferred_chunk_size_in_kibyte)
{
// A dummy vector which in real case will be the large data set's container
std::vector<int32_t> large_vector;
// irerate it now for 1024*10 times
for(int64_t i = 0; i < 1024 * 10; i++)
{
large_vector.push_back(1);
}
uint64_t preferred_chunk_size_in_kibyte_holds_integer_num = 0; // 1 chunk how many intger will contain that num will come here
// total chunk number will be updated here
uint32_t total_chunk = total_chunk_counter(large_vector.size(), preferred_chunk_size_in_kibyte, preferred_chunk_size_in_kibyte_holds_integer_num);
// A temp counter to trace the index of the large_vector
int32_t temp_count = 0;
// loop will start if the total num of chunk is greater than 0. After each iteration total_chunk will be decremented
while(total_chunk > 0)
{
for (int64_t i = temp_count * preferred_chunk_size_in_kibyte_holds_integer_num; i < preferred_chunk_size_in_kibyte_holds_integer_num + temp_count * preferred_chunk_size_in_kibyte_holds_integer_num; i++)
{
// the repeated field large_data_collection is taking value from the large_vector
request_.add_large_data_collection(large_vector[i]);
}
temp_count++;
total_chunk--;
std::string ip_address = "localhost:50051";
auto channel = grpc::CreateChannel(ip_address, grpc::InsecureChannelCredentials());
std::unique_ptr<demo_grpc::AddressBook::Stub> stub = demo_grpc::AddressBook::NewStub(channel);
grpc::ClientContext context;
std::shared_ptr<::grpc::ClientReaderWriter< ::demo_grpc::Large_Data, ::demo_grpc::Large_Data> > stream(stub->Stream_Chunk_Service(&context));
// While the size of each chunk is eached then this repeated field is cleared. I am not sure before this
// value can be transfered to server or not. But my assumption is saying that it should be done
request_.clear_large_data_collection();
}
}
int main(int argc, char* argv[])
{
std::string client_address = "localhost:50051";
std::cout << "Address of client: " << client_address << std::endl;
// The following part for the Unary RPC
demo_grpc::C_Request query;
demo_grpc::S_Response result;
Client_Request(query);
// This part for the streaming chunk data (Bi directional Stream RPC)
demo_grpc::Large_Data stream_chunk_request_;
demo_grpc::Large_Data stream_chunk_response_;
uint64_t preferred_chunk_size_in_kibyte = 64;
Stream_Data_Chunk_Request(stream_chunk_request_, stream_chunk_response_, preferred_chunk_size_in_kibyte);
// Call
auto channel = grpc::CreateChannel(client_address, grpc::InsecureChannelCredentials());
std::unique_ptr<demo_grpc::AddressBook::Stub> stub = demo_grpc::AddressBook::NewStub(channel);
grpc::ClientContext context;
grpc::Status status = stub->GetAddress(&context, query, &result);
// the following status is for unary rpc as far I have understood the structure
if (status.ok())
{
Server_Response(query, result);
}
else
{
std::cout << status.error_message() << std::endl;
}
return 0;
}
heper function total_chunk_counter
#include <cmath>
uint32_t total_chunk_counter(uint64_t num_of_container_content,
uint64_t preferred_chunk_size_in_kibyte,
uint64_t &preferred_chunk_size_in_kibyte_holds_integer_num)
{
uint64_t cotainer_size_in_kibyte = (32ULL * num_of_container_content) / 1024;
preferred_chunk_size_in_kibyte_holds_integer_num = (num_of_container_content * preferred_chunk_size_in_kibyte) / cotainer_size_in_kibyte;
float total_chunk = static_cast<float>(num_of_container_content) / preferred_chunk_size_in_kibyte_holds_integer_num;
return std::ceil(total_chunk);
}
server.cpp which is totally incomplete
#include <myproto/big_data.pb.h>
#include <myproto/addressbook.grpc.pb.h>
#include <grpcpp/grpcpp.h>
#include <grpcpp/server_builder.h>
#include <iostream>
class AddressBookService final : public demo_grpc::AddressBook::Service {
public:
virtual ::grpc::Status GetAddress(::grpc::ServerContext* context, const ::demo_grpc::C_Request* request, ::demo_grpc::S_Response* response)
{
switch (request->choose_area())
{
// do processing for unary rpc. Intentionally avoided here
std::cout << "Information of " << request->choose_area() << " is sent to Client" << std::endl;
return grpc::Status::OK;
}
// Bi-directional streaming chunk data
virtual ::grpc::Status Stream_Chunk_Service(::grpc::ServerContext* context, ::grpc::ServerReaderWriter< ::demo_grpc::Large_Data, ::demo_grpc::Large_Data>* stream)
{
// stream->Large_Data;
return grpc::Status::OK;
}
};
void RunServer()
{
std::cout << "grpc Version: " << grpc::Version() << std::endl;
std::string server_address = "localhost:50051";
std::cout << "Address of server: " << server_address << std::endl;
grpc::ServerBuilder builder;
builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
AddressBookService my_service;
builder.RegisterService(&my_service);
std::unique_ptr<grpc::Server> server(builder.BuildAndStart());
server->Wait();
}
int main(int argc, char* argv[])
{
RunServer();
return 0;
}
In summary my desire
I need to pass the content of large_vector with the repeated field large_data_collection of message Large_Data. I should chunk the size of the large_vector and populate the repeated field large_data_collection with that chunk size
In server side all chunk will be concatenate by keeping the exact order of the large_vector. Some processing will be done on them (eg: double the value of each index). Then again whole data will be sent to the client as a chunk stream
Would be great if the present unary rpc don't wait for the completion of the bi-directional rpc
Solution with example would be really helpful. Advance thanks. The github repo is available here.

Infiniband RDMA completion queues with multithreading

I'm learning how to use RDMA via Inifniband and one problem I'm having is using a connection with more than 1 thread because I cant figure out how to create another completion queue so the work completions get mixed up between the threads and it craps out, how do I create a queue for each thread using the connection?
Take this vomit for example:
void worker(struct ibv_cq* cq){
while(conn->peer_mr.empty()) Sleep(1);
struct ibv_wc wc{};
struct ibv_send_wr wr{};
memset(&wr, 0, sizeof wr);
struct ibv_sge sge{};
sge.addr = reinterpret_cast<unsigned long long>(conn->rdma_memory_region);
sge.length = RDMA_BUFFER_SIZE;
sge.lkey = conn->rdma_mr->lkey;
wr.wr_id = reinterpret_cast<unsigned long long>(conn);
wr.opcode = IBV_WR_RDMA_READ;
wr.sg_list = &sge;
wr.num_sge = 1;
wr.send_flags = IBV_SEND_SIGNALED;
struct ibv_send_wr* bad_wr = nullptr;
while(true){
if(queue >= maxqueue) continue;
for(auto i = 0ULL; i < conn->peer_mr.size(); ++i){
wr.wr.rdma.remote_addr = reinterpret_cast<unsigned long long>(conn->peer_mr[i]->mr.addr) + conn->peer_mr[i]->offset;
wr.wr.rdma.rkey = conn->peer_mr[i]->mr.rkey;
const auto err = ibv_post_send(conn->qp, &wr, &bad_wr);
if(err){
std::cout << "ibv_post_send " << err << "\n" << "Errno: " << std::strerror(errno) << "\n";
exit(err);
}
++queue;
conn->peer_mr[i]->offset += RDMA_BUFFER_SIZE;
if(conn->peer_mr[i]->offset >= conn->peer_mr[i]->mr.length) conn->peer_mr[i]->offset = 0;
}
int ne;
do{
ne = ibv_poll_cq(cq, 1, &wc);
} while(!ne);
--queue;
++number;
}
}
If I had more than one of them they would all be receiving each others work completions, I want them to receive only their own and not those of other threads.
The completion queues are created somewhere outside of this code (you are passing in an ibv_cq *). If you'd like to figure out how to create multiple ones, that's the area to focus on.
However, the "crapping out" is not (just) happening because completions are mixed up between threads: the ibv_poll_cq and ibv_post_send functions are thread safe. Instead, the likely problem is that your code isn't thread-safe: there are shared data structures that are accessed without locks (conn->peer_mr). You would have the same issues even without RDMA.
The first step is to figure out how to split up the work into pieces. Think about the pieces that each thread will need to make it independent from the others. It'll likely be a single peer_mr, a separate ibv_cq *, and a specific chunk of your rdma_mr. Then code that :)

toLocal8bit send over TCP

Im creating TCP Server/Client application in QT Creator framework. I want to get some data from UI input field and send it over TCP.
Im doing something like this in client application:
void MainWindow::on_btn_login_clicked()
{
QByteArray text = (ui->login_input->text()).toLocal8Bit();
char* out = text.data();
connection->ConnectAndSendData(out);
}
and in the ConnectAndSendData function:
void TcpConnect::ConnectAndSendData(const char* data)
{
socket = new QTcpSocket(this);
int port = 1234;
socket->connectToHost("localhost", port);
if(socket->waitForConnected(3000))
{
qDebug() << "connected to s. localhost at port " << port;
socket->flush();
socket->write(data, sizeof(data));
qDebug() << data << "\n";
socket->waitForReadyRead();
char* serverresponse;
socket->read(serverresponse, 128);
if(serverresponse == MESSAGE_LOGINRQ)
socket->write(data);
socket->flush();
socket->close();
}
else
{
/**/
}
}
and the data in line socket->write(data, sizeof(data)); is properly send to server, but when server echoes it, it looks like "something/x00/x00/x00/x00" or somethinglike that. Also when i to do something like this:
#define MESSAGE_WANTLOGIN "wanlogin"
socket->write(MESSAGE_WANTLOGIN, sizeof(MESSAGE_WANTLOGIN));
message is messed up with those null signs.
on the server side receiving data look as simple as:
void Thread::readyRead()
{
socket->flush();
QByteArray data = socket->readAll();
qDebug() << "data received: " << data;
if(data == MESSAGE_WANTLOGIN)
{
socket->write(MESSAGE_LOGINRQ);
} else
{
qDebug() << "error not messageloginrq";
}
}
and like u can assume, though i send "wanlogin" message, server receiving something like "wanlogin/x00/x00" and this if obviously returns false.
this trash is applied on the end of data, and this impossible to check what message was send. The other thing is that maximum size of send data is 8 chars, but also to data of this length trash is applied so it looks like "wanlogin/x00/x00"; however, when i type more chars, for example 10, the send data is just cut to 8 signs, with no /x00s.
So my question is how to clear data from those /x00s and how to send more than 1 byte of information(i need it e.g. to send login and password of user). Sorry if there's some stupid mistake, its my first client/server application which also using multithreading for each client.
sizeof(data) is 4 or 8 depending if you are on a 32-bit or 64-bit machine. It is not the size of your data, but the size (in byte) of a pointer.
So what happens is that your actual wanlogin is in fact a 6 character string, and you end up sending 2 more bytes. In this case you are lucky: the char array returned by data() is null-terminated, so you have one extra 0 that you can access, but accessing the second 0 is undefined behavior i.e anything can happen.
The solution is to use strlen() instead of sizeof. Or, better, to directly call write() with a QByteArray by changing ConnectAndSendData(const char* data) to ConnectAndSendData(const QByteArray &data).
void MainWindow::on_btn_login_clicked()
{
const QByteArray text = (ui->login_input->text()).toLocal8Bit();
connection->ConnectAndSendData(text);
}
void TcpConnect::ConnectAndSendData(const QByteArray & data)
{
socket = new QTcpSocket(this);
quint16 port = 1234;
socket->connectToHost("localhost", port);
if(socket->waitForConnected(3000))
{
qDebug() << "connected to s. localhost at port " << port;
socket->write(data);
...
}
...
}

How do I make an in-place modification on an array using grpc and google protocol buffers?

I'm having a problem with a const request with the google protocol buffers using grpc. Here is my problem:
I would like to make an in-place modification of an array's value. For that I wrote this simple example where I try to pass an array and sum all of it's content. Here's my code:
adder.proto:
syntax = "proto3";
option java_package = "io.grpc.examples";
package adder;
// The greeter service definition.
service Adder {
// Sends a greeting
rpc Add (AdderRequest) returns (AdderReply) {}
}
// The request message containing the user's name.
message AdderRequest {
repeated int32 values = 1;
}
// The response message containing the greetings
message AdderReply {
int32 sum = 1;
}
server.cc:
//
// Created by Eric Reis on 7/6/16.
//
#include <iostream>
#include <grpc++/grpc++.h>
#include "adder.grpc.pb.h"
class AdderImpl final : public adder::Adder::Service
{
public:
grpc::Status Add(grpc::ServerContext* context, const adder::AdderRequest* request,
adder::AdderReply* reply) override
{
int sum = 0;
for(int i = 0, sz = request->values_size(); i < sz; i++)
{
request->set_values(i, 10); // -> this gives an error caused by the const declaration of the request variable
// error: "Non-const function 'set_values' is called on the const object"
sum += request->values(i); // -> this works fine
}
reply->set_sum(sum);
return grpc::Status::OK;
}
};
void RunServer()
{
std::string server_address("0.0.0.0:50051");
AdderImpl service;
grpc::ServerBuilder builder;
// Listen on the given address without any authentication mechanism.
builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
// Register "service" as the instance through which we'll communicate with
// clients. In this case it corresponds to an *synchronous* service.
builder.RegisterService(&service);
// Finally assemble the server.
std::unique_ptr<grpc::Server> server(builder.BuildAndStart());
std::cout << "Server listening on " << server_address << std::endl;
// Wait for the server to shutdown. Note that some other thread must be
// responsible for shutting down the server for this call to ever return.
server->Wait();
}
int main(int argc, char** argv)
{
RunServer();
return 0;
}
client.cc:
//
// Created by Eric Reis on 7/6/16.
//
#include <iostream>
#include <grpc++/grpc++.h>
#include "adder.grpc.pb.h"
class AdderClient
{
public:
AdderClient(std::shared_ptr<grpc::Channel> channel) : stub_(adder::Adder::NewStub(channel)) {}
int Add(int* values, int sz) {
// Data we are sending to the server.
adder::AdderRequest request;
for (int i = 0; i < sz; i++)
{
request.add_values(values[i]);
}
// Container for the data we expect from the server.
adder::AdderReply reply;
// Context for the client. It could be used to convey extra information to
// the server and/or tweak certain RPC behaviors.
grpc::ClientContext context;
// The actual RPC.
grpc::Status status = stub_->Add(&context, request, &reply);
// Act upon its status.
if (status.ok())
{
return reply.sum();
}
else {
std::cout << "RPC failed" << std::endl;
return -1;
}
}
private:
std::unique_ptr<adder::Adder::Stub> stub_;
};
int main(int argc, char** argv) {
// Instantiate the client. It requires a channel, out of which the actual RPCs
// are created. This channel models a connection to an endpoint (in this case,
// localhost at port 50051). We indicate that the channel isn't authenticated
// (use of InsecureChannelCredentials()).
AdderClient adder(grpc::CreateChannel("localhost:50051",
grpc::InsecureChannelCredentials()));
int values[] = {1,2};
int sum = adder.Add(values, 2);
std::cout << "Adder received: " << sum << std::endl;
return 0;
}
My error happens when i try to call the method set_values() on the request object that is defined as const. I understand why this error is occurring but I just can't figure out a way to overcome it without making a copy of the array.
I tried to remove the const definition but the RPC calls fails when I do that.
Since I'm new to this RPC world and even more on grpc and the google protocol buffers I'd like to call for your help. What is the best way to solve this problem?
Please see my answer here. The server receives a copy of the AdderRequest sent by the client. If you were to modify it, the client's original AdderRequest would not be modified. If by "in place" you mean the server modifies the client's original memory, no RPC technology can truly accomplish that, because the client and server run in separate address spaces (processes), even on different machines.
If you truly need the server to modify the client's memory:
Ensure the server and client run on the same machine.
Use OS-specific shared-memory APIs such as shm_open() and mmap() to map the same chunk of physical memory into the address spaces of both the client and the server.
Use RPC to transmit the identifier (name) of the shared memory (not the actual data in the memory) and to invoke the server's processing.
When both client and server have opened and mapped the memory, they both have pointers (likely with different values in the different address spaces) to the same physical memory, so the server will be able to read what the client writes there (with no copying or transmitting) and vice versa.

How to stub a socket in C?

I've written client code that's supposed to send some data through a socket and read back an answer from the remote server.
I would like to unit-test that code. The function's signature is something along the lines of:
double call_remote(double[] args, int fd);
where fd is the file descriptor of the socket to the remote server.
Now the call_remote function will, after sending the data, block on reading the answer from the server. How can I stub such a remote server for unit-testing the code?
Ideally I would like something like:
int main() {
int stub = /* initialize stub */
double expected = 42.0;
assert(expected == call_remote(/* args */, stub);
return 0;
}
double stub_behavior(double[] args) {
return 42.0;
}
I would like stub_behavior to be called and send the 42.0 value down the stubbed file descriptor.
Any easy way I can do that?
If this is a POSIX system, you can use fork() and socketpair():
#define N_DOUBLES_EXPECTED 10
double stub_behaviour(double []);
int initialize_stub(void)
{
int sock[2];
double data[N_DOUBLES_EXPECTED];
socketpair(AF_UNIX, SOCK_STREAM, 0, sock);
if (fork()) {
/* Parent process */
close(sock[0]);
return sock[1];
}
/* Child process */
close(sock[1]);
/* read N_DOUBLES_EXPECTED in */
read(sock[0], data, sizeof data);
/* execute stub */
data[0] = stub_behaviour(data);
/* write one double back */
write(sock[0], data, sizeof data[0]);
close(sock[0]);
_exit(0);
}
int main()
{
int stub = initialize_stub();
double expected = 42.0;
assert(expected == call_remote(/* args */, stub);
return 0;
}
double stub_behavior(double args[])
{
return 42.0;
}
...of course, you will probably want to add some error checking, and alter the logic that reads the request.
The file descriptor created by socketpair() is a normal socket, and thus socket calls like send() and recv() will work fine on it.
You could use anything which can be accessed with a file descriptor. A file or, if you want simulate blocking behaviour, a pipe.
Note: obviosly socket specific calls (setsockopt, fcntl, ioctl, ...) wouldn't work.
I encountered the same situation and I'll share my approach. I created network dumps of exactly what the client should send, and what the server response should be. I then did a byte-by-byte comparison of the client request to ensure it matched. If the request is valid, I read from the response file and send it back to the client.
I'm happy to provide more details (when I'm at a machine with access to this code)
Here is a C++ implementation (I know, the original question was for C, but it is easy to convert back to C if desired). It probably doesn't work for very large strings, as the socket will probably block if the string can't be buffered. But it works for small unit tests.
/// Class creates a simple socket for testing out functions that write to a socket.
/// Usage:
/// 1. Call GetSocket() to get a file description socket ID
/// 2. write to that socket FD
/// 3. Call ReadAll() read back all the data that was written to that socket.
/// The sockets are all closed by ReadAll(), so this is a one-use object.
///
/// \example
/// MockSocket ms;
/// int socket = ms.GetSocket();
/// send(socket,"foo bar",7);
/// ...
/// std::string s = ms.ReadAll();
/// EXPECT_EQ("foo bar",s);
class MockSocket
{
public:
~MockSocket()
{
}
int GetSocket()
{
socketpair(AF_UNIX, SOCK_STREAM, 0, sockets_);
return sockets_[0];
}
std::string ReadAll()
{
close(sockets_[0]);
std::string s;
char buffer[256];
while (true)
{
int n = read(sockets_[1], buffer, sizeof(buffer));
if (n > 0) s.append(buffer,n);
if (n <= 0) break;
}
close(sockets_[1]);
return s;
}
private:
int sockets_[2];
};