My team is designing a scalable solution with micro-services architecture and planning to use gRPC as the transport communication between layers. And we've decided to use async grpc model. The design that example(greeter_async_server.cc) provides doesn't seem viable if I scale the number of RPC methods, because then I'll have to create a new class for every RPC method, and create their objects in HandleRpcs() like this.
Pastebin (Short example code).
void HandleRpcs() {
new CallDataForRPC1(&service_, cq_.get());
new CallDataForRPC2(&service_, cq_.get());
new CallDataForRPC3(&service, cq_.get());
// so on...
}
It'll be hard-coded, all the flexibility will be lost.
I've around 300-400RPC methods to implement and having 300-400 classes will be cumbersome and inefficient when I'll have to handle more than 100K RPC requests/sec and this solution is a very bad design. I can't bear the overhead of creation of objects this way on every single request. Can somebody kindly provide me a workaround for this. Can async grpc c++ not be simple like its sync companion?
Edit: In favour of making the situation more clear, and for those who might be struggling to grasp the flow of this async example, I'm writing what I've understood so far, please make me correct if wrong somewhere.
In async grpc, every time we have to bind a unique-tag with the completion-queue so that when we poll, the server can give it back to us when the particular RPC will be hit by the client, and we infer from the returned unique-tag about the type of the call.
service_->RequestRPC2(&ctx_, &request_, &responder_, cq_, cq_,this); Here we're using the address of the current object as the unique-tag. This is like registering for our RPC call on the completion queue. Then we poll down in HandleRPCs() to see if the client hits the RPC, if so then cq_->Next(&tag, &OK) will fill the tag. The polling code snippet:
while (true) {
GPR_ASSERT(cq_->Next(&tag, &ok));
GPR_ASSERT(ok);
static_cast<CallData*>(tag)->Proceed();
}
Since, the unique-tag that we registered into the queue was the address of the CallData object so we're able to call Proceed(). This was fine for one RPC with its logic inside Proceed(). But with more RPCs each time we'll have all of them inside the CallData, then on polling, we'll be calling the only one Proceed() which will contain logic to (say) RPC1(postgres calls), RPC2(mongodb calls), .. so on. This is like writing all my program inside one function. So, to avoid this, I used a GenericCallData class with the virtual void Proceed() and made derived classes out of it, one class per RPC with their own logic inside their own Proceed(). This is a working solution but I want to avoid writing many classes.
Another solution I tried was keeping all RPC-function-logics out of the proceed() and into their own functions and maintaining a global std::map<long, std::function</*some params*/>> . So whenever I register an RPC with unique-tag onto the queue, I store its corresponding logic function (which I'll surely hard code into the statement and bind all the parameters required), then the unique-tag as key. On polling, when I get the &tag I do a lookup in the map for this key and call the corresponding saved function. Now, there's one more hurdle, I'll have to do this inside the function logic:
// pseudo code
void function(reply, responder, context, service)
{
// register this RPC with another unique tag so to serve new incoming request of the same type on the completion queue
service_->RequestRPC1(/*params*/, new_unique_id);
// now again save this new_unique_id and current function into the map, so when tag will be returned we can do lookup
map.emplace(new_unique_id, function);
// now you're free to do your logic
// do your logic
}
You see this, code has spread into another module now, and it's per RPC based.
Hope it clears the situation.
I thought if somebody could have implemented this type of server in a more easy way.
This post is pretty old by now but I have not seen any answer or example regarding this so I will show how I solved it to any other readers. I have around 30 RPC calls and was looking for a way of reducing the footprint when adding and removing RPC calls. It took me some iterations to figure out a good way to solve it.
So my interface for getting RPC requests from my (g)RPC library is a callback interface that the recepiant need to implement. The interface looks like this:
class IRpcRequestHandler
{
public:
virtual ~IRpcRequestHandler() = default;
virtual void onZigbeeOpenNetworkRequest(const smarthome::ZigbeeOpenNetworkRequest& req,
smarthome::Response& res) = 0;
virtual void onZigbeeTouchlinkDeviceRequest(const smarthome::ZigbeeTouchlinkDeviceRequest& req,
smarthome::Response& res) = 0;
...
};
And some code for setting up/register each RPC method after the gRPC server is started:
void ready()
{
SETUP_SMARTHOME_CALL("ZigbeeOpenNetwork", // Alias that is used for debug messages
smarthome::Command::AsyncService::RequestZigbeeOpenNetwork, // Generated gRPC service method for async.
smarthome::ZigbeeOpenNetworkRequest, // Generated gRPC service request message
smarthome::Response, // Generated gRPC service response message
IRpcRequestHandler::onZigbeeOpenNetworkRequest); // The callback method to call when request has arrived.
SETUP_SMARTHOME_CALL("ZigbeeTouchlinkDevice",
smarthome::Command::AsyncService::RequestZigbeeTouchlinkDevice,
smarthome::ZigbeeTouchlinkDeviceRequest,
smarthome::Response,
IRpcRequestHandler::onZigbeeTouchlinkDeviceRequest);
...
}
This is all that you need to care about when adding and removing RPC methods.
The SETUP_SMARTHOME_CALL is a home-cooked macro which looks like this:
#define SETUP_SMARTHOME_CALL(ALIAS, SERVICE, REQ, RES, CALLBACK_FUNC) \
new ServerCallData<REQ, RES>( \
ALIAS, \
std::bind(&SERVICE, \
&mCommandService, \
std::placeholders::_1, \
std::placeholders::_2, \
std::placeholders::_3, \
std::placeholders::_4, \
std::placeholders::_5, \
std::placeholders::_6), \
mCompletionQueue.get(), \
std::bind(&CALLBACK_FUNC, requestHandler, std::placeholders::_1, std::placeholders::_2))
I think the ServerCallData class looks like the one from gRPCs examples with a few modifications. ServerCallData is derived from a non-templete class with an abstract function void proceed(bool ok) for the CompletionQueue::Next() handling. When ServerCallData is created, it will call the SERVICE method to register itself on the CompletionQueue and on every first proceed(ok) call, it will clone itself which will register another instance. I can post some sample code for that as well if someone is interested.
EDIT: Added some more sample code below.
GrpcServer
class GrpcServer
{
public:
explicit GrpcServer(std::vector<grpc::Service*> services);
virtual ~GrpcServer();
void run(const std::string& sslKey,
const std::string& sslCert,
const std::string& password,
const std::string& listenAddr,
uint32_t port,
uint32_t threads = 1);
private:
virtual void ready(); // Called after gRPC server is created and before polling CQ.
void handleRpcs(); // Function that polls from CQ, can be run by multiple threads. Casts object to CallData and calls CallData::proceed().
std::unique_ptr<ServerCompletionQueue> mCompletionQueue;
std::unique_ptr<Server> mServer;
std::vector<grpc::Service*> mServices;
std::list<std::shared_ptr<std::thread>> mThreads;
...
}
And the main part of the CallData object:
template <typename TREQUEST, typename TREPLY>
class ServerCallData : public ServerCallMethod
{
public:
explicit ServerCallData(const std::string& methodName,
std::function<void(ServerContext*,
TREQUEST*,
::grpc::ServerAsyncResponseWriter<TREPLY>*,
::grpc::CompletionQueue*,
::grpc::ServerCompletionQueue*,
void*)> serviceFunc,
grpc::ServerCompletionQueue* completionQueue,
std::function<void(const TREQUEST&, TREPLY&)> callback,
bool first = false)
: ServerCallMethod(methodName),
mResponder(&mContext),
serviceFunc(serviceFunc),
completionQueue(completionQueue),
callback(callback)
{
requestNewCall();
}
void proceed(bool ok) override
{
if (!ok)
{
delete this;
return;
}
if (callStatus() == ServerCallMethod::PROCESS)
{
callStatus() = ServerCallMethod::FINISH;
new ServerCallData<TREQUEST, TREPLY>(callMethodName(), serviceFunc, completionQueue, callback);
try
{
callback(mRequest, mReply);
}
catch (const std::exception& e)
{
mResponder.Finish(mReply, Status::CANCELLED, this);
return;
}
mResponder.Finish(mReply, Status::OK, this);
}
else
{
delete this;
}
}
private:
void requestNewCall()
{
serviceFunc(
&mContext, &mRequest, &mResponder, completionQueue, completionQueue, this);
}
ServerContext mContext;
TREQUEST mRequest;
TREPLY mReply;
ServerAsyncResponseWriter<TREPLY> mResponder;
std::function<void(ServerContext*,
TREQUEST*,
::grpc::ServerAsyncResponseWriter<TREPLY>*,
::grpc::CompletionQueue*,
::grpc::ServerCompletionQueue*,
void*)>
serviceFunc;
std::function<void(const TREQUEST&, TREPLY&)> callback;
grpc::ServerCompletionQueue* completionQueue;
};
Although the thread is old I wanted to share a solution I am currently implementing. It mainly consists templated classes inheriting CallData to be scalable. This way, each new rpc will only require specializing the templates of the required CallData methods.
Calldata header:
class CallData {
protected:
enum Status { CREATE, PROCESS, FINISH };
Status status;
virtual void treat_create() = 0;
virtual void treat_process() = 0;
public:
void Proceed();
};
CallData Proceed implementation:
void CallData::Proceed() {
switch (status) {
case CREATE:
status = PROCESS;
treat_create();
break;
case PROCESS:
status = FINISH;
treat_process();
break;
case FINISH:
delete this;
}
}
Inheriting from CallData header (simplified):
template <typename Request, typename Reply>
class CallDataTemplated : CallData {
static_assert(std::is_base_of<google::protobuf::Message, Request>::value,
"Request and reply must be protobuf messages");
static_assert(std::is_base_of<google::protobuf::Message, Reply>::value,
"Request and reply must be protobuf messages");
private:
Service,Cq,Context,ResponseWriter,...
Request request;
Reply reply;
protected:
void treat_create() override;
void treat_process() override;
public:
...
};
Then, for specific rpc's in theory you should be able to do things like:
template<>
void CallDataTemplated<HelloRequest, HelloReply>::treat_process() {
...
}
It's a lot of templated methods but preferable to creating a class per rpc from my point of view.
In my system, I have a juggle a bunch of TCP clients and I am bit confused on how to design it [most of my experience is in C, hence the insecurity]. I am using boost ASIO for managing connection. These are the components I have
A TCPStream class : thin wrapper over boost asio
an IPC protocol, which implement a protocol over TCP:
basically Each message starts with a type and length field
so we can read the individual messages out of the stream.
Connection classes which handle the messages
Observer class which monitors connections
I am writing pseudo C++ code to be concise. I think you will get the idea
class TCPStream {
boost::asio::socket socket_;
public:
template <typename F>
void connect (F f)
{
socket_.connect(f);
}
template <typename F>
void read (F f)
{
socket_.read(f);
}
};
class IpcProtocol : public TCPStream {
public:
template <typename F
void read (F f)
{
TCPStream::read(
[f] (buffer, err) {
while (msg = read_indvidual_message(buffer)) {
// **** this is a violation of how this pattern is
// supposed to work. Ideally there should a callback
// for individual message. Here the same callback
// is called for N no. of messages. But in our case
// its the same callback everytime so this should be
// fine - just avoids some function calls.
f(msg);
};
};
)
}
};
Lets say I have a bunch of TCP connections and there are a handler class
for each of the connection. Lets name it Connection1, Connection2 ...
class Connection {
virtual int type() = 0;
};
class Connection1 : public Connection {
shared_ptr<IpcProtocol> ipc_;
int type ()
{
return 1;
}
void start ()
{
ipc_.connect([self = shared_from_this()](){ self->connected(); });
ipc_.read(
[self = shared_from_this()](msg, err) {
if (!err)
self->process(msg);
} else {
self->error();
}
});
}
void connected ()
{
observer.notify_connected(shared_from_this());
}
void error ()
{
observer.notify_error(shared_from_this());
}
};
This pattern repeats for all connections one way or other.
messages are processed by the connection class itself. But it will let know of
other events [connect, error] to an observer. The reason -
Restart the connection, everytime it disconnect
Bunch of guys needs to know if the connection is established so that they can
send initial request/confguration to server.
There are things that needs be done based on connection status of muliple connections
Eg: if connection1 and connection2 are established, then start connection3 etc.
I added a middle Observer class is there so that the observers do have to directly connect to the connection everytime it is restarted. Each time connection breaks, the connection class is deleted and new one is created.
class Listeners {
public:
virtual void notify_error(shared_ptr<Connection>) = 0;
virtual void notify_connect(shared_ptr<Connection>) = 0;
virtual void interested(int type) = 0;
};
class Observer {
std::vector<Listeners *> listeners_;
public:
void notify_connect(shared_ptr<Connection> connection)
{
for (listener : listeners_) {
if (listener->interested(connection->type())) {
listener->notify_error(connection);
}
}
}
};
Now a rough prototype of this works. But I was wondering if this class design
any good. There are multiple streaming servers which will continuously produce states and send it to my module to program the state in h/w. This needs to be extensible as more clients will be added in future.
Threading
The legacy code had one thread per TCP connection and this worked fine. Here I am trying to handle multiple connections on same thread. Still there will be multiple threads calling ioservice. So the observer will run on multiple threads. I am planning to have a mutex per Listener, so that listeners wont get multiple events concurrently.
HTTP Implements a protocol over TCP so the HTTP Server asio examples are a good starting point for your design, especially: HTTP Server 2, HTTP Server 3 and HTTP Server 4.
Note: that connection lifetime is likely to be an issue, especially since you intend to use class member functions as handlers, see the question and answers here: How to design proper release of a boost::asio socket or wrapper thereof.
I'm trying to add some async operations deep within an existing codebase, which is being called within a web server implemented using pion (which itself uses boost::asio).
The current code needs to continue operating in contexts where there is no io_service available, so I did the following, where Foo::bar is the main entry point of the existing codebase, and handleRequest is the pion request handler:
class Foo
{
public:
void bar(std::string input, boost::asio::io_service* io = NULL)
{
ioService = io;
if ( io == NULL )
{
barCommon(input);
}
else
{
boost::asio::spawn(*io, boost::bind(&Foo::barAsync, this, input, _1));
}
}
void barAsync(std::string input, boost::asio::yield_context yc)
{
barCommon(input, &yc);
}
void barCommon(std::string input, boost::asio::yield_context* yieldContext = NULL)
{
// Existing code here, with some operations performed async
// using ioService and yieldContext if they are not NULL.
}
private:
boost::asio::io_service* ioService;
// Other member variables, which cause a crash when accessed
}
void handleRequest(pion::http::request_ptr request, pion::tcp::connection_ptr connection)
{
Foo* foo = acquireFooPointer();
foo->bar(std::string(request->get_content()), &connection->get_io_service());
}
This seems to work insofar as it ends up running Foo::barCommon inside a coroutine, but the existing code crashes as soon as it tries to access Foo member variables. What am I missing here?
EDIT: Just to be clear, the pointer acquired in handleRequest is to a heap-allocated Foo object whose lifetime matches that of the server process.
I am searching for a way to implement something like this, using boost
class GenBoost{
boost::signal<void(void)> m_signal;
std::function<void (bool)> m_function
public:
void setSignal(boost::signal<void(void)> sigArg)
{
m_signal = sigArg;
}
void setFunction(std::function<void (bool)> &functionArg)
{
m_function = functionArg;
m_signal.connect(boost::bind(&GebBoost::onSignal,this,_1));
}
void onSignal(){
//do something
}
};
How can this be achieved. signal copying is not possible!?
I am not 100% sure of your intent, but assuming that onSignal() will not need to interact with m_signal (and that you only need one connection to m_signal), it appears that you can decouple m_signal from your class entirely. For example, if you really do not want the body of 'onSignal()' to be called until a 'm_function' has been set, you could do something like:
class GenBoost{
std::function<void (bool)> m_function;
boost::signals::scoped_conection m_connection;
public:
void setSignal(boost::signal<void(void)>& sigArg)
{
m_connection = sigArg.connect(boost::bind(&GebBoost::onSignal,this));
}
void setFunction(std::function<void (bool)> &functionArg)
{
m_function = functionArg;
}
void onSignal()
{
if ( m_function )
do_work();
}
void do_work()
{
//do something
}
};
Note that I stripped the last _1 from m_signal.connect(boost::bind(&GebBoost::onSignal,this,_1)); because the signal is declared as type void(void) so should be connected to a void callback. I also added the scoped_connection so that if the object of type GenBoost is destroyed before the signal it is connected to, it will automatically disconnect rather than allow the signal to retain an invalid pointer.
Alternatively, if you needed to retain a reference to the original signal, you could add back your original signal member, but as a pointer boost::signal<void(void)>* m_signal;. Then setSignal becomes:
void setSignal(boost::signal<void(void)>& sigArg)
{
m_signal = &sigArg;
m_connection = m_signal->connect(boost::bind(&GebBoost::onSignal,this));
}
I still recommend using the scoped connection and connecting in setSignal so that you ensure you only have one connection to one signal (If setSignal is called twice, the connection to the first signal is automatically disconnected). If you go the pointer route, you must externally ensure that the passed-in signal has a lifetime longer than the GenBoost object.
I have a C++ API which has a certain defined functions and it's related callbacks.
All these functions are asynchronous in nature.
Now, using this API I want to construct an asynchronous system which sends
multiple request to the server for collecting different data items and then use
these data item for further use.
For example:
void functionA()
{
requestDataForA(); //asynchronous request to the server
//async wait for the callback
processDataForA();
}
void functionB()
{
requestDataForB(); //asynchronous request to the server
//async wait for the callback
processDataForB();
}
void functionC()
{
requestDataForC(); //asynchronous request to the server
//async wait for the callback
processDataForC();
}
Now my question is that when the callback gives the data item, how to use it for subsequent processing. It cannot be done in callback as callback doesn't know who will use the data.
Thanks
Shiv
You implicitly have this information, you just need to track it. Lets say that object A calls functionA, you should make A implement a particular interface that accepts data related that is the response from calling requestA. Lets say this response is DataA, then the interface would be
class InterfaceADataHandler
{
public:
virtual void handle(DataA const&) = 0; // this is the method that will process the data..
};
class A : public InterfaceADataHandler
{
public:
void handle(DataA const&) {} // do something with data
// Now I want to be called back
void foo()
{
functionA(this); // call function A with this instance
}
};
void functionA(InterfaceADataHandler* pHandler)
{
// store this handler against request (say some id)
request..();
// wait for callback
// when you have callback, lookup the handler that requested the data, and call that handler
}
In most API's, you the developer would be providing the callback which will be invoked by the API with the data that has been retrieved. You can then store the data and use it at a later time or use it within the callback (assuming that you won't take very long to process and promise not to block for I/O).
The model would look more like:
void functionA()
{
requestDataForA(processDataForA); //asynchronous request to the server
}
void processDataForA(void *someData)
{
// process "someData"
}