I am a relative newbie to using gRPC. I am creating a microservice that need to run a long computation and return the results to the client via gRPC. I am trying to work out how I run the gRPC server in a thread or threads and the computation in another thread.
Usually you would want to have the server instance running and when a request comes in do some data retrieval or some computation and then formulate the results and return them to the requester. In many cases the operation performed in non-trivial and might take time to process. Ideally you don't want the server thread to block waiting for the operation to complete. The C++ code examples I have found are rather trivial and was looking for more guidance on how to correctly implement this scenario.
The controller would look something like:
void dummyFunction() {
while(true) {
// do something
}
}
void start() {
thread main_thread = thread{dummyFunction};
main_thread.join();
...
mainGRPCServer->start(target_str);
}
In the MainServer implementation I have used a Synchronous server as in the greeter example
void RunServer(string &server_address) {
NServiceImpl n_service;
SServiceImpl s_service;
grpc::EnableDefaultHealthCheckService(true);
grpc::reflection::InitProtoReflectionServerBuilderPlugin();
ServerBuilder builder;
// Listen on the given address without any authentication mechanism.
builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
// Register "service" as the instance through which we'll communicate with
// clients. In this case it corresponds to an *synchronous* service.
builder.RegisterService(&n_service);
builder.RegisterService(&s_service);
// Finally assemble the server.
std::unique_ptr<Server> server(builder.BuildAndStart());
// Wait for the server to shutdown. Note that some other thread must be
// responsible for shutting down the server for this call to ever return.
server->Wait();
}
void MainServerImpl::start(string & target_str) {
worker_ = thread( RunServer, ref(target_str));
worker_.join();
}
Obviously this implementation is not going to work as I understand the grpc Server itself has it's own threading model. I have looked at using an async server implementation. Can anyone guide me on how to structure this ?
UPDATE: I found this on the GoogleGroups:
*The C++ server has two threading models available: sync and async. Most users will want to use the sync model: the server will have an (internal) threadpool that manages multiplexing requests onto some number of threads (reusing threads between requests). The async model allows you to bring your own threading model, but is a little trickier to use - in that mode you request new calls when your server is ready for them, and block in completion queues while there is no work to do. By arranging when you block on the completion queues, and on which completion queues you make requests, you can arrange a wide variety of threading models.*
But I still can't seem to find any good examples of implementation
Best intro found so far is https://grpc.io/docs/languages/cpp/async/
Yes, https://grpc.io/docs/languages/cpp/async/ is the best place to start with. And the helloworld example (https://github.com/grpc/grpc/tree/v1.35.0/examples/cpp/helloworld) using async would be a good reference, too.
Related
Recently I've been learning socket programming and finally I found some good examples from Beej's Guide to Network Programming.
There's a chat server example using poll under the poll section.
Code:
charsever.c (chat server receives clients message and send messages to all the other clients)
After I read the code line by line and fully understand the example, I suddenly realize how clever and neat the design is.
Basically, it uses a poll to monitor everything, the server listening socket accept for new incoming TCP connections and all the existing TCP sockets. No extra threads or processes are needed.
Then I started to ask myself:
It seems that I can use multiple processes (or multiple threads if that's too complicated) to achieve the same effect. Take the chat server as example, the design can be:
main process handle new incoming TCP connections and add new connection sockets to a global array all_sockets.
for each new connection in main process, fork a child process to block, write something like:
//pseudo-code
int bytes_from_client;
while(true){
if( (bytes_from_client = recv(xx,xx,xx,xx)) <= 0 ){
if(bytes_from_client == 0){
client_shutdown();
} else {
error_handle();
}
} else {
//recv data from client and send messages to all the other clients
for(int i = 0; i < all_sockets[x]; i++){
send(xx,xx,xx,xx);
}
}
}
Ok then I need to handle some synchronization issues for global variables. Use mutex or something else. (hard part)
So now for the questions:
What exactly do I benefit from the poll design pattern, compared to the mulithreads one that I laterly described? No need to handle synchronization? Only this 1 advantage?
(A more generic but meaningful question) Is this design pattern made by poll like functions (select, epoll) that makes them so different/unique and great? (I am a newbie and I asked this because I've seen so many people saying how great and significant the poll family functions are. But they never tell why, neither give examples or comparisons. )
One basic reason why a simple select() loop works so very well is that network packets always arrive one-at-a-time and, compared to the blinding speed of any CPU, very slowly. There simply isn't any advantage to a more complicated arrangement.
Often, one thread or process is dedicated to handling the network connection ... polling for incoming packets and sending outgoing messages on behalf of everyone. (I've heard it called "the telegrapher.") It then uses queues to distribute the work to a stable of "worker-bee" processes, which retrieve requests on one queue, then post the outgoing answers to another queue which is also listened-to by the telegrapher.
Sometimes, the telegrapher simply listens for connection-requests, places the handles on a queue, and the worker bees open the connection and do the networking themselves. But, once again, they can simply use polling and it works just fine. Even the fastest communication line is orders of magnitude slower than a CPU.
I'm trying to implement simple networking game (client - server) which uses UDP to transfer game events over network, and I have this working well, but now I would like to add to game chat over tcp in same console application. I've tried to implement multi client chat using select() and non-blocking master socket. Chat is working as standalone application but I have problems putting it together.
Basically my server loop looks like this:
while(true)
{
sendUDPdata()
...
while(true)
{
receiveUDPdata()
}
}
Problem is that when I want to add chat to server's main loop (handling UDP) like this:
while(true)
{
HandleTCPConnections();
sendUDPdata();
...
while(true)
{
receiveUDPdata();
}
}
calling select() in HandleTCPConnections() blocks whole server. Is there any way how to handle this?
There are two good ways to do this:
Use threads. Have a thread to handle your TCP sockets and a thread to handle your UDP sockets.
Use a reactor. Both the UDP code and the TCP code register their sockets with the reactor. The reactor blocks on all the sockets (typically using poll) and calls into the appropriate code when activity occurs on that socket.
There are lots of libraries out there for both of these options (such as libevent and boost.asio) that you can use if you don't want to reinvent the wheel.
select is a blocking call if there's no data available from the sockets, in your case.
Your chat can either run along with the server or in parallel with it: you've already got the first case; for the second, you'd better go for a separate thread that handles the chat. C++ has <thread>, which you may want to look into.
A separate thread is easier to implement in this case because you've a separate connection, and separate sockets therefore, that would otherwise need to be looked after for concurrent access.
I have a client-server model based problem in which there is one server process and multiple client processes and the client processes are in a wait state waiting for the server to signal.
When the server signals, clients do the required processing and go back to waiting
The problem is that i need a way using which I can notify all clients that come up (register for server's notification) i.e. wake up all the client processes waiting for the signal and not just one of them.
I was earlier using mach_semaphore to solve this problem on Mac OS X.
So my code used the semaphore_signal_all() API exposed by mach_semaphore but I was using a call to bootstrap_register() to achieve this but this system call has been deprecated and so is no longer useful.
I tried to look for a workaround and found here that bootstrap_check_in() could be the way to go but on further digging found that this needs an API bootstrap_create_service() which also is deprecated.
So in a nutshell, the problem is I can't find any IPC mechanism (related to semaphores or otherwise) using which I can wake all client processes waiting for the signal.
Below are some code snippets which would convey the issue better:
SERVER.cpp
void server()
{
signal_all_clients_using_semaphore();
/*
do some processing
*/
signal_all_clients_using_semaphore();
/* and so on */
}
CLIENT.cpp
void client()
{
RegisterForTheNotificationFromServer(); // internally looks up for the semaphore created by the server
while(1)
{
semaphore_wait(INFINTE_TIME);
/* do some small processing */
}
}
Needless to say that there are multiple instances of client and only one instance of server running.
It would be really useful if you could provide some code snippets to back your answer.Also, your answer need not be mac specific and can be for linux also (but not for windows as we already have manual reset events - CreateEvent/SetEvent APIs using which we can achieve the desired results)
I'm writing a forking chat server in C++, with each incoming client being its own process. The server-client interactions are done though normal sockets with ZeroMQ sockets handling message queuing and IPC. The basic problem is that there when the server forks to accommodate the new client, the client's process has a copy of the context (which is what fork does, right?), so when it binds sockets with the context, none of the other clients are aware of the socket. Long story short: how do I get each client thread to have the same context so that they can talk to each other through ZeroMQ?
I've looked at various ways to share the context between processes, and so far I'm found only this one. The problem with that is 1) it uses a thread pool, and from what I understand from what's written there, only 5 threads are created; this server is needs to support at least 256 and thus will have at least that many threads, and 2) it uses ZeroMQ for both talking to clients and for backend tasks; I'm limited to using ZeroMQ for backend only.
I've looked at the ZeroMQ mailing list and one message said that fork() is orthogonal to how ZeroMQ works. Does this mean that I can't share my context across forked child processes? If that's the case, how do I share the context across multiple processes while keeping in mind the requirement of supporting at least 256 clients and using ZeroMQ for only backend?
EDIT: Cleared up the thread/process confusion. Sorry about that.
EDIT2: The reason why I'm also favoring forking over threads is that I'm used to having a main process that accepts an incoming socket connection then forks, giving the new socket to the child. I'm not sure how to do that in a threading fashion (not really well-practiced, but not totally out of my league)
EDIT3: So, starting to rewrite this with threads. I guess this is the only way?
EDIT4: For further clarification, incoming connections to the server can be either TCP or UDP and I have to handle which type it is when the client connects, so I can't use a ZeroMQ socket to listen in.
Context Sharing
The reason to share ZMQ context in the example code from your link is, that the server(main()) uses inproc socket to communicate with workers(worker_routine()). Inproc sockets cannot communicate with each other unless they are created from the same ZMQ context, even they settle in the same process. In your case, I think it's not necessary to share it since no inproc sockets are supposed to be used. So, your code might look like:
void *worker_routine (void *arg)
{
// zmq::context_t *context = (zmq::context_t *) arg; // it's not necessary for now.
zmq::context_t context(1); // it's just fine to create a new context
zmq::socket_t socket (context, ZMQ_REP);
// socket.connect ("inproc://workers"); // inproc socket is useless here.
socket.connect("ipc:///tmp/workers"); // need some sockets who can cross process.
// handling code omitted.
}
int main ()
{
// omitted...
// workers.bind ("inproc://workers"); // inproc socket is useless here.
workers.bind("ipc:///tmp/workers");
// Launch pool of worker processes
for (int i = 0; i < 5; ++i) {
if (fork() == 0) {
// worker process runs here
worker_routine(NULL);
return 0;
}
}
// Connect work processes to client process via a queue
zmq::proxy (clients, workers, NULL);
return 0;
}
handling process per request
And now to talk about your requirement, one process per request. The last example code is just intended to illustrate the usage of zmq::proxy, which is provided to simplify the server code with ROUTER-DEALER pattern. But it can't fulfill your requirement. So, you have to implement it manually. It just looks like another example. The difference is that you need to invoke fork() when frontend socket is readable and put the while loop into sub process.
if (items[0].revents & ZMQ_POLLIN) {
if (fork() == 0) {
// sub process runs here
while (1) {
// forward frames here
}
// sub process ends here
return 0;
}
}
Suggestion
At the end, I have to say, it's too much heavy to create a process for one request exactly unless your scenario is really special. Please use thread, or consider the asynchronous IO like zmq::poll.
Hi I am working on an assignment writing multi threaded client server.
So far I have done is open a socket in a port and forked two thread for listening and writing to client. But I need to connect two type of clients to the server and service them differently. My question is what would be my best approach?
I am handling connection in a class which has a infinite loop to accept connection. When ever a connection is accepted, the class create two thread to read and write to client? Now if I wnat to handle another client of different type, what should we do?
Do I need to open another port? or is it possible to service through same port? May be if it is possible to identify the type of client in the socket than I can handle messages differently.
Or do you suggest like this?
Fork two thread for two type of client and monitor inbound connection in each thread in different port.
when a connection accepted each thread spawn another two thread for listening and writing.
please make a suggestion.
Perhaps you'll get a better answer from a Unix user, but I'll provide what I know.
Your server needs a thread that opens a 'listening' socket that waits for incoming connections. This thread can be the main thread for simplicity, but can be an alternate thread if you are concerned about UI interaction, for example (in Windows, this would be a concern, not sure about Unix). It sounds like you are at least this far.
When the 'listening' socket accepts a connection, you get a 'connected' socket that is connected to the 'client' socket. You would pass this 'connected' socket to a new thread that manages the reading from and writing to the 'connected' socket. Thus, one change I would suggest is managing the 'connected' socket in a single thread, not two separate threads (one for reading, one for writing) as you have done. Reading and writing against the same socket can be accomplished using the select() system call, as shown here.
When a new client connects, your 'listening' socket will provide a new 'connected' socket, which you will hand off to another thread. At this point, you have two threads - one that is managing the first connection and one that is managing the second connection. As far as the sockets are concerned, there is no distinction between the clients. You simply have two open connections, one to each of your two clients.
At this point, the question becomes what does it mean to "service them differently". If the clients are expected to interact with the server in unique ways, then this has to be determined somehow. The interactions could be determined based on the 'client' socket's IP address, which you can query, but this seems arbitrary and is subject to network changes. It could also be based on the initial block of data received from the 'client' socket which indicates the type of interaction required. In this case, the thread that is managing the 'connected' socket could read the socket for the expected type of interaction and then hand the socket off to a class object that manages that interaction type.
I hope this helps.
You can handle the read-write on a single client connection in one thread. The simplest solution based on multiple-threads will be this:
// C++ like pseudo-code
while (server_running)
{
client = server.accept();
ClientHandlingThread* cth = CreateNewClientHandlingThread(client);
cth->start();
}
class ClientHandlingThread
{
void start()
{
std::string header = client->read_protocol_header();
// We get a specific implementation of the ProtocolHandler abstract class
// from a factory, which create objects by inspecting some protocol header info.
ProtocolHandler* handler = ProtocolHandlerFactory.create(header);
if (handler)
handler->read_write(client);
else
log("unknown protocol")
}
};
To scale better, you can use a thread pool, instead of spawning a new thread for each client. There are many free thread pool implementations for C++.
while (server_running)
{
client = server.accept();
thread_pool->submit(client);
cth->start();
}
The server could be improved further by using some framework that implements the reactor pattern. They use select or poll functions under the hood. You can use these functions directly. But for a production system it is better to use an existing reactor framework. ACE is one of the most widely known C++ toolkits for developing highly scalable concurrent applications.
Different protocols are generally serviced on different ports. However, you could service both types of clients over the same port by negotiating the protocol to be used. This can be as simple as the client sending either HELO or EHLO to request one or another kind of service.