I'm using Linux and C++. What is the best way to start a remote process on another machine and verify that it is still running periodically programatically? I'll need to do this for multiple processes as well. Thank you
You can use MPI Interface to communicate between the master and slaves via send and recv methods. These can be used to track the status periodically. MPI tag
can be assigned to each slave and track the response from them. More info https://en.wikipedia.org/?title=Message_Passing_Interface
Related
I'm trying to enhance some client C++ code using gRPC to support failover between 2 LAN connections.
I'm unsure if I found a bug in gRPC, or more likely that I'm doing something wrong.
Both the server and client machines are on the same network with dual LAN connections, I'll call LAN-A and LAN-B.
The server is listening on 0.0.0.0:5214, so accepts connections on both LANs.
I tried creating the channel on the client with both IPs, and using various load balancing options, ex:
string all_endpoints = "ipv4:172.24.1.12:5214,10.23.50.123:5214";
grpc::ChannelArguments args;
args.SetLoadBalancingPolicyName("pick_first");
_chan = grpc::CreateCustomChannel(all_endpoints,
grpc::InsecureChannelCredentials(),
args);
_stub = std::move(Service::NewStub(_chan));
When I startup the client and server with all LAN connections functioning, everything works perfectly. However, if I kill one of the connections or startup the client with one of the connections down, gRPC seems to be blocking forever on that subchannel. I would expect it to use the subchannel that is still functioning.
As an experiment, I implemented some code to only try to connect on 1 channel (the non-functioning one in this case), and then wait 5 seconds for a connection. If the deadline is exceeded, then we create a new channel and stub.
if(!_chan->WaitForConnected(std::chrono::system_clock::now() +
std::chrono::milliseconds(5000)))
{
lan_failover();
}
The stub is a unique_ptr so should be destroyed, the channel is a shared_ptr. What I see is that I can successfully connect on my new channel but when my code returns, gRPC ends up taking over and indefinitely blocking on what appears to be trying to connect on the old channel. I would expect gRPC would be closing/deleting this no longer used channel. I don't see any functions available in the cpp version that I can call on the channel or globally that would for the shutdown/closure of the channel.
I'm at a loss on how to get gRPC to stop trying to connect on failed channels, any help would be greatly appreciated.
Thanks!
Here is some grpc debug output I see when I startup with the first load balancing implementation I mention, and 1 of the 2 LANs is not functioning (blocking forever):
https://pastebin.com/S5s9E4fA
You can enable keepaliaves. Example usage: https://github.com/grpc/grpc/blob/master/test/cpp/end2end/flaky_network_test.cc#L354-L358
Just wanted to let anyone know the problem wasn't with gRCP, but the way our systems were configured with a SAN that was being written too. A SAN was mounted through the LAN connection I was using the test failover through and the process was actually blocking because it was trying to access that SAN. The stack trace was misleading because it showed the gRPC thread.
I'm building a system that has 2 processes.
Process 1
This process is actually a Node.js program. This process is actually a Web Server handling the incoming request.
Process 2
This process is actually a C++ program.
Both the processes are started automatically at startup with help of rc.local
Now, for Process 1 there are some specific request that should be passed to Process 2.
For example, if Process 1 receives a post request at route /enqueue with a JSON body payload, Process 1 should stringify the JSON and pass to Process 2.
When Process 2 receives the JSON, it should kill a worker thread and start a new thread with that JSON to perform the actual task. The worker thread should be killed irrespective of whether the worker thread is still processing previous JSON
If both the processes were Node.js application, I could have forked Process 2 from Process 1 and used the following code.
process.on('message',function(message){
//implementation
}
...
process.send(data);
But my second process is a C++ app.
Any idea on how to implement it?
Note: Before flagging this question, please keep in mind I'm not looking for a full code. I just need the idea on how to do it.
You cannot use the Nodejs messaging/eventing facility for this purpose as it is specific to Node.
You will need to use the communications facilities of your operating system such as Unix, TCP, UDP sockets or an eventing system that both processes can communicate with, like Redis or ZeroMQ.
I'm trying to monitor the network sessions on server withe event driven programming (and not polling on /proc/net/tcp or udp).
I was able to find this article but it only provide one time look at the current state and not an event on each change (LISTEN, ESTABLISHED...).
Is it possible to use this like in this article that monitors processes changes but on network connections?
If not, is there any other API that I can use in order to achive this without polling /porc/net/* in interval?
I was reading about pipes today, it seems an interesting for me to use it or to start about it.
but it's written there that the processes should be like a parent and child or client server and the communication will be between them.
there are some things I don't understand and before start I should be sure about them.
If I have a running process for example on my machine, and I want to send message to it, of course I can't control it, so how can I make it respond to my messages??
should I create the client and server and run them both??
Or the running process application can be a server for example and I can send some events to it??
if you can help me in this :)
thanks :)
If the running process is reading from stdin, you may write to /proc/the_pid/fd/0 to send your message.
I have implemented a zmq library using push / pull on windows. There is a server and up to 64 clients running over loopback. Each client can send and receive to the server. There is a thread that waits for each client to connect on a pull zmq socket. Clients can go away at any time.
The server is expected to go down at times and when it comes back up the clients need to reconnect to it.
The problem is that when nothing is connected I have 64 receive threads waiting for a connection. This shows up as a lot of connections in tcpview and my colleagues inform me that this is appearing like a performance/d-dos sort of attack.
So in order to get around that issue I'd like the clients to send some sort of heart beat to the server "hey I'm here" on one socket. However I can't see how to do that with zmq.
Any help would be appreciated.
I think the basic design of having 64 threads on the server waiting for external connections is flawed. Why not have a single 'master' thread binding the socket, which the external clients would connect to?
Internal to the server, you could still have 64 worker threads. Work would be distributed to the worker threads by the master thread. The communication between the master and the worker threads would be using zmq messages over the inproc transport.
What I have described are simple fan-in and fan-out patterns which are covered in the zmq guide. If you adopt this, most of the zmq code in the clients and workers would remain unchanged. You would have to write code for the master thread, but the zproxy class of CZMQ may work well for you (if you're using CZMQ).
So my advice is to get the basic design right before trying to add heartbeats. [Actually, I'm not sure how heartbeats would help your current problem.]