I am interested in a c++ linux-only (possible relaxed to posix-only) IPC solution that would behave as follows; a program called 'calculator' is started, and can listen to messages. Calculator would have a loop that periodically checks for message strings and then acts on the based on their content.
Another program called 'send_msg' can send messages to its pid (ideally a hostname/pid, through tcp or udp).
$ calculator &
// awhile later
$ send_msg <calculator pid> show calculations
Calc1: 52% complete
Calc2: 21% complete
$ send_msg <calculator pid> alter Calc2 <numeric parameters>
Ok! I'm restarting my calculations!
$
I am very versed in c++, but know nothing about network programming and am not interested in spending much time right now to learn it. Is there an easy-to-use c++ package that does the above? I would rather not have to choose things like port numbers, file locations, etc.
I think you may like zeromq (spelled 0mq), or the forked crossroadsio, as they abstract away a lot of the handholding allowing you simply pub/sub, as well as many other patterns. 0mq has (had?) a bunch of examples starting with simple ping-pong.
The setup you are requesting is anything but simple, I think.
You'd probably do best with either a Unix-domain socket or a TCP socket (port number) for communication between the background calculator and the front end. So, for example, you might run:
calculator -p 3456 &
The calculator is then listening on port 3456. Your send_msg program can then be used to make the calculator do things:
send_msg -p 3456 show calculations
When the calculator receives the message, it acts according to the orders, sending the answer back to the send_msg progam on the socket, which then echoes it to its standard output.
Meanwhile, you have a calculator that may need to be multi-threaded. It also needs to be able to determine how much work is involved in each calculation, so that it can report on the progress of each calculation. Neither you nor I have specified how the calculation is set up, but it might be:
send_msg -p 3456 new calc.file
to indicate that the calculator should start a new calculation, reading the problem from the file calc.file. It might echo back:
Calc1: ETC = 3:15
where, by some more or less devious means, it has determined that the Estimated Time to Completion (ETC) is 3 minutes, 15 seconds. You can set up the second calculation in a similar way. To handle this, you need a controller thread that is listening for connections from send_msg. When it gets told to create a new job, it starts a new thread (or process) to do the calculation. There has to be some agreed mechanism between the master thread in the calculator and the actual calculating threads. This might be as simple as a location where each thread writes its progress and the master reads. But the calculation threads need to keep track of how much work they've done, how much there is left to do, and whether the estimates need to be changed.
Now, I might be making things too complicated, but the interface you showed suggests that something similar to that might be necessary. If you single-thread the calculator, it has to do some sort of round-robin scheduling of its work on each calculation you set it, as well as periodically checking to see whether the send_msg program has sent a new message.
Have a look at RCF - it's native C++ and has publish/subscribe support which should make this pretty easy.
Related
Got a large C++ function in Linux that calls a whole lot of other functions, making up an algorithm. At various points given certain bad inputs, the algorithm can get "stuck" and go on forever. Adding a timeout seems appropriate as all potential "stuck" points cannot be predicted. But despite scouring the Internet for timeout examples I've only found how to apply timeouts when either the thing your timing is a separate thread or it's reading inputs. My code is a single thread and does not modify file descriptors, so not coming up with any luck. Do I basically have no choice but to thread it?
I am not sure about the situation, actually server applications or embedded applications often run for years in background without stopping. I think one option is to let your program run in background and log to a file(or screen) timely, and, if you really want to stop the program after certain time, you can use timeout command or a script to kill your program after that time, say, timeout 15s your-prog.
I am writing a gtk application (in C++) to communicate with motor controller through serial port. I am working with Linux Ubuntu and termios lib.
I need advice on the best solution to do it.
So here are the constraints I have:
1- when i send a request, the controler send me back a message
2- the controler can send me error notification at any time if an error occurs
3- request is ANSII characters string finished with [CR]
4- controller answer is ANSII characters string finished with [CR][NL]
Because of (3) and (4), I thought it was appropriate to configure serial port in CANONICAL mode.
Because of the GUI + (2), I thought about multi-threading: a main thread who write user request on serial port and an other infinite thread to read controller answer. Do you think it is a good idea?
Second question: if I am using multi-threading I want to be able to write data when I need it so I have to find a way to stop/sleep the reading thread during writing maybe with pthread_cond_wait. Am I right? I've seen poll and select functions but I don't really understand them and I am not sure they are compatible with canonical mode.
I am getting started with multi-threading and serial port. I read lots of things on google, forum...but the large amount of information is a little overwhelming for a beginner.
Thank you for your help.
The main thing to consider here for separating the GUI from the serial port is going to be your delays. Are you ever going to be performing any actions that will cause you to need to poll the port for a specific amount of time that would be noticeable to the user? If you are just doing request/reply and the latency of those is really low your user probably wouldn't notice any of those delays. Additionally receiving those asynchronous error messages would also not cause any sort of noticeable delay I would imagine. Unless you know for a fact that there could be numerous seconds of delay after an Init message or something like that gets sent to the controller it will probably make your life much simpler to keep the application single threaded.
On the other hand if there will be those large latencies or you just want to mess around with multi thread I would just start with 1 thread that does all the GUI work and another thread that handles all the serial IO. Use message passing or event notification between those two threads for coordinating your activities and it should be pretty straight forward.
i got a very specific question about server programming in UNIX (Debian, kernel 2.6.32). My goal is to learn how to write a server which can handle a huge amount of clients. My target is more than 30 000 concurrent clients (even when my college mentions that 500 000 are possible, which seems QUIIITEEE a huge amount :-)), but i really don't know (even whats possible) and that is why I ask here. So my first question. How many simultaneous clients are possible? Clients can connect whenever they want and get in contact with other clients and form a group (1 group contains a maximum of 12 clients). They can chat with each other, so the TCP/IP package size varies depending on the message sent.
Clients can also send mathematical formulas to the server. The server will solve them and broadcast the answer back to the group. This is a quite heavy operation.
My current approach is to start up the server. Than using fork to create a daemon process. The daemon process binds the socket fd_listen and starts listening. It is a while (1) loop. I use accept() to get incoming calls.
Once a client connects I create a pthread for that client which will run the communication. Clients get added to a group and share some memory together (needed to keep the group running) but still every client is running on a different thread. Getting the access to the memory right was quite a hazzle but works fine now.
In the beginning of the programm i read out the /proc/sys/kernel/threads-max file and according to that i create my threads. The amount of possible threads according to that file is around 5000. Far away from the amount of clients i want to be able to serve.
Another approach i consider is to use select () and create sets. But the access time to find a socket within a set is O(N). This can be quite long if i have more than a couple of thousands clients connected. Please correct me if i am wrong.
Well, i guess i need some ideas :-)
Groetjes
Markus
P.S. i tag it for C++ and C because it applies to both languages.
The best approach as of today is an event loop like libev or libevent.
In most cases you will find that one thread is more than enough, but even if it isn't, you can always have multiple threads with separate loops (at least with libev).
Libev[ent] uses the most efficient polling solution for each OS (and anything is more efficient than select or a thread per socket).
You'll run into a couple of limits:
fd_set size: This is changable at compile time, but has quite a low limit by default, this affects select solutions.
Thread-per-socket will run out of steam far earlier - I suggest putting the longs calculations in separate threads (with pooling if required), but otherwise a single thread approach will probably scale.
To reach 500,000 you'll need a set of machines, and round-robin DNS I suspect.
TCP ports shouldn't be a problem, as long as the server doesn't connection back to the clients. I always seem to forget this, and have to be reminded.
File descriptors themselves shouldn't be too much of a problem, I think, but getting them into your polling solution may be more difficult - certainly you don't want to be passing them in each time.
I think you can use the event model(epoll + worker threads pool) to solve this problem.
first listen and accept in main thread, if the client connects to the server, the main thread distribute the client_fd to one worker thread, and add epoll list, then this worker thread will handle the reqeust from the client.
the number of worker thread can be configured by the problem, and it must be no more the the 5000.
I have a simple python server script which forks off multiple instances (say N) of C++ program. The C++ program generates some events that need to be captured.
The events are currently being captured in a log file (1 logfile per forked process). In addition, i need to periodically (T minutes) get the rate at which the events are being produced across all child processes to either the python server or some other program listening for these events (still not sure). Based on rate of these events, some "re-action" may be taken by the server (say reduce the number of forked instances)
Some pointers i have briefly looked at:
grep log files - go through the running process log files (.running), filter those entries generated in the last T minutes, analyse the data and report
socket ipc - add code to c++ program to send the events to some server program which analyses the data after T minutes, reports and starts all over again
redis/memcache (not sure completely) - add code to c++ program to use some distributed store to capture all the generated data, analyses the data after T minutes, reports and starts all over again
Please let me know your suggestions.
Thanks
if time is not of the essence (T minutes sounds like it is long compared to whatever events are happening in the C++ programs that are kicked off) then dont make things any more complicated than they need to be. forget IPC (sockets, shared mem, etc), just have each C++ program log what you need to know about time/performance and let the python script check logs every T minutes that you need the data. dont waste time overcomplicating something that you can do in a simple manner
As a alternative to your socket IPC suggestion, how about 0mq. It's a library (in C with python bindings available) that can do message transfer on an inter-thread, inter-process or inter-machine level. Pretty simple to get going, and pretty quick.
I'm not affiliated with it. I'm just evaluating it for other uses and thought it might be a fit for you as well.
I have two applications running on my machine. One is supposed to hand in the work and other is supposed to do the work. How can I make sure that the first application/process is in wait state. I can verify via the resources its consuming, but that does not guarantee so. What tools should I use?
Your 2 applications shoud communicate. There are a lot of ways to do that:
Send messages through sockets. This way the 2 processes can run on different machines if you use normal network sockets instead of local ones.
If you are using C you can use semaphores with semget/semop/semctl. There should be interfaces for that in other languages.
Named pipes block until there is both a read and a write operation in progress. You can use that for synchronisation.
Signals are also good for this. In C it is called sendmsg/recvmsg.
DBUS can also be used and has bindings for variuos languages.
Update: If you can't modify the processing application then it is harder. You have to rely on some signs that indicate the progress. (I am assuming you processing application reads a file, does some processing then writes the result to an output file.) Do you know the final size the result should be? If so you need to check the size repeatedly (or whenever it changes).
If you don't know the size but you know how the processing works you may be able to use that. For example the processing is done when the output file is closed. You can use strace to see all the system calls including the close. You can replace the close() function with the LD_PRELOAD environment variable (on windows you have to replace dlls). This way you can sort of modify the processing program without actually recompiling or even having access to its source.
you can use named pipes - the first app will read from it but it will be blank and hence it will keep waiting (blocked). The second app will write into it when it wants the first one to continue.
Nothing can guarantee that your application is in waiting state. You have to pass it some work and get back a response. It might be transactions or not - application can confirm that it got the message to process before it starts to process it or after it was processed (successfully or not). If it does not wait, passing a piece of work should fail. Whether when trying to write to a TCP/IP socket or other means, or if timeout occurs. This depends on implementation, what kind of transport you are using and other requirements.
There is actually a way of figuring out if the process (thread) is in blocking state and waiting for data on a socket (or other source), but that means that client should be on the same computer and have access privileges required to do that, but that makes no sense other than debugging, which you can do using any debugger anyway.
Overall, the idea of making sure that application is waiting for data before trying to pass it that data smells bad. Not to mention the racing condition - what if you checked and it was OK, and when you actually tried to send the data, you found out that application is not waiting at that time (even if that is microseconds).