Simple but fast IPC method for a Python and C++ application? - c++

I have a GNU Radio application which utilizes both Python and C++ code. I want to be able to signal the C++ code of an event. If they were in the same scope I would normally use a simple boolean, but the code is separate to the point where some form of shared memory is required. The code in question is performance-critical so an efficient method is required.
I was initially thinking about a shared memory segment that is accessible by both Python and C++. Therefore I could set a flag in the python code and check it from C++. Since I just need a simple flag to pause the C++ code, would a semaphore suffice?
To be clear, I need to set a flag from Python and the C++ code will simply check this flag, and if it is set enter a busy loop.
So would trying to implement a shared memory segment between Python/C++ be a reasonable approach? How about a semaphore? On Linux, which is easier to implement?
Thanks!

Assuming this is two separate applications on one machine and you need decent real time performance you don't want to go with sockets. I would use a flag in shared memory, and probably use a semaphore to make sure both programs can't be accessing the flag at once. This library provides access to the semaphores and shared memory with Python and supports Python versions 2.4-3.1 (not 3.0): http://semanchuk.com/philip/posix_ipc
EDIT: Changed recommendation to using a semaphore protecting the flag in shared memory

Why not open a unix socket? Or use DBus

If Boost is an option, you could use Boost.Python and Boost.Interprocess. Boost.Python gives you a way for Python & C++ objects to interact and Boost.Interprocess gives you plenty of options for shared memory or synchronization primitives across process boundaries.

DBus looks promising. It supports signals, so you should be able to stop an application on demand. However, I'm not sure if it's performance will be enough for you.

You can try using custom signals. I don't know about Python code being able to send custom signals, but your C/C++ can certainly define custom signals with SIGIO.
If you have stringent response-time requirements, you might need to look beyond your application code and into some time of OS with support for real-time signals (rt-linux, muOs, etc.)

Related

GNU pth vs. pthread

I want to build a portable and efficient server in C++; it will have lots of clients trying to connect at the same time, so it must be able of handling each request parallel.
I have been trying to find documentation, guides... etc. for multithreading. I have found a lot about POSIX Pthread, but almost nothing for GNU Pth (apart from the official manual in gnu.org).
So, can anyone explain me the difference between POSIX Pthread and GNU Pth? Please, I want the response not to be a copy of Wikipedia's contents (keep in mind that I'm an absolute newbie to multithreading). I want my server to be portable and efficient between all *nix-based systems, keeping away of using heavy fork()s.
Thanks for your help.
PS: I think it's better to ask this here: what about Windows? Are Pthreads or Pth an option there? If not, what is the API for that operating system?
Use Pthreads, it's much more widely used, so there is far more information and support available for it. I've never met anyone who actually uses GNU Pth. Or better yet if you are using C++11 use std::thread and if not then use boost::thread.
So, can anyone explain me the difference between POSIX Pthread and GNU Pth?
Pthreads is a cross-platform standard for pre-emptible multithreading, meaning (usually) the OS kernel manages the threads and the OS scheduler decides when each thread gets to run (if you have a single core only one thread can run at a time, if you have multiple cores multiple threads can run at a time). The OS scheduler could pause any thread at (almost) any time and let another thread run, so each thread gets a limited "time slice" and then other threads get to run.
GNU Pth is a non-preemptible user-space threading library, meaning the threads and which ones run at which time are decided in user-space not by the kernel. Some people say programs using non-preemptible threading libraries are easier to understand, because your thread won't get paused at arbitrary times for another thread to run.
I want my server to be portable and efficient between all *nix-based systems, keeping away of using heavy fork()s.
fork is not heavy on UNIX.
what about W*ndows? Are Pthreads or Pth an option there? If not, what is the API for that operating system?
There are pthreads APIs for Windows, but they're not native to the Windows OS. I don't know if GNU Pth works on Windows - I doubt it, unless you use Cygwin. Windows has its own Win32 thread model.
Using std::thread or boost::thread is portable to POSIX platforms and Windows, and makes certain parts of the API easier to use (specifically, locking and unlocking mutexes can be easily done in an exception safe way and condition variables are easier to use.)
Gnu PTH is for a very limited use case: you want to use a multi-threaded implementation paradigm but you don't want to use multiple CPUs or cores and you don't want to rely on any OS or kernel-level support. Since almost all general-purpose CPUs now have multiple cores, this use case is increasingly irrelevant.
Windows has a separate threading model from POSIX; if you want your application to be cross-platform it is best to use a cross-platform threading library such as boost::thread.
I think GNUs PTH is meaned for C in the first place. You can use it on C++ too but C++ have its own anyway.
There are quite some applications using pth like low-level burning tools (and so GUI-Tools like K3B and Brasero depend on pth), also GnuPG uses PTH, the package management of Archlinux and some multimedia stuff.
On Windows its always a bit complicated. Microsoft did never get over the fact that C is the Programming Language from/for UNIX-Systems and so is suffering the NIH Symptome (Not Invented Here)
So they do a lot of stuff without any advantage just to be different.
If you use an Application which should run everywhere and its not low-level, use Qt with its QThreads and QThreadPool
Its 100% the same on all operating systems
You need much less code
If you write an "low-level" application i recommend to split your applications into backends and frontends and write a own backend for each OS and use the library which will do the least problems.

Multiplatform multiprocessing?

I was wondering why in the new C++11 they added threads and not processes.
Couldn't have they done a wrapper around platform specific functions?
Any suggestion about the most portable way to do multiprocessing? fork()? OpenMP?
If you could use Qt, QProcess class could be an elegant platform independent solution.
If you want to do this portably I'd suggest you avoid calling fork() directly and instead write your own library function that can be mapped on to a combination of fork() and exec() on systems where that's available. If you're careful you can make your function have the same or similar semantics as CreateProcess() on Win32.
UNIX systems tend to have a quite different approach to processes and process management compared to Windows based systems so it's non-trivial to make all but the simplest wrappers portable.
Of course if you have C++11 or Boost available I'd just stick with that. If you don't have any globals (which is a good thing generally anyway) and don't set up and shared data any other way then the practical differences between threads and processes on modern systems is slim. All the threads you create can make progress independently of each other in the same way the processes can.
Failing that you could look at An MPI implementation if message passing suits your task, or a batch scheduler system.
I am using Boost Interprocess.
It does not provide the possibility to create new processes, but once they are there, it allows them to communicate.
In this particular case I can create the processes I need from a shell script.

Getting user input in C++ without halting the program

I have a C++ program that captures videos, and I would like to be able create a command-line program to update its frame rate, image format, etc on the fly.
How can I do this without halting the entire program? I need it to be able to wait for user input, but still capture videos at the same time. I know this will probably involve some kind of multi-threading, which I am entirely new to. Some suggestions/links would be nice.
Than you all,
Are you developing this for a specific platform or does it need to be platform independent?
If you are developing for windows you should look into the win32 API. specifically beginthread or _beginthreadex on msdn
I'm not too familiar with *nix development but pthreads i believe would do the trick and can be used in Windows and *nix
Another option would be to use the BOOST libraries. BOOST can be used on Windows and *nix systems. Below is a link to the BOOST Thread documentation.
http://www.boost.org/doc/libs/1_44_0/doc/html/thread.html
I find BOOST Threads a lot easier to use than WIN32 Threads and at the same time you're not tied down to a specific platforms API.
Create a thread to handle video, whilst using the main thread to wait for input. Thread creation depends on platform, and can be a little overwhelming to those who are new. You will need a mutex on variables that can be altered through the command line, and you'll need to look up of how to make your code "thread safe".
In the days before multi-threading it was possible to solve this as well by regularly peeking into the keyboard buffer from time to time. I mention this just as an alternative to opening the multi-threading box which often gives you more than you bargain for.
EDIT: I read now a bit more carefully what you want to achieve, having a console program to update another program with new settings. I think what you then need is for the programs to communicate with one another. Look at boost::interprocess for that.

Using ptrace to write a program supervisor in userspace

I'll looking for advice/resources to write a program that can intercept system calls from a programm to supervise it's filesystem, network, etc access.
The aim of this is to write an online judge, so that untrusted code can be run safely on a server.
This is on linux, and I would prefer to write C++ or a scripting langauge (ruby, python, etc), and a library would be great!
Thanks.
This looks like a good place to start.
http://www.linuxjournal.com/article/6100
You can't safely use ptrace() to sandbox a hostile application.
The application can always use multiple threads with deliberate race conditions to alter syscall arguments passed via pointers (eg. a filename) after you've inspected them but before the kernel looks at them.

C++ master/worker

I am looking for a cross-platform C++ master/worker library or work queue library. The general idea is that my application would create some sort of Task or Work objects, pass them to the work master or work queue, which would in turn execute the work in separate threads or processes. To provide a bit of context, the application is a CD ripper, and the the tasks that I want to parallelize are things like "rip track", "encode WAV to Mp3", etc.
My basic requirements are:
Must support a configurable number of concurrent tasks.
Must support dependencies between tasks, such that tasks are not executed until all tasks that they depend on have completed.
Must allow for cancellation of tasks (or at least not prevent me from coding cancellation into my own tasks).
Must allow for reporting of status and progress information back to the main application thread.
Must work on Windows, Mac OS X, and Linux
Must be open source.
It would be especially nice if this library also:
Integrated with Qt's signal/slot mechanism.
Supported the use of threads or processes for executing tasks.
By way of analogy, I'm looking for something similar to Java's ExecutorService or some other similar thread pooling library, but in cross-platform C++. Does anyone know of such a beast?
Thanks!
I haven't used it in long enough that I'm not positive whether it exactly meets your needs, but check out the Adaptive Communications Environment (ACE). This library allows you to construct "active objects" which have work queues and execute their main body in their own threads, as well as thread pools that can be shared amoung objects. Then you can pass queue work objects on to active objects for them to process. Objects can be chained in various ways. The library is fairly heavy and has a lot to it to learn, but there have been a couple of books written about it and theres a fair amount of tutorial information available online as well. It should be able to do everything you want plus more, my only concern is whether it possesses the interfaces you are looking for 'out of the box' or if you'd need to build on top of it to get exactly what you are looking for.
I think this calls for intel's Threading Building Blocks, which pretty much does what you want.
Check out Intels' Thread Building Blocks library.
Sounds like you require some kind of "Time Sharing System".
There are some good open source ones out there, but I don't know
if they have built-in QT slot support.
This is probably a huge overkill for what you need but still worth mentioning -
BOINC is a distributed framework for such tasks. There's a main server that gives out tasks to perform and a cloud of workers that do its bidding. It is the framework behind projects like SETI#Home and many others.
See this post for creating threads using the boost library in C++:
Simple example of threading in C++
(it is a c++ thread even though the title says c)
basically, create your own "master" object that takes a "runnable" object and starts it running in a new thread.
Then you can create new classes that implement "runnable" and throw them over to your master runner any old time you want.