I'm integrating a 3rd party C++ package to a python application using SWIG. The package connects to a proprietary API over a network and receives updates. The overall flow is that python instantiates a C++ object, calls its functions to set it up and then waits for the updates.
I implemented a callback mechanism for the updates using SWIG's directors feature, and while testing from python, or from C++ functions called by python, it works well. Namely I'm able to inherit a C++ class in Python, call its virtual functions from C++ and see the python code take priority and execute.
The problem:
When I receive the updates from the network I get:
The thread 'Win32 Thread' (0x1f78) has exited with code 0 (0x0).
Unhandled exception at 0x1e0650cb in python.exe: 0xC0000005: Access violation writing location 0x0000000c.
This exception is thrown from within python27.dll while calling the callback function.
My suspicion is this: I've violated the GIL
AFAIU the updates come from a different thread and call python's code using that thread.
At this point I'm at a loss. Is SWIG's director feature limited only to flows initiated within python (i.e. from python managed threads)?
How do I circumvent this? How do I induce updates from C++ to python? Is it even possible using SWIG?
Should I use a completely different approach?
I'm open for any suggestions on this matter...
If your SWIG-wrapped C++ code invokes the callback routine in-thread, then there're probably no GIL problems - SWIG-generated code does not perform any GIL management that I've seen, which means that when Python code calls in to your C++ code, you retain the GIL throughout the call.
However, if your C++ code defers the callback to another thread, then you very likely have violated the GIL. This is simple enough to work around: Before you invoke the callback, call PyGILState_Ensure(), and when the callback completes, invoke PyGILState_Release. Refer to http://docs.python.org/c-api/init.html, the section "Non-Python created threads". (If you're using C++ exception handling here, you might need to take extra care to ensure that you can release the GIL.)
If you haven't yet looked at the stack trace, it's worth verifying that that NULL pointer deref isn't something silly happening in your code. (You can attach to the Python process running your code with VS/GDB/WinDBG; the Python execution will remain inscrutable, but you can trace your C++ code this way.)
Related
I am developing a plug-in that runs in several host applications, and which maintains its own threads using the boost::thread library, version 1.53.0.
When running my plug-in in a particular application on Mac, I get a null access error when calling boost::condition_variable::timed_wait(). At the top of the call stack is a call to pthread_getspecific(), which is called from inside the boost thread library.
If I replace timed_wait() with a call to boost::this_thread_sleep(), same behaviour: exception is thrown when pthread_getspecific() is called internally.
This application is the only one which exhibits this kind of behaviour; if I run my plug-in in other hosts, it works as expected.
I don't have much experience with pthreads, but I think the exception must be caused by some properties being set up by the host application. Does anyone have any better idea of what might be going on here?
Thanks!
It turns out that the pthread API calls where not thread safe in my application, causing null pointer crashes whenever I called them in a separate thread.
I wrote a tool in C++ using wxWidgets for the GUI and IBM ILOG Cplex to solve an optimization problem.
In one of the functions called by the wx event handler, I invoke the IBM ILOG Cplex Optimizer which is itself multi-threaded code.
I realize that this causes indererministic bugs with non-sensical memory contents.
Since I have no experince in writing multi-threaded code and would like to get away without spending three weeks learning how to do it, I would like to know:
Is there is some safe, possibly inelegant way to avoid problems here? (More elegant, maybe, than writing a file to disc, calling a different task through the OS and reading the output back in).
Is it a bad idea to launch Cplex threads from a wx thread?
Is it generally a bad idea to use two libraries that might use different libraries internally to implement multi-threading? (I have no idea what there is except pthreads and what is used by either cplex or wx).
Any help and background information is appreciated.
Based on my experience, the rule is:
every wxWdiget function call that change the display must be made in the wxWidget thread
I don't know much about Cplex, but if you say it's multithreaded, chances are you are calling an asynchronous function and you handle the results in a call back. The callback is most definitely not called withing the wxWidget thread. If you then try to display the results within the callback, you are breaking the rule stated above. That's when you'll get nice little bugs, which in my case usually materialize as heap corruption.
To fix that you must pass the results of your callback to the wxWidget thread and display them in that thread. There's many way to do it, but the global mechanism is to trigger a custom event on wxWigdet that get passed to the wxWidget thread.
Check this link, http://wiki.wxwidgets.org/Custom_Events you need to use
wxEvtHandler::AddPendingEvent(wxEvent& event)
I'm working on a C++ application which uses a library written in C by another team. The writers of the library like to call exit() when errors happen, which ends the program immediately without calling the destructors of objects on the stack in the C++ application. The application sets up some system resources which don't automatically get reclaimed by the operating system after the process ends (shared memory regions, interprocess mutexes, etc), so this is a problem.
I have complete source code for both the app and the library, but the library is very well-established and has no unit tests, so changing it would be a big deal. Is there a way to "hook" the calls to exit() so I can implement graceful shutdown for my app?
One possibility I'm considering is making one big class which is the application - meaning all cleanup would happen either in its destructor or in the destructor of one of its members - then allocating one of these big objects on the heap in main(), setting a global pointer to point to it, and using atexit() to register a handler which simply deletes the object via the global pointer. Is that likely to work?
Is there a known good way to approach this problem?
In the very worst case, you can always write your own implementation of exit and link it rather than the system's own implementation. You can handle the errors there, and optionally call _exit(2) yourself.
Since you have the library source, it's even easier - just add a -Dexit=myExit flag when building it, and then provide an implementation of myExit.
install exit handler with atexit and implement the desired behavior
If you want to make the C library more usable from C++, you could perhaps run it in a separate process. Then make sure (with an exit handler or otherwise) that when it exits, your main application process notices and throws an exception to unwind its own stack. Perhaps in some cases it could handle the error in a non-fatal way.
Of course, moving the library use into another process might not be easy or particularly efficient. You'll have some work to do to wrap the interface, and to copy inputs and outputs via the IPC mechanism of your choice.
As a workaround to use the library from your main process, though, I think the one you describe should work. The risk is that you can't identify and isolate everything that needs cleaning up, or that someone in future modifies your application (or another component you use) on the assumption that the stack will get unwound normally.
You could modify the library source to call a runtime- or compile-time-configurable function instead of calling exit(). Then compile the library with exception-handling and implement the function in C++ to throw an exception. The trouble with that is that the library itself probably leaks resources on error, so you'd have to use that exception only to unwind the stack (and maybe do some error reporting). Don't catch it and continue even if the error could be non-fatal as far as your app is concerned.
If the call exit and not assert or abort, there are a few points to get control again:
When calling exit, the destructors for objects with static lifetime (essentially: globals and objects declared with static) are still executed. This means you could set up a (few) global "resource manager" object(s) and release the resources in their destructor(s).
As you already found, you can register hooks with atexit. This is not limited to one. You can register more.
If all else fails, because you have the source of the library, you can play some macro tricks to effectively replace the calls to exit with a function of your own that could, for example, throw an exception.
I'm currently wrapping a lua class in c++ and it's going pretty well so far. But I'm wondering if there some way to break a lua script from running(could be in the middle of the script) for another thread. So if I run my lua script on thread 1, can I break it from thread 2? Would lua_close(...) do that?
Thanks.
If this is an expected occurrence and most of the Lua script's time is spent inside Lua functions (i.e., not lengthy, blocking C calls), you could install a debug hook that checks for your "break flag" every N instructions and aborts the script. See the "debug library" section of Programming in Lua.
Then there must be some way to force the script to halt?
No, there doesn't.
Lua provides exactly one thread safety guarantee: two separate lua_States (separate being defined as different objects returned by lua_open) can be called freely from two different CPU threads. Thread A can call lua_State X, and thread B can call lua_State Y.
That is the only guarantee Lua gives you. Lua is inherently single-threaded; it only provides this guarantee because all of the global information for Lua is contained in a self-contained object: lua_State.
The rules of Lua require that only one thread talk to a lua_State at any one time. Therefore, it is impossible for Lua code to be executing and then be halted or paused by external C++ code. That would require breaking the rule: some other thread would have to call functions on this thread's lua_State.
If you want to abort a script in-progress, you will have to use debug hooks. This will dramatically slow down the performance of your script (if that matters to you). You will also need thread synchronization primitives, so that thread B can set a flag that thread A will read and halt on.
As for how you "abort" the script, that... is impossible in the most general case. And here's why.
The lua_State represents a Lua thread of execution. It's got a stack, a pointer to instructions to interpret, and some code it's currently executing. To "abort" this is not a concept that Lua has. What Lua does have is the concept of erroring. Calling lua_error from within C code being called by Lua will cause it to longjmp out of that function. This will also unwind the Lua thread, and it will return an error to the most recent function that handles errors.
Note: if you use lua_error in C++ code, you need to either have compiled Lua as C++ (thus causing errors to work via exception handling rather than setjmp/longjmp) or you need to make sure that every object with a destructor is off the stack when you call it. Calling longjmp in C++ code is not advisable without great care.
So if you issue lua_pcall to call some script, and in your debug hook, you issue lua_error, then you will get an error back in your lua_pcall statement. Lua's stack will be unwound, objects destroyed, etc. That's good.
The reason why this does not work in the general case (besides the fact that I have no idea how Lua would react to a debug hook calling lua_error) is quite simple: Lua scripts can execute pcall too. The error will go to the nearest pcall statement, whether it's in Lua or in C.
This Lua script will confound this method:
local function dostuff()
while true do end
end
while true do
pcall(dostuff)
end
So issuing lua_error does not guarantee that you'll "abort the script." Think of it like throwing exceptions; anyone could catch them, so you can't ensure that they are only caught in one place.
In general, you should never want to abort a Lua scripts execution in progress. This is not something you should be wanting to do.
OK, so this may be an odd situation but please bear with me.
I have a Python program which calls up a C++ class via a SWIG interface. I came to a point where I must asynchronously signal (to update a status) the Python code from the C++ library. Originally I had inefficient busy loops which polled a flag. I wanted to replace this by using SIGUSR1.
So, the problem is I discovered that even though these are separate 'threads', they share the same PID. That is the Python and C++ program both report the same PID. I have sent the signal using kill, and the Python code caught it in its handler, however it did not interrupt the code as I expected. I tried two methods of waiting for the signal in the Python code, first py calling Python's signal.pause which I read was supposed to be preempted on the reception of a signal. The other one was a simple time.sleep which was supposed to do basically the same thing - return when the signal comes through.
For some reason this isn't working - my C++ code sends the signal, the Python code receives it and calls the handler, however, the pause/sleep calls never return.
If it is possible to correctly signal the same process, how would you do it?
(and if this is just dumb forgive me and move on)
Signals are not the right tool for the job here. Normally, this would be a job for inter-thread synchronization primitives, such as
the locks from the thread module
the Event objects from the threading module
However, it's not easy to manipulate Python thread locks from C++. So I would use the old-fashioned, but very simple, approach of
a pipe, which the Python thread reads from, and the C++ thread writes exactly one byte to when it wants to wake up the Python.
If you are in the same program I am not sure what you would need signals for in this case. Take a look at the observer pattern. If you have your python event handlers subscribe to events in your C++ library, you can avoid signals all together.