I'm trying to use C++ to wrap some simple C++ and encountered an issue when trying to wrap even the most basic stuff.
SWIG doesn't seem to try and catch and exception that might occurr inside a typemap.
For example for the following SIWG interface file:
%module badswig
std::string func();
SWIG Generates this:
SWIGINTERN PyObject *_wrap_func(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
std::string result;
if (!PyArg_ParseTuple(args,(char *)":func")) SWIG_fail;
result = func();
resultobj = SWIG_NewPointerObj((new std::string(static_cast< const std::string& >(result))), SWIGTYPE_p_std__string, SWIG_POINTER_OWN | 0 );
return resultobj;
fail:
return NULL;
}
Notice that new std::string is unprotected at all so that if std::bad_alloc will be thrown it will propogate to Python and crash.
There are likely other cases as well not only SWIG using new.
There is %exception which handles this for the wrapped function (If func() throws) but I couldn't find anything to handle this for typemaps, especially for ones supplied by SWIG.
Am I missing something? is there someway to correctly handle this?
EDIT:
In response to an answer and to clarify my question, %exception does not achive what I want:
%module badswig
%exception %{
try {
$action
} catch (const std::bad_alloc&) {
PyErr_NoMemory();
SWIG_fail;
}
%}
std::string func();
Generates this:
SWIGINTERN PyObject *_wrap_func(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
std::string result;
if (!PyArg_ParseTuple(args,(char *)":func")) SWIG_fail;
try {
result = func();
} catch (const std::bad_alloc&) {
PyErr_NoMemory();
SWIG_fail;
}
resultobj = SWIG_NewPointerObj((new std::string(static_cast< const std::string& >(result))), SWIGTYPE_p_std__string, SWIG_POINTER_OWN | 0 );
return resultobj;
fail:
return NULL;
}
Notice how new std::string has no try catch around it.
The typemap just tells SWIG what interface code to generate to convert call parameters and return value of the function. The %exception generates code immediately before and after the function to be wrapped, to transition exceptions from C++ to target language or to convert the C++ exception to the target language equivalent. If you use %exception as per the SWIG docs, you should find that your wrapper function contains the typemap code you specified for that function, and the exception code around your C++ function:
void swig_function(PyObj*) {
typemap-related code to convert args from Python to C++
exception-related code
your-c++-func(args)
exception-related code (catch clauses etc)
typemap-related code to convert return value from C++ to Python
}
Related
I have a native C++ library and a SWIG-generated wrapper for C#. the .i file contains an %exception that wraps all the $actions inside regular try/catchblocks to convert exceptions to our desired target C# exceptions using SWIG_CSharpSetPendingExceptionCustom. Something like this
%exception {
try
{
$action
}
catch(const our::Exception& exc)
{
//blahblah
SWIG_CSharpSetPendingExceptionCustom(/*blah*/);
return $null;
}
catch(const std::exception& exc)
{
/*blah*/
}
catch(...)
{
/*blah*/
}
}
This works great, all regular C++ exceptions are nicely wrapped and forwarded to C#.
But what I also need is to wrap all my function with the SEH __try/__except blocks in order to catch potential access violations and whatnot and print the native stack trace. Something like
%exception {
__try
{
$action
}
__except(print_my_stack_trace(GetExceptionInformation()))
{
}
}
Unfortunately it is not allowed to combine regular try with SEH __try in one function in any way. So if my library has an arbitrary function myclass::f, what I would like to achieve is for SWIG to first create a f_temp function whose body will contain try/catch, then the true wrapper of f that would call f_temp wrapped with __try/__catch. The result for f would look something like
void SWIGSTDCALL CSharp_blah_myclass_f_TEMP__(void * jarg1) {
myclass *arg1 = (myclass *) 0 ;
arg1 = (myclass*)jarg1;
{
try
{
(arg1)->f();
}
catch(const our::Exception& exc)
{
...
SWIG_CSharpSetPendingExceptionCustom(...);
return ;
}
catch(const std::exception& exc)
{
...
}
}
SWIGEXPORT void SWIGSTDCALL CSharp_blah_myclass_f__(void * jarg1) {
__try
{
CSharp_blah_myclass_f_TEMP__(jarg1)
}
__except(print_my_stack)
{
}
}
I am thinking of resorting to writing a perl script that would manipulate the swig-generated wrapper .cpp file textually, but I first want to make sure that this is not possible to achieve with swig magic. So, can anyone think of a SWIG solution that does not involve ad-hoc manipulation of generated sources?
I am writing analyser for natural language and I have a wrapper of c++ code in python 3 created with swig. I'd like to use a function which is some kind of stream writer and it takes std::ostream & os as the parameter. So I guess it would work if I somehow import ostringstream(read as which lib.so I should use in my ctypes.CDLL) in my python code then pass it to this function, lest call it create_stream_writer(stream), and then use stream.str() to get string. Is it any way to do this using ctypes or any other library?
I am using docker container running Ubuntu 18.04, python3.6
code should look like this I guess:
def analyse(text, config):
reader = PlainTextReader.create_string_reader(text, config)
stream = ctypes.ostringstream() # some magic hear
writer = TokenWriter.create_stream_writer('plain', stream, reader.tagset())
for sentence in sentences(reader):
writer.write_sentence(sentence)
return stream.str()
You can do this (and make it nice for Python developers too). This answer is essentially a Python 3 version of my older answer on wrapping iostreams.
To simplify things here I used boost's iostreams library. If you can't/don't use boost then you can write this all from standard C++ library components, it's just far more verbose.
I've also aimed higher than mapping io.StringIO to std::stringstream and instead gone for mapping any 'file like' Python object to any iostream. That is to say we use aim to use duck typing on the Python object to just call read() and write() sensibly as and when needed for our C++ stream objects.
%module test
%{
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/categories.hpp>
// This is just a helper that we can use with boost iostreams to proxy everything to a Python object
struct python_stream_device {
typedef char char_type;
typedef boost::iostreams::bidirectional_device_tag category;
std::streamsize read(char* s, std::streamsize n) {
PyObject *p = PyObject_CallMethod(o, "read", "l", static_cast<long int>(n));
if (PyErr_Occurred()) {
// TODO: throw a C++ exception to back out of wherever we are and then re-throw the Python one...
assert(false);
}
assert(p);
char *ptr = nullptr;
Py_ssize_t len = 0;
PyObject *str = PyUnicode_AsUTF8String(p);
PyBytes_AsStringAndSize(str, &ptr, &len);
if (PyErr_Occurred()) {
assert(false); // Let's just pretend this is error handlng...
}
memcpy(s, ptr, len);
Py_DECREF(str);
Py_DECREF(p);
return len;
}
std::streamsize write(const char* s, std::streamsize n) {
PyObject *ret = PyObject_CallMethod(o, "write", "s#", s, static_cast<Py_ssize_t>(n));
if (PyErr_Occurred()) {
// See above
assert(false);
}
std::streamsize r = PyLong_AsSsize_t(ret);
Py_DECREF(ret);
return r;
}
// Using this means we can rely on the default synthesised operator= + copy ctor etc. and saves us some code.
swig::SwigPtr_PyObject o;
python_stream_device(PyObject *o) : o(o) {}
};
typedef boost::iostreams::stream<python_stream_device> python_stream;
%}
// Here is the stuff that wraps it neatly
%typemap(in) std::iostream& (python_stream tmp) {
// Writing the typemap this way lets us get RAII semantics despite the goto in the SWIG macros in the simplest way
tmp.open(python_stream_device($input));
$1 = &tmp;
}
// We can just use the same typemaps for other cases too:
%apply std::iostream& { std::istream&, std::ostream& };
// Below is just for testing:
%{
#include <iostream>
%}
%inline %{
// This is the function you want to call
void fun1(std::ostream& out) {
assert(out.good());
out << "Hello world, from C++";
assert(out.good());
}
// This one is here for completeness because once you've got this far you may as well support this too.
void fun2(std::istream& in) {
std::string tmp;
//in >> tmp;
std::getline(in, tmp);
assert(in.good());
std::cout << "fun2 got: " << tmp << std::endl;
}
%}
This is enough that you can then use some Python like this:
import io
import test
i=io.StringIO()
test.fun1(i)
print('After fun1: %s' % i.getvalue())
i=io.StringIO('hello world, from Python!\n')
test.fun2(i)
As Mark Tolonen pointed in comments it is impossible to do this with ctypes. So I just wrote c++ function that do everything I need, and then created a wrapper using SWIG. Because using typemap in SWIG to map StringIO(Python) to ostreingstream(C++) looks like black magic and I couldn't find the way to do this.
I have some trouble handling custom C++ exceptions when calling from Cython.
My situation is the following: I have a library that uses CustomLibraryException for all exceptions. What I want is basically get the error message and raise a Python error with it.
The user guide has some hints but it is a bit unspecific.
The first possibility is to do:
cdef int bar() except +ValueError
This converts the CustomLibraryException to a ValueError, but loses the error message.
The other possibility is to explicitly convert the error using
cdef int raise_py_error()
cdef int something_dangerous() except +raise_py_error
I don't really understant this solution. I understood that raise_py_error has to be a C++ function that somehow handles the error. I am not sure how to handle it though. The function doesn't get an argument and is called inside the catch block in C++.
If any one has an working example of handling a C++ exception in Cython, that would be of great help.
Agreed the wording in the doc page leaves something to be desired. While "Cython cannot throw C++ exceptions", here is a raise_py_error that does what we want.
First, define the custom exception class in cython and make a handle to it using the "public" keyword
from cpython.ref cimport PyObject
class JMapError(RuntimeError):
pass
cdef public PyObject* jmaperror = <PyObject*>JMapError
Then write the exception handler (the docs aren't super clear this must be written in C++ and imported):
#include "Python.h"
#include "jmap/cy_utils.H"
#include "jmap/errors.H"
#include <exception>
#include <string>
using namespace std;
extern PyObject *jmaperror;
void raise_py_error()
{
try {
throw;
} catch (JMapError& e) {
string msg = ::to_string(e.code()) +" "+ e.what();
PyErr_SetString(jmaperror, msg.c_str());
} catch (const std::exception& e) {
PyErr_SetString(PyExc_RuntimeError, e.what() );
}
}
Finally, bring the handler into cython with an extern block, and use it:
cdef extern from "jmap/cy_utils.H":
cdef void raise_py_error()
void _connect "connect"() except +raise_py_error
Done. I now see new exception, constructed with the error code as intended:
JMapError: 520 timed connect failed: Connection refused
The default C++ exception handler in Cython should illustrate exactly how to accomplish what you are trying to do:
static void __Pyx_CppExn2PyErr() {
// Catch a handful of different errors here and turn them into the
// equivalent Python errors.
try {
if (PyErr_Occurred())
; // let the latest Python exn pass through and ignore the current one
else
throw;
} catch (const std::bad_alloc& exn) {
PyErr_SetString(PyExc_MemoryError, exn.what());
} catch (const std::bad_cast& exn) {
PyErr_SetString(PyExc_TypeError, exn.what());
} catch (const std::domain_error& exn) {
PyErr_SetString(PyExc_ValueError, exn.what());
} catch (const std::invalid_argument& exn) {
PyErr_SetString(PyExc_ValueError, exn.what());
} catch (const std::ios_base::failure& exn) {
// Unfortunately, in standard C++ we have no way of distinguishing EOF
// from other errors here; be careful with the exception mask
PyErr_SetString(PyExc_IOError, exn.what());
} catch (const std::out_of_range& exn) {
// Change out_of_range to IndexError
PyErr_SetString(PyExc_IndexError, exn.what());
} catch (const std::overflow_error& exn) {
PyErr_SetString(PyExc_OverflowError, exn.what());
} catch (const std::range_error& exn) {
PyErr_SetString(PyExc_ArithmeticError, exn.what());
} catch (const std::underflow_error& exn) {
PyErr_SetString(PyExc_ArithmeticError, exn.what());
} catch (const std::exception& exn) {
PyErr_SetString(PyExc_RuntimeError, exn.what());
}
catch (...)
{
PyErr_SetString(PyExc_RuntimeError, "Unknown exception");
}
}
So you can either #define __Pyx_CppExn2PyErr your_custom_exn_handler in an included .h file to override the generic behavior, or use a one-off custom handler as
cdef extern from "...":
void your_exn_throwing_fcn() except +your_custom_exn_handler
If CustomLibraryException derives from std::runtime_error (as a well-behaved C++ exception should), then the behavior you're seeing is a bug in Cython.
If it doesn't, then the easiest thing to do is to wrap the C++ function you're calling in a C++ helper function that translates the exception:
double foo(char const *, Bla const &); // this is the one we're wrapping
double foo_that_throws_runtime_error(char const *str, Bla const &blaref)
{
try {
return foo(str, blaref);
} catch (CustomLibraryException const &e) {
throw std::runtime_error(e.get_the_message());
}
}
This will cause a RuntimeError to be raised on the Python side. Alternatively, throw an std::invalid_argument to raise a ValueError, etc. (see the table in the page you linked to).
In Cython's sources, https://github.com/cython/cython/blob/master/tests/run/cpp_exceptions.pyx they actually implement the raise_py_error in a .pyx file. This makes it a lot easier to share error handling between other .pyx files.
A quick solution involves simply 2 files : myerror.pyx :
class MyError(RuntimeError):
"Base class for errors raised from my C++."
pass
cdef int raise_my_py_error() except *:
raise MyError("There was a problem")
and myerror.pxd :
cdef int raise_my_py_error() except *
which allows you to add an except +my_py_error in all your files.
However, this "loses" the e.what() of C++ exceptions. So a more interesting solution needs a couple more helper files :
my_error_helper.h :
extern const char* get_my_py_error_message();
my_error_helper.cxx :
#include <exception>
const char* get_my_py_error_message()
{
try {
throw;
} catch (const my_custom_cxx_exception& e) {
return e.what();
}
}
my_error_helper.pxd :
cdef extern from "my_error_helper.h":
const char* get_my_py_error_message()
my_error.pxd :
cdef int raise_my_py_error() except *
my_error.pyx :
cimport my_error_helper
class MyError(RuntimeError):
"Base class for errors raised from my C++."
pass
cdef int raise_my_py_error() except *:
msg = my_error_helper.get_my_py_error_message().decode('utf-8')
raise MyError(msg)
I'm trying to add embedded python to a OpenGL/SDL application.
So far everything works fine, like entering a string via SDL keyboard events and executing it with the embedded python interpreter.
I'm now into adding functions to call C/C++ functions like
void set_iterations(int c);
is invoked on the python interpreter with
>>> test.iterations(23)
The parsing of the integer parameter works like charm
static PyObject* test_iterations(PyObject *self, PyObject *args) {
int iterations;
PyArg_ParseTuple(args,"i",&iterations);
set_iterations(iterations);
return Py_None;
}
But when I try this: >>> test.addignore('something')
static PyObject* test_addignore(PyObject *self, PyObject *args) {
char* ignorestring;
//PyArg_ParseTuple(args,"s",ignorestring);
PyArg_ParseTuple(args,"s",&ignorestring);
add_global_ignore(ignorestring); // C function
return Py_None;
}
Python gives me this error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-3: invalid data
The string should be UTF-8, as SDL is set to grab UNICODE from the keyboard, and everything else works perfectly.
Does anyone have an idea on what I might be doing wrong here?
I also inspected the args object passed to the function with
std::string args_str;
PyObject* repr = PyObject_Repr(args);
if (repr != NULL) {
args_str = PyBytes_AsString(PyUnicode_AsEncodedString(repr, "utf-8", "Error"));
Py_DECREF(repr);
}
std::cout << args_str << "\n";
And it gives me this: ('somestring',)
Solution:
The error pointed out by rodrigo, was originally causing be believe that my debug code that should print the resulting string as PyObject was wrong. But the problem was that I was passing the parser the wrong pointer, leading to said undefined behaviour in the memory and therefore leading me to believe the parser was the problem.
The occurring last parser error was then the debug output itself, which was pointing to the wrong memory address.
Thanks rodrigo, since your answer lead to solving the problem: accepted. Thank you for your help and patience.
Try:
static PyObject* test_addignore(PyObject *self, PyObject *args) {
char* ignorestring;
if (!PyArg_ParseTuple(args,"s", &ignorestring)) //Note the &
return NULL; //Never ignore errors
add_global_ignore(ignorestring);
Py_RETURN_NONE; //Always use helper macros
}
I want to embed python in my C++ application. I'm using Boost library - great tool. But i have one problem.
If python function throws an exception, i want to catch it and print error in my application or get some detailed information like line number in python script that caused error.
How can i do it? I can't find any functions to get detailed exception information in Python API or Boost.
try {
module=import("MyModule"); //this line will throw excetion if MyModule contains an error
} catch ( error_already_set const & ) {
//Here i can said that i have error, but i cant determine what caused an error
std::cout << "error!" << std::endl;
}
PyErr_Print() just prints error text to stderr and clears error so it can't be solution
Well, I found out how to do it.
Without boost (only error message, because code to extract info from traceback is too heavy to post it here):
PyObject *ptype, *pvalue, *ptraceback;
PyErr_Fetch(&ptype, &pvalue, &ptraceback);
//pvalue contains error message
//ptraceback contains stack snapshot and many other information
//(see python traceback structure)
//Get error message
char *pStrErrorMessage = PyString_AsString(pvalue);
And BOOST version
try{
//some code that throws an error
}catch(error_already_set &){
PyObject *ptype, *pvalue, *ptraceback;
PyErr_Fetch(&ptype, &pvalue, &ptraceback);
handle<> hType(ptype);
object extype(hType);
handle<> hTraceback(ptraceback);
object traceback(hTraceback);
//Extract error message
string strErrorMessage = extract<string>(pvalue);
//Extract line number (top entry of call stack)
// if you want to extract another levels of call stack
// also process traceback.attr("tb_next") recurently
long lineno = extract<long> (traceback.attr("tb_lineno"));
string filename = extract<string>(traceback.attr("tb_frame").attr("f_code").attr("co_filename"));
string funcname = extract<string>(traceback.attr("tb_frame").attr("f_code").attr("co_name"));
... //cleanup here
This is the most robust method I've been able to come up so far:
try {
...
}
catch (bp::error_already_set) {
if (PyErr_Occurred()) {
msg = handle_pyerror();
}
py_exception = true;
bp::handle_exception();
PyErr_Clear();
}
if (py_exception)
....
// decode a Python exception into a string
std::string handle_pyerror()
{
using namespace boost::python;
using namespace boost;
PyObject *exc,*val,*tb;
object formatted_list, formatted;
PyErr_Fetch(&exc,&val,&tb);
handle<> hexc(exc),hval(allow_null(val)),htb(allow_null(tb));
object traceback(import("traceback"));
if (!tb) {
object format_exception_only(traceback.attr("format_exception_only"));
formatted_list = format_exception_only(hexc,hval);
} else {
object format_exception(traceback.attr("format_exception"));
formatted_list = format_exception(hexc,hval,htb);
}
formatted = str("\n").join(formatted_list);
return extract<std::string>(formatted);
}
In the Python C API, PyObject_Str returns a new reference to a Python string object with the string form of the Python object you're passing as the argument -- just like str(o) in Python code. Note that the exception object does not have "information like line number" -- that's in the traceback object (you can use PyErr_Fetch to get both the exception object and the traceback object). Don't know what (if anything) Boost provides to make these specific C API functions easier to use, but, worst case, you could always resort to these functions as they are offered in the C API itself.
This thread has been very useful for me, but I had problems with the Python C API when I tried to extract the error message itself with no traceback. I found plenty of ways to do that in Python, but I couldn't find any way to do this in C++. I finally came up with the following version, which uses the C API as little as possible and instead relies much more on boost python.
PyErr_Print();
using namespace boost::python;
exec("import traceback, sys", mainNamespace_);
auto pyErr = eval("str(sys.last_value)", mainNamespace_);
auto pyStackTrace = eval("'\\n'.join(traceback.format_exception(sys.last_type, sys.last_value, sys.last_traceback))", mainNamespace_);
stackTraceString_ = extract<std::string>(pyStackTrace);
errorSummary_ = extract<std::string>(pyErr);
The reason this works is because PyErr_Print() also sets the value for sys.last_value, sys.last_type, and sys.last_traceback. Those are set to the same values as sys.exc_info would give, so this is functionally similar to the following python code:
import traceback
import sys
try:
raise RuntimeError("This is a test")
except:
err_type = sys.exc_info()[0]
value = sys.exc_info()[1]
tb = sys.exc_info()[2]
stack_trace = "\n".join(traceback.format_exception(err_type, value, tb))
error_summary = str(value)
print(stack_trace)
print(error_summary)
I hope someone finds this useful!
Here's some code based on some of the other answers and comments, nicely formatted with modern C++ and comments. Minimally tested but it seems to work.
#include <string>
#include <boost/python.hpp>
#include <Python.h>
// Return the current Python error and backtrace as a string, or throw
// an exception if there was none.
std::string python_error_string() {
using namespace boost::python;
PyObject* ptype = nullptr;
PyObject* pvalue = nullptr;
PyObject* ptraceback = nullptr;
// Fetch the exception information. If there was no error ptype will be set
// to null. The other two values might set to null anyway.
PyErr_Fetch(&ptype, &pvalue, &ptraceback);
if (ptype == nullptr) {
throw std::runtime_error("A Python error was detected but when we called "
"PyErr_Fetch() it returned null indicating that "
"there was no error.");
}
// Sometimes pvalue is not an instance of ptype. This converts it. It's
// done lazily for performance reasons.
PyErr_NormalizeException(&ptype, &pvalue, &ptraceback);
if (ptraceback != nullptr) {
PyException_SetTraceback(pvalue, ptraceback);
}
// Get Boost handles to the Python objects so we get an easier API.
handle<> htype(ptype);
handle<> hvalue(allow_null(pvalue));
handle<> htraceback(allow_null(ptraceback));
// Import the `traceback` module and use it to format the exception.
object traceback = import("traceback");
object format_exception = traceback.attr("format_exception");
object formatted_list = format_exception(htype, hvalue, htraceback);
object formatted = str("\n").join(formatted_list);
return extract<std::string>(formatted);
}
Btw I was curious why everyone is using handle<> instead of handle. Apparently it disables template argument deduction. Not sure why you'd want that here but it isn't the same anyway, and the Boost docs say to use handle<> too so I guess there is a good reason.