As I understand the relationship between C and C++, the latter is essentially an extension of the former and retains a certain degree of backwards compatibility. Is it safe to assume that the python C API can be called with C++ code?
More to the point, I notice that the official python documentation bundles C and C++ extensions together on the same page. Nowhere am I able to find a C++ API. This leads me to believe that the same API is safe to use in both languages.
Can someone confirm or deny this?
EDIT:
I think I made my question much more complicated than it needs to be. The question is this: what must I do in order to write a python module in C++? Do I just follow the same directions as listed here, substituting C code for C++? Is there a separate API?
I can confirm that the same Python C API is safe to be used in both languages, C and C++.
However, it is difficult to provide you with more detailed answer, unless you will ask more specific question. There are numerous caveats and issues you should be aware of. For example, your Python extensions are defined as C types struct, not as C++, so don't expect to have their constructor/destructor implicitly defined and called.
For example, taking the sample code from Defining New Types in the Python manual, it can be written in C++ way and you can even blend-in C++ types:
// noddy.cpp
namespace {
struct noddy_NoddyObject
{
PyObject_HEAD
// Type-specific fields go here.
std::shared_ptr<int> value; // WARNING
};
PyObject* Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
try {
Noddy *self = (Noddy *)type->tp_alloc(type, 0);
if (self) {
self->value = std::make_shared(7);
// or more complex operations that may throw
// or extract complex initialisation as Noddy_init function
return self;
}
}
catch (...) {
// do something, log, etc.
}
return 0;
}
PyTypeObject noddy_NoddyType =
{
PyObject_HEAD_INIT(NULL)
// ...
}
} // unnamed namespace
But, neither constructor nor destructor of the std::shared_ptr will be called.
So, remember to define dealloc function for your noddy_NoddyType where you will reset the value with nullptr. Why even bother with having value defined as shared_ptr, you may ask. It is useful if you use your Python extension in C++, with exceptions, to avoid type conversions and casts, to have more seamless integration inside definitions of your implementation, error handling based on exception may be easier then, etc.
And in spite of the fact that your objects of the noddy_NoddyType are managed by machinery implemented in pure C, thanks to dealloc function the value will be released according to well-known RAII rules.
Here you can find interesting example of nearly seamless integration of Python C API with the C++ language: How To catch Python stdout in c++ code
Python C API can be called within C++ code.
Python C++ extensions are written using the same C API as C extensions use, or using some 3rd party API, such as boost::python.
Related
I am working on a plugin system to replace shared libs.
I am aware of ABI issues when designing an API for shared libs and entry points in the libs, such as exported classes, should be carefully design.
For example, adding, removing or reordering private member variables of an exported class may lead to different memory layout and runtime errors (from my understanding, that's why the Pimpl pattern might be useful). Of course there are many other pitfalls to avoid when modifying exported classes.
I have built a small example here to illustrate my question.
First, i provide the following header for the plugin developer :
// character.h
#ifndef CHARACTER_H
#define CHARACTER_H
#include <iostream>
class Character
{
public:
virtual std::string name() = 0;
virtual ~Character() = 0;
};
inline Character::~Character() {}
#endif
Then the plugin is built as a shared lib "libcharacter.so" :
#include "character.h"
#include <iostream>
class Wizard : public Character
{
public:
virtual std::string name() {
return "wizard";
}
};
extern "C"
{
Wizard *createCharacter()
{
return new Wizard;
}
}
And finally the main application that uses the plugin :
#include "character.h"
#include <iostream>
#include <dlfcn.h>
int main(int argc, char *argv[])
{
(void)argc, (void)argv;
using namespace std;
Character *(*creator)();
void *handle = dlopen("../character/libcharacter.so", RTLD_NOW);
if (handle == nullptr) {
cerr << dlerror() << endl;
exit(1);
}
void *f = dlsym(handle, "createCharacter");
creator = (Character *(*)())f;
Character *character = creator();
cout << character->name() << endl;
dlclose(handle);
return 0;
}
Is it sufficient to define an abstract class to get rid of all ABI issues?
Is it sufficient to define an abstract class to get rid of all ABI issues?
Short answer:
No.
I wouldn't recommend using C++ for a plugin API (see longer answer below), but if you do decide to stick with C++ then:
Don't use any standard library types in your plugin API.
For instance, Character::name() returns a std::string. If the implementation of std::string ever changes (and it has in the past in GCC) then that will result in Undefined Behavior. Really, anything that you don't control (any third-party library) shouldn't be used in the API.
Don't use exceptions or RTTI across the plugin boundary. On Linux exceptions and RTTI might work if you load the plugin with RTLD_GLOBAL (not a good idea for plugins) and both the host and the plugin use the same runtime. But in general you either won't be able to catch exceptions from another module, or they might even cause heap corruption (if they are allocated by different runtimes).
Only add functions to the end of your abstract classes, or everything will silently break because of the vtable layout changing (and that can be really hard to diagnose).
Always allocate and deallocate an object from the same module. I noticed you don't have a destroyCharacter() function (main() actually leaks the character but that's another question). Always provide symmetric create and destroy functions for resources created by a different module (shared library or plugin).
I believe on Linux with GCC the host application's operator new and operator delete get correctly propagated to the loaded plugin (through weak symbols), but if you want it to work on Windows then don't assume that operator new and operator delete in the host application and the plugin are the same. A statically linked runtime, especially built with LTO, might also mess with this.
Longer answer:
There are a lot of possible issues when exporting a C++ API from a plugin.
Generally speaking, there are no guarantees about it working if anything about the toolchains used to build the host application and the plugin differs. This can include (but is not limited to) compilers, versions of the language, compiler flags, preprocessor definitions, etc.
The common wisdom regarding plugins is to use a pure C89 API, because the C ABI on all common platforms is very stable.
Keeping to the common subset of C89 and C++ will mean that the host and plugin can use different language standards, standard libraries, etc. Unless the host or the plugin are built with some weird (and probably non-standard-conforming) APIs, this should be reasonably safe. Obviously, you still have to be careful with data structure layouts.
You can then provide a rich C++ header-only wrapper for the C API that handles lifetime and error code/exception conversions, etc.
As a nice bonus, C APIs are producible and consumable by most languages, which could allow the plugin authors to use not just C++.
There are actually quite a few pitfalls even in a C API. If we're being pedantic then the only safe things are functions with fixed-size arguments and return types (pointers, size_t, [u]intN_t) - not even necessarily built-in types (short, int, long, ...), or enums. E.g. in GCC: -fshort-enums can change the size of enums, -fpack-struct[=n] can change the padding within structs.
So, if you really want to be safe then don't use enums and either pack all your structs or don't expose them directly (instead expose accessor functions).
Other considerations:
These aren't strictly related to the question but should definitely be considered before committing to a specific style of API.
Error handling: Whether or not you use C++, you'll need an alternative to exceptions.
This will probably be some form of error code. std::error_code in C++ can be then used to wrap the raw enum/int as soon as you're in C++ land, and if the API uses C++ then a std::expected-like or Boost.Outcome-like type with a stable ABI could be used.
Loading the plugin and importing symbols: With abstract classes it's easy - a simple factory function is all you need. With a traditional C API you might end up needing to import hundreds of symbols. One way of dealing with that would be to emulate a vtable in C. Make each object that has associated functions start with a pointer to a dispatch table, e.g.
typedef struct game_string_view { const char *data; size_t size; } game_string_view;
typedef enum game_plugin_error_code { game_plugin_success = 0, /* ... */ } game_plugin_error_code;
typedef struct game_plugin_character_impl *GamePluginCharacter; // handle to a Character
typedef struct game_plugin_character_dispatch_table { // basically a vtable
void (*destroy)(GamePluginCharacter character); // you could even put destroy() here
game_string_view (*name)(GamePluginCharacter character);
void (*update)(GamePluginCharacter character, /*...*/, game_plugin_error_code *ec); // might fail
} game_plugin_character_dispatch_table;
typedef struct game_plugin_character_impl {
// every call goes through this table and takes GamePluginCharacter as it's first argument
const game_plugin_character_dispatch_table *dispatch;
} game_plugin_character_impl;
Future extensibility and compatibility: You should design the API, knowing that you'll want to change it in the future and keep compatibility. IMO, a C API lends itself well to this because it forces you to be very precise in what is exposed. The plugin should be able to expose it's API version to the host in a way that is forward and backward compatible.
It's a good idea to think about extensibility when designing each function signature. E.g. if a struct is passed by pointer (instead of by value), then it's size can be extended without breaking compatibility (so long as at run time both the caller and the callee agree on it's size).
Visibility: Maybe look into visibility on Linux and other platforms. This isn't really a question of API design, just helps clean up the symbols exported from a shared library.
All of the above is by no means extensive.
I would suggest the talk "Hourglass Interfaces for C++ APIs" as further "reading".
And of course there other good talks and articles on the matter (that I can't remember of the top of my head).
Since I am fairly new to Objective-C programming language, I'm facing a huge problem: how can I call a method of my application (made in Objective-C) from my dynamically loaded library (made in C++), by passing it some parameters?
I've been digging the web for a couple of days with no luck. The only thing I found is about IMP, but I'm not sure if that's what I need.
You actually have a plethora of options here.
If the dylib is one of your own, you can compile it as Objective-C++ for Mac OS X and #ifdef the objective-C calls (or not if you are only targeting Mac OS)
Both Obj-C and C++ can make use of C interfaces, so you could write an adapter layer in C (remember Obj-c is a strict superset of C) and expose it for the dylib to call the C functions which then call the Obj-C code.
You can use Blocks, which work in C, C++, and of course Obj-C
you can include the objective-c runtime (see documentation) and muck with that (This is where you would use the *IMP thing you mentioned).
Yet another option might be to use Objective C++ from the Cocoa side to setup C++ objects and expose those to the dylib. How you would go about this really depends on what the library is and how it is used etc; we need to know more about the dylib and how it is being used to elaborate on this.
Since you specifically mention using an IMP lets talk a bit more in depth about that. The declaration is typedef void (*IMP)(id self, SEL _cmd, ...); which you can see takes a pointer to an Obj-C objects, and a SEL (selector), which is just a special C-String representation of the method. You can read more about both SEL and IMP in the documentation.
You can also make use of the runtime's C functions such as objc_msgSend to call a method by passing a pointer to the object and a SEL just like with IMP.
This should be enough information to get you started. Thanks for this question BTW, I never really sat down and thought about all the possible ways to combine C++ with Objective-C before. Odds are I even missed something ;)
You can use objective c runtime
include <objc/runtime.h>
objc_msgSend(id, SEL, arg0, ...)
where
id - is the object where you want to send message
SEL - is struct pointer, describing message you send.
arg0,... are the arguments that you pass to selector.
For more understanding of runtime, see the source code http://www.opensource.apple.com/source/objc4/
Also you can cast IMP address, and call the function.
int(* foo)(id, SEL, arg) = IMP;
foo(someObject, #selector(someMessage), arg);
When we write a program in C, it is possible that we will call some libraries that were wrote in C++ but had a C interface. Then it may happen that when we call these libraries, a C++ exception will occur. So my question is how we can handle this situation. I am more interested in this problem from a C++ developer's perspective. Suppose I am developing a C++ library that will be invoked by a C program, should I stop using exception and instead return error codes? Another situation is that if I have already a fully developed C++ library that uses exception, how can I transfer this library in a quick way that will only use error returning method?
You have to catch all exceptions on the C++ side and convert them to appropriate error returns in C, which may include specific error codes where appropriate. This doesn't mean that you stop using exceptions — you can still use them in C++ — but you can't expose them to C, they become an implementation detail.
A typical wrapper can be structured as follows:
// thingy-wrapper.h, typically included from C code:
#ifdef __cplusplus
extern "C" {
#endif
// create a typedef visible to C that doesn't expose the layout of
// the implementation.
typedef void *thingy_t;
// wrapper for std::string Thingy::get_name()
char *thingy_get_name(thingy_t t);
#ifdef __cplusplus
}
#endif
// thingy-wrapper.cpp, implements the wrapper, compiled with C++:
#include <thingy-wrapper.h>
#include <thingy.hpp> // declares Thingy class
char *thingy_get_name(thingy_t t_)
{
try {
Thingy& t = *static_cast<Thingy*>(t_);
std::string name = t.get_name();
return strdup(name.c_str());
}
catch(...) {
return NULL;
}
}
In this simple example, the caller of thingy_get_name can detect that an error occurred, but cannot find out the details of the error. A more realistic example would catch specific exceptions, and set a last_error variable to an error code or message before returning NULL. ... would only be caught as a last resort, and would set last_error to a generic UNKNOWN_ERROR value. A separate API for querying the last error, such as thingy_last_error(), would be available for the more careful callers of thingy_get_name().
The separation between error and non-error results enables code that doesn't care about the cause of errors to simply check if it received NULL, while allowing more conscientious code to properly propagate or report the error. If your library is multi-threaded, make sure that last_error uses thread-local storage.
Then it may happen that when we call these libraries, a C++ exception will occur. So my question is how we can handle this situation.
The C code cannot handle this situation. C code cannot deal with C++ exceptions.
Suppose I am developing a C++ library that will be invoked by a C program, should I stop using exception and instead return error codes?
No. If you want the C++ library to be consumed by C++ code you should use native C++ error handling. Which means exceptions.
However, the interface that you expose to the C code must not throw exceptions. Typically this means writing an adaptor layer that catches exceptions raised by the C++ library, and converts them into error codes to be consumed by the C code.
Another situation is that if I have already a fully developed C++ library that uses exception, how can I transfer this library in a quick way that will only use error returning method?
There's really no shortcut here. You have to write the adaptor that converts C++ exceptions into C error codes. You'll be writing an adaptor anyway if you want the library to expose interfaces for both C and C++ consumers. So the aspect of error handling is just another thing to take care of with this adaptor.
Exception are not caught in C so if you want to catch them then in your C code or your C++ code you have to write the wrappers very carefully.
Also make sure that in your C++ functions you have your functions declared as:
extern "C"
You may also check How to mix C and C++
is it possible to wrap a c++ library into c?
how could i do this?
are there any existing tools?
(need to get access to a existing c++ library but only with C)
You can write object-oriented code in C, so if it's an object-oriented C++ library, it's possible to wrap it in a C interface. However, doing so can be very tedious, especially if you need to support inheritance, virtual functions and such stuff.
If the C++ library employs Generic Programming (templates), it might get really hairy (you'd need to provide all needed instances of a template) and quickly approaches the point where it's just not worth doing it.
Assuming it's OO, here's a basic sketch of how you can do OO in C:
C++ class:
class cpp {
public:
cpp(int i);
void f();
};
C interface:
#ifdef __cplusplus
extern "C" {
#endif
typedef void* c_handle;
c_handle c_create(int i)
{
return new cpp(i);
}
void c_f(c_handle hdl)
{
static_cast<cpp*>(hdl)->f();
}
void c_destroy(c_handle hdl)
{
delete static_cast<cpp*>(hdl);
}
#ifdef __cplusplus
}
#endif
Depending on your requirements, you could amend that. For example, if this is going to be a public C interface to a private C++ API, handing out real pointers as handles might make it vulnerable. In that case you would hand out handles that are, essentially, integers, store the pointers in a handle-to-pointer map, and replace the cast by a lookup.
Having functions returning strings and other dynamically sized resources can also become quite elaborate. You would need the C caller provide the buffer, but it can't know the size before-hand. Some APIs (like parts of the WIn32 API) then allow the caller to call such a function with a buffer of the length 0, in which case they return the length of the buffer required. Doing so, however, can make calling through the API horribly inefficient. (If you only know the length of the required buffer after the algorithm executed, it needs to be executed twice.)
One thing I've done in the past is to hand out handles (similar to the handle in the above code) to internally stored strings and provide an API to ask for the required buffer size, retrieve the string providing the buffer, and destroy the handle (which deletes the internally stored string).
That's a real PITA to use, but such is C.
Write a c++ wrapper that does an extern c, compile that with c++, and call your wrapper.
(don't “extern c”)
extern C only helps you to have a names in dll like you see them.
You can use
dumpbin /EXPORTS your.dll
to see what happens with names with extern C or without it.
http://msdn.microsoft.com/en-us/library/c1h23y6c(v=vs.71).aspx
To answer your question... It depends... But it is highly unlikely that you can use it without wrappings. If this C++ library uses just a simple functions and types you can just use it. If this C++ library uses a complex classes structure - probably you will be unable to use it from C without wrapping. It is because the internal of classes may be structured one way or another depending on many conditions (using inference with virtual tables or abstracting. Or in example complex C++ library may have its own object creation mechanisms so you HAVE to use it in the way it is designed or you will get unpredictable behavior).
So, I think, you have to prepare yourself for doing dome wrappings.
And here is a good article about wrapping C++ classes. It the article the Author tells about wrapping C++ classes to C# but he uses C at first step.
http://www.codeproject.com/KB/cs/marshalCPPclass.aspx
If the C++ library is written which can be compiled with C compiler with slight editting (such as changing bool to int, false to 0 and true to 1 etc), then that can be done.
But not all C++ code can be wrapped in C. Template is one feature in C++ that cannot be wrapped, or its nearly impossible.
Wrap it in C++ cpp that calls that dll, and "extern C" in that file you made.
I've got a library written in C++ which I wrap using SWIG and use in python. Generally there is one class with few methods. The problem is that calling these methods may be time consuming - they may hang my application (GIL is not released when calling these methods). So my question is:
What is the simplest way to release GIL for these method calls?
(I understand that if I used a C library I could wrap this with some additional C code, but here I use C++ and classes)
Not having any idea what SWIG is I'll attempt an answer anyway :)
Use something like this to release/acquire the GIL:
class GILReleaser {
GILReleaser() : save(PyEval_SaveThread()) {}
~GILReleaser() {
PyEval_RestoreThread(save);
}
PyThreadState* save;
};
And in the code-block of your choosing, utilize RAII to release/acquire GIL:
{
GILReleaser releaser;
// ... Do stuff ...
}
The real problem is that SWIG is not documented well (I saw hints to use changelog for searching ;) ).
Ok, I found out that I can do inline functions in SWIG and use macros to release/acquire GIL, it looks like this:
%inline %{
void wrappedFunction(OriginalObject *o, <parameters>) {
Py_BEGIN_ALLOW_THREADS
o->originalFunction(<parameters>);
Py_END_ALLOW_THREADS
}
%}
This function is not present in original C++, but available in python module. This is (almost) exactly what I wanted. (what I would like is to wrap original method like python decorator does)
You can use the same API call as for C. No difference. Include "python.h" and call the appoproate function.
Also, see if SWIG doesn't have a typemap or something to indicate that the GIL shuold not be held for a specific function.