I have a big C project and I want to expose it's API to python by using SWIG.
For that, I have a script for generating automatically the interface file for swig (.i file).
For each type defined in the project, the script adds to the interface file %pointer_functions and %array_functions for that type.
Those directives create set of functions including a "new" function which in turn allocates memory.
Is there a way to automatically deallocate that memory when the object is garbage collected instead of calling explicitly the "delete" function?
I have a solution for structs, using the %newobject directive and %extend with destructor definition which free the allocated memory - similar to the solution in this post SWIG - Garbage Collection using %newobject .
But it doesn't work on primitive types like int, enum, etc...
Any ideas how to solve it for such types?
Related
I've been reading the Apache Arrow docs, and I've figured out how to use it in Java and C++. But what I'd like to do is offload some work to JNI (C/C++) code from Java, and the documentation (e.g. https://arrow.apache.org/docs/java/cdata.html) just doesn't seem to cover my use cases, and methods in the example (e.g. getMemoryAddress on IntVector) just don't seem to exist like they do in the examples. I want to start simple, so here's what I'd like to do:
Allocate two Arrow IntVector's in Java and fill them with data
Allocate space for another IntVector in Java for the result
Get whatever native pointers I need from those vectors and pass them through a JNI call
Wrap those vectors in C++ so I can access them.
Do whatever work I want to offload and finalize the result vector
Return to Java and have the result accessible.
Can anyone point me to an example or some tips on how to do this?
BTW, the examples also use JavaCPP instead of JNI. But I already have a bunch of JNI code in this project, and I'd rather not mix in another kind of bridge if it's not necessary.
Thanks.
I tried allocating IntVector objects in Java, but I can't tell which naive pointers I have to retrieve to pass to C++ to provide proper access to those vectors.
JavaCPP is merely a convenience for the example, JNI is fine.
The C Data Interface is still what you want. When you say "get whatever native pointers I need": that is exactly what a struct ArrowArray is in the C Data Interface. Use the C Data Interface module in Java to export your Java arrays and get the address of a struct ArrowArray, and then pass that address to your C++ code via JNI. Then, use libarrow's C Data Interface implementation to import the arrays and work with them.
When the C++ side is done, it does the same thing: it exports the result vector and returns an address to Java via JNI; the Java code then imports the vector from that address.
---Context---
I want to have a class called "fileProcessor". This class is completely static and merely serves as a convinient namespace (within my normal library namespace) for some global function. This is a basic blueprint of the class with only the relevant stuff
class fileProcessor{
private:
lua_State* LUA_state;
public:
static std::variant<type1,type2> processFile(const char* filePath,const char* processorScript);
}
Please note again that I ommitted most of the stuff from the class so if anything seems odd ignore it.
What process file is supposed to do is:
Read the filePath file, storing all directives including it (this is my own filetype or style of syntax. This is already handeled correctly). The directives are stored with strings, one for the command and one for everything after it.
Read the script file and check if it has a commented out fileProcessor line at the top. This is to make sure that the lua script loaded is relevant and not some random behaviour script
Load and compile the lua script.
Make all read directives available (they are saved in a struct of 2 strings as mentioned before)
Run the file and recieve a object back. The object should only be of types that I listed in the return type (variant)
I am having problems with step 4 and one vital part of the scripting.
---Question---
How can I make the creation of a full new object of type1 or type2 possible within lua, write to it from within lua and then get it back from the lua stack into c++ and still know if its type1 or type2?
---No example provided since this question is more general and the only reason I provided my class is for context.---
It seems like you are trying to do it the other way around. I quote a part of this answer:
...you are expecting Lua to be the primary language, and C++ to be the client. The problem is, that the Lua C interface is not designed to work like that, Lua is meant to be the client, and all the hard work is meant to be written in C so that Lua can call it effortlessly.
If you are convinced there is no other way that doing it other way around you can follow the workaround that answer has given. Otherwise I think you can achieve what you need by using LUA as it meant to be.
LUA has 8 basic types (nil, boolean, number, string, userdata, function, thread, and table). But you can add new types as you require by creating a class as the new type in native C++ and registering it with LUA.
You can register by either:
Using some LUA helper for C++ like luna.h (as shown in this tutorial).
Pushing a new lua table with the C++ class (check this answer).
Class object instance is created in your native C++ code and passed to LUA. LUA then makes use of the methods given by the class interface.
In file gperftools-2.2.1/src/gperftools/malloc_extension.h, it reads:
// Extra extensions exported by some malloc implementations. These
// extensions are accessed through a virtual base class so an
// application can link against a malloc that does not implement these
// extensions, and it will get default versions that do nothing.
//
// NOTE FOR C USERS: If you wish to use this functionality from within
// a C program, see malloc_extension_c.h.
My question is how exactly can I access these extensions through a virtual base class?
Usually to load a class from a dynamic library, I would need to write a base class which allows me to get an instance of the wanted class and its functions through polymorphism, as described here.
However to do so there must be some class factory functions available in the API, but there are no such functions in any tcmalloc files. Moreover I would also need to load the tcmalloc library with dlopen(), which is not recommended according to the install note:
...loading a malloc-replacement library via dlopen is
asking for trouble in any case: some data will be allocated with one malloc, some with another.
So clearly accessing the extensions through the typical way as mentioned above is not an option. I can get away with using the C versions as declared in malloc_extensions_c.h but just wonder if there is any better solution.
I managed to load the malloc extensions via some 'hack', which is not as clean as I would prefer, but it gets the job done. Here is the (temporary) solution for those who are interested in.
First, most of these malloc extension functions are similar to static functions in a way that they are mostly called on the current instance only, e.g. to call the GetMemoryReleaseRate() function on the current process you just call MallocExtension::instance()->GetMemoryReleaseRate(). Therefore we don't need to create a base class and get an instance of MallocExtension class to call these functions.
For the example above, I'd just create a standalone function getMemoryReleaseRate() which simply calls the required function when it gets called, as below:
getMemoryReleaseRate()
{
MallocExtension::instance()->GetMemoryReleaseRate();
}
This function can be inserted directly to a source file, e.g. tcmalloc.cc, or, if you prefer not to edit the tcmalloc source every time there is a new version, added to your makefile, to be attached to the source file when it is compiled.
Now in your code, you can call the MallocExtension function via the 'facade' function you have created via dlsym(), e.g. as below:
typedef void (*getMemoryReleaseRate)();
((getMemoryReleaseRate)dlsym(RTLD_DEFAULT, "getMemoryReleaseRate"))();
Simply including this header and doing MallocExtension::instance()->GetMemoryReleaseRate(); would work too. No need to modify tcmalloc for that.
I'm exporting some c++ classes to Lua with SWIG. I have declared boost::filesystem::path in the SWIG interface file like this:
namespace boost
{
namespace filesystem
{
class path {};
}
}
Now I want to call a function declared in a lua script which should take a boost::filesystem::path& as parameter to pass it to another object. I only need to be able to pass the path to the object. I don't need to use any functionality from the path object.
function on_path_selected(the_path)
another_object:set_path(the_path)
end
I am going to call the Lua function from c++ using it's index.
lua_rawgeti(L, LUA_REGISTRYINDEX, m_function_index);
lua_push[SOMETHING](L, path_object); // <-- HOW TO ?
lua_pcall(L,1,0,0)
THE QUESTION: How to push a boost::filesystem::path as a parameter to the Lua function?
This is actually fairly complicated. The expected use of SWIG is to create modules for Lua. The Lua script should be the one deciding what gets called and what doesn't. It isn't really meant for embedded use, where you use SWIG to expose some C++ objects and then call Lua code directly from your application.
That's not to say that it's impossible, just complicated.
All SWIG-based C++ objects are passed through Lua as pointers. Thus ownership is a question; you can't just shove a pointer to a stack object into Lua.
The safest way to do this is to pass a new copy of the object to Lua. That way, Lua owns the pointer. SWIG will know that Lua owns the pointer, and will attach a proper garbage collection mechanism to it to clean it up. So everything should be fine, memory wise.
But doing this requires properly "boxing" (for want of a better term) that object the way that SWIG wants it done. This requires using certain SWIG macros.
Given how you have bound the path type to SWIG, you would do something like this to stick it onto a Lua stack:
swig_type_info *pathType = SWIG_TypeQuery("boost::filesystem::path *");
boost::filesystem::path *pArg = new boost::filesystem::path(the_path);
SWIG_NewPointerObj(L, pArg, pathType, 1);
SWIG_TypeQuery fetches the type of any object that has been bound by SWIG to Lua. This type info object is needed for SWIG_NewPointerObj, which takes a pointer to that type. Both of these are macros. SWIG_NewPointerObj gives Lua ownership of the pointer; Lua's garbage collector will delete it thanks to SWIG's metatables. Also SWIG_NewPointerObj pushes the object onto the lua_State stack.
Once it's on the stack, you can pretty much do whatever you want with it. Return it from a function to Lua, pass it to a Lua function as an argument, stick it in a global variable, etc. It's a Lua value.
Now, if you type this code into your project, odds are good that you'll get a compile error when the compiler sees swig_type_info. This type is defined internally within the source file generated by SWIG's command-line.
You have two options:
Put this source code into the .swig file itself. Yes, really. You can define regular C++ functions there, within verbatum sections (the %{ %} delimited blocks). These functions will be copied directly into SWIG's generated code. You can access them by putting prototypes in headers. This is the simplest and easiest way to work. This is often used for creating special interfaces, where a pre-existing C++ function isn't appropriate for a Lua API (or simply doesn't exist).
You can generate an appropriate header that contains these definitions with the -external-runtime argument. This has to be a different SWIG execution step from the step that generates the .cpp file. See, it doesn't actually process the SWIG file or anything. All it needs is the target language (-lua) and whether you're using C++ (-c++).So just have a command that does swig -c++ -lua -external-runtime someheader.h, and that's all you need to get the types and macros.
Include that header in whatever source you want to attach SWIG-bound objects to Lua in.
How is Python able to call C++ objects when the interpreter is C and has been built w/ a C compiler?
Boost.Python has special macros that declare functions with extern "C" so the Python interpreter will be able to call them. It's kind of complicated, but you can look at the Boost documentation for more info.
Python declares a C-API (see http://docs.python.org/2/c-api/ or http://docs.python.org/3/c-api/). This API defines a generic object type called PyObject which is just a normal C struct. This structure defines (nearly) everything a python object can do, e.g., what happens when do additions or comparisons on this object or simply call it like a function.
Because python types are also objects (and therefore are represented in C by a PyObject structure), defining a new type is a simple matter of defining a new PyObject struct like that one. When methods are called in Python, the interpreter forwards the call to C functions associated with this structure.
As long as a given (compiled) extension provides the correct entry points such that the Python interpreter can introspect it and find out what is available (the documentation I indicated above does explain this in details), then it can use these objects like any other object you normally have available at the prompt - which BTW, are constructed using the very same C-API. It suffices you import the compiled extension.
I hope it is somewhat clear how the Python interpreter calls stuff from compiled extensions from the above. The sole missing gap is how the C-API calls the C++ code.
Boost.Python does this by declaring C entry points in code along the lines as explained here: Elegantly call C++ from C. Every time you call, e.g., boost::python::class_, it does this for the type you declare to python, creating therefore a PyObject that represents your class, with the name you choose. As you call .def on this class you go filling in the internal slots of that structure, declaring more methods, operators and attributes of your new type. Each of these internal slots points to a C-style function that is nothing but a wrapper to the equivalent C++ call.
C++ can interoperate with C by extern "C" declarations.