Why does PyCXX handle new-style classes in the way it does? - c++

I'm picking apart some C++ Python wrapper code that allows the consumer to construct custom old style and new style Python classes from C++.
The original code comes from PyCXX, with old and new style classes here and here. I have however rewritten the code substantially, and in this question I will reference my own code, as it allows me to present the situation in the greatest clarity that I am able. I think there would be very few individuals capable of understanding the original code without several days of scrutiny... For me it has taken weeks and I'm still not clear on it.
The old style simply derives from PyObject,
template<typename FinalClass>
class ExtObj_old : public ExtObjBase<FinalClass>
// ^ which : ExtObjBase_noTemplate : PyObject
{
public:
// forwarding function to mitigate awkwardness retrieving static method
// from base type that is incomplete due to templating
static TypeObject& typeobject() { return ExtObjBase<FinalClass>::typeobject(); }
static void one_time_setup()
{
typeobject().set_tp_dealloc( [](PyObject* t) { delete (FinalClass*)(t); } );
typeobject().supportGetattr(); // every object must support getattr
FinalClass::setup();
typeobject().readyType();
}
// every object needs getattr implemented to support methods
Object getattr( const char* name ) override { return getattr_methods(name); }
// ^ MARKER1
protected:
explicit ExtObj_old()
{
PyObject_Init( this, typeobject().type_object() ); // MARKER2
}
When one_time_setup() is called, it forces (by accessing base class typeobject()) creation of the associated PyTypeObject for this new type.
Later when an instance is constructed, it uses PyObject_Init
So far so good.
But the new style class uses much more complicated machinery. I suspect this is related to the fact that new style classes allow derivation.
And this is my question, why is the new style class handling implemented in the way that it is? Why is it having to create this extra PythonClassInstance structure? Why can't it do things the same way the old-style class handling does? i.e. Just type convert from the PyObject base type? And seeing as it doesn't do that, does this mean it is making no use of its PyObject base type?
This is a huge question, and I will keep amending the post until I'm satisfied it represents the issue well. It isn't a good fit for SO's format, I'm sorry about that. However, some world-class engineers frequent this site (one of my previous questions was answered by the lead developer of GCC for example), and I value the opportunity to appeal to their expertise. So please don't be too hasty to vote to close.
The new style class's one-time setup looks like this:
template<typename FinalClass>
class ExtObj_new : public ExtObjBase<FinalClass>
{
private:
PythonClassInstance* m_class_instance;
public:
static void one_time_setup()
{
TypeObject& typeobject{ ExtObjBase<FinalClass>::typeobject() };
// these three functions are listed below
typeobject.set_tp_new( extension_object_new );
typeobject.set_tp_init( extension_object_init );
typeobject.set_tp_dealloc( extension_object_deallocator );
// this should be named supportInheritance, or supportUseAsBaseType
// old style class does not allow this
typeobject.supportClass(); // does: table->tp_flags |= Py_TPFLAGS_BASETYPE
typeobject.supportGetattro(); // always support get and set attr
typeobject.supportSetattro();
FinalClass::setup();
// add our methods to the extension type's method table
{ ... typeobject.set_methods( /* ... */); }
typeobject.readyType();
}
protected:
explicit ExtObj_new( PythonClassInstance* self, Object& args, Object& kwds )
: m_class_instance{self}
{ }
So the new style uses a custom PythonClassInstance structure:
struct PythonClassInstance
{
PyObject_HEAD
ExtObjBase_noTemplate* m_pycxx_object;
}
PyObject_HEAD, if I dig into Python's object.h, is just a macro for PyObject ob_base; -- no further complications, like #if #else. So I don't see why it can't simply be:
struct PythonClassInstance
{
PyObject ob_base;
ExtObjBase_noTemplate* m_pycxx_object;
}
or even:
struct PythonClassInstance : PyObject
{
ExtObjBase_noTemplate* m_pycxx_object;
}
Anyway, it seems that its purpose is to tag a pointer onto the end of a PyObject. This will be because Python runtime will often trigger functions we have placed in its function table, and the first parameter will be the PyObject responsible for the call. So this allows us to retrieve the associated C++ object.
But we also need to do that for the old-style class.
Here is the function responsible for doing that:
ExtObjBase_noTemplate* getExtObjBase( PyObject* pyob )
{
if( pyob->ob_type->tp_flags & Py_TPFLAGS_BASETYPE )
{
/*
New style class uses a PythonClassInstance to tag on an additional
pointer onto the end of the PyObject
The old style class just seems to typecast the pointer back up
to ExtObjBase_noTemplate
ExtObjBase_noTemplate does indeed derive from PyObject
So it should be possible to perform this typecast
Which begs the question, why on earth does the new style class feel
the need to do something different?
This looks like a really nice way to solve the problem
*/
PythonClassInstance* instance = reinterpret_cast<PythonClassInstance*>(pyob);
return instance->m_pycxx_object;
}
else
return static_cast<ExtObjBase_noTemplate*>( pyob );
}
My comment articulates my confusion.
And here, for completeness is us inserting a lambda-trampoline into the PyTypeObject's function pointer table, so that Python runtime can trigger it:
table->tp_setattro = [] (PyObject* self, PyObject* name, PyObject* val) -> int
{
try {
ExtObjBase_noTemplate* p = getExtObjBase( self );
return ( p -> setattro(Object{name}, Object{val}) );
}
catch( Py::Exception& ) { /* indicate error */
return -1;
}
};
(In this demonstration I'm using tp_setattro, note that there are about 30 other slots, which you can see if you look at the doc for PyTypeObject)
(in fact the major reason for working this way is that we can try{}catch{} around every trampoline. This saves the consumer from having to code repetitive error trapping.)
So, we pull out the "base type for the associated C++ object" and call its virtual setattro (just using setattro as an example here). A derived class will have overridden setattro, and this override will get called.
The old-style class provides such an override, which I've labelled MARKER1 -- it is in the top listing for this question.
The only the thing I can think of is that maybe different maintainers have used different techniques. But is there some more compelling reason why old and new style classes require different architecture?
PS for reference, I should include the following methods from new style class:
static PyObject* extension_object_new( PyTypeObject* subtype, PyObject* args, PyObject* kwds )
{
PyObject* pyob = subtype->tp_alloc(subtype,0);
PythonClassInstance* o = reinterpret_cast<PythonClassInstance *>( pyob );
o->m_pycxx_object = nullptr;
return pyob;
}
^ to me, this looks absolutely wrong.
It appears to be allocating memory, re-casting to some structure that might exceed the amount allocated, and then nulling right at the end of this.
I'm surprised it hasn't caused any crashes.
I can't see any indication anywhere in the source code that these 4 bytes are owned.
static int extension_object_init( PyObject* _self, PyObject* _args, PyObject* _kwds )
{
try
{
Object args{_args};
Object kwds{_kwds};
PythonClassInstance* self{ reinterpret_cast<PythonClassInstance*>(_self) };
if( self->m_pycxx_object )
self->m_pycxx_object->reinit( args, kwds );
else
// NOTE: observe this is where we invoke the constructor, but indirectly (i.e. through final)
self->m_pycxx_object = new FinalClass{ self, args, kwds };
}
catch( Exception & )
{
return -1;
}
return 0;
}
^ note that there is no implementation for reinit, other than the default
virtual void reinit ( Object& args , Object& kwds ) {
throw RuntimeError( "Must not call __init__ twice on this class" );
}
static void extension_object_deallocator( PyObject* _self )
{
PythonClassInstance* self{ reinterpret_cast< PythonClassInstance* >(_self) };
delete self->m_pycxx_object;
_self->ob_type->tp_free( _self );
}
EDIT: I will hazard a guess, thanks to insight from Yhg1s on the IRC channel.
Maybe it is because when you create a new old-style class, it is guaranteed it will overlap perfectly a PyObject structure.
Hence it is safe to derive from PyObject, and pass a pointer to the underlying PyObject into Python, which is what the old-style class does (MARKER2)
On the other hand, new style class creates a {PyObject + maybe something else} object.
i.e. It wouldn't be safe to do the same trick, as Python runtime would end up writing past the end of the base class allocation (which is only a PyObject).
Because of this, we need to get Python to allocate for the class, and return us a pointer which we store.
Because we are now no longer making use of the PyObject base-class for this storage, we cannot use the convenient trick of typecasting back to retrieve the associated C++ object.
Which means that we need to tag on an extra sizeof(void*) bytes to the end of the PyObject that actually does get allocated, and use this to point to our associated C++ object instance.
However, there is some contradiction here.
struct PythonClassInstance
{
PyObject_HEAD
ExtObjBase_noTemplate* m_pycxx_object;
}
^ if this is indeed the structure that accomplishes the above, then it is saying that the new style class instance is indeed fitting exactly over a PyObject, i.e. It is not overlapping into the m_pycxx_object.
And if this is the case, then surely this whole process is unnecessary.
EDIT: here are some links that are helping me learn the necessary ground work:
http://eli.thegreenplace.net/2012/04/16/python-object-creation-sequence
http://realmike.org/blog/2010/07/18/introduction-to-new-style-classes-in-python
Create an object using Python's C API

to me, this looks absolutely wrong. It appears to be allocating memory, re-casting to some structure that might exceed the amount allocated, and then nulling right at the end of this. I'm surprised it hasn't caused any crashes. I can't see any indication anywhere in the source code that these 4 bytes are owned
PyCXX does allocate enough memory, but it does so by accident. This appears to be a bug in PyCXX.
The amount of memory Python allocates for the object is determined by the first call to the following static member function of PythonClass<T>:
static PythonType &behaviors()
{
...
p = new PythonType( sizeof( T ), 0, default_name );
...
}
The constructor of PythonType sets the tp_basicsize of the python type object to sizeof(T). This way when Python allocates an object it knows to allocate at least sizeof(T) bytes. It works because sizeof(T) turns out to be larger that sizeof(PythonClassInstance) (T is derived from PythonClass<T> which derives from PythonExtensionBase, which is large enough).
However, it misses the point. It should actually allocate only sizeof(PythonClassInstance) . This appears to be a bug in PyCXX - that it allocates too much, rather than too little space for storing a PythonClassInstance object.
And this is my question, why is the new style class handling implemented in the way that it is? Why is it having to create this extra PythonClassInstance structure? Why can't it do things the same way the old-style class handling does?
Here's my theory why new style classes are different from the old style classes in PyCXX.
Before Python 2.2, where new style classes were introduced, there was no tp_init member int the type object. Instead, you needed to write a factory function that would construct the object. This is how PythonExtension<T> is supposed to work - the factory function converts the Python arguments to C++ arguments, asks Python to allocate the memory and then calls the constructor using placement new.
Python 2.2 added the new style classes and the tp_init member. Python first creates the object and then calls the tp_init method. Keeping the old way would have required that the objects would first have a dummy constructor that creates an "empty" object (e.g. initializes all members to null) and then when tp_init is called, would have had an additional initialization stage. This makes the code uglier.
It seems that the author of PyCXX wanted to avoid that. PyCXX works by first creating a dummy PythonClassInstance object and then when tp_init is called, creates the actual PythonClass<T> object using its constructor.
... does this mean it is making no use of its PyObject base type?
This appears to be correct, the PyObject base class does not seem to be used anywhere. All the interesting methods of PythonExtensionBase use the virtual self() method, which returns m_class_instance and completely ignore the PyObject base class.
I guess (only a guess, though) is that PythonClass<T> was added to an existing system and it seemed easier to just derive from PythonExtensionBase instead of cleaning up the code.

Related

Downcasting trouble

This is my first experience with downcasting in C++ and I just can't understand the problem.
AInstruction and CInstruction inherit from AssemblerInstruction.
Parser takes the info in its ctor and creates one of those derived instruction types for its mInstruction member (accessed by getInstruction). In the program, a method of the base AssemblerInstruction class is used, for happy polymorphism.
But when I want to test that the Parser has created the correct instruction, I need to query the derived instruction members, which means I need to downcast parser.getInstruction() to an AInstruction or CInstruction.
As far as I can tell this needs to be done using a bunch of pointers and references. This is how I can get the code to compile:
TEST(ParserA, parsesBuiltInConstants)
{
AssemblerInstruction inst = Parser("#R3", 0).getInstruction();
EXPECT_EQ(inst.getInstructionType(), AssemblerInstruction::InstructionType::A);
AssemblerInstruction* i = &(inst);
AInstruction* a = dynamic_cast<AInstruction*>(i);
EXPECT_EQ(a->getLine(), "R3");
}
Running this gives this error:
unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
And stepping through the code, when the debugger is on the final line of the function, a is pointing to
0x00000000 <NULL>.
I imagine this is an instance where I don't have a full enough understanding of C++, meaning that I could be making a n00b mistake. Or maybe it's some bigger crazy problem. Help?
Update
I've been able to make this work by making mInstruction into a (dumb) pointer:
// in parser, when parsing
mInstructionPtr = new AInstruction(assemblyCode.substr(1), lineNumber);
// elsewhere in AssemblerInstruction.cpp
AssemblerInstruction* AssemblyParser::getInstructionPtr() { return mInstructionPtr; }
TEST(ParserA, parsesBuiltInConstants)
{
auto ptr = Parser("#R3", 0).getInstructionPtr();
AInstruction* a = dynamic_cast<AInstruction*>(ptr);
EXPECT_EQ(a->getLine(), "R3");
}
However I have trouble implementing it with a unique_ptr:
(I'm aware that mInstruction (non-pointer) is redundant, as are two types of pointers. I'll get rid of it later when I clean all this up)
class AssemblyParser
{
public:
AssemblyParser(std::string assemblyCode, unsigned int lineNumber);
AssemblerInstruction getInstruction();
std::unique_ptr<AssemblerInstruction> getUniqueInstructionPtr();
AssemblerInstruction* getInstructionPtr();
private:
AssemblerInstruction mInstruction;
std::unique_ptr<AssemblerInstruction> mUniqueInstructionPtr;
AssemblerInstruction* mInstructionPtr;
};
// in AssemblyParser.cpp
// in parser as in example above. this works fine.
mUniqueInstructionPtr = make_unique<AInstruction>(assemblyCode.substr(1), lineNumber);
// this doesn't compile!!!
unique_ptr<AssemblerInstruction> AssemblyParser::getUniqueInstructionPtr()
{
return mUniqueInstructionPtr;
}
In getUniqueInstructionPtr, there is a squiggle under mUniqueInstructionPtr with this error:
'std::unique_ptr<AssemblerInstruction,std::default_delete>::unique_ptr(const std::unique_ptr<AssemblerInstruction,std::default_delete> &)': attempting to reference a deleted function
What!? I haven't declared any functions as deleted or defaulted!
You can not downcast an object to something which doesn't match it's dynamic type. In your code,
AssemblerInstruction inst = Parser("#R3", 0).getInstruction();
inst has a fixed type, which is AssemblerInstruction. Downcasting it to AInstruction leads to undefined behavior - manifested as crash - because that is not what it is.
If you want your getInstruction to return a dynamically-typed object, it has to return a [smart] pointer to base class, while constructing an object of derived class. Something like that (pseudo code):
std::unique_ptr<AssemblerInstruction> getInstruction(...) {
return std::make_unique<AInstruction>(...);
}
Also, if you see yourself in need of downcasting object based on a value of a class, you are doing something wrong, as you are trying to home-brew polymorphism. Most of the times it does indicate a design flaw, and should instead be done using built-in C++ polymorphic support - namely, virtual functions.

In the V8 javascript engine, how to make a constructor function that re-uses an ObjectTemplate for each instance?

I have working code where I can create as many Point objects as I want, but it re-creates the object template each time the constructor is called, which seems like it's probably wrong.
Local<ObjectTemplate> global_templ = ObjectTemplate::New(isolate);
// make the Point constructor function available to JS
global_templ->Set(v8::String::NewFromUtf8(isolate, "Point"), FunctionTemplate::New(isolate, v8_Point));
and then the constructor itself:
void v8_Point(const v8::FunctionCallbackInfo<v8::Value>& args) {
HandleScope scope(args.GetIsolate());
// this bit should probably be cached somehow
Local<ObjectTemplate> point_template = ObjectTemplate::New(args.GetIsolate());
point_template->SetInternalFieldCount(1);
point_template->SetAccessor(String::NewFromUtf8(args.GetIsolate(), "x"), GetPointX, SetPointX);
point_template->SetAccessor(String::NewFromUtf8(args.GetIsolate(), "y"), GetPointY, SetPointY);
// end section to be cached
Local<Object> obj = point_template->NewInstance();
Point * p = new Point(1,1);
obj->SetInternalField(0, External::New(args.GetIsolate(), p));
args.GetReturnValue().Set(obj);
}
But it seems like I should be able to pass in the point_template object instead of re-creating it each time. I saw that there's a Data() field in args, but that only allows for a Value type and an ObjectTemplate is of type Template, not Value.
Any help on the right way to do this would be greatly appreciated.
I figured it out finally.
In javascript, when you add a function via a FunctionTemplate and then call it as a constructor (e.g. new MyFunction), then in your c++ callback the args.This() will be a new object created by the using the FunctionTemplate's InstanceTemplate object template.
// Everything has to go in a single global template (as I understand)
Local<ObjectTemplate> global_templ = ObjectTemplate::New(isolate);
// create the function template and tell it the callback to use
Local<FunctionTemplate> point_constructor = FunctionTemplate::New(isolate, v8_Point);
// set the internal field count so our actual c++ object can tag along
// with the javascript object so our accessors can use it
point_constructor->InstanceTemplate()->SetInternalFieldCount(1);
// associate getters and setters for the 'x' field on point
point_constructor->InstanceTemplate()->SetAccessor(String::NewFromUtf8(isolate, "x"), GetPointX, SetPointX);
... add any other function and object templates to the global template ...
// add the global template to the context our javascript will run in
Local<Context> x_context = Context::New(isolate, NULL, global_templ);
Then, for the actual function:
void v8_Point(const v8::FunctionCallbackInfo<v8::Value>& args) {
// (just an example of a handy utility function)
// whether or not it was called as "new Point()" or just "Point()"
printf("Is constructor call: %s\n", args.IsConstructCall()?"yes":"no");
// create your c++ object that will follow the javascript object around
// make sure not to make it on the stack or it won't be around later when you need it
Point * p = new Point();
// another handy helper function example
// see how the internal field count is what it was set to earlier
// in the InstanceTemplate
printf("Internal field count: %d\n",args.This()->InternalFieldCount()); // this prints the value '1'
// put the new Point object into the internal field
args.This()->SetInternalField(0, External::New(args.GetIsolate(), p));
// return the new object back to the javascript caller
args.GetReturnValue().Set(args.This());
}
Now, when you write the getter and setter, you have access to your actual c++ object in the body of them:
void GetPointX(Local<String> property,
const PropertyCallbackInfo<Value>& info) {
Local<Object> self = info.Holder();
// This is where we take the actual c++ object that was embedded
// into the javascript object and get it back to a useable c++ object
Local<External> wrap = Local<External>::Cast(self->GetInternalField(0));
void* ptr = wrap->Value();
int value = static_cast<Point*>(ptr)->x_; //x_ is the name of the field in the c++ object
// return the value back to javascript
info.GetReturnValue().Set(value);
}
void SetPointX(Local<String> property, Local<Value> value,
const PropertyCallbackInfo<void>& info) {
Local<Object> self = info.Holder();
// same concept here as in the "getter" above where you get access
// to the actual c++ object and then set the value from javascript
// into the actual c++ object field
Local<External> wrap = Local<External>::Cast(self->GetInternalField(0));
void* ptr = wrap->Value();
static_cast<Point*>(ptr)->x_ = value->Int32Value();
}
Almost all of this came from here: https://developers.google.com/v8/embed?hl=en#accessing-dynamic-variables
except it doesn't talk about the proper way to make your objects in a repeatable fashion.
I figured out how to clean up the c++ object in the internal field, but I don't have time to put the whole answer here. You have to pass in a Global object into your weak callback by creating a hybrid field (a struct works well) on the heap that has both the global object and a pointer to your c++ object. You can then delete your c++ object, call Reset() on your Global and then delete the whole thing. I'll try to add actual code, but may forget.
Here is a good source: https://code.google.com/p/chromium/codesearch#chromium/src/v8/src/d8.cc&l=1064 lines 1400-1441 are what you want. (edit: line numbers seem to be wrong now - maybe the link above has changed?)
Remember, v8 won't garbage collect small amounts of memory, so you may never see it. Also, just because your program ends doesn't mean the GC will run. You can use isolate->AdjustAmountOfExternalAllocatedMemory(length); to tell v8 about the size of the memory you've allocated (it includes this in its calculations about when there's too much memory in use and GC needs to run) and you can use isolate->IdleNotificationDeadline(1); to give the GC a chance to run (though it may choose not to).

Convert Objective C "code blocks" to equivalent in C++

I need a help on converting some Objective C "code block" methods to the equivalent in C++.
Please advise.
A is being used as code block...
Defined in .h file..
typedef void (^A)(void*); //argument is ptr to B
Used in one .mm file..
[[hello getInstance] getB]->queueLoadImageWithBlock([self.str UTF8String], (A
)^(void* img)
{
//some code...
});
The most direct analogy is std::function. This is a value type that is given a signature (e.g. std::function<int(int)> and can be any function object of the appropriate signature. A lambda can be used in place of the block at the call site.
obj->queueLoadImageWithBlock(self.url, [](void* img)
{
UIImage* img2 = (UIImage*)img;
UIImageView* iv = [[UIImageView alloc] initWithImage:img2];
iv.backgroundColor = [UIColor clearColor];
[self.iconSlot addSubview:iv];
iconLoaded(iv);
[iv release];
});
With Apple's version of clang you can use blocks in C and C++ as well as Objective-C. This is non-standard C++, obviously, but it works.
You can use C++ lambdas without changing the called function since lambdas are assignable to blocks (but not the other way around). See this question for more information.
As long as the requirement for a block is yours as opposed to system.
Like I said, there are several approaches. The function pointers require the least boilerplate, but they need an extra argument to pass the context from the caller (the self stuff in your case). Functors and pointer-to-members typically require template machinery to work, let's not go there. So with a function pointer, here's how it would go:
//Let's define a callback datatype
typedef void (*ResourceLoadObjFuncPtr)(void *, void*);
//argument 1 is ptr to ResourceLoadDescriptor, argument 2 is iconSlot, whatever it is
//Function that implements that type:
void MyLoad(void *img, void *iconSlot)
{
UIImage* img2 = (UIImage*)img;
UIImageView* iv = [[UIImageView alloc] initWithImage:img2];
iv.backgroundColor = [UIColor clearColor];
[(TheTypeOfIconslot*)iconSlot addSubview:iv];
iconLoaded(iv);
[iv release];
}
And you'd have to modify the prototype of queueLoadImageWithBlock to accept a ResourceLoadObjFuncPtr parameter instead of ResourceLoadObjCBlockCB, and another parameter for the context (just the iconSlot in our case).
And invoke:
[[GameViewController getInstance] getResourceLoadMediator]->
queueLoadImageWithFunction([self.url UTF8String], MyLoad, self.iconSlot);
Blocks are closures - they capture the variables of the function where they're declared. C++ provides no closures that GCC on iOS supports (other than, well, blocks). So you'd have to pass the variables from the function scope to the function parameter by hand. In our case, if my assumptions are right, there's just one variable; in a more complex case, you'd have to wrap them in a structure and pass a pointer to one.
An alternative to that would be using an abstract base class and a concrete implementation that captures the context via its constructor. This would go like this:
//Callback type
class ResourceLoader
{
public:
virtual void Load(void *) = 0;
};
//A callback implementation - not a function, but a class
class MyResourceLoader : public ResourceLoader
{
IconSlotType *iconSlot;
void Load(void *img)
{
//Same loader stuff as above
}
public:
MyResourceLoader(IconSlotType *isl)
:iconSlot(isl)
{}
};
The queueLoadImageWithBlock equivalent would now take a second parameter of type ResourceLoader* and no third parameter. As for the invokation, there's the issue of callback object lifetime. Is queueLoadImageWithBlock asynchronous - that is, does it return before invoking the callback? If so, then a local instance of MyResourceLoader won't do, you'd have to create one dynamically and somehow dispose it. Assuming it's synchronous (i. e. does not invoke the callback after it returns):
MyResourceLoader ResLoader(self.iconSlot);
[[GameViewController getInstance] getResourceLoadMediator]->
queueLoadImageWithLoader([self.url UTF8String], &ResLoader);
If it's not:
[[GameViewController getInstance] getResourceLoadMediator]->
queueLoadImageWithLoader([self.url UTF8String], new MyResourceLoader(self.iconSlot));

How can I bind a C/C++ structure to Ruby?

I need some advice how can I bind a C/C++ structure to Ruby. I've read some manuals and I found out how to bind class methods to a class, but I still don't understand how to bind structure fields and make them accessible in Ruby.
Here is the code I'm using:
myclass = rb_define_class("Myclass", 0);
...
typedef struct nya
{
char const* name;
int age;
} Nya;
Nya* p;
VALUE vnya;
p = (Nya*)(ALLOC(Nya));
p->name = "Masha";
p->age = 24;
vnya = Data_Wrap_Struct(myclass, 0, free, p);
rb_eval_string("def foo( a ) p a end"); // This function should print structure object
rb_funcall(0, rb_intern("foo"), 1, vnya); // Here I call the function and pass the object into it
The Ruby function seems to assume that a is a pointer. It prints the numeric value of the pointer instead of it's real content (i.e., ["Masha", 24]). Obviously the Ruby function can't recognize this object —I didn't set the object's property names and types.
How can I do this? Unfortunately I can't figure it out.
You have already wrapped your pointer in a Ruby object. Now all you have to do is define how it can be accessed from the Ruby world:
/* Feel free to convert this function to a macro */
static Nya * get_nya_from(VALUE value) {
Nya * pointer = 0;
Data_Get_Struct(value, Nya, pointer);
return pointer;
}
VALUE nya_get_name(VALUE self) {
return rb_str_new_cstr(get_nya_from(self)->name);
}
VALUE nya_set_name(VALUE self, VALUE name) {
/* StringValueCStr returns a null-terminated string. I'm not sure if
it will be freed when the name gets swept by the GC, so maybe you
should create a copy of the string and store that instead. */
get_nya_from(self)->name = StringValueCStr(name);
return name;
}
VALUE nya_get_age(VALUE self) {
return INT2FIX(get_nya_from(self)->age);
}
VALUE nya_set_age(VALUE self, VALUE age) {
get_nya_from(self)->age = FIX2INT(age);
return age;
}
void init_Myclass() {
/* Associate these functions with Ruby methods. */
rb_define_method(myclass, "name", nya_get_name, 0);
rb_define_method(myclass, "name=", nya_set_name, 1);
rb_define_method(myclass, "age", nya_get_age, 0);
rb_define_method(myclass, "age=", nya_set_age, 1);
}
Now that you can access the data your structure holds, you can simply define the high level methods in Ruby:
class Myclass
def to_a
[name, age]
end
alias to_ary to_a
def to_s
to_a.join ', '
end
def inspect
to_a.inspect
end
end
For reference: README.EXT
This is not a direct answer to your question about structures, but it is a general solution to the problem of porting C++ classes to Ruby.
You could use SWIG to wrap C/C++ classes, structs and functions. In the case of a structure, it's burning a house to fry an egg. However, if you need a tool to rapidly convert C++ classes to Ruby (and 20 other languages), SWIG might be useful to you.
In your case involving a structure, you just need to create a .i file which includes (in the simplest case) the line #include <your C++ library.h>.
P.S. Once more, it's not a direct answer to your question involving this one struct, but maybe you could make use of a more general solution, in which case this may help you.
Another option is to use RubyInline - it has limited support for converting C and Ruby types (such as int, char * and float) and it also has support for accessing C structurs - see accessor method in the API.

Python C-API Object Allocation

I want to use the new and delete operators for creating and destroying my objects.
The problem is python seems to break it into several stages. tp_new, tp_init and tp_alloc for creation and tp_del, tp_free and tp_dealloc for destruction. However c++ just has new which allocates and fully constructs the object and delete which destructs and deallocates the object.
Which of the python tp_* methods do I need to provide and what must they do?
Also I want to be able to create the object directly in c++ eg "PyObject *obj = new MyExtensionObject(args);" Will I also need to overload the new operator in some way to support this?
I also would like to be able to subclass my extension types in python, is there anything special I need to do to support this?
I'm using python 3.0.1.
EDIT:
ok, tp_init seems to make objects a bit too mutable for what I'm doing (eg take a Texture object, changing the contents after creation is fine, but change fundamental aspects of it such as, size, bitdept, etc will break lots of existing c++ stuff that assumes those sort of things are fixed). If I dont implement it will it simply stop people calling __init__ AFTER its constructed (or at least ignore the call, like tuple does). Or should I have some flag that throws an exception or somthing if tp_init is called more than once on the same object?
Apart from that I think ive got most of the rest sorted.
extern "C"
{
//creation + destruction
PyObject* global_alloc(PyTypeObject *type, Py_ssize_t items)
{
return (PyObject*)new char[type->tp_basicsize + items*type->tp_itemsize];
}
void global_free(void *mem)
{
delete[] (char*)mem;
}
}
template<class T> class ExtensionType
{
PyTypeObject *t;
ExtensionType()
{
t = new PyTypeObject();//not sure on this one, what is the "correct" way to create an empty type object
memset((void*)t, 0, sizeof(PyTypeObject));
static PyVarObject init = {PyObject_HEAD_INIT, 0};
*((PyObject*)t) = init;
t->tp_basicsize = sizeof(T);
t->tp_itemsize = 0;
t->tp_name = "unknown";
t->tp_alloc = (allocfunc) global_alloc;
t->tp_free = (freefunc) global_free;
t->tp_new = (newfunc) T::obj_new;
t->tp_dealloc = (destructor)T::obj_dealloc;
...
}
...bunch of methods for changing stuff...
PyObject *Finalise()
{
...
}
};
template <class T> PyObjectExtension : public PyObject
{
...
extern "C" static PyObject* obj_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds)
{
void *mem = (void*)subtype->tp_alloc(subtype, 0);
return (PyObject*)new(mem) T(args, kwds)
}
extern "C" static void obj_dealloc(PyObject *obj)
{
~T();
obj->ob_type->tp_free(obj);//most of the time this is global_free(obj)
}
...
};
class MyObject : PyObjectExtension<MyObject>
{
public:
static PyObject* InitType()
{
ExtensionType<MyObject> extType();
...sets other stuff...
return extType.Finalise();
}
...
};
The documentation for these is at http://docs.python.org/3.0/c-api/typeobj.html and
http://docs.python.org/3.0/extending/newtypes.html describes how to make your own type.
tp_alloc does the low-level memory allocation for the instance. This is equivalent to malloc(), plus initialize the refcnt to 1. Python has it's own allocator, PyType_GenericAlloc, but a type can implement a specialized allocator.
tp_new is the same as Python's __new__. It's usually used for immutable objects where the data is stored in the instance itself, as compared to a pointer to data. For example, strings and tuples store their data in the instance, instead of using a char * or a PyTuple *.
For this case, tp_new figures out how much memory is needed, based on the input parameters, and calls tp_alloc to get the memory, then initializes the essential fields. tp_new does not need to call tp_alloc. It can for example return a cached object.
tp_init is the same as Python's __init__. Most of your initialization should be in this function.
The distinction between __new__ and __init__ is called two-stage initialization, or two-phase initialization.
You say "c++ just has new" but that's not correct. tp_alloc corresponds a custom arena allocator in C++, __new__ corresponds to a custom type allocator (a factory function), and __init__ is more like the constructor. That last link discusses more about the parallels between C++ and Python style.
Also read http://www.python.org/download/releases/2.2/descrintro/ for details about how __new__ and __init__ interact.
You write that you want to "create the object directly in c++". That's rather difficult because at the least you'll have to convert any Python exceptions that occurred during object instantiation into a C++ exception. You might try looking at Boost::Python for some help with this task. Or you can use a two-phase initialization. ;)
I don't know the python APIs at all, but if python splits up allocation and initialization, you should be able to use placement new.
e.g.:
// tp_alloc
void *buffer = new char[sizeof(MyExtensionObject)];
// tp_init or tp_new (not sure what the distinction is there)
new (buffer) MyExtensionObject(args);
return static_cast<MyExtensionObject*>(buffer);
...
// tp_del
myExtensionObject->~MyExtensionObject(); // call dtor
// tp_dealloc (or tp_free? again I don't know the python apis)
delete [] (static_cast<char*>(static_cast<void*>(myExtensionObject)));