manually building a python extension in cygwin with boost.python - c++

Question

Sorry for such a general title, but i'm not quite sure what exactly i'm missing or what i'm doing wrong. My aim is to build a python extension using boost.python under cygwin and avoiding boost.build tools, that is using make instead of bjam. The latter way was working for me quite good, but now i want to do it this way. I solved many issues by googling and looking for similar topics and that helped me to figure out some tricks and to move forward. Yet at the last step there seems to be some problem. I'll try to describe all my steps in some details in hope that this post may be useful to others in the future and also to better describe the setup.
Because i was not quite sure about the original (from various cygwin repositories) installations of both python and boost i decided to install them from scratch in my home directory, so here is what i do:
first install python. I'll skip the details for this, it is more-or-less straightforward.Important for the latter description is just the path:
/home/Alexey_2/Soft/python2.6 - this is PYTHONPATH, also included in PATH
working on boost:
a) unpack boost source into
/home/Alexey_2/Soft/boost_1_50_0 - this is BOOST_ROOT
b) making bjam. first go into directory:
/home/Alexey_2/Soft/boost_1_50_0/tools/build/v2
next, invoke bootstrap.sh this eventually creates b2 and bjam executables in this directory. In .bash_profile add this directory to PATH, so we can call bjam. Here and after each future edits of .bash_profile i restart cygwin to make the changes
come to the effect
c) still in the
/home/Alexey_2/Soft/boost_1_50_0/tools/build/v2
directory - edit user-config.jam, to let bjam know which python to use. So in my case i only add one line:
using python : 2.6 : /home/Alexey_2/Soft/python2.6/bin/python2.6 : /home/Alexey_2/Soft/python2.6/include/python2.6 : /home/Alexey_2/Soft/python2.6/bin ;
in the lib-path (last entry) i put /home/Alexey_2/Soft/python2.6/bin because it contains libpython2.6.dll
d) ok. now we can make boost-python libraries. go to BOOST_ROOT directory and execute command
bjam --with-python toolset=gcc link=shared
this creates necessary libraries (cygboost_python.dll and libboost_python.dll.a) and places them into
/home/Alexey_2/Soft/boost_1_50_0/stage/lib
building python extension.
here is my simple test program (actually part of the example code)
// file xyz.cpp
#include <boost/python.hpp>
#include <iostream>
#include <iomanip>
using namespace std;
using namespace boost::python;
class Hello
{
std::string _msg;
public:
Hello(std::string msg){_msg = msg;}
void set(std::string msg) { this->_msg = msg; }
std::string greet() { return _msg; }
};
BOOST_PYTHON_MODULE(xyz)
{
class_<Hello>("Hello", init<std::string>())
.def("greet", &Hello::greet)
.def("set", &Hello::set)
;
}
And here is the Makefile:
FLAGS= -fno-for-scope -O2 -fPIC
CPP=c++
# BOOST v 1.50.0
p1=/home/Alexey_2/Soft/boost_1_50_0
pl1=/home/Alexey_2/Soft/boost_1_50_0/stage/lib
# PYTHON v 2.6
p2=/home/Alexey_2/Soft/python2.6/include/python2.6
pl2=/home/Alexey_2/Soft/python2.6/bin
I=-I${p1} -I${p2}
L=-L${pl1} -lboost_python -L${pl2} -lpython2.6
all: xyz.so
xyz.o: xyz.cpp
${CPP} ${FLAGS} ${I} -c xyz.cpp
xyz.so: xyz.o
${CPP} ${FLAGS} -shared -o xyz.so xyz.o ${L}
clean:
rm *.o
rm xyz.so
Some comments:
library paths are set and i compile against proper libraries (see more: compile some code with boost.python by mingw in win7-64bit ).
The link above explains why it is important to configure user-config.jam - i did it in step 1c.
To avoid possible problems (as mentioned in above link and also in Cannot link boost.python with mingw (although for mingw) ) with boost.python libraries liked statically i use
link=shared
as the argument to bjam (see 1d)
As explained here: MinGW + Boost: undefined reference to `WSAStartup#8' the libraries with which one wants to compile something should be listed after object files, that is why we have:
${CPP} ${FLAGS} -shared -o xyz.so xyz.o ${L}
and not
${CPP} ${FLAGS} -shared ${L} -o xyz.so xyz.o
And here is a piece of my .bash_profile (eventually), where i define environmental variables:
# Python
export PATH=/home/Alexey_2/Soft/python2.6/bin:$PATH
export PYTHONPATH=/home/Alexey_2/Soft/python2.6
export LD_LIBRARY_PATH=/home/Alexey_2/Soft/python2.6/lib:/home/Alexey_2/Soft/python2.6/bin:$LD_LIBRARY_PATH
# Boost
export BOOST_ROOT=/home/Alexey_2/Soft/boost_1_50_0
export LD_LIBRARY_PATH=/home/Alexey_2/Soft/boost_1_50_0/stage/lib:$LD_LIBRARY_PATH
export PATH=/home/Alexey_2/Soft/boost_1_50_0/stage/lib:$PATH
# bjam
export PATH=/home/Alexey_2/Soft/boost_1_50_0/tools/build/v2:$PATH
Finally, to the problem. With the above setup i was able to successfully build the python extension object file:
xyz.so
However, when i test it by simple script:
# this is a test.py script
import xyz
the ImportError comes:
$ python test.py
Traceback (most recent call last):
File "test.py", line 1, in <module>
import xyz
ImportError: No module named xyz
It has been noted that the reason for such a problem may be the wrong python executable being used, but this is not the case:
$ which python
/home/Alexey_2/Soft/python2.6/bin/python
what is expected (note python is a symbolic link to python2.6 from that directory)
Here is one more useful piece of information i have:
$ ldd xyz.so
ntdll.dll => /cygdrive/c/Windows/SysWOW64/ntdll.dll (0x76fa0000)
kernel32.dll => /cygdrive/c/Windows/syswow64/kernel32.dll (0x76430000)
KERNELBASE.dll => /cygdrive/c/Windows/syswow64/KERNELBASE.dll (0x748e0000)
cygboost_python.dll => /home/Alexey_2/Soft/boost_1_50_0/stage/lib/cygboost_python.dll (0x70cc0000)
cygwin1.dll => /usr/bin/cygwin1.dll (0x61000000)
cyggcc_s-1.dll => /usr/bin/cyggcc_s-1.dll (0x6ff90000)
cygstdc++-6.dll => /usr/bin/cygstdc++-6.dll (0x6fa90000)
libpython2.6.dll => /home/Alexey_2/Soft/python2.6/bin/libpython2.6.dll (0x67ec0000)
??? => ??? (0x410000)
I'm wondering what
??? => ??? (0x410000)
could possibly mean. May be this what i'm missing. But what is that?
Any comments and suggestions (not only about the last question) are very much appreciated.
EDIT:
Following suggestion of (twsansbury) examining the python module search path with -vv option:
python -vv test.py
gives
# trying /home/Alexey_2/Programming/test/xyz.dll
# trying /home/Alexey_2/Programming/test/xyzmodule.dll
# trying /home/Alexey_2/Programming/test/xyz.py
# trying /home/Alexey_2/Programming/test/xyz.pyc
...
# trying /home/Alexey_2/Soft/python2.6/lib/python2.6/site-packages/xyz.dll
# trying /home/Alexey_2/Soft/python2.6/lib/python2.6/site-packages/xyzmodule.dll
# trying /home/Alexey_2/Soft/python2.6/lib/python2.6/site-packages/xyz.py
# trying /home/Alexey_2/Soft/python2.6/lib/python2.6/site-packages/xyz.pyc
Traceback (most recent call last):
File "test.py", line 1, in <module>
import xyz
ImportError: No module named xyz
The first directory is wherefrom i call python to run the script. The main conclusion is that cygwin python is looking for modules (libraries) with the standard Windows extension - dll (among other 3 types), not the .so as i originally expected from the Linux-emulation-style of cygwin. So changing the following lines in the previous Makefile to:
all: xyz.dll
xyz.o: xyz.cpp
${CPP} ${FLAGS} ${I} -c xyz.cpp
xyz.dll: xyz.o
${CPP} ${FLAGS} -shared -o xyz.dll xyz.o ${L}
clean:
rm *.o
rm xyz.dll
produces xyz.dll, which can successfully be loaded:
python -vv test.py
now gives:
Python 2.6.8 (unknown, Mar 21 2013, 17:13:04)
[GCC 4.5.3] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
# trying /home/Alexey_2/Programming/test/xyz.dll
dlopen("/home/Alexey_2/Programming/test/xyz.dll", 2);
import xyz # dynamically loaded from /home/Alexey_2/Programming/test/xyz.dll

Answer 1

ThatImportError is generally not Boost.Python related. Rather, it normally indicates that xyz is not in the Python Module Search Path.
To debug this, consider running python with -vv arguments. This will cause python to print a message for each file that is checked for when trying to import xyz. Regardless, the build process look correct, so the problem is likely the result of either the file extension or the module is not in the search path.
I am unsure as to how Cygwin will interact with Python's run-time loading behavior. However:
On Windows, python extensions have a .pyd extension.
On Linux, python extensions have a .so extension.
Additionally, verify that the xyz library is located in one of the following:
The directory containing the test.py script (or the current directory).
A directory listed in the PYTHONPATH environment variable.
The installation-dependent default directory.
If the unresolved library shown in ldd causes errors, it will generally manifest as an ImportError with a message indicating undefined references.