Compile Pybind with header files in Makefile (not using cmake) - c++

Question

I am trying to compile a Pybind11 module in C++ which calls several header files (.h) on top. As I have a lot of header files, I decided to do a Makefile, which works without problem, EXCEPT for creating the target shared object file (s.o). I need this shared object file in order to be able to call the Pybind module in Python.
However, when compiling, I get:
g++ -shared -fPIC neat.o network.o nnode.o link.o trait.o gene.o innovation.o organism.o species.o genome.o population.o example.o -o example.so
/usr/bin/ld: example.o: relocation R_X86_64_PC32 against symbol `_ZTI3Pet' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
Makefile:2: recipe for target 'example.so' failed
make: *** [example.so] Error 1
My question is basically: What I am doing wrong when compiling the object files in order to create a target ?
Makefile
example.so: neat.o network.o nnode.o link.o trait.o gene.o innovation.o organism.o species.o genome.o population.o example.o
g++ neat.o network.o nnode.o link.o trait.o gene.o innovation.o organism.o species.o genome.o population.o example.o -shared -o example.so
neat.o: neat.cpp neat.h
g++ -c -O3 -Wall -fPIC neat.cpp
network.o: network.cpp network.h
g++ -c -O3 -Wall -fPIC network.cpp
nnode.o: nnode.cpp nnode.h
g++ -c -O3 -Wall -fPIC nnode.cpp
link.o: link.cpp link.h
g++ -c -O3 -Wall -fPIC link.cpp
trait.o: trait.cpp trait.h
g++ -c -O3 -Wall -fPIC trait.cpp
gene.o: gene.cpp gene.h
g++ -c -O3 -Wall -fPIC gene.cpp
innovation.o: innovation.cpp innovation.h
g++ -c -O3 -Wall -fPIC innovation.cpp
organism.o: organism.cpp organism.h genome.h genome.cpp species.h species.cpp
g++ -c -O3 -Wall -fPIC organism.cpp
species.o: species.h species.cpp organism.cpp organism.h genome.h genome.cpp
g++ -c -O3 -Wall -fPIC species.cpp
genome.o: genome.cpp genome.h
g++ -c -O3 -Wall -fPIC genome.cpp
population.o: population.cpp population.h organism.h
g++ -c -O3 -Wall -fPIC population.cpp
experiments.o: experiments.cpp experiments.h
g++ -c -O3 -Wall -fPIC experiments.cpp
example.o:
g++ -O3 -Wall -std=c++11 -fopenmp -I -fPIC `python3 -m pybind11 --includes` -c example.cpp
clean:
rm *.o *.so
example.cpp
#include <pybind11/pybind11.h>
#include <iostream>
#include <string>
#include "population.h"
namespace py = pybind11;
int create_neat(){
Population *the_pop=0;
return 0;
}
PYBIND11_MODULE(example, m){
m.def("create_neat", &create_neat, "create a pop object");
}

Answer 1

The standard doesn't completely define the difference between #include "..." and #include <...>, but in all compilers I'm familiar with the <> syntax for include typically means, "this is a system-type header file" and the "" syntax means "this is a local-type header file".
In practice this usually means that "" are looked for in the working directory, then in directories specified by -I options to the compiler and finally in system default directories, while <> is the same except it does not look in the working directory.
So, you either need to change your code to use:
#include "population.h"
or else you need to add -I. to your compile line so the compiler knows to look in the current directory:
population.o: population.cpp population.h organism.h
g++ -I. -c population.cpp
Personally I would do both, because it's good hygiene to add the directories you need to your compile line, and because it's good practice to use "" for local headers so people reading the code understand immediately that this is your header and not a system header.
For your second question, Is the compilation in order to make a shared object file with the object files correct I'm sorry but I don't understand what you're asking.
If you're asking about the order of targets defined in the makefile, then they can be in any order at all except that the first target will be the default target (if you run make with no target name then make will build the first target--and any prerequisites needed for the first target).
ETA
OK, now that we can see that the actual error is that example.o does not exist. Why does it not exist? Again, because you've removed important details from the example in your question we can't say for sure. When you write in your question:
example.so: //some .o files which are not important// HERE IS THE PROBLEM
what exactly are the .o files that appear as prerequisites of example.so?
You must list the file example.o as a prerequisite of example.so if you want make to build example.o before trying to build example.so. You need to list all the object files that are needed to create the shared library. So this should be:
example.so: population.o example.o neat.o network.o nnode.o link.o trait.o gene.o innovation.o organism.o species.o genome.o
It doesn't matter what order they appear in the prerequisites list, but they must all be there.

Answer 2

After several hours of reading, trying, and also thanks to the hints of MadScientist, I got it working but the reasons why are still unclear. I just changed the order of the prerequisites in the compilation of the target, by putting the -shared -o example.so in front and not at the end of the line, and it worked.
The whole file is as follows:
example.so: neat.o network.o nnode.o link.o trait.o gene.o innovation.o organism.o species.o genome.o population.o example.o
g++ -shared -o example.so neat.o network.o nnode.o link.o trait.o gene.o innovation.o organism.o species.o genome.o population.o example.o
neat.o: neat.cpp neat.h
g++ -c -O3 -Wall -fPIC neat.cpp
network.o: network.cpp network.h
g++ -c -O3 -Wall -fPIC network.cpp
nnode.o: nnode.cpp nnode.h
g++ -c -O3 -Wall -fPIC nnode.cpp
link.o: link.cpp link.h
g++ -c -O3 -Wall -fPIC link.cpp
trait.o: trait.cpp trait.h
g++ -c -O3 -Wall -fPIC trait.cpp
gene.o: gene.cpp gene.h
g++ -c -O3 -Wall -fPIC gene.cpp
innovation.o: innovation.cpp innovation.h
g++ -c -O3 -Wall -fPIC innovation.cpp
organism.o: organism.cpp organism.h genome.h genome.cpp species.h species.cpp
g++ -c -O3 -Wall -fPIC organism.cpp
species.o: species.h species.cpp organism.cpp organism.h genome.h genome.cpp
g++ -c -O3 -Wall -fPIC species.cpp
genome.o: genome.cpp genome.h
g++ -c -O3 -Wall -fPIC genome.cpp
population.o: population.cpp population.h organism.h
g++ -c -O3 -Wall -fPIC population.cpp
experiments.o: experiments.cpp experiments.h
g++ -c -O3 -Wall -fPIC experiments.cpp
example.o:
g++ -O3 -Wall -std=c++11 -fopenmp `python3 -m pybind11 --includes` -fPIC -c example.cpp
clean:
rm *.o *.so