Dispatching SIMD instructions + SIMDPP + qmake - c++

I'm developing a QT widget that makes use of SIMD instruction sets. I've compiled 3 versions: SSE3, AVX, and AVX2(simdpp allows to switch between them by a single #define).
Now, what I want is for my widget to switch automatically between these implementations, according to best supported instruction set. Guide that is provided with simdpp makes use of some makefile magic:
CXXFLAGS=""
test: main.o test_sse2.o test_sse3.o test_sse4_1.o test_null.o
g++ $^ -o test
main.o: main.cc
g++ main.cc $(CXXFLAGS) -c -o main.o
test_null.o: test.cc
g++ test.cc -c $(CXXFLAGS) -DSIMDPP_EMIT_DISPATCHER \
-DSIMDPP_DISPATCH_ARCH1=SIMDPP_ARCH_X86_SSE2 \
-DSIMDPP_DISPATCH_ARCH2=SIMDPP_ARCH_X86_SSE3 \
-DSIMDPP_DISPATCH_ARCH3=SIMDPP_ARCH_X86_SSE4_1 -o test_null.o
test_sse2.o: test.cc
g++ test.cc -c $(CXXFLAGS) -DSIMDPP_ARCH_X86_SSE2 -msse2 -o test_sse2.o
test_sse3.o: test.cc
g++ test.cc -c $(CXXFLAGS) -DSIMDPP_ARCH_X86_SSE3 -msse3 -o test_sse3.o
test_sse4_1.o: test.cc
g++ test.cc -c $(CXXFLAGS) -DSIMDPP_ARCH_X86_SSE4_1 -msse4.1 -o test_sse4_1.o
Here is a link to the guide: http://p12tic.github.io/libsimdpp/v2.0~rc2/libsimdpp/arch/dispatch.html
I have no idea how to implement such behavior with qmake. Any ideas?
First that comes to mind is to create a shared library with dispatched code, and link it to the project. Here I'm stuck again. App is cross-platform, which means it has to compile with both GCC and MSVC(vc120, to be exact), which forces using nmake in Windows, and I tried, really, but it was like the worst experience in my whole programmer life.
Thanks in advance, programmers of the world!

sorry if this is a bit late. Hope I can still help.
You need to consider 2 areas: Compile time and run time.
Compile time - need to create code to support different features.
Run time - need to create code to decide which features you can run.
What you are wanting to do is create a dispatcher...
FuncImpl.h:
#pragma once
void execAvx2();
void execAvx();
void execSse();
void execDefault();
FuncImpl.cpp:
// Compile this file once for each variant with different compiler settings.
#if defined(__AVX2__)
void execAvx2()
{
// AVX2 impl
...
}
#elif defined (__AVX__)
void execAvx()
{
// AVX impl
...
}
#elif defined (__SSE4_2__)
void execSse()
{
// Sse impl
...
}
#else
void execDefault()
{
// Vanilla impl
...
}
#endif
DispatchFunc.cpp
#include "FuncImpl.h"
// Decide at runtime which code to run
void dispatchFunc()
{
if(CheckCpuAvx2Flag())
{
execAvx2();
}
else if(CheckCpuAvxFlag())
{
execAvx();
}
else if(CheckCpuSseFlags())
{
execSse();
}
else
{
execDefault();
}
}
What you can do is create a set of QMAKE_EXTRA_COMPILERS.
SampleCompiler.pri (Do this for each variant):
MyCompiler.name = MyCompiler # Name
MyCompiler.input = MY_SOURCES # Symbol of the source list to compile
MyCompiler.dependency_type = TYPE_C
MyCompiler.variable_out = OBJECTS
# EXTRA_CXXFLAGS = -mavx / -mavx2 / -msse4.2
# _var = creates FileName_var.o => replace with own variant (_sse, etc)
MyCompiler.output = ${QMAKE_VAR_OBJECTS_DIR}${QMAKE_FILE_IN_BASE}_var$${first(QMAKE_EXT_OBJ)}
MyCompiler.commands = $${QMAKE_CXX} $(CXXFLAGS) $${EXTRA_CXXFLAGS} $(INCPATH) -c ${QMAKE_FILE_IN} -o${QMAKE_FILE_OUT}
QMAKE_EXTRA_COMPILERS += MyCompiler # Add my compiler
MyProject.pro
...
include(SseCompiler.pri)
include(AvxCompiler.pri)
include(Avx2Compiler.pri)
..
# Normal sources
# Will create FuncImpl.o and DispatchFunc.o
SOURCES += FuncImpl.cpp \
DispatchFunc.cpp
# Give the other compilers their sources
# Will create FuncImpl_avx2.o FuncImpl_avx.o FuncImpl_sse.o
AVX2_SOURCES += FuncImpl.cpp
AVX_SOURCES += FuncImpl.cpp
SSE_SOURCES += FuncImpl.cpp
# Link all objects
...
All you need now is to call dispatchFunc()!
Checking cpu flags is another exercise for you:
cpuid

These are just project defines. You set them with DEFINES += in your .pro file.You set the flags for the instructions sets you want to support and simdpp takes care of selecting the best one for the processor at runtime.
See for example, Add a define to qmake WITH a value?

Here is a qmake .pro file for use with SIMD dispatchers. It is quite verbose, so for more instruction sets, it is better to generate the dispatched blocks by a script, write it to a .pri file and then include it from your main .pro file.
TEMPLATE = app
TARGET = simd_test
INCLUDEPATH += .
QMAKE_CXXFLAGS = -O3 -std=c++17
SOURCES += main.cpp
SOURCES_dispatch = test.cpp
{
# SSE2
DISPATCH_CXXFLAGS = -msse2
DISPATCH_SUFFIX = _sse2
src_dispatch_sse2.name = src_dispatch_sse2
src_dispatch_sse2.input = SOURCES_dispatch
src_dispatch_sse2.dependency_type = TYPE_C
src_dispatch_sse2.variable_out = OBJECTS
src_dispatch_sse2.output = ${QMAKE_VAR_OBJECTS_DIR}${QMAKE_FILE_IN_BASE}$${DISPATCH_SUFFIX}$${first(QMAKE_EXT_OBJ)}
src_dispatch_sse2.commands = $${QMAKE_CXX} $(CXXFLAGS) $${DISPATCH_CXXFLAGS} $(INCPATH) -c ${QMAKE_FILE_IN} -o ${QMAKE_FILE_OUT}
QMAKE_EXTRA_COMPILERS += src_dispatch_sse2
}
{
# SSE3
DISPATCH_CXXFLAGS = -msse3
DISPATCH_SUFFIX = _sse3
src_dispatch_sse3.name = src_dispatch_sse3
src_dispatch_sse3.input = SOURCES_dispatch
src_dispatch_sse3.dependency_type = TYPE_C
src_dispatch_sse3.variable_out = OBJECTS
src_dispatch_sse3.output = ${QMAKE_VAR_OBJECTS_DIR}${QMAKE_FILE_IN_BASE}$${DISPATCH_SUFFIX}$${first(QMAKE_EXT_OBJ)}
src_dispatch_sse3.commands = $${QMAKE_CXX} $(CXXFLAGS) $${DISPATCH_CXXFLAGS} $(INCPATH) -c ${QMAKE_FILE_IN} -o ${QMAKE_FILE_OUT}
QMAKE_EXTRA_COMPILERS += src_dispatch_sse3
}
{
# SSE41
DISPATCH_CXXFLAGS = -msse4.1
DISPATCH_SUFFIX = _sse41
src_dispatch_sse41.name = src_dispatch_sse41
src_dispatch_sse41.input = SOURCES_dispatch
src_dispatch_sse41.dependency_type = TYPE_C
src_dispatch_sse41.variable_out = OBJECTS
src_dispatch_sse41.output = ${QMAKE_VAR_OBJECTS_DIR}${QMAKE_FILE_IN_BASE}$${DISPATCH_SUFFIX}$${first(QMAKE_EXT_OBJ)}
src_dispatch_sse41.commands = $${QMAKE_CXX} $(CXXFLAGS) $${DISPATCH_CXXFLAGS} $(INCPATH) -c ${QMAKE_FILE_IN} -o ${QMAKE_FILE_OUT}
QMAKE_EXTRA_COMPILERS += src_dispatch_sse41
}

Related

function define in .h declared in .cu

I made a project split in three folders : src, include, obj
I am using CImg.h and cuda.h library.
I declare void convolve(cimg_library::CImg<float>&, cimg_library::CImg<float> const &);in convolve.h
and define it in convolve.cu
my files :
main.cpp :
#include "CImg.h"
#include "../include/convolve.h"
using namespace cimg_library;
int main(){
CImg<float> var1("/*path*/");
CImg<float> var2("/*path2*/");
convolve(var1,var2);
//some code
}
convolve.h :
1 #ifndef CONVOLVE_H
2 #define CONVOLVE_H
//some define
10 void convolve(cimg_library::CImg<float>&, cimg_library::CImg<float> const &);
11 #endif //CONVOLVE_H
convolve.cu :
1 void convolve(CImg<float>& img, const CImg<float>& kernel){
//some code
24 kernel<<<dimGrid,dimBlocks>>>(/*some arg*/);
}
and my new makefile :
1 CC=nvcc
2 CX=g++
3 IDIR =../include
4 special_IDIR = /usr/local/cuda-9.0/include
5 LDIR = /usr/local/cuda-9.0/lib64
6 CFLAGS=-I$(special_IDIR) -L$(LDIR)
7
8 LIBS = -lX11 -lpthread -lcudart
9 ODIR = ../obj
10
11
12 _DEPS = convolve.h kernel.cuh
13 DEPS = $(patsubst %,$(IDIR)/%,$(_DEPS))
14
15 _OBJ = main.o convolve.o kernel.o
16 OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))
17
18 $(ODIR)/%.o: %.cpp $(DEPS)
19 $(CX) -x c++ -o $# $< $(CFLAGS) $(LIBS)
20
21 $(ODIR)/%.o: %.cu $(DEPS)
22 $(CC) -cu -o $# $< $(CFLAGS)
23
24 all: $(OBJ)
25 $(CC) -o $# $^ $(CFLAGS)
26
27
28 .PHONY: clean
29
30 clean:
31 rm -f $(ODIR)/*.o *~ core $(INCDIR)/*~
Here is my issue : it says the reference to void convolve(cimg_library::CImg<float>&, cimg_library::CImg<float> const &) is undefined.
I think this is because I never clearly say to the compiler that the definition of convolve is in convolve.cu. But I do not find how to make this link properly.
Thanks for your help !
please don't post code (or Makefile contents) with line numbers. It just gets in the way, usually.
The compilation process here follows a compile...link process. The first stage is the compile stage, and you should be using -c for both g++ and nvcc during this stage (i.e. for both types of Makefile targets)
nvcc has no -cu option. I think maybe you meant -x cu
I also removed the ampersands on the function call.
Here is a simplified example based on what you have shown, primarily removing the CImg stuff and some other unnecessary items:
$ cat main.cpp
#include "convolve.h"
int main(){
int var1 = 0;
int var2 = 1;
convolve(var1, var2);
}
$ cat convolve.h
void convolve(int &, int &);
$ cat convolve.cu
#include "convolve.h"
__global__ void kernel(){};
void convolve(int & i1, int & i2){
kernel<<<1,1>>>();
}
$ cat Makefile
CC=nvcc
CX=g++
IDIR = .
special_IDIR = /usr/local/cuda-9.1/include
LDIR = /usr/local/cuda-9.1/lib64
CFLAGS=-I$(special_IDIR) -L$(LDIR)
LIBS =
ODIR = .
_DEPS = convolve.h
DEPS = $(patsubst %,$(IDIR)/%,$(_DEPS))
_OBJ = main.o convolve.o
OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))
$(ODIR)/%.o: %.cu $(DEPS)
$(CC) -c -x cu -o $# $< $(CFLAGS)
$(ODIR)/%.o: %.cpp $(DEPS)
$(CX) -c -x c++ -o $# $< $(CFLAGS) $(LIBS)
all: $(OBJ)
$(CC) -o $# $^ $(CFLAGS)
.PHONY: clean
clean:
rm -f $(ODIR)/*.o *~ core $(INCDIR)/*~
$ make clean
rm -f ./*.o *~ core /*~
$ make
g++ -c -x c++ -o main.o main.cpp -I/usr/local/cuda-9.1/include -L/usr/local/cuda-9.1/lib64
nvcc -c -x cu -o convolve.o convolve.cu -I/usr/local/cuda-9.1/include -L/usr/local/cuda-9.1/lib64
nvcc -o all main.o convolve.o -I/usr/local/cuda-9.1/include -L/usr/local/cuda-9.1/lib64
$
change the instances of cuda-9.1 to cuda-9.0 in the above Makefile if you have CUDA 9.0 in your setup. The linker libs (-L) are not really needed for the compile commands, however. When linking with nvcc as we are doing here, it's also unnecessary to pass the -I and -L switches that refer to the CUDA includes and CUDA libraries that are part of the toolkit. nvcc already knows how to find those.
There may be any number of additional recommendations or tweaks to the Makefile, for instance the -x cu and -x c++ switches are probably unnecessary (at least they are unnecessary for my simplified example) but my objective here is not to create the perfect Makefile, but to give you a roadmap to get past the issue you are currently witnessing.

Makefile compilation error from cuda/cpp program [duplicate]

I made a project split in three folders : src, include, obj
I am using CImg.h and cuda.h library.
I declare void convolve(cimg_library::CImg<float>&, cimg_library::CImg<float> const &);in convolve.h
and define it in convolve.cu
my files :
main.cpp :
#include "CImg.h"
#include "../include/convolve.h"
using namespace cimg_library;
int main(){
CImg<float> var1("/*path*/");
CImg<float> var2("/*path2*/");
convolve(var1,var2);
//some code
}
convolve.h :
1 #ifndef CONVOLVE_H
2 #define CONVOLVE_H
//some define
10 void convolve(cimg_library::CImg<float>&, cimg_library::CImg<float> const &);
11 #endif //CONVOLVE_H
convolve.cu :
1 void convolve(CImg<float>& img, const CImg<float>& kernel){
//some code
24 kernel<<<dimGrid,dimBlocks>>>(/*some arg*/);
}
and my new makefile :
1 CC=nvcc
2 CX=g++
3 IDIR =../include
4 special_IDIR = /usr/local/cuda-9.0/include
5 LDIR = /usr/local/cuda-9.0/lib64
6 CFLAGS=-I$(special_IDIR) -L$(LDIR)
7
8 LIBS = -lX11 -lpthread -lcudart
9 ODIR = ../obj
10
11
12 _DEPS = convolve.h kernel.cuh
13 DEPS = $(patsubst %,$(IDIR)/%,$(_DEPS))
14
15 _OBJ = main.o convolve.o kernel.o
16 OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))
17
18 $(ODIR)/%.o: %.cpp $(DEPS)
19 $(CX) -x c++ -o $# $< $(CFLAGS) $(LIBS)
20
21 $(ODIR)/%.o: %.cu $(DEPS)
22 $(CC) -cu -o $# $< $(CFLAGS)
23
24 all: $(OBJ)
25 $(CC) -o $# $^ $(CFLAGS)
26
27
28 .PHONY: clean
29
30 clean:
31 rm -f $(ODIR)/*.o *~ core $(INCDIR)/*~
Here is my issue : it says the reference to void convolve(cimg_library::CImg<float>&, cimg_library::CImg<float> const &) is undefined.
I think this is because I never clearly say to the compiler that the definition of convolve is in convolve.cu. But I do not find how to make this link properly.
Thanks for your help !
please don't post code (or Makefile contents) with line numbers. It just gets in the way, usually.
The compilation process here follows a compile...link process. The first stage is the compile stage, and you should be using -c for both g++ and nvcc during this stage (i.e. for both types of Makefile targets)
nvcc has no -cu option. I think maybe you meant -x cu
I also removed the ampersands on the function call.
Here is a simplified example based on what you have shown, primarily removing the CImg stuff and some other unnecessary items:
$ cat main.cpp
#include "convolve.h"
int main(){
int var1 = 0;
int var2 = 1;
convolve(var1, var2);
}
$ cat convolve.h
void convolve(int &, int &);
$ cat convolve.cu
#include "convolve.h"
__global__ void kernel(){};
void convolve(int & i1, int & i2){
kernel<<<1,1>>>();
}
$ cat Makefile
CC=nvcc
CX=g++
IDIR = .
special_IDIR = /usr/local/cuda-9.1/include
LDIR = /usr/local/cuda-9.1/lib64
CFLAGS=-I$(special_IDIR) -L$(LDIR)
LIBS =
ODIR = .
_DEPS = convolve.h
DEPS = $(patsubst %,$(IDIR)/%,$(_DEPS))
_OBJ = main.o convolve.o
OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))
$(ODIR)/%.o: %.cu $(DEPS)
$(CC) -c -x cu -o $# $< $(CFLAGS)
$(ODIR)/%.o: %.cpp $(DEPS)
$(CX) -c -x c++ -o $# $< $(CFLAGS) $(LIBS)
all: $(OBJ)
$(CC) -o $# $^ $(CFLAGS)
.PHONY: clean
clean:
rm -f $(ODIR)/*.o *~ core $(INCDIR)/*~
$ make clean
rm -f ./*.o *~ core /*~
$ make
g++ -c -x c++ -o main.o main.cpp -I/usr/local/cuda-9.1/include -L/usr/local/cuda-9.1/lib64
nvcc -c -x cu -o convolve.o convolve.cu -I/usr/local/cuda-9.1/include -L/usr/local/cuda-9.1/lib64
nvcc -o all main.o convolve.o -I/usr/local/cuda-9.1/include -L/usr/local/cuda-9.1/lib64
$
change the instances of cuda-9.1 to cuda-9.0 in the above Makefile if you have CUDA 9.0 in your setup. The linker libs (-L) are not really needed for the compile commands, however. When linking with nvcc as we are doing here, it's also unnecessary to pass the -I and -L switches that refer to the CUDA includes and CUDA libraries that are part of the toolkit. nvcc already knows how to find those.
There may be any number of additional recommendations or tweaks to the Makefile, for instance the -x cu and -x c++ switches are probably unnecessary (at least they are unnecessary for my simplified example) but my objective here is not to create the perfect Makefile, but to give you a roadmap to get past the issue you are currently witnessing.

The executable test file is not running the tests

I have created a project:
// func.h :
#ifndef FUNCS_H_
#define FUNCS_H_
int addInts(int i, int j);
#endif /* FUNCS_H_ */
// func.cpp
#include "func.h"
int addInts(int i, int j) {
return i + j;
}
// main.cpp
#include "func.h"
#include <iostream>
int main() {
std::cout << addInts(5, 7) << std::endl;
return 0;
}
//makefile
OBJS := \
main.o \
func.o
CXX := g++
CXX_FLAGS := -Wall -Werror -Wextra -std=c++11
all: main_prj
main_prj: $(OBJS)
$(CXX) $(OBJS) -lm -o main
-include $(OBJS:.o=.d)
%.o: %.cpp
$(CXX) -c $(CXX_FLAGS) $*.cpp -o $*.o
clean:
rm -f main $(OBJS)
And I created also a test (boost test) for that function:
// test/main_test.cpp
#define BOOST_TEST_MODULE main_test
#include <boost/test/included/unit_test.hpp>
//____________________________________________________________________________//
// FIXTURES ...
//____________________________________________________________________________//
// test/addInts/addInts_test.cpp
#include <boost/test/unit_test.hpp>
#include "../../func.h"
BOOST_AUTO_TEST_CASE(addInts_5_7) {
int addInts_5_7 = 5 + 7;
BOOST_CHECK_EQUAL(addInts_5_7, addInts(5, 7));
}
// test/makefile
OBJS_TEST := \
main_test.o \
addInts/addInts_test.o \
../func.o
CXX_TEST := g++
CXX_FLAGS_TEST := -Wall -Werror -Wextra -std=c++11
all_test: main_test_prj
main_test_prj: $(OBJS_TEST)
$(CXX_TEST) $(OBJS_TEST) -lm -o main_test
-include $(OBJS_TEST:.o=.d)
%.o: %.cpp
$(CXX_TEST) -c $(CXX_FLAGS_TEST) $*.cpp -o $*.o
clean_test:
rm -f main_test $(OBJS_TEST)
Now, the make commands work, so they clean all the created files, or create the o files and the executables. When I run the main file, the output is correct: 12 ; but when I run the main_test file, the output is a little strange:
Running 1 test case...
*** No errors detected
I would expect the output with running tests and OK/KO, or pass/not pass... I cannot figure what am I doing wrong. Can anyone help me, please?
You have one test.
The output says that one test was run, and that no errors were found.
This output appears to be as documented.
There is no problem here.
Though, sadly, the documentation on how to change this output is rather lacking…

avr32-gcc: Global variable not initialized with function output

I try to initialize a global variable with the output of a function. It works as expected for gnu-gcc, but when compiled with avr32-gcc, the variable is initialized with 0. I want the variable to be initialized by a function since in the real scenario it is a reference (MyClass& myclass = MyClass::getInstance().
Here is the code:
extern "C"
{
#include "led.h"
}
int getInteger()
{
return 10;
}
int my_int = getInteger();
int main (void)
{
LED_On(LED2);
//my_int = getInteger(); //* This line really sets my_int = 10;
if(my_int == 10)
{
LED_On(LED1);
}
while(1){
__asm__ ("nop");
}
return 0;
}
and here is my Makefile:
PROJ_NAME=TEST
# Include paths for project folder
SRCS_INC += -I. -I./preprocessor
# Sources files for project folder (*.c and *.cpp)
SRCS += main.cpp exception_noNMI.S startup_uc3.S trampoline_uc3.S intc.c led.c
AS = avr32-gcc
ASFLAGS = -x assembler-with-cpp -c -mpart=uc3c1512c -mrelax
ASFLAGS += ${SRCS_INC}
CC = avr32-gcc
CFLAGS += -mpart=uc3c1512c
CFLAGS += ${SRCS_INC}
CXX = avr32-g++
LINKER = avr32-g++
OBJCOPY = avr32-objcopy
LDFLAGS = -nostartfiles -mpart=uc3c1512c
LDFLAGS += ${SRCS_INC}
OBJS += $(addsuffix .o, $(basename $(SRCS)))
# Main rule
all: $(PROJ_NAME).elf
# Linking
${PROJ_NAME}.elf: ${OBJS}
#echo Linking...
#$(CXX) $(LDFLAGS) $^ -o $#
#$(OBJCOPY) -O ihex ${OBJCOPYFLAGS} $(PROJ_NAME).elf $(PROJ_NAME).hex
#$(OBJCOPY) -O binary $(PROJ_NAME).elf $(PROJ_NAME).bin
# Assembly files in source folder
%.o: ./%.S
#mkdir -p $(dir $#)
#$(AS) $(ASFLAGS) -c $< -o $#
# C files in source folder
%.o: ./%.c
#mkdir -p $(dir $#)
#$(CC) -c $< -o $# $(CFLAGS)
# CPP files in SRC folder
%.o: ./%.cpp
#mkdir -p $(dir $#)
#$(CXX) -c $< -o $# $(CFLAGS)
.PHONY: clean
clean:
#rm -f $(OBJS) $(PROJ_NAME).elf $(PROJ_NAME).hex $(PROJ_NAME).bin
I also tried to make an init function with __attribute(constructor) as explained in this answer.
Still it is not excuted.
As far as I can see, you're not having optimization enabled. Thus, the compiler may actually make a call to getInteger during startup. Now, you're replacing the default startup with something that you do not show. Maybe you just need to make sure that this is done during startup? Also, you could try to enable optimization and see if the call is resolved.

C++ Ruby Extension with External Libraries

I started a little experiment today: I wrote a C++ class which depends on some other libraries (ALGLIB, Eigen, internal tools) and I wanted to create a Ruby wrapper for that class. I'm currently using Rice to do that. First I wrote a very simple C++ wrapper for my class:
// #file MLPWrapper.h
#pragma once
#include "mlp/MLP.h"
#include <ruby.h>
class MLPWrapper
{
MLP mlp; // <- my C++ class
public:
...
void fit()
{
...
mlp.fit(stop);
}
};
The library's cpp-file is this:
// #file cmlp.cpp
#include "rice/Data_Type.hpp"
#include "rice/Constructor.hpp"
#include "MLPWrapper.h"
using namespace Rice;
extern "C"
void Init_cmlp()
{
Data_Type<MLPWrapper> rb_cMLPWrapper = define_class<MLPWrapper>("MLP")
.define_constructor(Constructor<MLPWrapper>())
...
.define_method("fit", &MLPWrapper::fit);
}
And this is the extconf.rb:
require "mkmf-rice"
$CFLAGS << "-O3"
HEADER_DIRS = [
"..",
"../../external/libraries/eigen-eigen-3.0.1",
"../../external/libraries/alglib/cpp/src",
"../../external/libraries/CMA-ESpp"
]
LIB_DIRS = [
"../../build/external/libraries/alglib/cpp/src",
"../../build/external/libraries/CMA-ESpp/cma-es",
"../../build/src/tools"
]
dir_config("libs", HEADER_DIRS, LIB_DIRS)
have_library("libtools")
have_library("libalglib")
create_makefile("cmlp")
Everything works fine except linking. Obviously the header files of the libraries are included, otherwise it would not compile. But when I run a little test program ("require "cmlp"; mlp = MLP.new") in Ruby it does not find the symbol _ZN6LoggerC1ENS_6TargetESs, which is part of libtools (an enum). This is what happens when I build the C++ extension and execute the test program:
$ ruby extconf.rb
checking for main() in -lrice... yes
checking for main() in -llibtools... no
checking for main() in -llibalglib... no
checking for main() in -llibcmaes... no
creating Makefile
$ make
g++ -I. -I. -I/usr/lib/ruby/1.8/x86_64-linux -I. -I.. -I../../external/libraries/eigen-eigen-3.0.1 -I../../external/libraries/alglib/cpp/src -I../../external/libraries/CMA-ESpp -I/var/lib/gems/1.8/gems/rice-1.4.3/ruby/lib/include -fPIC -fno-strict-aliasing -g -g -O2 -fPIC -O3 -Wall -g -c cmlp.cpp
g++ -shared -o cmlp.so cmlp.o -L. -L/usr/lib -L../../build/external/libraries/alglib/cpp/src -L../../build/external/libraries/CMA-ESpp/cma-es -L../../build/src/tools -L. -Wl,-Bsymbolic-functions -rdynamic -Wl,-export-dynamic -L/var/lib/gems/1.8/gems/rice-1.4.3/ruby/lib/lib -lrice -lruby1.8 -lpthread -lrt -ldl -lcrypt -lm -lc
$ ruby test.rb
ruby: symbol lookup error: ./cmlp.so: undefined symbol: _ZN6LoggerC1ENS_6TargetESs
The libraries are all compiled with CMake (add_library(...)) and are located at
../../build/src/tools/libtools.so
../../build/external/libraries/alglib/cpp/src/libalglib.so
../../build/external/libraries/CMA-ESpp/cma-es/libcmaes.so
I don't know how to solve this problem on my own and I could not find any helpful documentation for my problem. How do i fix this extconf.rb? I appreciate every hint.
edit: OK, I changed the extconf.rb:
require "rubygems"
require "mkmf-rice"
BASE_DIR = "/bla/"
$CFLAGS << " -O3"
dir_config("tools", [BASE_DIR + "src", BASE_DIR + "external/libraries/eigen-eigen-3.0.1"], BASE_DIR + "build/src/tools")
unless have_library("tools")
abort "tools are missing. please compile tools"
end
dir_config("alglib", BASE_DIR + "external/libraries/alglib/cpp/src", BASE_DIR + "build/external/libraries/alglib/cpp/src")
unless have_library("alglib")
abort "alglib is missing. please compile alglib"
end
dir_config("cmaes", BASE_DIR + "external/libraries/CMA-ESpp", BASE_DIR + "build/external/libraries/CMA-ESpp/cma-es")
unless have_library("cmaes")
abort "cmaes is missing. please compile cmaes"
end
create_makefile("cmlp")
The generated Makefile is:
SHELL = /bin/sh
#### Start of system configuration section. ####
srcdir = .
topdir = /usr/lib/ruby/1.8/x86_64-linux
hdrdir = $(topdir)
VPATH = $(srcdir):$(topdir):$(hdrdir)
exec_prefix = $(prefix)
prefix = $(DESTDIR)/usr
sharedstatedir = $(prefix)/com
mandir = $(prefix)/share/man
psdir = $(docdir)
oldincludedir = $(DESTDIR)/usr/include
localedir = $(datarootdir)/locale
bindir = $(exec_prefix)/bin
libexecdir = $(prefix)/lib/ruby1.8
sitedir = $(DESTDIR)/usr/local/lib/site_ruby
htmldir = $(docdir)
vendorarchdir = $(vendorlibdir)/$(sitearch)
includedir = $(prefix)/include
infodir = $(prefix)/share/info
vendorlibdir = $(vendordir)/$(ruby_version)
sysconfdir = $(DESTDIR)/etc
libdir = $(exec_prefix)/lib
sbindir = $(exec_prefix)/sbin
rubylibdir = $(libdir)/ruby/$(ruby_version)
docdir = $(datarootdir)/doc/$(PACKAGE)
dvidir = $(docdir)
vendordir = $(libdir)/ruby/vendor_ruby
datarootdir = $(prefix)/share
pdfdir = $(docdir)
archdir = $(rubylibdir)/$(arch)
sitearchdir = $(sitelibdir)/$(sitearch)
datadir = $(datarootdir)
localstatedir = $(DESTDIR)/var
sitelibdir = $(sitedir)/$(ruby_version)
CC = gcc
LIBRUBY = $(LIBRUBY_SO)
LIBRUBY_A = lib$(RUBY_SO_NAME)-static.a
LIBRUBYARG_SHARED = -l$(RUBY_SO_NAME)
LIBRUBYARG_STATIC = -lruby1.8-static
RUBY_EXTCONF_H =
CFLAGS = -fPIC -fno-strict-aliasing -g -g -O2 -fPIC $(cflags) -O3
INCFLAGS = -I. -I. -I/usr/lib/ruby/1.8/x86_64-linux -I.
DEFS =
CPPFLAGS = -I/bla/external/libraries/CMA-ESpp -I/bla//external/libraries/alglib/cpp/src -I//bla/src -I/bla/external/libraries/eigen-eigen-3.0.1 -I/var/lib/gems/1.8/gems/rice-1.4.3/ruby/lib/include
CXXFLAGS = $(CFLAGS) -Wall -g
ldflags = -L. -Wl,-Bsymbolic-functions -rdynamic -Wl,-export-dynamic -L/var/lib/gems/1.8/gems/rice-1.4.3/ruby/lib/lib
dldflags =
archflag =
DLDFLAGS = $(ldflags) $(dldflags) $(archflag)
LDSHARED = g++ -shared
AR = ar
EXEEXT =
RUBY_INSTALL_NAME = ruby1.8
RUBY_SO_NAME = ruby1.8
arch = x86_64-linux
sitearch = x86_64-linux
ruby_version = 1.8
ruby = /usr/bin/ruby1.8
RUBY = $(ruby)
RM = rm -f
MAKEDIRS = mkdir -p
INSTALL = /usr/bin/install -c
INSTALL_PROG = $(INSTALL) -m 0755
INSTALL_DATA = $(INSTALL) -m 644
COPY = cp
#### End of system configuration section. ####
preload =
CXX = g++
libpath = . $(libdir) /bla/external/libraries/CMA-ESpp/cma-es /bla/build/external/libraries/alglib/cpp/src /bla/build/src/tools
LIBPATH = -L. -L$(libdir) -L/bla/build/external/libraries/CMA-ESpp/cma-es -L/bla/build/external/libraries/alglib/cpp/src -L/bla/build/src/tools
DEFFILE =
CLEANFILES = mkmf.log
DISTCLEANFILES =
extout =
extout_prefix =
target_prefix =
LOCAL_LIBS =
LIBS = -lcmaes -lalglib -ltools -lrice -lruby1.8 -lpthread -lrt -ldl -lcrypt -lm -lc
SRCS = cmlp.cpp
OBJS = cmlp.o
TARGET = cmlp
DLLIB = $(TARGET).so
EXTSTATIC =
STATIC_LIB =
BINDIR = $(bindir)
RUBYCOMMONDIR = $(sitedir)$(target_prefix)
RUBYLIBDIR = $(sitelibdir)$(target_prefix)
RUBYARCHDIR = $(sitearchdir)$(target_prefix)
TARGET_SO = $(DLLIB)
CLEANLIBS = $(TARGET).so $(TARGET).il? $(TARGET).tds $(TARGET).map
CLEANOBJS = *.o *.a *.s[ol] *.pdb *.exp *.bak
all: $(DLLIB)
static: $(STATIC_LIB)
clean:
#-$(RM) $(CLEANLIBS) $(CLEANOBJS) $(CLEANFILES)
distclean: clean
#-$(RM) Makefile $(RUBY_EXTCONF_H) conftest.* mkmf.log
#-$(RM) core ruby$(EXEEXT) *~ $(DISTCLEANFILES)
realclean: distclean
install: install-so install-rb
install-so: $(RUBYARCHDIR)
install-so: $(RUBYARCHDIR)/$(DLLIB)
$(RUBYARCHDIR)/$(DLLIB): $(DLLIB)
$(INSTALL_PROG) $(DLLIB) $(RUBYARCHDIR)
install-rb: pre-install-rb install-rb-default
install-rb-default: pre-install-rb-default
pre-install-rb: Makefile
pre-install-rb-default: Makefile
$(RUBYARCHDIR):
$(MAKEDIRS) $#
site-install: site-install-so site-install-rb
site-install-so: install-so
site-install-rb: install-rb
.SUFFIXES: .c .m .cc .cxx .cpp .C .o
.cc.o:
$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
.cxx.o:
$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
.cpp.o:
$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
.C.o:
$(CXX) $(INCFLAGS) $(CPPFLAGS) $(CXXFLAGS) -c $<
.c.o:
$(CC) $(INCFLAGS) $(CPPFLAGS) $(CFLAGS) -c $<
$(DLLIB): $(OBJS) Makefile
#-$(RM) $#
$(LDSHARED) -o $# $(OBJS) $(LIBPATH) $(DLDFLAGS) $(LOCAL_LIBS) $(LIBS)
$(OBJS): ruby.h defines.h
../../build/src/tools/libtools.so
../../build/external/libraries/alglib/cpp/src/libalglib.so
../../build/external/libraries/CMA-ESpp/cma-es/libcmaes.so
could be the problem. I would try absolute paths here, I could imagine that the pwd is a different one than you expected while running ruby extconf.rb.
Also, I assume you need a dir_config entry for each library you'd like to link in. So
dir_config('libs', HEADER_DIRS, LIB_DIRS)
should be replaced by
dir_config('tools', '<Path to include dir>', '<Path to lib dir>')
dir_config('alglib', '<Path to include dir>', '<Path to lib dir>')
Note that the leading 'lib' should be omitted, just like you would in the -l linker option.
Next, if you want to be absolutely sure that a library is found, replace
have_library('libtools')
by
have_library('tools') or raise
Again, 'lib' is omitted.
If you still can't solve the problem, please post the Makefile generated by `ruby extconf.rb'.