Working on a code that enables Python to call a C++ code compiled into shared library file (.so file) via Python's ctypes module, using the standard ctypes.CDLL method. The C++ code performs the numerical calculations and the Python code mainly serves as a controller and performs data analysis. Everything worked fine. However, as soon as I included jsoncpp library in C++ code, Python started to complain about undefined symbol errors such as undefined symbol: _ZN4Json5ValueaSES0_ etc. I used jsoncpp library in C++ mainly to export data to a JSON file. Having searched the internet for a while, it appears this hasn't been discussed that much. Any idea how to handle this?
Well, looks like the solution is to also include -ljsoncpp flag while compiling towards the shared .so file:
$(CC) -shared -o testcode.so testcode.o -ljsoncpp
Related
I am making a c++ shared library to include in an IDL program on linux. I'm using call_external and that is all working fine. However I am used to dynamically linking fftw in the code I wish to include in the library. The code compiles into a library file fine but when I try to run it the idl program crashes with the error 'symbol lookup error.. undefined symbol: fftw_plan...'
I suspect it is due to the dynamic link not working in the shared library but am not sure how to deal with that. I've looked at the fftw documentation and IDL help. I've tried googling the subject but it is a little niche.
So in my makefile I'm linking the shared library using 'FFTWFLAG=-lfftw3 -lm'
And making the library file with
OneTurn.so: $(OBJECTS)
$(CC) $(INCFLAGS) $(LDFLAGS) $(FFTWFLAG) -o OneTurn.so $(OBJECTS)"
I'm then calling the c++ library from idl with
IDL> a=double(1)
IDL> t=call_external('OneTurn.so', 'OneTurn', a, a, /D_VALUE, /UNLOAD,/CDECL)
Making bunch
/usr/local/harris/idl87/bin/bin.linux.x86_64/idl: symbol lookup error: ./OneTurn.so: undefined symbol: fftw_plan_r2r_2d
If I don't include the fftw stuff everything works as expected.
Update
I had to pass the fftw linker arguments through to the shared library with -Wl,-lfftw3,lm.
This now compiles and runs however the fftw code still does not work! Working on it...
Thanks for your answers and suggestions so far!
Update 2
I was being a muppet: hadn't included a bit of code for one function, that was the undefined symbol... I didn't recognise it as it had a pre- and post-fix. Thanks for your help!
I haven't done this with C++ recently. Could there be a name mangling problem? Look at the .so file with nm to see make sure you have the right symbols and to check their names.
I have 'correctly' installed gsl on mac 10.13.2.
In my c++ program, I am calling like usual, for example:
#include <gsl/gsl_math.h>
However while running the code, it can not find the gsl.
fatal error: 'gsl/gsl_math.h' file not found
I was wondering how to correctly link gsl PATH and libraries.
I have tried,
setting PATH and LD_LIBRARY_PATH in .bash_profile
setting PKG_CONFIG_PATH to .../Gsl2.3/lib/pkgconfig
$which gsl-config returns
/Users/gkdgoutam/Softwares/HEP_Softwares/Install/Gsl2.3/bin/gsl-config
$pkg-config --libs gsl returns
-L/Users/gkdgoutam/Softwares/HEP_Softwares/Install/Gsl2.3/lib -lgsl -lgslcblas -lm
The only solution I can find is to run everytime with gsl linked.
Like:
g++ $(gsl-config --cflags) mycode.cc $(gsl-config --libs) && ./a.out
But I was wondering if the GSL PATH can be set globally so that I can simply run
g++ mycode.cc && ./a.out
This is how c++ code is compiled and built:
COMPILATION
A compilation unit will take each cpp file and work its way through included headers to locate forward declaration of implementations of symbol signatures of used functionality in your code. In your case this involves gsl/gsl.h. If the file cannot be found in the search directories, which you can expand by specifying C_INCLUDE_PATH and or CPLUS_INCLUDE_PATH. If you omit #include <gsl/gsl_math.h>, your code will not compile as there are signatures, which cannot be found for GSL functions, which you use.
LINKING
Once you have compiled all cpp/cc files you need to link the binary, which can be executed. The linking process consists of a search through all symbols in your .o/.obj... files and a matching of the same to what it can find in your object files and the libraries, which you have specified using for example -lgsl. If all goes well, every lookup finds an according binary implementation for your machine's hardware model (i.e. 64bit/32bit ..., Arm, Intel, ... etc). If some are not found you will find linkage errors.
What you are asking is, is there a way that C++ does not work as above? No!
If you leave out #include <gsl/gsl.h> or if said file is not found in your search paths, compilation will not work or. If you omit -lgsl, linking will fail. If you find it annoying to write all the above stuff in the command line, write a Makefile to reduce the building process to ideally a simple command: make.
Don't forget, that if you are linking against the shared library version of GSL, you might need specifying LD_LIBARAY_PATH on Linux and DYLD_LIBRARY_PATH on Macs as well.
TLDR: you cannot ask a c++ compiler / linker to work differently as designed.
I'm trying to use Ruby's FFI library to link functions from the bitcoin-core secp256k1 library.
To make the secp256k1_ecdsa_sign function accessible, I built libsecp256k1 using autotools (as directed in README.md). Then I created a shared object to use in FFI by running g++ -shared secp256k1/src/.libs/libsecp256k1_la-secp256k1.o. Importing this into my Ruby file using FFI let me use the function and everything worked perfectly.
I'm trying to do the exact same with the secp256k1_ecdsa_sign_recoverable function, which is in the same C project, just a different header file. However, I get the error Function 'secp256k1_ecdsa_sign_recoverable' not found in [bin/secp256k1.so] (FFI::NotFoundError).
I think this is either because I'm not creating the shared object properly (created with the aforementioned g++ command), or because this function is for some reason not a public-facing one in the C project (though I don't know enough about C to know how to figure out if this is the case).
If someone could help me figure out how to use this function it would be greatly appreciated.
It ends up the secp256k1_recovery.h file is only included if you specify that when building the libsecp256k1 library. Specifically, I needed to run ./configure --enable-module-recovery instead of ./configure.
I am trying to incorporate a C library into some Rcpp code.
I can use the C library in a C++ program easily. I 'make' the C library, which creates the .a and .dll files in the /lib folder. I can then use the package by including the header in the program and running something like this from command line:
cc myfile.cpp -o myfile -Ipath.to.header path.to.lib.a -lz
This essentially tells the compiler to take the .cpp program, include headers from -I, and to link to two libraries.
Getting this to work with Rcpp shouldn't be overly difficult if I understand makevars correctly (which I unfortunately don't seem to).
I add the library to a folder in my package, and in src I add a makevars and makevars.win that look like this:
PKG_CFLAGS=
# specify header location
PKG_CPPFLAGS=-Ipath.to.lib/include
# specify libs to link to
PKG_LIBS=path.to.lib/lib/file.a -lz
# make library
path.to.lib/lib/file.a:
cd path.to.lib;$(MAKE)
This correctly 'makes' the .a and .dll files for the library, however none of the Rcpp magic runs (i.e. in the build I never see the g++ system call that compiles the files in src), so "no Dll was created".
I am fairly certain this is a problem in my makevars target that makes the library. When I remove that portion from the makevars, and 'make' the library from the command line myself before building the package, I get the proper g++ calls with my -I and -l statements, but I get errors about undefined references.
I notice that the -l statements are only included in the final g++ call where the final .dll is made, but isn't included in the earlier g++ calls where the files with the library headers are compiled.
So I have two problems:
How do I fix my makevars so that it 'makes' the library, but doesn't stop Rcpp from compiling the files in src?
How do I deal with the undefined references? The library is clearly not header-only, so I am guessing it needs the -l statement in the earlier g++ calls, but that may not even be possible.
The best approach is to avoid complicated src/Makevars file altogether.
One easy-ish approach around this: use configure to build your static library, then once you actually build just refer to it in src/Makevars.
I use that scheme in Rblpapi (where we copy an externally supplied library in) and in nloptr where we download nlopt sources and build it 'when needed' (ie when no libnlopt is on the system).
I'm adding two classes and libraries to a system, parent.so and child.so deriving from it.
The problem is when the program is loading child.so it cannot find parent's virtual function's definition from parent.so.
What happens,
nm -D child.so will gives something like (I just changed the names)
U _ZN12PARENT15virtualFunctionEv
The program will crash running
_handle = dlopen(filename, RTLD_NOW|RTLD_GLOBAL); //filename is child.so
it'll give an error with LD_DEBUG = libs
symbol lookup error: undefined symbol: _ZN12PARENT15virtualFunctionEv (fatal)
The thing I cannot explain is, I tried LD_DEBUG = symbols using GDB, when running dlopen, the log shows it tried to look up basically in all libaries in the system except parent.so, where the symbol is defined. But from libs log parent.so is already loaded and code is run, and it is at the same path of all other libraries.
......
27510: symbol=_ZN12PARENT15virtualFunctionEv; lookup in file=/lib/tls/libm.so.6
27510: symbol=_ZN12PARENT15virtualFunctionEv; lookup in file=/lib/tls/libc.so.6
27510: symbol=_ZN12PARENT15virtualFunctionEv; lookup in file=/lib/ld-linux.so.2
27510: child.so: error: symbol lookup error: undefined symbol: _ZN12PARENT15virtualFunctionEv(fatal)
How the program or system is managing which library to look for a symbol's definition?
I'm new to Linux, can anybody point me some directions to work on?
Thanks.
EDIT
The command used to generate parent.so file is
c++ -shared -o parent.so parent.o
Similar for child.so. Is any information missing for linking here? Looks like child is only including parent's header file.
EDIT2
After another test, calling
_handle = dlopen("parent.so", RTLD_NOW|RTLD_GLOBAL);
before the crashing line will solve the problem, which I think means originally parent.so was not loaded. But I'm still not very clear about the cause.
You need to tell the linker that your library libchild.so uses functionality in libparent.so. You do this when you are creating the child library:
g++ -shared -o libchild.so child_file1.o child_file2.o -Lparent_directory -lparent
Note that order is important. Specify the -lparent after all of your object files. You might also need to pass additional options to the linker via the -Wl option to g++.
That still might not be good enough. You might need to add the library that contains libparent.so to the LD_LIBRARY_PATH environment variable.
A couple of gotchas: If you aren't naming those libraries with a lib prefix you will confuse the linker big time. If you aren't compiling your source files with either -fPIC or -fpic you will not have relocatable objects.
Addendum
There's a big potential problem with libraries that depend on other libraries. Suppose you use version 1.5 of the parent package when your compile your child library source files. You manage to get past all of the library dependencies problems. You've specified that your libchild.so depends on libparent.so. Your stuff just works. That is until version 2.0 of the parent package comes out. Now your stuff breaks everywhere it's used, and you haven't changed one line of code.
The way to overcome this problem is to specify at the time you build your child library that the resultant shared library depends specifically on version 1.5 of libparent.so`.
To do this you will need to pass options from g++/gcc to the linker via the -Wl option. Use -Wl,<linker_option>,<linker_option>,... If those linker options need spaces you'll need to backslash-escape them in the command to g++. A couple of key options are -rpath and -soname. For example, -rpath=/path/to/lib,-soname=libparent.so.1.5.
Note very well: You need to use the -soname=libparent.so.1.5 option when you are building libparent.so. This is what lets the system denote that your libchild.so (version 1.0) depends on libparent.so (version 1.5). And you don't build libparent.so. You build libparent.so.1.5. What about libparent.so? That needs to exist to, but it should be a symbolic link to some numbered numbered version (preferably the most recent version) of libparent.so.
Now suppose non-backward compatible parent version 2.0 is compiled and built into a shiny new libparent.so.2.0 and libparent.so is symbolically linked to this shiny new version. An application that uses your clunky old libchild.so (version 1.0) will happily use the clunky old version of libparent.so instead of the shiny new one that breaks everything.
It looks like you're not telling the linker that child.so needs parent.so, use something like the following:
g++ -shared -o libparent.so parent.o
g++ -shared -o libchild.so -lparent child.o
When you build your main program, you have to tell the compiler that it links with those libraries; that way, when it starts, linux will load them for it.
Change their names to libparent.so and libchild.so.
Then compile with something like this:
g++ <your files and flags> -L<folder where the .so's are> -lparent -lchild
EDIT:
Maybe it would be a smaller change to try loading parent.so before child.so. Did you try that already?