CMake "OBJECT" library: clang not linking properly - c++

I have a shared library (which currently compiles, loads and runs) mylib.so. From within this library, I want to use a new function (register it in another external library). The signature is bool my_function(const QVariant *, PyObject **).
This new function is defined in a separate .cpp file which is compiled to an object and then linked to mylib.so.
So I create a new OBJECT with my custom function
ADD_LIBRARY(helper_lib OBJECT helper_lib.cpp)
And include this when building my library
ADD_LIBRARY(mylib SHARED source.cpp $<TARGET_OBJECTS:helper_lib>)
It fails with an "undefined reference to `my_function'"
I can see that
The helper_lib.o file is generated
nm helper_lib.o shows
0000000000000000 T _Z11my_functionPK8QVariantPP7_object
nm mylib.o shows
U my_function
The helper_lib.o is passed to clang++ :
clang++ -fPIC [...] -o my_lib.so mylib.o helper_lib.o [...]
I struggle to see where the mistake happens. I can imagine that there is something wrong in mylib.o which shows an unmangled symbol name which cannot be matched to the helper_lib.o symbol name, but I may as well be totally on the wrong track with this.
helper_lib.h
void my_function();
helper_lib.cpp
#include "helper_lib.h"
void my_function()
{
return;
}
source.cpp is more complicated, as it contains mainly code automatically generated by sip.

It works for me with a simple source.cpp. So it must be that something gets messed up during inclusion, you can try moving #include "helper_lib.h to the top of your source.cpp.
To verify that this has nothing to do with your toolchain, you can try from a clean build directory the following project:
CMakeLists.txt:
cmake_minimum_required(VERSION 3.3)
project(dummy)
ADD_LIBRARY(helper_lib OBJECT helper_lib.cpp)
ADD_LIBRARY(mylib SHARED source.cpp $<TARGET_OBJECTS:helper_lib>)
source.cpp:
#include "helper_lib.h"
void dummy() {
my_function();
}
helper_lib.h:
#pragma once
void my_function();
helper_lib.cpp:
#include "helper_lib.h"
void my_function() {
}
Some documentation.

Related

relocation against xxx in read-only section '.text' - wrong compiler or linux setup in SUSE?

I'm not a frequent user of Linux and I think I did something wrong.
This is the code for a test dynamic library ".so" I'm generating.
class InternalClass
{
public:
int Function(){ return 10; }
};
extern "C"
{
int WrapperFunctionSimple() { return 10; }
void WrapperCreateInstance() {InternalClass* item = new InternalClass(); delete item; }
}
The compilation fails with the following error:
g++ -Wall -fexceptions -O2 -c /home/lidia/compartida/TestLibrary/TestLibrary/main.cpp -o obj/Release/main.o
g++ -shared obj/Release/main.o -o bin/Release/libTestLibrary.so -s
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: obj/Release/main.o: warning: relocation against `_Znwm##GLIBCXX_3.4' in read-only section `.text'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: obj/Release/main.o: relocation R_X86_64_PC32 against symbol `_Znwm##GLIBCXX_3.4' can not be used when making a shared object; recompile with -fPIC
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
I tried with -fPIC as suggested and it compiles. But when using this library, It cannot be loaded when I add that last function:
void WrapperCreateInstance() {InternalClass* item = new InternalClass(); delete item; }
The problem is using the InternalClass, without this function everything works.
I'm using VirtualBox. I installed OpenSUSE 64bit, the app that uses the library is also 64bit. In another linux distribution (Mint), with exactly the same project and settings (without fPIC), it can be compiled. When I use that library (.so) it works in SUSE.
I'm also using :
gcc (SUSE Linux) 7.5.0
g++ (SUSE Linux) 7.5.0
My IDE is Code::Blocks 20 (last version). Settings are empty except for the -m64 flag.
What am I doing wrong? This seems like something advanced Linux users could help me understand.
EDIT:
To add more information, this can compile in Ubuntu with the same settings. Not in SUSE
To me it happened, if one library (A) depends on another one (B) and library A was linked before library B. The solution is to link library B first and then A.
It happened to me when I used GCC to compile a CPP file. So for C++ files use g++ only, and obviously not GCC, which is meant for c files.
I am using cmake and GCC in ubuntu and I got the same error.
In my case, I added the XX.h file to include directories, and used in my main.cpp a "typedef struct x" which is defined in XX.h header file. However I forgot to add XX.c to the executable sources.
This error is cleared when I added XX.c to the target executable sources in add_executable() of Cmake.
I hope I can clearly state my case.
In my case this happened when I had an abstract class A and two derived classes B and C like below.
class BaseAbstractClass
{
public:
virtual void doNothing(void) = 0;
};
class A : public BaseAbstractClass
{
public:
void doNothing(void){
return;
}
};
class B : public BaseAbstractClass
{
public:
void doNothing(void); // Only declaration, definition is nowhere
};
But one of derived classes (class B here) has only declaration of doNothing(), but lacks definition anywhere in project files.
I had a similar issue and the order of linking answer was useful for me. It ended up being I was including the cpp file rather than the header (include was called at wrong time.)

C wrapper around C++ library without unnecessary header files

C++ newbie here. I'm creating a C wrapper around C++ library which expose just foo() function.
wrapper.h
#include "SomeLibrary.h"
#include "SomeAnotherLibrary.h"
#ifdef __cplusplus
extern "C" {
#endif
void foo();
#ifdef __cplusplus
}
#endif
wrapper.cpp
#include "wrapper.h"
void foo() {
// calls to `SomeLibrary.h` and `SomeAnotherLibrary.h` functions...
}
I would like to compile this code just to be able to call foo() from a different C code. Note that I care just about the foo() function. I would like to completely ignore SomeLibrary.h and SomeAnotherLibrary.h header files.
So I tried to compilethe wrapper into object file wrapper.o as follows:
g++ -c wrapper.cpp -o wrapper.o -I../some_library/include -I../some_other_library/include -L../some_library/lib -lSomeFirstLibrary -lSomeSecondLibrary
Problem:
When I used wrapper.o in my C project, it still require me to load bunch of header files from SomeLibrary.h and SomeAnotherLibrary.h (which I do not care about at all). Here is my C project code:
my_project.c:
#include "wrapper.h"
void main() {
foo();
}
And compile it:
gcc my_project.c wrapper.o -o my_project
Which yields following error:
my_program.c:3:28: fatal error: SomeLibrary.h: No such file or directory
Question:
How I should compile the wrapper to ignore all other header files except wrapper.h?
Remove
#include "SomeLibrary.h"
#include "SomeAnotherLibrary.h"
from wrapper.h and put those lines in wrapper.cpp next to #include "wrapper.h".
Then remove the -L../some_library/lib -lSomeFirstLibrary -lSomeSecondLibrary linker related flags from the
g++ -c wrapper.cpp -o wrapper.o -I../some_library/include -I../some_other_library/include
command (-c means no linking is done here so there's no point in passing linker flags)
and move it to the
gcc my_project.c wrapper.o -o my_project -L../some_library/lib -lSomeFirstLibrary -lSomeSecondLibrary
command like this.
Then it should work.
When you get undefined reference to... messages, it means that you are declaring functions, and calling them, but never defining them. The error is coming from the fact that you are linking your libraries in the wrong place. You do not need to link libraries to the .o file, you need to link them to the executable file.

Why doesn't the linker complain of duplicate symbols?

I have a dummy.hpp
#ifndef DUMMY
#define DUMMY
void dummy();
#endif
and a dummy.cpp
#include <iostream>
void dummy() {
std::cerr << "dummy" << std::endl;
}
and a main.cpp which use dummy()
#include "dummy.hpp"
int main(){
dummy();
return 0;
}
Then I compiled dummy.cpp to three libraries, libdummy1.a, libdummy2.a, libdummy.so:
g++ -c -fPIC dummy.cpp
ar rvs libdummy1.a dummy.o
ar rvs libdummy2.a dummy.o
g++ -shared -fPIC -o libdummy.so dummy.cpp
When I try compile main and link the dummy libs
g++ -o main main.cpp -L. -ldummy1 -ldummy2
There is no duplicate symbol error produced by linker. Why does this happen when I link two identical libraries statically?
When I try
g++ -o main main.cpp -L. -ldummy1 -ldummy
There is also no duplicate symbol error, Why?
The loader seems always to choose dynamic libs and not the code compiled in the .o files.
Does it mean the same symbol is always loaded from the .so file if it is both in a .a and a .so file?
Does it mean symbols in the static symbol table in static library never conflict with those in the dynamic symbol table in a .so file?
There's no error in either Scenario 1 (dual static libraries) or Scenario 2 (static and shared libraries) because the linker takes the first object file from a static library, or the first shared library, that it encounters that provides a definition of a symbol it has not yet got a definition for. It simply ignores any later definitions of the same symbol because it already has a good one. In general, the linker only takes what it needs from a library. With static libraries, that's strictly true. With shared libraries, all the symbols in the shared library are available if it satisfied any missing symbol; with some linkers, the symbols of the shared library may be available regardless, but other versions only record the use a shared library if that shared library provides at least one definition.
It's also why you need to link libraries after object files. You could add dummy.o to your linking commands and as long as that appears before the libraries, there'll be no trouble. Add the dummy.o file after libraries and you'll get doubly-defined symbol errors.
The only time you run into problems with this double definitions is if there's an object file in Library 1 that defines both dummy and extra, and there's an object file in Library 2 that defines both dummy and alternative, and the code needs the definitions of both extra and alternative — then you have duplicate definitions of dummy that cause trouble. Indeed, the object files could be in a single library and would cause trouble.
Consider:
/* file1.h */
extern void dummy();
extern int extra(int);
/* file1.cpp */
#include "file1.h"
#include <iostream>
void dummy() { std::cerr << "dummy() from " << __FILE__ << '\n'; }
int extra(int i) { return i + 37; }
/* file2.h */
extern void dummy();
extern int alternative(int);
/* file2.cpp */
#include "file2.h"
#include <iostream>
void dummy() { std::cerr << "dummy() from " << __FILE__ << '\n'; }
int alternative(int i) { return -i; }
/* main.cpp */
#include "file1.h"
#include "file2.h"
int main()
{
return extra(alternative(54));
}
You won't be able to link the object files from the three source files shown because of the double-definition of dummy, even though the main code does not call dummy().
Regarding:
The loader seems always to choose dynamic libs and not compiled in the .o files.
No; the linker always attempts to load object files unconditionally. It scans libraries as it encounters them on the command line, collecting definitions it needs. If the object files precede the libraries, there's not a problem unless two of the object files define the same symbol (does 'one definition rule' ring any bells?). If some of the object files follow libaries, you can run into conflicts if libraries define symbols that the later object files define. Note that when it starts out, the linker is looking for a definition of main. It collects the defined symbols and referenced symbols from each object file it is told about, and keeps adding code (from libraries) until all the referenced symbols are defined.
Does it means the same symbol is always loaded from .so file, if it is both in .a and .so file?
No; it depends which was encountered first. If the .a was encountered first, the .o file is effectively copied from the library into the executable (and the symbol in the shared library is ignored because there's already a definition for it in the executable). If the .so was encountered first, the definition in the .a is ignored because the linker is no longer looking for a definition of that symbol — it's already got one.
Does it mean that symbols in static symbol table in a static library are never in conflict with those in dynamic symbol table in .so file?
You can have conflicts, but the first definition encountered resolves the symbol for the linker. It only runs into conflicts if the code that satisfies the reference causes a conflict by defining other symbols that are needed.
If I link 2 shared libs, can I get conflicts and the link phase failed?
As I noted in a comment:
My immediate reaction is "Yes, you can". It would depend on the content of the two shared libraries, but you could run into problems, I believe. […cogitation…] How would you show this problem? … It's not as easy as it seems at first sight. What is required to demonstrate such a problem? … Or am I overthinking this? … […time to go play with some sample code…]
After some experimentation, my provisional, empirical answer is "No, you can't" (or "No, on at least some systems, you don't run into a conflict"). I'm glad I prevaricated.
Taking the code shown above (2 headers, 3 source files), and running with GCC 5.3.0 on Mac OS X 10.10.5 (Yosemite), I can run:
$ g++ -O -c main.cpp
$ g++ -O -c file1.cpp
$ g++ -O -c file2.cpp
$ g++ -shared -o libfile2.so file2.o
$ g++ -shared -o libfile1.so file1.o
$ g++ -o test2 main.o -L. -lfile1 -lfile2
$ ./test2
$ echo $?
239
$ otool -L test2
test2:
libfile2.so (compatibility version 0.0.0, current version 0.0.0)
libfile1.so (compatibility version 0.0.0, current version 0.0.0)
/opt/gcc/v5.3.0/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.21.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/opt/gcc/v5.3.0/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
$
It is aconventional to use .so as the extension on Mac OS X (it's usually .dylib), but it seems to work.
Then I revised the code in the .cpp files so that extra() calls dummy() before the return, and so does alternative() and main(). After recompiling and rebuilding the shared libraries, I ran the programs. The first line of output is from the dummy() called by main(). Then you get the other two lines produced by alternative() and extra() in that order because the calling sequence for return extra(alternative(54)); demands that.
$ g++ -o test2 main.o -L. -lfile1 -lfile2
$ ./test2
dummy() from file1.cpp
dummy() from file2.cpp
dummy() from file1.cpp
$ g++ -o test2 main.o -L. -lfile2 -lfile1
$ ./test2
dummy() from file2.cpp
dummy() from file2.cpp
dummy() from file1.cpp
$
Note that the function called by main() is the first one that appears in the libraries it is linked with. But (on Mac OS X 10.10.5 at least) the linker does not run into a conflict. Note, though, that the code in each shared object calls 'its own' version of dummy() — there is disagreement between the two shared libraries about which function is dummy(). (It would be interesting to have the dummy() function in separate object files in the shared libraries; then which version of dummy() gets called?) But in the extremely simple scenario shown, the main() function manages to call just one of the dummy() functions. (Note that I'd not be surprised to find differences between platforms for this behaviour. I've identified where I tested the code. Please let me know if you find different behaviour on some platform.)

Static *template* class member across dynamic library

Edit: the comments below the accepted answer show that it might be an issue with the Android dynamic loader.
I have a header for a template class with a static member. At runtime the address of the static member is used in the library and in the client code. The template is implicitly instantiated both in the library and in the client code. It works fine on Linux and OSX, the symbol is duplicated but marked as "uniqued" as shown by nm (see below).
However when I compile for ARM (Android), the symbol is marked weak in both the DSO and the executable. The loader does not unify and the symbol is effectively duplicated at runtime!
I read these:
two instances of a static member, how could that be?
Static template data members storage
and especially this answer:
https://stackoverflow.com/a/2505528/2077394
and:
http://gcc.gnu.org/wiki/Visibility
but I am still a little bit puzzled. I understand that the attributes for visibility helps to optimize, but I thought it should work by default. I know the C++ standard does not care about shared library, but does it means that using shared libraries breaks the standard? (or at least this implementation is not C++ standard conform?)
Bonus: how can I fix it? (and not using template is not an acceptable answer:))
Header:
template<class T>
struct TemplatedClassWithStatic {
static int value;
};
template<class T>
int TemplatedClassWithStatic<T>::value = 0;
shared.cpp:
#include "TemplateWithStatic.hpp"
int *addressFromShared() {
return &TemplatedClassWithStatic<int>::value;
}
main.cpp:
#include "TemplateWithStatic.hpp"
#include <cstdio>
int *addressFromShared();
int main() {
printf("%p %p\n", addressFromShared(), &TemplatedClassWithStatic<int>::value);
}
And building, looking at the symbols definitions:
producing .so:
g++-4.8 -shared src/shared.cpp -o libshared.so -I include/ -fPIC
compiling and linking main:
g++-4.8 src/main.cpp -I include/ -lshared -L.
symbols are marked as "unique":
nm -C -A *.so a.out | grep 'TemplatedClassWithStatic<int>::value'
libshared.so:0000000000200a70 u TemplatedClassWithStatic<int>::value
a.out:00000000006012b0 u TemplatedClassWithStatic<int>::value
producing .so
~/project/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-g++ -o libshared.so src/shared.cpp -I include/ --sysroot=/Users/amini/project/android-ndk-r9/platforms/android-14/arch-arm/ -shared
compiling and linking main
~/project/android-ndk-r9/toolchains/arm-linux-androideabi-4.8/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-g++ src/main.cpp libshared.so -I include/ --sysroot=${HOME}/project/android-ndk-r9/platforms/android-14/arch-arm/ -I ~/project/android-ndk-r9/sources/cxx-stl/gnu-libstdc++/4.8/include -I ~/project/android-ndk-r9/sources/cxx-stl/gnu-libstdc++/4.8/libs/armeabi-v7a/include -I ~/project/android-ndk-r9/sources/cxx-stl/gnu-libstdc++/4.8/include/backward -I ~/project/android-ndk-r9/platforms/android-14/arch-arm/usr/include ~/project/android-ndk-r9/sources/cxx-stl/gnu-libstdc++/4.8/libs/armeabi-v7a/libgnustl_static.a -lgcc
symbols are weak!
nm -C -A *.so a.out | grep 'TemplatedClassWithStatic<int>::value'
libshared.so:00002004 V TemplatedClassWithStatic<int>::value
a.out:00068000 V TemplatedClassWithStatic<int>::value
Edit, note for the context: I was playing with OOLua, a library helping binding C++ to Lua and my unittests were failing when I started to target Android. I don't "own" the code and I would rather modifying it deeply.
Edit, to run it on Android:
adb push libshared.so data/local/tmp/
adb push a.out data/local/tmp/
adb shell "cd data/local/tmp/ ; LD_LIBRARY_PATH=./ ./a.out"
0xb6fd7004 0xb004
Android does not support unique symbols. It is a GNU extension of ELF format that only works with GLIBC 2.11 and above. Android does not use GLIBC at all, it employs a different C runtime called Bionic.
(update) If weak symbols don't work for you (end update) I'm afraid you would have to modify the code such that it does not rely on static data.
There may be some compiler/linker settings that you can tweak to enable this (have you looked at the -fvisibility flag?).
Possibly a GCC attribute modifier may be worth trying (explicitly set __attribute__ ((visibility ("default"))) on the variable).
Failing that, the only workarounds I could suggest are: (all are somewhat ugly):
Explicitly instantiate all forms of the template that are created in the shared library and provide the initializers in its implementation (not in the header). This may or may not work.
Like (1) but use a shim function as a myers singleton for the shared variable (example below).
Allocate a variable in a map for the class based upon rtti (which might also fail across a shared library boundary).
e.g.
template<class T>
struct TemplatedClassWithStatic {
static int& getValue() { return TemplatedClassWithStatic_getValue((T const*)0); }
};
// types used by the shared library.. can be forward declarations here but you run the risk of violating ODR.
int& TemplatedClassWithStatic_getValue(TypeA*);
int& TemplatedClassWithStatic_getValue(TypeB*);
int& TemplatedClassWithStatic_getValue(TypeC*);
shared.cpp
int& TemplatedClassWithStatic_getValue(TypeA*) {
static int v = 0;
return v;
}
int& TemplatedClassWithStatic_getValue(TypeB*) {
static int v = 0;
return v;
}
int& TemplatedClassWithStatic_getValue(TypeC*) {
static int v = 0;
return v;
}
The executable would also have to provide implementations for any types that it uses to instantiate the template.

Undefined Reference To Member function

I'm trying to solve a compiling problem regarding outside defined constructor.
Here's 2 of the classes that have this problem:
//Username.h
#ifndef USERNAME_H
#define USERNAME_H
#include <string>
using namespace std;
class Username{
private:
string Name;
public:
Username(string = "");
string getName() const;
void setName(string);
};
#endif
...
//Username.cpp
#include "Username.h"
Username::Username(string n):Name(n){}
string Username::getName() const{return Name;}
void Username::setName(string n){Name = n;}
...
//User.h
#ifndef USER_H
#define USER_H
#include <string>
#include "Username.h"
using namespace std;
class User{
protected:
Username* UserUsername;
public:
User(string s);
virtual ~User();
Username* getUsername() const;
void setUsername(Username*);
};
#endif
...
//User.cpp
#include "User.h"
User::User(string s):UserUsername(new Username(s)){}
User::~User(){}
Username* User::getUsername() const{return UserUsername;}
void User::setUsername(Username* u){UserUsername=u;}
int main(){return 0;}
If I compile using "g++ User.cpp" I get this error:
/tmp/ccEmWmfl.o: In function `User::User(std::basic_string<char, std::char_traits<char>, std::allocator<char> >)':
User.cpp:(.text+0x3e): undefined reference to `Username::Username(std::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
collect2: ld returned 1 exit status
If I use "g++ User.cpp Username.cpp -o main" or if I use inline constructors/destructors I get no error.
These classes are very easy, but I have tons more to compile that require more than one class.
I'm new at compiling in Ubuntu shell with g++, so please, can someone help me to understand?
g++ User.cpp
This compiles User.cpp and attempts to create an executable from it (ie the linker is invoked). Since User.cpp has a symbol not fully defined in User.cpp (the Username constructor here:
User::User(string s):UserUsername(new Username(s)){}
the symbol is expected to be defined at the link stage. Linking is done by combining all the generated object file output of all the cpps you generated and piecing together the missing symbols. Here you're not telling g++ about where to find the full definition of the Username constructor other than the cpp with main, so its failing.
This, however:
g++ User.cpp Username.cpp -o main
Tells the linker where to find full Username definitions (in the object file generated by compiling Username.cpp). So linking can succeed in filling in the missing pieces by using whats defined in Username.cpp to match up identifiers not defined in User.cpp.
You may think -- "well I've told the compiler about it by including the header file, it should know!". What you've done is declared the symbol. You've made a promise that something will eventually be defined, either during compilation of that one cpp or by later linking it with another object file generated by compiling another cpp. g++ needs to know where you intend to pull all your definitions from so it can build a final executable correctly.
You already answered your question, if you don't add Username.cpp at the compilation process User can't knwo it.
You can use partial compilation, with the -c flag:
g++ -c User.cpp
This produces a User.o file.
You do this partial compilation thing on each of your files
g++ -c Username.cpp
And then you link all the object files (the *.o files) together:
g++ User.o Username.o -o main
Usually, you'd use some build system to automate this process. With that you'd also get other advantages, like not skipping the recompilation of files that didn't change since the last compilation.
This can also happen, if you are trying to call a member function of a template class from the source file other than the implementation file of the template class. For example: you have clas ABC with ABC.h and ABC.cpp (implementation file).
As long as your calls to generic objects lies in ABC.cpp, all works well. However, if you try to include ABC.h in another class say BCD and call any member function of object ABC, you will get the Undefined Reference To Member function error.
Solution: Create a single header file for your template classes including the implementation to avoid this error which is misleading.
User.cpp add:
#include "Username.h"
It's not really a C++ problem, but more a GCC issue. Normally, you'd build a larger program with Make. You then have a makefile rule that builds all .cpp files in a directory and links hem together. An alternative is that your makefile specifically states which .cpp files should be compiled together.
You have to pass all of your implementation files to the g++ executable. The first compilation attempt fails because you're only compiling User.cpp into an object file and linking it into an executable object called main.
In the second attempt you correctly pass both necessary implementation files to the g++ executable. In this case, User.cpp and Username.cpp are compiled into object files and linked together to form the main executable. In this case, the implementation of the Username::Username() constructor is present.