libtool linkage - global state initalization of convenience libraries - c++

I have a setup that does not work, and I have no idea what I am doing wrong here -
I am trying to convert a project from handcrafted Makefiles to autotools, and I think I have most of it set up correctly, as the application and all its convenience libraries builds and links correctly, but there is some trouble with the global state initializers of the convenience libraries.
Some of the libraries follow a pattern like this in the code:
// in global scope of somemodule.cpp
namespace {
bool registered = ModuleShare::registerModule<SomeModule>("SomeModule");
}
this code, along with the actual module source, is compiled into a convenience library using libtool
// libsomething Makefile.am
noinst_LTLIBRARIES = libsomething.la
libsomething_la_SOURCES = \
[ ... ]
moduleshare.cpp moduleshare.h \
somemodule.cpp somemodule.h \
[ ... ]
and this library is built, and referenced in the application Makefile.am as follows:
// someapp Makefile.am
bin_PROGRAMS = someapp
someapp_SOURCES = someapp.c someapp.h
someapp_CPPFLAGS = -I ${top_srcdir}/something
someapp_LDADD = ${top_srcdir}/something/libsomething.la
I have modified ModuleShare::registerModule to verify it is not called:
template<typename T>
static bool registerModule(const std::string &module){
printf("%s\n", module.c_str());
[ ... ]
return true;
}
What could be the reason for this?
EDIT:
At this point, I have figured out that this problem is related to the linker that is allowed to remove unused symbols during linkage. If I link manually using --whole-archive, everything works as expected.
Coming from a C background, I also tried
static void
__attribute__((constructor))
register (void)
{
ModuleShare::registerModule<SomeModule>("SomeModule");
}
but that also does not produce the behaviour that I expected, which is wierd, considering that I rely on this construct a lot in my private C projects.
At this point, I am open to suggestions in any direction. I know that libtool does not provide per-library flags in LDADD, and I can't, and simply don't want to compile everything with --whole-archive just to get rid of these symptoms. I have only limited control over the codebase, but I think what I really need to ask here is, what is a good and reliable way to initialize program state in a convenience library using autotools?
EDIT2:
I think I am a step closer - it seems that the application code has no calls to the convenience library, and hence the linker omits it. The application calls into the library only via a templated function defined in a header file of SomeModule, which relies on the static initializers called in the convenience libraries.
This dependency reversion is screwing over the whole build.
Still, I am unsure how to solve this :/
Thanks,
Andy

Since you're using autotools, you might be in a situation where exclusive use of gcc is feasible. In that case you can apply the `used' attribute to tell gcc to force the linker to include the symbol:
namespace {
__attribute__ ((used))
bool registered = ModuleShare::registerModule<SomeModule>("SomeModule");
}

Instead of using --whole-archive, have you tried to just add -u registered?
As ld manual states:
-u symbol
--undefined=symbol
Force symbol to be entered in the output file as an undefined symbol. Doing this may, for example, trigger linking of additional modules from standard libraries. -u may be repeated with different option arguments to
enter additional undefined symbols. This option is equivalent to the "EXTERN" linker script command.
Edit: I think that what you try to achieve is quite similar to Qt's plugin management. This article details it a bit. And this is 4.8's official documentation.
Taking into account that convenience libraries are statically build, maybe, it would be enough to create a dummy instance inside this convenience library (or a dummy use of registered) and declare it as extern where you use it (this is what Q_EXPORT_PLUGIN2 and Q_IMPORT_PLUGIN are doing).

This answer might require too many code changes for you, but I'll mention it. As I mentioned before, the convenience library model is a kind of static linkage where libraries are collated together before final linkage to an encapsulating library. Only in this case the "library" is an executable. So I'd imagine that stuff in the unnamed namespace (static variables essentially), especially unreferenced variables would be removed. With the above code, it worked as I kind of expected it to.
I was able to get registerModule to print it's message without special linker tricks in a convenience library like this:
somemodule.cpp
// in global scope
namespace registration {
bool somemodule = ModuleShare::registerModule<SomeModule>("SomeModule");
}
someapp.cpp
// somewhere
namespace registration {
extern bool somemodule;
... // and all the other registration bools
}
// in some code that does initialization, possibly
if (!(registration::somemodule && ...)) {
// consequences of initialization failure
}

Related

Linker optimizing out shared library

Using CMake and gcc/clang compilers I want to create a shared library out of this file (Registrator.cpp):
#include ...
namespace
{
struct Registrator
{
Registrator()
{
...
}
~Registrator()
{
...
}
} g_registrator [[gnu::used, gnu::visibility("default")]];
} // namespace
This object is not used anywhere else. Its responsibility is solely to do some things during its (de)construction.
When defining it as:
add_library(${libname} OBJECT Registrator.cpp)
It is a way to prevent the linker from optimizing out this TU as in OBJECT case all sources get into final executable.
However I want to be able to use this as a SHARED library as well. And in case of:
add_library(${libname} SHARED Registrator.cpp)
the library is not linked at all, and gnu::visibility("default") unfortunately doesn't quite cut it.
Is there any way to achieve this? Thanks for help in advance.
Cheers :)
Assuming use of g++, you can disable the --as-needed link behavior. This is enabled by default, which can lead to the issue you are seeing. Try passing the link flag to disable it using CMake's target_link_options() command:
add_library(${libname} SHARED Registrator.cpp)
target_link_options(${libname} PUBLIC "-Wl,--no-as-needed")

I receive different results on UNIX and WIN when use static members with static linking of shared library to executable. Please explain why?

Please consider following peace of code:
// 1. Single header file. Imagine that it is some static library.
// Counter.h
#pragma once
struct Counter
{
Counter()
{
++getCount();
}
static int& getCount()
{
static int counter = 0;
return counter;
}
};
// 2. Shared library (!) :
// main_DLL.cpp
#include <iostream>
#include "counter.h"
extern "C"
{
__declspec(dllexport) // for WIN
void main_DLL()
{
Counter c;
std::cout << "main_DLL : ptr = " << &Counter::getCount()<< " value = " << Counter::getCount() << std::endl;
}
}
// 3. Executable. Shared library statically (!) linked to the executable file.
// main.cpp
#include "counter.h"
#include <iostream>
extern "C"
{
__declspec(dllimport) // for WIN
void main_DLL();
}
int main()
{
main_DLL();
Counter c;
std::cout << "main_EXE : ptr = " << &Counter::getCount() << " value = " << Counter::getCount() << std::endl;
}
Results:
Results for WIN (Win8.1 gcc 5.1.0):
main_DLL : ptr = 0x68783030 value = 1
main_EXE : ptr = 0x403080 value = 1
// conclusion: two different counters
Results for UNIX (Red Hat <I don’t remember version exactly> gcc 4.8.3):
main_DLL : ptr = 0x75693214 value = 1
main_EXE : ptr = 0x75693214 value = 2
// conclusion: the same counter addressed
Building:
Building for WIN:
g++ -c -Wall -Werror -o main_DLL.o main_DLL.cpp
g++ -shared -Wl,--out-implib=libsharedLib.a -o libsharedLib.so main_DLL.o
g++ -Wall –Werror -o simpleExample main.cpp -L./ -lsharedLib
Building for UNIX:
g++ -c -Wall -Werror -fPIC -o main_DLL.o main_DLL.cpp
g++ -shared -fPIC -o libsharedLib.so main_DLL.o
g++ -Wall –Werror -fPIC -o simpleExample main.cpp -L./ -lsharedLib
So, you see that I added –fPIC on UNIX and there is no need to create import library for UNIX, because all exports symbols are included inside shared library. On Windows I use __declspec for it.
For me, results on Windows are pretty much expected. Because shared library and executable are building separately and they should know about static variable in Counter::getCount. They should simply allocate memory for it, that’s why they have different static counters.
I did quite some analysis using tools like nm, objdump. Although I’m not a big expert in them, so I haven’t found anything suspicious. I can provide their output if needed.
Using ldd tool I can see that library linked statically in both cases.
Why I can’t see the same results on Unix for me it’s strange. Could the root cause lie in building options (–fPIC, for example), or I’m missing something?
In windows, A DLL is not exporting global and static symbols unless you add the dllexport statement, therefore, the linker doesn't even know they exists, so it allocate new instance for the static member.
In linux/unix a shared lib is exporting all the global and static symbols, so when the linker find the existence of the static member in the shared lib, it just use its address.
That is the reason for the different result.
EDIT: This is a complete rewrite of the answer. With much more details.
I think that this question deserves more elaborated answer. Especially that there are things that were not mentioned so far.
Dependency Walker
Let me start with referring to the “Dependency Walker” program.
It is a nice program (although these days a bit old-schoolish in its look & feel) that allows analyzing Windows binaries (both EXE and DLL) for symbols that they export/import and their own dependencies to other DLLs. Also it allows showing undecorated symbol names but this seems to be working only with MSVC build binaries. (And some more but that is not important here.)
Thanks to this program crucial information (for this question) can be uncovered. So I encourage you to use it during experiments.
Exporting policy on Linux vs. Windows
SHR already pointed this out but I will mention it also for completeness of the answer. And some extra details.
On Linux every symbol gets exported from a shared library by default. On the other hand on Windows you have to explicitly state which symbols to export from a shared library.
GCC seems however to provide some means of controlling exports in "Windows style". See for example Visibility entry on GCC Wiki.
Also note that there are various ways of exporting on both Linux and Windows. For example both seem to support exporting selectively by providing linker with a list of names for symbols to export. But it also seems that nowadays (on Windows at least) this isn't really used much. __declspec approach seems to be preferred.
What can be exported?
After that general introduction let's now stick to Windows case. Nowadays you export/import symbols from shared libraries by using the __declspec. Just as shown in the question. (Well maybe not exactly that - typically you use a #define to handle bi-directionality as shown in already mentioned Visibility entry on GCC Wiki.)
But the declaration can be applied not only to functions, methods and global variables. It can also be applied to types. For example you can have:
class __declspec(dllexport) Counter { /* ... */ };
Such exporting/importing means in general that all members get exported/imported.
Not so easy!
But it would be too easy, wouldn't it? The complication is that GCC and MSVC handle exporting types differently.
My notes here are based mostly on experiments (checks done using Dependency Walker) so I can be wrong or not precise enough. But I did observe differences in behavior.
In tests I used MSVC 2013 from the Express Edition with update 5. For GCC I used MinGW distro from nuwen.net, version 13.0.
MSVC, when exporting entire type, exports each and every member. Including implicitly defined members (like compiler generated copy constructor). And including inlined functions. Furthermore if inlined function has some static local variables they get exported to (!).
GCC on the other hand seems to be far more restrictive. It doesn't export implicitly defined members. Nor it doesn't export inlined members.
Exporting/Importing inline functions
If instead of exporting entire type you would explicitly export an inlined function then and only then will GCC really export it. But still it will not export static local variables in that function.
Further more if you try to import an inlined function GCC will error. With GCC you cannot define symbols that you are importing. And this happens when you import inlined (and so defined) symbol. So in fact it doesn't make any sense to export inlined functions with GCC.
MSVC allows to import inlined functions. In all cases I checked it didn't seem to actually inline the function but instead called the imported version.
Yet note that because MSVC in case of inlined function exports also its static local variables it would be possible for it to really inline the function (rather than import it) while maintaining a single copy of static local variables. For ordinary programs such behavior is mandated by the Standard (N3337, C++11), in point 7.1.2 ([dcl.fct.spec]) at $4 we can read:
(…) A static local variable in an extern inline function always refers to the same object. (…)
But a program and a shared library are actually more like two programs so they are out of scope for the Standard. Yet MSVC even in that case acts (or better to say: could act) as one would expect from a single program.
Solution
Denis Bakhvalov in a comment provided solution for his own question. The solution is to move getCount function from header to source file and export/import it.
This seems to be the only solution portable between GCC and MSVC. Or to be more precise MSVC allows more solutions to this problem but none of them will work when program is build under GCC.
The variable trick
The above is not entirely true. There is another workaround that will work consistently between GCC and MSVC.
This is to stop using static local variable. Instead make it a global variable (most likely by making it static variable in the class) and export it. This will make the trick as well.
Sadly there is no way (or I don't know any) to directly force exporting/importing static local variables. You have to change them to global variables to do that.
MSVC solutions
With MSVC you have more options.
As mentioned before exporting/importing the inlined function itself (whether directly or through type) will do the job.
Summary
As described above even consistency between GCC and MSVC on Windows only requires care. You have to limit yourself to stay in common subset of allowed solutions.
Keeping the program (source) interoperable between Linux and Windows even if with same compiler (GCC) also requires care.
Luckily there is a common subset for all three environments: GCC on Linux, GCC on Windows and MSVC on Windows. That common subset is described already by mentioned Denis' comment.
So do not inline functions that you intend to export/import. Keep them in sources. And on Windows builds (regardless of compiler) export them explicitly (otherwise you will get linker error anyway since the functions in sources of a shared library will not be available when building program).
Note that this is actually a reasonable approach on its own. Inlining function from shared library doesn't seem wise. It freezes not only the interface but also implementation (of that function). You can no longer change this function freely (and deliver new version of your shared library) since all clients would have to be rebuild since they could have inlined that function. So it is a wise approach by itself not to inline from shared library. And as a bonus it assures that your sources are multi-platform friendly.
Also do have a look into the mentioned Visibility entry on GCC Wiki. It might be reasonable to use that approach (of explicit exports) on Linux as well since it seems cleaner (from design point of view) and more efficient at runtime. While it fits well what you have to do for Windows anyway.

C++ mangling name for use in Emscripten

I'm trying to compile a simple HelloWorld Prgramm from C++ to Javascript using emscripten.
It works fine when I include a main function which call's e.g. the multi function.
Here is my code (HelloWorld.cpp).
#include <stdio.h>
class HelloWorld {
public: void sayHello() {
printf("Hello World Klasse! %f", multi(7));
}
public: double multi(double x){
return x * x;
}
};
However if I don't include a main function the emcc compile always put's out
ERROR root: No functions to process. Make sure you prevented LLVM
from eliminating them as dead (use EXPORTED_FUNCTIONS if necessary,
see the FAQ)
I know about the 'EXPORTED_FUNCTIONS' option which tells what functions should be included into the compile .js file.
I tried various diffrent things:
Using the mangling name, as far as I understood this the name should be '_multi_d10HelloWorldd'. I also tried without classname and some other combinations.
emcc -s HelloWorld.cpp -s EXPORTED_FUNCTIONS='["_multi_d10HelloWorldd"]'
Using the modifier EXPORT_ALL
emcc -s HelloWorld.cpp -s EXPORT_ALL=1
Whatever I do the functions won't be included in the final js file.
From what I understand from the EMCC FAQ I need to use EXPORTED_FUNCTIONS so I can later on call the desired function e.g. 'sayHello' from JS unsing the same method name.
And this is exactly what I need to do later on.
Could someone please point me to a solution or any other possible option which I may have not thought of ?
Is the mangling name I thought of correct ?
Create an "extern c" block. Inside this block define the functions you want to expose to javascript. These functions should be prefixed with an underscore. Inside one of these functions you can instantiate your C++ class.
This is the same approach as one would take when writing a dynamic library, which has the advantage that you can reuse your library in a native program should you wish.

How to test a C++ library usability in configure.in?

I'm working on a C++ project on GNU/Linux and I'm looking for a way to test the existence and usability of IBM Informix's library with the Autotools - namely, editing a configure.in. I don't have experience with Autotools, so basically I'm picking up from the project's configure.in et al. scripts and copying&changing where I feel needs to be changed. IOW, I've been adapting from the existing text in configure.in.
So far I've been using successfully the AC_CHECK_LIB in configure.in to test whether a certain library both exists and is usable. But this only seems to work with libraries with functions, not e.g. classes. Namely, this fails when testing Informix's libifc++.so library:
AC_CHECK_LIB(ifc++, ITString,
INFORMIX_LIB="-L$INFORMIX_LIB_LOCATION/c++ -lifc++ -L$INFORMIX_LIB_LOCATION -L$INFORMIX_LIB_LOCATION/dmi -L$INFORMIX_LIB_LOCATION/esql -lifdmi -lifsql -lifasf -lifgen -lifos -lifgls -lifglx $INFORMIX_LIB_LOCATION/esql/checkapi.o -lm -ldl -lcrypt -lnsl",
echo "* WARNING: libifc++.so not found!"
INFORMIX_INC=""
INFORMIX_LIB=""
)
I've also tried using other combinations, like ITString::ITString, etc.
I haven't found a "pure" function in Informix's API (i.e., one that isn't contexted in a C++ class). So I'm hoping that either there's a way to use AC_CHECK_LIB in this context, or there's another autoconf/configure.in "command" for this specific use.
Thanks in advance for your feedback.
You've discovered a shortcoming of autotools, but one that can't really be helped. Autotools checks for symbol names in the library binary, and unlike C where symbol names of functions are identical to the function names, C++ "mangles" function's symbol names to accomplish things like function overloading. What's worse is that C++ doesn't really even have a "standard" mangling convention, so different C++ compilers may produce different symbol names for the same function. Thus, autotools can't check for C++ symbol names in a reliable manner.
Does the library you are trying to use have any functions that are declared with extern "C"? This causes the C++ compiler to generate standardized C-style symbol names, and autotools will be able to find them.
I ran into this issue trying to detect gtest and gmock (the Google unit testing and object mocking frameworks) with Autotools, and here's what I came up with:
# gtest has a main function in the gtest_main library with C linkage, we can test for that.
AC_CHECK_LIB([gtest_main], [main], [HAVE_GTEST=1] [TEST_LIBS="$TEST_LIBS -lgtest_main"],
AC_MSG_WARN([libgtest (Google C++ Unit Testing Framework) is not installed. Will not be able to make check.]))
# gmock has no functions with C linkage, so this is a roundabout way of testing for it. We create a small test
# program that tries to instantiate one of gmock's objects, and try to link it with -lgmock and see if it works.
if test "$HAVE_GTEST"
then
saved_ldflags="${LDFLAGS}"
LDFLAGS="${LDFLAGS} -lgtest -lgmock"
AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <gmock/gmock.h>], [testing::Cardinality dummy])],
[TEST_LIBS="$TEST_LIBS -lgmock"] [HAVE_GMOCK=1],
[AC_MSG_WARN([libgmock (Google C++ Object Mocking Framework) is not installed. Will not be able to make check.])])
LDFLAGS="${saved_ldflags}"
fi
There might be a cleaner way of achieving this, but I think your problem is that C++ methods get "mangled" to allow additional information about the method (argument & return types etc) to be encoded. For example; the method int A::foo(void) will get mangled to something like __ZN1A3fooEv.
So you need to find the mangled name of a method in the library. You can do this by using the nm command on Unix-like OSs:
$ nm libifc++.so | grep ITString
It's worth mentioning that the exact mangling format varies across different compilers; and so by embedding a certain compiler's mangled symbol in your configure.in it may not work on other platforms - YMMV.
Note: you can use the c++filt utility to demangle a name back to it's human-readable form; so for the example I gave previously:
$ c++filt __ZN1A3fooEv
A::foo()
See Name Mangling in C++ on Wikipedia for more information.
If the library you are checking for supports pkg-config, this becomes very easy. Here is all I added to my configure.in to check for and enable gtest and gmock:
dnl ************************************
dnl Check for googletest and googlemock
dnl ************************************
PKG_CHECK_MODULES(gtestmock, libgtest >= 0.4.0, libgmock >= 0.4.0)
AC_SUBST(gtestmock_LIBS)
AC_SUBST(gtestmock_CFLAGS)
And then in my Makefile.am somewhere:
sometarget_CXXFLAGS = $(gtestmock_CFLAGS) $(AM_CXXFLAGS)
sometarget_LDADD = $(gtestmock_LIBS)
Pretty trivial, eh?
AC_LANG_CPLUSPLUS
AC_CHECK_LIB(Sockets, main)
Caveat: http://lists.gnu.org/archive/html/autoconf/2006-09/msg00019.html

linker error: undefined reference c++

I've tried looking at similar problems, but could not easily find one that helped my problem.
I've created a project in C++ and am working on UNIX to compile, link, and run it. My specific problem is an undefined reference to a method I declare in a separate file.
In the file SharedCache.cpp, I have the following method:
int SharedCache::replaceLine(ullong address){
int evictPID = -1;
int cacheSet = calcCacheSet( address );
//random uniformly-distributed value for cache line
float numLines = static_cast<float>(CACHE_LINES_PER_SET);
uint cacheLine = static_cast<uint>(uniformDistr( numLines ));
if(cache[cacheSet][cacheLine] != NULL){
evictPID = cache[cacheSet][cacheLine]->PID;
}
uint PID= calcPID(address);
uint tag= calcTag(address);
cache[cacheSet][cacheLine]->setLine(PID, tag); //mutex method
return PID;
}
The line uint cacheLine = static_cast<uint>( uniformDistr( numLines )); makes a call to the function I want to use from another file. The linker error is an undefined reference to this method.
uniformDistr( float ); is declared in the header distributions.h and defined in distributions.cpp. In my project options I set the linker flag -distributions and I also compiled the distributions.cpp to make sure a distributions.o file exists to link with.
From here, I don't know where to go, because this has not solved the problem.
Without more precise information on which compiler/linker commands were invoked and the exact error outputs, it is difficult to provide a good answer.
However, from your description of what you did, it seems that you are not passing distributions.o to the linker. Unlike other languages where the compiler/linker search for object files to link in automatically, C++ linkers require an explicit specification of the objects to link together.
Your use of -ldistributions here is incorrect: the -l flag is used to link to a static or dynamic library (respectively .a and .so files on Linux), whereas you want to specify another object file that the linker should consider. Specifying -ldistributions makes the linker look for distributions.a or distributions.so in the standard library locations.
Basically, your linker invocation now looks something like this (probably with many more flags and libraries):
gcc -o my_program SharedCache.o -ldistributions
To correctly link the distributions code in, you need to make it look something like (again, many flags and libraries probably missing compared to the real thing):
gcc -o my_program SharedCache.o distributions.o
This should resolve the missing symbols issue and produce a working binary (or at the very least a different error ;-) ). How to do this in KDevelop however I do not know.
I've not used KDevelop, however, on the command line you would just add distributions.o as an input file to the linking process. No need for dashes or leaving off the .o extension.
Alternatively, can you just add distributions.cpp to your KDevelop project? That way it should get compiled and linked automatically (this is how it works in things like Visual Studio or Eclipse).
Did you add the distributions.cpp to your makefile? Also I believe the required linker flag is -ldistributions.