Why the linker is looking for non-needed libs - c++

I have two C++ modules: A and B, where B links to a set of static libs lib*.a (I use * here to mean a set of lib files) and A links to B.
I have CMakeLists.txt for B:
add_library(B STATIC B.cpp)
target_include_directories(B PUBLIC ${CMAKE_CURRENT_SOURCE_DIR} "/path/to/headers/directory/for/lib*.a")
link_directories("Path/to/directory/contains/lib*.a")
target_link_libraries(B PRIVATE lib*)
and CMakeLists.txt for A:
add_library(A STATIC A.cpp)
target_include_directories(A PUBLIC ${CMAKE_CURRENT_SOURCE_DIR} "/path/to/headers/for/libB.a") # compiler outputs libB.a when target name is B
link_directories("Path/to/directory/contains/libB.a")
target_link_libraries(A PRIVATE B)
Everything works fine, until I try to make A a shared lib, then I got error from the linker saying cannot find -l*. I believe the reason is that when I set A to be shared, the compiler looks for shared lib, which is not available.
The thing I don't understand:
B is already a static lib, why the linker needs lib*?
why compiler looks for shared lib when I set A to be shared? I thought even a shared A can be linked to static libs
I do want to note that in A.cpp, I have #include B.hh at the top, and B.hh further includes the headers for lib*.a, that is why I have target_include_directories(B PUBLIC...)

Your initial setup was already not working, but you did not see the error until switching to a shared library.
The reason for this is that all your target_link_libraries dependencies on A and B were not actually processed at all so long as both were static libraries. This has to do with what a static library is: Just a bunch of object files packed together into a single file. A static library is not linked, hence any build dependencies that concern the linker (such as linking in additional static libraries as dependencies) will not get resolved. The build system will keep track of them, but if they are missing, you will not notice unless you actually pass those libraries to something that is actually being linked - such as an executable or, in your case now, a shared library.
Since it can be a bit embarassing to have invisible unresolved dependencies like this, idiomatic CMake will have you model things differently. You never pull in static dependencies directly, instead you use an approach like the following: At the lowest layer, you have a find_library call that finds any external pre-built dependencies on the system (the lib* in your example). You then wrap that up into an imported target so that you never have to touch the raw library path again. To smoothen things out, you can hide all of this inside a find package script, so that your code only ever sees a find_package call that provides a lib* target and don't worry about the internals.
You then model all of your library interdepencies through target dependencies only. Such an approach has several advantages that make it more robust: If CMake can't find the lib*, it will tell you right away when the find_package call fail, instead of waiting until you're trying to link like it does now. By only specifying target-based dependencies, you ensure that the build is actually consuming the libraries that it was configured with (which is very much not ensured with the link_directories based approach you're having now).

Related

CMake: Absolute lib-pathnames given to target_link_library( PRIVATE ) are exported, appear in generated INTERFACE_LINK_LIBRARIES. How do I?

In a large CMake/C++ project our major library target Foo is being added. It has many statically linked dependencies that are provided as absolute-paths to target_link_libraries(). Now I'm writing the CMake to export this library once built, including the generated CMake so a library consumer can simply use CMake find_package(). However the generated FooTargets.cmake is embeding the absolute-paths of the link-libraries instead of just the library names.
Additionally, I've already created an add_library(BarCommon INTERFACE), to wrap up the referenced 3rd-party static libs separately and that seems to be working and the client-application consumed it. That accounts for 90% of the libraries also mentioned by FOO. So really FOO's exported INTERFACE_LINK_LIBRARIES should be about 3 file names instead of 40 absolute-pathnames.
I'm confused, why is the target_link_libraries(FOO PRIVATE ${libs} is defining the exported 'INTERFACE_LINK_LIBRARIES' property?
I would like that generated INTERFACE_LINK_LIBRARIES property to be only library filenames, and optimally have finer control over which filenames end up in it. How do I control this?
Code:
Inside a large CMake Macro for generating targets we have this:
-
I call it to create library/target FOO.
_TARGET_ARGS_LIB_DEPENDS will resolve to ABSOLUTE FILE PATHNAMES for about 40 statically linked library dependencies.
add_library(${TargetName} STATIC ${AllSources} ${_TARGET_ARGS_OBJ_FILES})
<snip>
target_link_libraries(${TargetName}
PRIVATE
$<${GCC}:-Wl,--start-group>
${_TARGET_ARGS_LIB_DEPENDS}
$<${GCC}:-Wl,--end-group>
)
where that longish function returns I have just added these blocks:
target_link_directories(FOO
INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>
$<INSTALL_INTERFACE:foo/lib>
)
target_include_directories(FOO
INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>
$<INSTALL_INTERFACE:foo/include>
)
install(TARGETS Foo
EXPORT
FooTargets
DESTINATION
foo/lib
)
There is some more 'install' boilerplate at the end of my CMakeLists.txt
After I run CMAKE, build the project and run the 'INSTALL' target CMake generated from Visual Studio I can examine generated file: local_install\Release\Windows\Foo\lib\cmake\Foo\FooTargets.cmake shows:
set_target_properties(Nexidia.Workbench.StaticLib.StaticLib PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/Foo/include"
INTERFACE_LINK_DIRECTORIES "${_IMPORT_PREFIX}/Foo/lib"
INTERFACE_LINK_LIBRARIES "\$<LINK_ONLY:\$<0:-Wl,--start-group>>;C:/full/path/to/libA.lib;C:/full/path/to/libB.lib \$<LINK_ONLY:\$<0:-Wl,--end-group>>"
)
(with the full paths to about 40 libraries).
PS. I'm reading Professional CMake: A Practical Guide, so if someone could cite the chapter/sub-section that answers this it might help.
This is because your library is static and so it cannot be correctly linked to consumers without including its dependencies on consumers' link lines. The same thing happens with the shared dependencies of shared libraries. Static dependencies of shared libraries are fully "absorbed" into the shared library and so they don't appear.
CMake is aware of all this, and so to respect the other semantics of PRIVATE, it inserts the $<LINK_ONLY:...> generator expression. This ensures that the libraries are used for linking only and that any associated INTERFACE_* properties are not propagated.
In other words, CMake is behaving correctly here.
You should resolve this by creating and linking to targets for libB and friends that you can either export as part of your main build or import into both your main build and via find_dependency in your distributed package, along with the libraries themselves, of course.
The best advice I can give you is to always link to targets in CMake, never to paths or raw library flags.

Switching away from mixing static and shared libraries

I have a project where I mix between static and shared libs. The shared libs are delivered. Each shared lib is built with its correspondence static lib. A shared lib can depend on other shared libs as well. For example, I have: sharedA, sharedB, sharedC, staticA, staticB and staticC. SharedA is built with staticA inside and similar for other shared. Let's say sharedC, besides its dependency on staticC, also depends on sharedB.
For the static libraries, if they need dependency on another static one, I link against the one's correspondence shared lib. For example:
staticB depends on staticA. Hence, I link staticB against sharedA. The reason for this is to avoid size increase if statically declare staticB's dependency on staticA.
This system has been working. However, problem arises when I build the static libs with hidden visibility. The linker links with no error. But more than one symbols of a specific class might end up in the final symbol map. Therefore, the program can pick up the undesired one, which causes weird behaviour.
I have been reading about cmake object library, and probably switching the static libs to object libs could help in this case. I change the static libs to object, and get undefined symbol errors. I define dependency of an object lib on other, but still could not get rid of all the errors. My questions are:
I am using target_link_libraries(obj2 PUBLIC obj) to declare dependency between 2 object libs. Can this be the reason for undefined symbol error I get? Because according to cmake documentation:
Object Libraries may "link" to other object libraries to get usage requirements, but since they do not have a link step nothing is done with their object files.
Maybe I should switch to target_link_libraries(obj2 $<TARGET_OBJECTS:obj>) so obj2 is built with object files from obj.c? Another thing with this is: obj needs to be a library (created with add_library). In the function where I define all the links, obj is a variable and could not be put after $<TARGET_OBJECTS:. I have been looking around but not able to find anything. Is there a way I can put variable name after $<TARGET_OBJECTS:, or I have to refractor somehow to make this possible?
Any other suggestion on how to do this would be helpful to me

Allow mutual dependencies in Cmake

I am trying to migrate a huge base of code from a custom in-house toolchain to Cmake. This code is comprised of lots of shared libraries and some executable.
Some of these libraries are interdependent. Let's take as examples Lib1 and Lib2, where Lib1 uses symbols from Lib2, and Lib2 uses symbols from Lib1.
In order to to create the Lib1 DLL, we need exported symbols from Lib2, which in turn need exported symbols from Lib1. Currently we resolved this by linking in two phases, as described here : Mutual Imports
). This gives us the following steps :
Create lib1.lib and lib1.exp
Create lib2.lib and lib2.exp
Create lib1.DLL using lib1.obj and lib2.lib/lib2.exp
Create lib2.DLL using lib2.obj and lib1.lib/lib1.exp
I know that this is a code smell, but the code base is way too huge to try and correct all of these cycles. So I would like to get a similar result using Cmake. I did not find any way of doing this from the documentation, except using only custom commands which would completely defeat the purpose of this tool.
The legacy toolchain does the following calls :
# compile sources
cl lib1\SRC\lib1help.cpp ... /Fo"build\cl\lib1.obj"
cl lib2\SRC\lib2help.cpp ... /Fo"build\cl\lib2.obj"
# create lib + exp symbols
lib /DEF build\cl\lib1.obj /OUT:build\link1\lib1.lib
lib /DEF build\cl\lib2.obj /OUT:build\link1\lib2.lib
# create mutual dependant dll using exp
link build\cl\lib1.obj build\link1\lib2.lib /DLL /OUT:build\link2\lib1.dll
link build\cl\lib2.obj build\link1\lib1.lib /DLL /OUT:build\link2\lib2.dll
As of now I have the following CmakeLists.txt:
add_library(lib2 SHARED lib2/src/lib2help.cpp)
add_library(lib1 SHARED lib1/src/lib1help.cpp)
target_include_directories(lib1 PUBLIC lib1/include/)
target_compile_definitions(lib1 PRIVATE __LIB1)
target_link_libraries(lib1 PRIVATE lib2)
target_include_directories(lib2 PUBLIC lib2/include/)
target_compile_definitions(lib2 PRIVATE __LIB2)
target_link_libraries(lib2 PRIVATE lib1)
This gives me the following result :
[cmake] CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle)
[cmake] "lib2" of type SHARED_LIBRARY
[cmake] depends on "lib1" (weak)
[cmake] "lib1" of type SHARED_LIBRARY
[cmake] depends on "lib2" (weak)
Do you guys know a way I could implement this using Cmake ?
For now I plan on only targeting windows systems, but a OS agnostic would be awesome !
Thanks
I was so fixated on making a SHARED library that I didn't explore the STATIC+SHARED way. Thanks to #arrowd and #tsyvaref for the suggestion.
I got it working using the following CmakeLists.txt
add_library(lib1_obj OBJECT src/lib1help.cpp)
target_include_directories(lib1_obj PUBLIC include/)
target_include_directories(lib1_obj PRIVATE $<TARGET_PROPERTY:lib2_obj,INCLUDE_DIRECTORIES>)
target_compile_definitions(lib1_obj PRIVATE __LIB1)
add_library(lib1_static STATIC $<TARGET_OBJECTS:lib1_obj>)
target_link_libraries(lib1_static PUBLIC lib2_static)
set_target_properties(lib1_static PROPERTIES STATIC_LIBRARY_OPTIONS "/DEF")
set_target_properties(lib1_static PROPERTIES OUTPUT_NAME lib1)
add_library(lib1_shared SHARED $<TARGET_OBJECTS:lib1_obj>)
target_link_libraries(lib1_shared PRIVATE lib1_static)
set_target_properties(lib1_shared PROPERTIES RUNTIME_OUTPUT_NAME lib1)
add_library(lib1 ALIAS lib1_shared)
The exact same file is used for lib2, except of course switching names.
I am not sure this is the best way, but I am happy enough for now.
Here is what's happening :
Compile sources into an OBJECT library
Link the mutually dependants libs as static, with the "/DEF" to create export file
Rename the static libs output to facilitate their usage for linker
Link the shared library using .obj, .lib, and .exp previsouly created
Rename the shared library dll to be used easier
Here we can see the content of each DLL using Dependency Walker :

stop cmake target_link_libraries linking both object files of static lib in addtion to the static lib itself

I've tried to build a rather large shared library on Windows with cmake + ninja + msvc, that is composed of multiple static libs from subfolders. So a root CMakeLists.txt looks like:
project (sharedlib CXX)
include(${CMAKE_CURRENT_LIST_DIR}/staticlib1/CMakeLists.txt)
include(${CMAKE_CURRENT_LIST_DIR}/staticlib2/CMakeLists.txt)
add_library(sharedlib SHARED)
target_link_libraries(sharedlib
staticlib1
staticlib2
)
set_target_properties(wux PROPERTIES LINK_FLAGS "/WHOLEARCHIVE")
Where the CMakeLists.txt in sub-folders staticlib1 and staticlib2 both look something like:
add_library(staticlib1 STATIC)
target_sources(staticlib1 PUBLIC
${CMAKE_CURRENT_LIST_DIR}/sourceA.cpp
${CMAKE_CURRENT_LIST_DIR}/sourceB.cpp
)
target_include_directories(staticlib1 PUBLIC
${CMAKE_CURRENT_LIST_DIR}/inc
)
target_compile_options(staticlib1 PUBLIC
/flag1
/flag2
)
When I run cmake --build both staticlib1 and staticlib2 get built no problem. Cool. But when the linker tries to build sharedlib, the cmake-generated rsp file has:
CMakeFiles\sharedlib.dir\staticlib1\sourceA.cpp.obj
CMakeFiles\sharedlib.dir\staticlib1\sourceB.cpp.obj
CMakeFiles\sharedlib.dir\staticlib2\sourceC.cpp.obj
CMakeFiles\sharedlib.dir\staticlib2\sourceD.cpp.obj
staticlib1.lib
staticlib2.lib
So I get linker errors because symbols are defined twice. How do I get cmake to stop adding both the objects and the final libs to the linker rsp?
Technically I don't need the static libs after the fact. However, I cannot simply switch to using OBJECT libraries. The actual project has ~250 static libs, comprising ~3500 object files. The linker dies (out of memory) with just a fraction of the object files.
I must build up the static libs first and then link just them into the shared lib after the fact, to get around the memory limitations of the linker. Our current build scripts follow this pattern just fine, so I know it works. I just need cmake to follow the same pattern.
This is the effect of target_sources command: with PUBLIC keyword it adds sources both for the library (static) and for anyone who links that library.
You should instead use PRIVATE keyword, or, better, add sources in the add_library call itself:
add_library(staticlib1 STATIC
${CMAKE_CURRENT_LIST_DIR}/sourceA.cpp
${CMAKE_CURRENT_LIST_DIR}/sourceB.cpp
)

Static libraries linked against other static libraries with CMake - one works, one doesn't. Why?

Background
I have a project that uses other smaller projects. These projects themselves are made of other projects. A lot of this is legacy or has other managerial-fiat reasons for being arranged as is, so rolling everything into a single project is not an option. Some libraries are pre-compiled on remote shares.
I have 2 main subprojects that are giving me a headache:
Project Foo is an executable and library that links several static subprojects (foo_subproject_1, foo_subproject_n). These subprojects are further linked against static libraries in remote locations (some_lib, some_other_lib). Project Foo's executable compiles, links, and runs correctly
Project Bar is an executable that links several other projects, including libFoo. Linking fails with "undefined reference to " foo_subproject functions
As far as I can tell, the two projects are arranged similarly with their linkage instructions. Looking on SO, I discover that linking static libraries against static libraries shouldn't work, but then I'm confused as to how Project Foo is compiled successfully.
gcc and g++ 4.9.2 are the compilers (extern "C" problems of having some parts in C and some in C++ have already been checked for)
Question
I have misunderstood something about either or both how CMake add_subdirectory works, or about how the linker works. Can someone please explain how Project Foo works successfully, and Project Bar (doesn't) works as expected?
Update I looked closer at foo_lib.a and foo_runtime.
I should have determined that something was off to start with, because foo_runtime is nearly 100MB in size, and foo_lib is only 10KB.
nm reveals that foo_lib.a references a few dozen symbols, most of which are undefined. foo_runtime meanwhile references everything.
Equally confusing is that foo_subproject_1.a is similarly mostly undefined. Again, this is what I expect to see; but I don't understand how foo_runtime can be built from this?
I'm still unclear as to why some_library -> subproject -> foo_runtime is successful, but some_library -> subproject -> foo_lib -> bar isn't. At this stage of my investigations, I am expecting both commands to fail.
Project Foo is arranged (using CMake) thusly:
cmake_minimum_required(VERSION 2.6)
project(foo)
set(FOO_SRCS
# source and headers for main foo project
)
# Project Foo libraries are subdirectories within this project
add_subdirectory(subs/foo_subproject_1)
add_subdirectory(subs/foo_subproject_2)
# Runtime executable
add_executable(foo_runtime main.c ${FOO_SRCS})
target_link_libraries(foo_runtime foo_subproject_1 foo_subproject_2)
# Library version (static library)
add_library(foo_lib STATIC ${FOO_SRCS})
target_link_libraries(foo_lib foo_subproject_1 foo_subproject_2)
Project Foo's subdirectories loosely have the following architecture:
cmake_minimum_required(VERSION 2.6)
project(foo_subproject_<n>)
set(FOO_SUBPROJECT_<N>_SRCS
# source and headers for subproject
)
# foo_subproject's remote libraries are all static
add_library(some_lib STATIC IMPORTED)
set_target_properties(some_lib PROPERTIES IMPORTED_LOCATION /path/to/libsome_lib.a)
add_library(some_other_lib STATIC IMPORTED)
set_target_properties(some_other_lib PROPERTIES IMPORTED_LOCATION /path/to/libsome_other_lib.a)
include_directories(/paths/to/libs/include/)
# Static library for foo_subproject_N, links against static libs above
add_library(foo_subproject_<N> STATIC ${FOO_SUBPROJECT_<N>_SRCS})
target_link_libraries(foo_subproject_<N> some_library some_other_library)
Project Bar is arranged thusly:
cmake_minimum_required(VERSION 2.6)
project(bar)
set(BAR_SRCS
# source and headers for main bar project
)
# Project Bar libraries are remote from Bar's perspective
add_library(foo_lib STATIC IMPORTED)
set_target_properties(foo_lib PROPERTIES IMPORTED_LOCATION /path/to/foo/libfoo_lib.a)
include_directories(/path/to/foo/include/)
# Runtime executable
add_executable(bar main.c ${BAR_SRCS} foo_lib)
Project Bar fails to link (compiles ok) with multiple errors of the form:
bar_frobulator.cpp:123: undefined reference to 'foo_subproject_1_init_frobulation'
where foo_subproject_1_init_frobulation lives in foo_subproject_1
Can someone please explain how Project Foo works successfully, and Project Bar (doesn't) works as expected?
In short: Creating STATIC library doesn't involve linking step!
Details
In the Foo project you have an executable foo_runtime, which "works" because it is linked with proper libraries (e.g. with library foo_subproject_1 which defines foo_subproject_1_init_frobulation symbol).
An executable bar from Bar project doesn't perform that linking, so it fails. The line
target_link_libraries(bar foo_lib)
links with foo_lib, but this library doesn't defines the needed symbol foo_subproject_1_init_frobulation.
Note, that the line
target_link_libraries(foo_lib foo_subproject_1 foo_subproject_2)
in the Foo project doesn't perform actual linking: in general, building a static library doesn't involve linking step.
Given line just propagates include directories (and other compile-features) from foo_subproject_* libraries to the foo_lib one.
How to make it work
Because static library foo_lib doesn't track its dependency, you need to link bar with a library, which knows that. E.g., make foo_lib shared, or combine foo_subproject_* libraries into archive library, as suggested by the referenced question How to combine several C/C++ libraries into one?.
Alternatively, you may build Foo subproject within Bar one and, instead of creation of IMPORTED foo_lib target, use "normal" foo_lib target, created within Foo project. In that case, line
target_link_libraries(bar foo_lib)
would mean for CMake to (actually) link bar with foo_subproject_* libraries, because those libraries are "linked" (in CMake sense) into foo_lib. Again, the last "linking" has a meaning only for CMake: the file foo_lib.a doesn't aware about needing of foo_subproject_* libraries.
Tsyvarev's answer described well-enough what I was actually doing (rather than what I thought I was doing) to get me to look up the right things to answer the root question that I had ("Why does foo_runtime work but bar_runtime doesn't, when both link against static libraries linked against static libraries?")
From the CMake documentation for target_link_libraries:
Library dependencies are transitive by default with this signature. When this target is linked into another target then the libraries linked to this target will appear on the link line for the other target too.
target_link_libraries does not cause linking in the static libraries foo_subproject_1 and foo_subproject_2 (static libraries don't invoke the linker). What is does is makes the list of required libraries available for anything that tries to link with foo_subproject_1 or foo_subproject_2
So, effectively, my foo_runtime and foo_lib target_link_libraries command is, as far as CMake is concerned:
target_link_libraries(foo_runtime foo_subproject_1 some_library some_other_library foo_subproject_2 some_library some_other_library)
target_link_libraries(foo_lib foo_subproject_1 some_library some_other_library foo_subproject_2 some_library some_other_library)
foo_lib, being static, doesn't invoke the linker. foo_runtime, being an executable, does.
bar is a totally different project, so doesn't get to take advantage of the transitive dependencies (and linking fails, because I'm missing all the symbols from the lower-level libraries).
This behaviour of target_link_libraries wasn't expected, as such the behaviour of the project as a whole was somewhat misleading (since it looked like I was bundling a bunch of libraries at the subproject level, and then bundling those at the upper level. In reality, I had just told the upper level what all the libraries it needed were).
To summerise:
Q "Why does foo_runtime work but bar_runtime doesn't, when both link against static libraries linked against static libraries?"
A "foo_runtime is not linking against static libraries linked against static libraries. foo_runtime is linked against more static libraries than you originally thought it was linking against.
bar_runtime doesn't work because your foo_lib is basically empty"