I am trying to migrate a huge base of code from a custom in-house toolchain to Cmake. This code is comprised of lots of shared libraries and some executable.
Some of these libraries are interdependent. Let's take as examples Lib1 and Lib2, where Lib1 uses symbols from Lib2, and Lib2 uses symbols from Lib1.
In order to to create the Lib1 DLL, we need exported symbols from Lib2, which in turn need exported symbols from Lib1. Currently we resolved this by linking in two phases, as described here : Mutual Imports
). This gives us the following steps :
Create lib1.lib and lib1.exp
Create lib2.lib and lib2.exp
Create lib1.DLL using lib1.obj and lib2.lib/lib2.exp
Create lib2.DLL using lib2.obj and lib1.lib/lib1.exp
I know that this is a code smell, but the code base is way too huge to try and correct all of these cycles. So I would like to get a similar result using Cmake. I did not find any way of doing this from the documentation, except using only custom commands which would completely defeat the purpose of this tool.
The legacy toolchain does the following calls :
# compile sources
cl lib1\SRC\lib1help.cpp ... /Fo"build\cl\lib1.obj"
cl lib2\SRC\lib2help.cpp ... /Fo"build\cl\lib2.obj"
# create lib + exp symbols
lib /DEF build\cl\lib1.obj /OUT:build\link1\lib1.lib
lib /DEF build\cl\lib2.obj /OUT:build\link1\lib2.lib
# create mutual dependant dll using exp
link build\cl\lib1.obj build\link1\lib2.lib /DLL /OUT:build\link2\lib1.dll
link build\cl\lib2.obj build\link1\lib1.lib /DLL /OUT:build\link2\lib2.dll
As of now I have the following CmakeLists.txt:
add_library(lib2 SHARED lib2/src/lib2help.cpp)
add_library(lib1 SHARED lib1/src/lib1help.cpp)
target_include_directories(lib1 PUBLIC lib1/include/)
target_compile_definitions(lib1 PRIVATE __LIB1)
target_link_libraries(lib1 PRIVATE lib2)
target_include_directories(lib2 PUBLIC lib2/include/)
target_compile_definitions(lib2 PRIVATE __LIB2)
target_link_libraries(lib2 PRIVATE lib1)
This gives me the following result :
[cmake] CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle)
[cmake] "lib2" of type SHARED_LIBRARY
[cmake] depends on "lib1" (weak)
[cmake] "lib1" of type SHARED_LIBRARY
[cmake] depends on "lib2" (weak)
Do you guys know a way I could implement this using Cmake ?
For now I plan on only targeting windows systems, but a OS agnostic would be awesome !
Thanks
I was so fixated on making a SHARED library that I didn't explore the STATIC+SHARED way. Thanks to #arrowd and #tsyvaref for the suggestion.
I got it working using the following CmakeLists.txt
add_library(lib1_obj OBJECT src/lib1help.cpp)
target_include_directories(lib1_obj PUBLIC include/)
target_include_directories(lib1_obj PRIVATE $<TARGET_PROPERTY:lib2_obj,INCLUDE_DIRECTORIES>)
target_compile_definitions(lib1_obj PRIVATE __LIB1)
add_library(lib1_static STATIC $<TARGET_OBJECTS:lib1_obj>)
target_link_libraries(lib1_static PUBLIC lib2_static)
set_target_properties(lib1_static PROPERTIES STATIC_LIBRARY_OPTIONS "/DEF")
set_target_properties(lib1_static PROPERTIES OUTPUT_NAME lib1)
add_library(lib1_shared SHARED $<TARGET_OBJECTS:lib1_obj>)
target_link_libraries(lib1_shared PRIVATE lib1_static)
set_target_properties(lib1_shared PROPERTIES RUNTIME_OUTPUT_NAME lib1)
add_library(lib1 ALIAS lib1_shared)
The exact same file is used for lib2, except of course switching names.
I am not sure this is the best way, but I am happy enough for now.
Here is what's happening :
Compile sources into an OBJECT library
Link the mutually dependants libs as static, with the "/DEF" to create export file
Rename the static libs output to facilitate their usage for linker
Link the shared library using .obj, .lib, and .exp previsouly created
Rename the shared library dll to be used easier
Here we can see the content of each DLL using Dependency Walker :
Related
In a large CMake/C++ project our major library target Foo is being added. It has many statically linked dependencies that are provided as absolute-paths to target_link_libraries(). Now I'm writing the CMake to export this library once built, including the generated CMake so a library consumer can simply use CMake find_package(). However the generated FooTargets.cmake is embeding the absolute-paths of the link-libraries instead of just the library names.
Additionally, I've already created an add_library(BarCommon INTERFACE), to wrap up the referenced 3rd-party static libs separately and that seems to be working and the client-application consumed it. That accounts for 90% of the libraries also mentioned by FOO. So really FOO's exported INTERFACE_LINK_LIBRARIES should be about 3 file names instead of 40 absolute-pathnames.
I'm confused, why is the target_link_libraries(FOO PRIVATE ${libs} is defining the exported 'INTERFACE_LINK_LIBRARIES' property?
I would like that generated INTERFACE_LINK_LIBRARIES property to be only library filenames, and optimally have finer control over which filenames end up in it. How do I control this?
Code:
Inside a large CMake Macro for generating targets we have this:
-
I call it to create library/target FOO.
_TARGET_ARGS_LIB_DEPENDS will resolve to ABSOLUTE FILE PATHNAMES for about 40 statically linked library dependencies.
add_library(${TargetName} STATIC ${AllSources} ${_TARGET_ARGS_OBJ_FILES})
<snip>
target_link_libraries(${TargetName}
PRIVATE
$<${GCC}:-Wl,--start-group>
${_TARGET_ARGS_LIB_DEPENDS}
$<${GCC}:-Wl,--end-group>
)
where that longish function returns I have just added these blocks:
target_link_directories(FOO
INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>
$<INSTALL_INTERFACE:foo/lib>
)
target_include_directories(FOO
INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>
$<INSTALL_INTERFACE:foo/include>
)
install(TARGETS Foo
EXPORT
FooTargets
DESTINATION
foo/lib
)
There is some more 'install' boilerplate at the end of my CMakeLists.txt
After I run CMAKE, build the project and run the 'INSTALL' target CMake generated from Visual Studio I can examine generated file: local_install\Release\Windows\Foo\lib\cmake\Foo\FooTargets.cmake shows:
set_target_properties(Nexidia.Workbench.StaticLib.StaticLib PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/Foo/include"
INTERFACE_LINK_DIRECTORIES "${_IMPORT_PREFIX}/Foo/lib"
INTERFACE_LINK_LIBRARIES "\$<LINK_ONLY:\$<0:-Wl,--start-group>>;C:/full/path/to/libA.lib;C:/full/path/to/libB.lib \$<LINK_ONLY:\$<0:-Wl,--end-group>>"
)
(with the full paths to about 40 libraries).
PS. I'm reading Professional CMake: A Practical Guide, so if someone could cite the chapter/sub-section that answers this it might help.
This is because your library is static and so it cannot be correctly linked to consumers without including its dependencies on consumers' link lines. The same thing happens with the shared dependencies of shared libraries. Static dependencies of shared libraries are fully "absorbed" into the shared library and so they don't appear.
CMake is aware of all this, and so to respect the other semantics of PRIVATE, it inserts the $<LINK_ONLY:...> generator expression. This ensures that the libraries are used for linking only and that any associated INTERFACE_* properties are not propagated.
In other words, CMake is behaving correctly here.
You should resolve this by creating and linking to targets for libB and friends that you can either export as part of your main build or import into both your main build and via find_dependency in your distributed package, along with the libraries themselves, of course.
The best advice I can give you is to always link to targets in CMake, never to paths or raw library flags.
I did wrote some CMake code to transitively get all library dependencies of a given executable target and copy dlls next to it (yes Windows World)
Then I got an external library that does not support CMake. That have a single implib lets call it a.lib with an a.dll, this library also depends on another dll without any available implib (b.dll)
So I wrote a aConfig.cmake like this:
set(a_FOUND TRUE)
add_library(a SHARED IMPORTED)
set_target_properties(a PROPERTIES
IMPORTED_LINK_INTERFACE_LANGUAGES "CXX"
IMPORTED_LOCATION "${a_DIR}/bin/a.dll"
IMPORTED_IMPLIB "${a_DIR}/lib/a.lib"
)
add_library(b SHARED IMPORTED)
set_target_properties(b PROPERTIES
IMPORTED_LOCATION "${a_DIR}/bin/b.dll"
)
# to express a dependency without requiring implib at link time
target_link_library(a PRIVATE b)
I end up with this (expected) error
IMPORTED library can only be used with the INTERFACE keyword of
target_link_libraries
Then I modified b to become an INTERFACE (It feels strange here):
add_library(b INTERFACE IMPORTED)
set_target_properties(b PROPERTIES
IMPORTED_LOCATION "${a_DIR}/bin/b.dll"
)
target_link_library(a INTERFACE b)
with CMake 3.18 my transitive loop does not want to ask for an IMPORTED_LOCATION on an INTERFACE
INTERFACE_LIBRARY targets may only have whitelisted properties. The
property "IMPORTED_LOCATION" is not allowed.
but with CMake 3.19 it works and do exactly what I am expecting. So I read the Release Note and except this statement :
Interface Libraries may now have source files added via add_library() or target_sources(). Those with sources will be generated as part of the build system.
nothing seems related to this.
Since I found the CMake 3.18 error, I feel that what I am doing is wrong but I cannot find a better way to express this runtime dependency.
Does anyone have a better idea on how to express this dependency or does it feels right like this ?
In CMake 3.19 and after you could create b target for a dll without a lib in a single line:
add_library(b INTERFACE "${a_DIR}/bin/b.dll")
So when you link a with b, a will get a dependency from the file ${a_DIR}/bin/b.dll.
This is what CMake 3.19 introduces: a dependency of INTERFACE library from the file.
For implement that dependency in CMake pre-3.19, one need to create additional custom target:
add_custom_target(b_files DEPENDS "${a_DIR}/bin/b.dll")
add_library(b INTERFACE)
add_dependencies(b b_files)
I have two C++ modules: A and B, where B links to a set of static libs lib*.a (I use * here to mean a set of lib files) and A links to B.
I have CMakeLists.txt for B:
add_library(B STATIC B.cpp)
target_include_directories(B PUBLIC ${CMAKE_CURRENT_SOURCE_DIR} "/path/to/headers/directory/for/lib*.a")
link_directories("Path/to/directory/contains/lib*.a")
target_link_libraries(B PRIVATE lib*)
and CMakeLists.txt for A:
add_library(A STATIC A.cpp)
target_include_directories(A PUBLIC ${CMAKE_CURRENT_SOURCE_DIR} "/path/to/headers/for/libB.a") # compiler outputs libB.a when target name is B
link_directories("Path/to/directory/contains/libB.a")
target_link_libraries(A PRIVATE B)
Everything works fine, until I try to make A a shared lib, then I got error from the linker saying cannot find -l*. I believe the reason is that when I set A to be shared, the compiler looks for shared lib, which is not available.
The thing I don't understand:
B is already a static lib, why the linker needs lib*?
why compiler looks for shared lib when I set A to be shared? I thought even a shared A can be linked to static libs
I do want to note that in A.cpp, I have #include B.hh at the top, and B.hh further includes the headers for lib*.a, that is why I have target_include_directories(B PUBLIC...)
Your initial setup was already not working, but you did not see the error until switching to a shared library.
The reason for this is that all your target_link_libraries dependencies on A and B were not actually processed at all so long as both were static libraries. This has to do with what a static library is: Just a bunch of object files packed together into a single file. A static library is not linked, hence any build dependencies that concern the linker (such as linking in additional static libraries as dependencies) will not get resolved. The build system will keep track of them, but if they are missing, you will not notice unless you actually pass those libraries to something that is actually being linked - such as an executable or, in your case now, a shared library.
Since it can be a bit embarassing to have invisible unresolved dependencies like this, idiomatic CMake will have you model things differently. You never pull in static dependencies directly, instead you use an approach like the following: At the lowest layer, you have a find_library call that finds any external pre-built dependencies on the system (the lib* in your example). You then wrap that up into an imported target so that you never have to touch the raw library path again. To smoothen things out, you can hide all of this inside a find package script, so that your code only ever sees a find_package call that provides a lib* target and don't worry about the internals.
You then model all of your library interdepencies through target dependencies only. Such an approach has several advantages that make it more robust: If CMake can't find the lib*, it will tell you right away when the find_package call fail, instead of waiting until you're trying to link like it does now. By only specifying target-based dependencies, you ensure that the build is actually consuming the libraries that it was configured with (which is very much not ensured with the link_directories based approach you're having now).
I've tried to build a rather large shared library on Windows with cmake + ninja + msvc, that is composed of multiple static libs from subfolders. So a root CMakeLists.txt looks like:
project (sharedlib CXX)
include(${CMAKE_CURRENT_LIST_DIR}/staticlib1/CMakeLists.txt)
include(${CMAKE_CURRENT_LIST_DIR}/staticlib2/CMakeLists.txt)
add_library(sharedlib SHARED)
target_link_libraries(sharedlib
staticlib1
staticlib2
)
set_target_properties(wux PROPERTIES LINK_FLAGS "/WHOLEARCHIVE")
Where the CMakeLists.txt in sub-folders staticlib1 and staticlib2 both look something like:
add_library(staticlib1 STATIC)
target_sources(staticlib1 PUBLIC
${CMAKE_CURRENT_LIST_DIR}/sourceA.cpp
${CMAKE_CURRENT_LIST_DIR}/sourceB.cpp
)
target_include_directories(staticlib1 PUBLIC
${CMAKE_CURRENT_LIST_DIR}/inc
)
target_compile_options(staticlib1 PUBLIC
/flag1
/flag2
)
When I run cmake --build both staticlib1 and staticlib2 get built no problem. Cool. But when the linker tries to build sharedlib, the cmake-generated rsp file has:
CMakeFiles\sharedlib.dir\staticlib1\sourceA.cpp.obj
CMakeFiles\sharedlib.dir\staticlib1\sourceB.cpp.obj
CMakeFiles\sharedlib.dir\staticlib2\sourceC.cpp.obj
CMakeFiles\sharedlib.dir\staticlib2\sourceD.cpp.obj
staticlib1.lib
staticlib2.lib
So I get linker errors because symbols are defined twice. How do I get cmake to stop adding both the objects and the final libs to the linker rsp?
Technically I don't need the static libs after the fact. However, I cannot simply switch to using OBJECT libraries. The actual project has ~250 static libs, comprising ~3500 object files. The linker dies (out of memory) with just a fraction of the object files.
I must build up the static libs first and then link just them into the shared lib after the fact, to get around the memory limitations of the linker. Our current build scripts follow this pattern just fine, so I know it works. I just need cmake to follow the same pattern.
This is the effect of target_sources command: with PUBLIC keyword it adds sources both for the library (static) and for anyone who links that library.
You should instead use PRIVATE keyword, or, better, add sources in the add_library call itself:
add_library(staticlib1 STATIC
${CMAKE_CURRENT_LIST_DIR}/sourceA.cpp
${CMAKE_CURRENT_LIST_DIR}/sourceB.cpp
)
I have a project using 3 CMakeLists.txt:
CMakeLists.txt C is my executable and depends on
CMakeLists.txt B which is a static lib and that depends on
CMakeLists.txt A which is also a static lib and depends on an external lib
In CMakeLists.txt C I specify my dependency against B using using target_link_libraries() and I do the same for the dependency of B against A. In CMakeLists.txt A I specify the dependency against the external lib.
I would expect this to work but C actually complains at link-time and I can only get it to work by specifying a dependency in C against the external lib.
Note that the external lib is dynamic (a .so file).
This looks weird to me, no? Anyone understands what is going on?
Thanks,
Antoine.
That should work. I bet there's a bug in the CMakeLists.txt.
View real dependencies
Check it with:
cmake .. --graphviz=deps.dot
xdot deps.dot
It will show a pretty picture of the dependency tree that cmake sees.
If you don't have xdot, export it to a png:
dot -Tpng deps.dot -o deps.png
firefox deps.png
Library not found ?
Another possibility is that the external library can't actually be found. Use find_library rather than just putting the library name:
find_library(FAIL failingmadly)
if (NOT FAIL)
message(FATAL_ERROR "Couldn't find the failingmadly library")
endif()
target_link_libraries(my_lib_a ${FAIL})
Position independent code?
Another possibility when linking static libs with dependencies on shared libs is the PIC complaints. You could add this in cmake before compiling anything:
add_definitions(-fPIC)
Good luck.