The Problem:
I'm in the process of redoing the make system in our legacy project, changing from its present arcane version to CMake. At the moment, I've got CMake treating the whole thing as one large CMake project, but our code base is so large that it's breaking most of the IDEs we throw at it.
We'd like to break it apart, and CMake's find_package "module mode" seems to be ideal for breaking it into "feature-sized" chunks. The prime candidate is a large chunk of the code that only needs to be maintained rarely, and even then its usually by another team. That would allow us to maintain the code, but not continuously recompile it when updating different code.
That said, this chunk of code uses Boost's shared pointer in the API, and while different versions of shared pointer probably will work together, I'd rather not take the chance. So, ideally, the package will be aware of what version of "boost" the system is using, what version of boost was used when the module was compiled, and be able to recompile -- or, at the very least, throw an error or warning in CMake -- if the two don't match.
So ... how does one go about ensuring that versions of common dependencies match in CMake find_package modules? The only thing I can think of is testing the appropriate VERSION variable, but that seems ... bulky. Is there something I'm missing?
Additional Information:
We're using CMake 3.5.1, but we can upgrade to 3.5.2 if that would make a difference. This project is actually a Software Product Line (q.v.), so we are planning to use more modern SPL Software Engineering techniques at some point in the future (yet another reason to choose CMake). The codebase is currently in Redhat Linux, but ideally the technique(s) would be cross-platform.
You may use config mode of the find_package for allow you modules to expose some internal properties to their user (root project).
If each your module provides library target, you may expose that target with property containing Boost version attached and list this property in a special COMPATIBLE_INTERFACE_STRING property.
Your root project will include modules via find_package() calls and will read these properties. When it will try to link libraries, provided by such modules, version compatibility will be automatically performed by CMake:
modA/CMakeLists.txt:
...
find_package(Boost)
add_library(modA_lib ...)
... # Link modA_lib with Boost
# Install modA_lib target and exports it for use in other project.
install(TARGETS modA_lib EXPORT modA_lib)
# Configured -config file is shown below
configure(modA-config.cmake.in modA-config.cmake)
install(EXPORT modA_lib
DESTINATION share/cmake/modA)
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/modA-config.cmake
DESTINATION share/cmake/modA)
modA/modA-config.cmake.in:
include(#CMAKE_INSTALL_PREFIX#/share/cmake/modA/modA_lib.cmake) # Include file described library target
# Expose linked version of Boost via target's property.
set_property(TARGET modA_lib PROPERTY INTERFACE_BOOST_VERSION #Boost_VERSION#)
# Mark this property as compatibility requirement
set_property(TARGET modA_lib PROPERTY APPEND COMPATIBLE_INTERFACE_STRING BOOST_VERSION)
(modB is implemented in similar manner)
root/CMakeLists.txt:
find_package(modA) # This imports target modA_lib
find_package(modB) # This imports target modB_lib
add_executable(root_exe <...>)
# Boost version check will be performed here
target_link_libraries(root_exe modA_lib modB_lib)
Additionally, an executable created in the root project may request specific Boost version via setting appropriate property:
add_executable(root_exe <...>)
set_property(TARGET root_exe PROPERTY BOOST_VERSION <...>)
In this case it will be prohibited (by CMake) for its dependencies to use Boost library with other versions.
More info and usage examples see in CMake build system description.
Related
I'm working on a project with a lot of external dependencies, which are included in the project tree. I would like to pre-build all of the dependencies and share the importable targets within the project tree.
I was planning to use cmake install with a CMAKE_INSTALL_PREFIX in the source tree, and use CMAKE_PREFIX_PATH to reference it for find_package. However, I'm beginning to wonder how maintainable this strategy is going to be? For example here's something I noticed in one of the installed cmake scripts:
${CMAKE_PREFIX_PATH}/lib/cmake/glfw3/glfw3Targets.cmake:
set_target_properties(glfw PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include"
INTERFACE_LINK_LIBRARIES "/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/x86_64-linux-gnu/libm.so;dl;/usr/lib/x86_64-linux-gnu/libX11.so;-lpthread"
)
It seems really suspicious to me that all of those link libraries are fully resolved to paths on the host machine.
I guess the question is: Are cmake installs to a prefix meant to be distributable and this is just a bad example, or are they meant to be tied to the machine you "install" them on? Ie. Is prefix really meant to just relocate where on the system things are supposed to be "installed", and my hope to use it as a shared package manager likely to be problematic?
Yes, EXPORT'ed CMake targets can be "distributable", but the project should follow some principles for achieve that.
If you link with a (external) library, but do not want export file to contain absolute path to it, then do not pass absolute path directly to target_link_libraries.
In case a linked library is shipped with a compiler (e.g. m or rt), things are simple: just pass the library's name to the target_link_libraries.
In case a linked library YYY comes from other package and is detected by find_package(YYY), this implies several things:
Script FindYYY.cmake or YYYConfig.cmake should return IMPORTED target. If this is not true, you may try to wrap its results into IMPORTED target.
target_link_libraries should use this IMPORTED target.
Script XXXConfig.cmake, shipped with your project, should use find_dependency(YYY) for discover a library on a user machine.
For find_dependency(YYY) work, a user machine should have FindYYY.cmake or YYYConfig.cmake script. Alternatively, you may ship FindYYY.cmake with your project, and adjust CMAKE_MODULE_PATH variable before find_dependency() call (in your XXXConfig.cmake).
I'm creating a very small project that depends on the following library: https://github.com/CopernicaMarketingSoftware/AMQP-CPP
I'm doing what i always do with third-party libraries: i add their git repo as a submodule, and build them along with my code:
option(COOL_LIBRARY_OPTION ON)
add_subdirectory(deps/cool-library)
include_directories(deps/cool-library/include)
target_link_libraries(${PROJECT_NAME} coollib)
This has worked perfectly for libraries like Bullet, GLFW and others. However, this AMQP library does quite an ugly hack. Their include directory is called include, but in their CMake install() command, they rename it to amqpcpp. And their main header, deps/cool-library/amqpcpp.h, is referencing all other headers using that "fake" directory.
What happens is: when CMake tries to compile my sources which depend on deps/cool-library/amqpcpp.h, it fails because it's not finding deps/cool-library/amqpcpp/*.h, only deps/cool-library/include.
Does anyone have any idea how i can fix this without having to bundle the library into my codebase?
This is not how CMake is supposed to work.
CMake usually builds an entire distributive package of a library once and then installs it to some prefix path. It is then accessible for every other build process on the system by saying "find_package()". This command finds the installed distibution, and all the libs, includes etc. automagically. Whatever weird stuff library implementers did, the resulting distros are more or less alike.
So, in this case you do a lot of unnecessary work by adding includes manually. As you see it can also be unreliable.
What you can do is:
to still have all the dependencies source distributions in submodules (usually people don't bother doing this though)
build and install each dependency package into another (.gitignored) folder within the project or outside by using their own CMakeLists.txt. Let's say with a custom build step in your CMakeLists.txt
use "find_package()" in your CMakeLists.txt when build your application
Two small addition to Drop's answer: If the library set up their install routines correctly, you can use find_package directly on the library's binary tree, skipping the install step. This is mostly useful when you make changes to both the library and the dependent project, as you don't have to run the INSTALL target everytime to make library changes available downstream.
Also, check out the ExternalProject module of CMake which is very convenient for having external dependencies being built automatically as part of your project. The general idea is that you still pull in the library's source as a submodule, but instead of using add_subdirectory to pull the source into your project, you use ExternalProject_Add to build it on its own and then just link against it from your project.
I have the following situation:
I'm working on an application that depends on a number of third party libs, among them wxwidgets
I build the application for multiple target configurations (x86, arm, Linux, Windows) using Linux as my build host system
Due to the above mentioned multiple target configurations, I have chosen to build those third-party libs from source, using CMake's ExternalProject_Add function.
The third-party libs are built 'on-demand' at a location separate from my application's CMAKE_BINARY_DIR so that I can wipe the build tree for my application without having to rebuild the third-party libs (takes a looooong time).
The location of the third-party libs is different depending on what target configuration I build them for (obviously).
I'm quite new to CMake and the problem I currently face is this:
The source files in my application can't find the wx include files and I need to set the correct linker flags to be able to link my application against wxwidgets.
This seems to be handled by a utility 'wx-config' that provides exactly that info as output when run with either the --cppflags or --libs flag. I can not however, figure out how to catch that output and append it to the include dirs and linked libraries I setup from my CMakeLists.txt files.
So basically what I want is.
Build wxwidgets (if it doesn't exist) for the current target configuration
Run wx-config --cppflags and --libs to find out the correct include dirs and linker flags for the current target configuration
Use the info from step 2 when building targets that are my own application
So far I've tried something like this:
# Set a target-configuration-specific location
set(wxwidgetsTop ${MYPROJECT_EXTERNAL_DIR}/wxwidgets/wxwidgets_${MYPROJECT_CURRENT_TARGET_CFG})
# Build the project
ExternalProject_Add( wxWidgetsExternal
PREFIX ${wxwidgetsTop}
URL ${MYPROJECT_EXTERNAL_DIR}/tarballs/wxWidgets-3.0.2.tar.bz2
SOURCE_DIR ${wxwidgetsTop}/src/wxwidgets
CONFIGURE_COMMAND ${configure_cmdline}
BUILD_COMMAND make -j${MYPROJECT_NCPU}
INSTALL_COMMAND make install
)
# Create a wxwidgets target to be used as a dependency from other code
add_library(wxWidgets IMPORTED STATIC GLOBAL)
add_dependencies(wxWidgets wxWidgetsExternal)
# (non-working) attempt to get the correct include dirs and linker
# flags for wxwidgets
add_custom_command(TARGET wxWidgetsExternal
POST_BUILD
COMMAND ${INSTALL_DIR}/bin/wx-config ARGS --cppflags
COMMENT "Running wx-config"
)
but the above does not provide a way to actually use the result from the custom command to append the cppflags and linker options when building the targets that make up my application.
What is a good way to achieve what I want?
I see three different ways of doing this:
Method 1: use find_package
Use wxWidgets as a standalone requirement for your project, and expect the devs to install it before building your project. In your CMakeLists.txt you will need to call find_package(wxWidgets), like this:
find_package(wxWidgets COMPONENTS net gl core base)
if(wxWidgets_FOUND)
include(${wxWidgets_USE_FILE})
# and for each of your dependent executable/library targets:
target_link_libraries(<YourTarget> ${wxWidgets_LIBRARIES})
endif()
This has the advantage of not rebuilding the lib if you rebuild your project, however it requires some work for your user (they need to handle the installation of wxWidgets by hand) and for you (you need to setup include paths / compile definitions / ... by hand).
Method 2: embed wxWidgets
The second option is to bundle wxWidgets in your repo (svn external or git submodule) and usually (re)write the CMakeLists.txt of this lib to be target-oriented. Then, in your top-most CMakeLists.txt, you can do the following:
# for example, if you just need core and net:
target_link_librairies(my_app PUBLIC wxWidgetsCore wxWidgetsNet)
# No need to manually setup include dirs, etc...
To make a CMakeLists.txt target-oriented, you define include directories and other compilation properties for a target, not a directory. Example:
# When defining wxWidgetsCore, for example
add_library(wxWidgetsCore ...)
target_include_directories(wxWidgetsCore PUBLIC someDir)
target_compile_definitions(wxWidgetsCore PUBLIC -pedantic)
target_link_libraries(wxWidgetsCore PUBLIC someLib)
The drawback of this approach is that rebuilding your project will trigger a rebuild of wxWidgets. However, it is possible to trick this by not using "rebuild" but "clean just my app, then build". Here is some insight on how to achieve this.
Method 3: some sort of hybrid
The big drawback of method 2 leads to the third approach: don't put wxWidgets in your project, but create a CMakeLists.txt that will "import" the lib. The idea: you ask your user for the directory where wxWidgets is installed, then this script will setup everything for your project. First, put the CMakeLists.txt here:
/your-project-root
/thirdparty
/wxWidgets
CMakeLists.txt
/dir-where-wxwidgets-is-installed
...
Now, you define an imported target:
# When defining wxWidgetsCore, for example
set(WX_INCLUDE_DIR ${USER_SPECIFIED_WX_ROOT}/include)
add_library(wxWidgetsCore IMPORTED GLOBAL)
set_property(TARGET wxWidgetsCore APPEND PROPERTY
INTERFACE_INCLUDE_DIRECTORIES ${WX_INCLUDE_DIR})
See INTERFACE_INCLUDE_DIRECTORIES and INTERFACE_LINK_LIBRARIES. You need your user to have build wxWidgets somewhere in his system, but from your point of view you just do target_link_libraries(your_app PUBLIC wxWidgets...), as in method 2. The advantage is that this approach is interchangeable with method 2 transparently, and you don't put the whole dependency in your project.
Setting cppflags and linker flags has to be done at CMake time, but you are trying to run wx-config at build time and you are not capturing its output anyway, so your add_custom_command() isn't doing anything useful other than printing things to the build tool's output.
Ideally, you would use the FindwxWidgets module CMake already provides. It requires wxWidgets to already be built (but see further below). Have a look at the CMake documentation for it and see if that at least sounds like what you are trying to achieve manually by using wx-config. If you can get FindwxWidgets to do the job for you, that would be a much cleaner approach.
Getting something to build at configure time so you can use it later on in your CMakeLists.txt file is a bit more tricky. ExternalProject_Add() downloads and builds things at build time, but you need wxWidgets to be built earlier at configure time. I wrote an article recently for how to do at least the downloading part at configure time and you should be able to adapt it to do the whole build at configure time instead. The article uses Google Test as its example and can be found here:
https://crascit.com/2015/07/25/cmake-gtest/
It would be trivial to make it put the wxWidgets build wherever you like, not just in the CMAKE_BINARY_DIR area. That would allow you to have different wxWidgets builds for each build configuration and to be able to wipe out your application's build tree independently of the wxWidgets builds.
Hope that points you in the right direction.
The solution I use checks for wxWidgets installation in the system using find_package, if it's not found, then the script downloads wxWidgets from github and links the program against downloaded library. The lib is installed in the build directory, so only the first build is slow - subsequent builds do not even check wxWidgets sources timestamps, so the process is as fast as building using preinstalled wxWidgets library.
Here's how my script does it:
It quietly checks for wxWidgets installation using find_package(wxWidgets QUIET),
If it's found, the script adds a dummy library wxWidgets_external,
If it's not, then it creates an ExternalProject named wxWidgets_external which downloads, builds and installs the library in the build dir, setting wxWidgets_ROOT_DIR to point to the wxWidgets installation dir,
Then we add another ExternalProject pointing to a folder with the main program's source files and CMakeLists.txt build script. This external projects depends on wxWidgets_external which is either a dummy library in case wxWidgets is preinstalled in the system, or an external project set up to download the library from github,
In the aforementioned CMakeLists.txt we again call find_package, this time with REQUIRED parameter and use the library the standard way (https://docs.wxwidgets.org/trunk/overview_cmake.html). Because we set up the dependencies and variables correctly, this call will use either preinstalled wxWidgets (if it's available) or the one downloaded from github.
There are more quirks to it, but that's the gist of it. The full sample code (tested on Linux, Windows and Mac) is available on github (https://github.com/lszl84/wx_cmake_template).
Also see full blog post which explains this in more detail: https://justdevtutorials.medium.com/wxwidgets-cmake-multiplatform-superbuild-4ea86c4e6eda
I've just been reading about CMake's Config-File Package "concept" which sounds very promising. What I like very much about it is that if I create a Config-File Package myself I can specify other packages on which it depends. My Question is: How can I create a Config-File package that is "relocatable" and depends on a Find-Module Package (e.g. boost)?
In more detail: Suppose I want to create a package named HyDi. The cmake documentation explains then very nicely how I can create the corresponding HydiConfig.cmake and HydiTargets.cmake files automatically. A very simple version of the CMakeLists.txt that does this is:
project(HyDi)
find_package(Boost COMPONENTS program_options)
add_library(HyDi foo.cpp foo.hpp)
target_include_directories(HyDi PUBLIC INTERFACE ${Boost_INCLUDE_DIRS})
target_link_libraries(HyDi ${Boost_LIBRARIES})
target_compile_options(HyDi INTERFACE PUBLIC "-std=c++11")
install(TARGETS HyDi EXPORT HyDiTarget
LIBRARY DESTINATION lib
ARCHIVE DESTINATION lib
RUNTIME DESTINATION bin
INCLUDES DESTINATION include
)
install(FILES foo.hpp DESTINATION include)
configure_file(cmake/HyDiConfig.cmake
"${CMAKE_CURRENT_BINARY_DIR}/HyDi/HyDiConfig.cmake"
COPYONLY
)
set(ConfigPackageLocation lib/cmake/HyDi)
install(EXPORT HyDiTarget FILE HyDiTargets.cmake
NAMESPACE Upstream:: DESTINATION ${ConfigPackageLocation} )
install(FILES cmake/HyDiConfig.cmake DESTINATION ${ConfigPackageLocation})
The corresponding HydiConfig.cmake is:
include(CMakeFindDependencyMacro)
find_dependency(Boost COMPONENTS program_options)
include("${CMAKE_CURRENT_LIST_DIR}/HyDiTargets.cmake")
However if I install this library, the HyDiTargets.cmake file will contain the include path to the Boost Libraries hardcoded and is thus not relocatable.
Note that the cmake documentation gives an example of how not to include the boost libraries that is essentially my version. But they unfortunately don't explain how to do it better.
I understand that I could build boost using cmake and could then import boost as a Config-file package so that my HydiTargets.cmake would relocatable. But this approach doesn't work with every other library that provides a Findxxx.cmake file.
Actually CMake doing this correct, when inject a "hardcoded" path to boost libraries (and you doing it wrong). Because after your library gets compiled and installed, it should "links" w/ very particular boost version (when it was at the moment of your library compilation) -- i.e. its header files and static/dynamic libraries.
Consider scenario: after successfull installation of your library, someone installs a new version of boost library (or any other third party library you depending on) in parallel (yeah, boost and some other libs could coexists in the same install prefix). To make things looks like a real world example, assume it is ABI incompatible w/ the previous version. Now if that "lucky" developer wants to use your (already compiled and installed) library (using exported targets and provided HyDiConfig.cmake) he'll get a trouble:
your library already linked to a "previous" boost version (remember ABI incompatible w/ a newer one);
so when you somehow replace that "hardcoded" paths and would find a newer version (as you trying to do in HyDiConfig.cmake) your "lucky" customer would angry on you for that mess!
That is not only about boost… same policy for all third party libraries: they should remains the same as it was at the moment of compilation (or at least ABI compatible if we are talking about dynamic linking at run time, but this is a separate story).
Moreover, your user probably even do not use boost in his app (but have more than one version installed) -- why you should find smth? You already know (thanks to CMake and hardcoded paths) what boost version "required" for your library! So finding smth new is completely wrong in this case! It is "too late… Your library already compiled, linked and installed!
Another case: he wants to use some other (newer) version of boost… Depending on order of find_package(boost) and find_package(yourLib) results may vary… but he'll be angry on you anyway!
Say I am creating a project that uses a certain library and I have to provide the path for that library while linking. In the command line or makefile I might have:
g++ ... -L/path/to/mylibrary
I'm also going to send this project to someone else who wants to use it. The path on their system might not necessarily be the same as mine. They could be using a different file path all together.
How do I make sure that the path to the library works for both my computer and the recipient of my project?
This is the role of a build system or build configuration tool. There are many of those around. The main one is probably CMake as it has a very extensive feature set, cross-platform, and widely adopted. There are others like Boost.Jam, autoconf, and others.
The way that these tools will be used is that they have automated scripts for looking into the file-system and finding the headers or libraries that you need, i.e., the dependencies required to compile your code. They can also be used to do all sorts of other fancy things, like checking what features the OS supports and reconfiguring the build as a consequence of that. But the point is, you don't hard-code any file-paths into the build configuration, everything is either relative to your source folder or it is found automatically by the build script.
Here is an example CMake file for a project that uses Boost:
cmake_minimum_required (VERSION 2.8)
project (ExampleWithBoost)
find_package(Boost 1.46 COMPONENTS thread program_options filesystem REQUIRED)
# Add the boost directory to the include paths:
include_directories(SYSTEM ${Boost_INCLUDE_DIR})
# Add the boost library directory to the link paths:
link_directories(${Boost_LIBRARY_DIRS})
# Add an executable target (for compilation):
add_executable(example_with_boost example_with_boost.cpp)
# Add boost libraries to the linking on the target:
target_link_libraries(example_with_boost ${Boost_LIBRARIES})
The find_package cmake function is simply a special script (specialized for Boost, and installed with CMake) that finds the latest version of boost (with some minimal version) installed on the system, and it does so based on the file-name patterns that the library uses. You can also write your own equivalents of find_package, or even your own package finders, using the functions that CMake provides for searching the file system for certain file-name patterns (e.g., regular expressions).
As you see, the build configuration file above only refer directly to your source files, like "example_with_boost.cpp", and it's only relative to the source folder. If you do things right, the configuration scripts will work on virtually any system and any OS that CMake supports (and that the libraries you depend on support). This is how most major cross-platform projects work, and when you understand how to work with these systems, it's very powerful and very easy to use (in general, far easier to use and trouble-free than build configurations that you do by point-and-click within IDE menus like in Visual Studio).
You can use premake that generates cross platform makefiles: Visual Studio, Gcc and others
http://industriousone.com/what-premake
CMake is another alternative
http://www.cmake.org/
I'm not sure if there's a single universal way of doing this, but people often provide different config files and let the main Makefile detect which one to include: linux.make, darwin.make, cygwin.make etc.
there are also various tools like CMake that allow to automate this, but all in all it's just scripting that hides the system-dependency from the developer.