I have been trying to figure out the best way to handle dependencies in my C++ projects. For smaller libraries, like fmt, it is not too difficult to just add the following:
include(FetchContent)
FetchContent_Declare(FormatLib
GIT_REPOSITORY "https://github.com/..."
GIT_TAG "..."
)
FetchContent_MakeAvailable(FormatLib)
However, it is undesirable to have the project downloaded into the build directory every time with larger dependencies, for example, the Vulkan Loader which depends on other things.
What I would like would be the ability to do something like, before or at configure time of my main project, to download, configure, build and install a library system wide. I do not know if this is a feature possible with CMake but any help would be appreciated.
I also know that the ExternalProject module has existed in CMake for a long time now, and would be perfect for what I want, I think... if I were able to use it to install a library before I would then call find_package inside the same CMakeLists.txt
I purchased and read a considerable amount of the "Professional CMake 8th Edition" but still don't know if such a feature or technique exists in CMake.
I should clarify that I don't need the Vulkan loader, I was just giving an example. In the case of the Vulkan loader, if you have the SDK already installed, you can use the built in find_package command. My question is whether there's a way to populate such a library before hand. There are specific libraries that I haven't been able to find using package managers such as Conan and it seems like the best possible solution would be to actually just pull the source straight from an online git repo, build and install.
So in conclusion, what I would like to be able to do is the following: Write a CMakeLists.txt that details all the dependencies of a project, possibly searches for the dependencies using find_package, if they're not found, it installs them system wide so that from there on out, every build of said project or similar project on the system has access to the package installed.
Any advice, insight or suggestions on how I can mitigate this problem or even how my desired development strategy is flawed would be appreciated.
Related
I would like to know how to add external libraries into my project. Is there a standard way of doing so?
The way I do it and that I don't like is:
Have a folder called vendors where I add submodules e.g. boost, openssl...
I build the external libraries (as they come with a cmake to build in general).
I add a premake (I could have used a cmake) to each external library and I configure so I can see the project in VS as well as the cpp and the hpp files.
I don't like this because I do copy the binaries of the external libraries manually, hence if I delete the bin folder I can't build my solution just by clicking build but I have to build the external libraries first using there cmake and then I copy the binaries manually to the bin/ folder of my solution.
Could you please give me a "standard" way I can do this? I feel like there could be better ways by just using to the max the CMake that comes with the external library. Also, I don't like changing the external libs too much, I just want to be able to update them anytime and everything works without me touching stuff.
How can this be done?
One approach is to use CMake's FetchContent functionality.
The FetchContent module allows specifying Git repositories of CMake projects to fetch at configure time. The default setting assumes that that repository has a CMakeLists.txt file at the repo's root directory. It clones the repo to a default (but configurable) location, and then just calls add_subdirectory() on the cloned directory.
You can read about how to use it in the reference documentation, and you can read about how that approach compares with some other CMake-supported approaches for using dependencies in the official Using Dependencies Guide. Do brace yourself when reading the reference docs, though. They're not designed to be like a beginner-friendly tutorial, and since FetchContent is built upon another module called ExternalProject, some of the docs for FetchContent just point you to go read sections from the ExternalProject docs. Be prepared to do a bit of digging.
Here's a basic example I used in a project at one point.
include(FetchContent)
FetchContent_Declare(
range-v3
GIT_REPOSITORY git#github.com:ericniebler/range-v3.git
GIT_TAG "0.12.0" # https://github.com/ericniebler/range-v3/releases
GIT_SHALLOW TRUE
GIT_PROGRESS ON
SYSTEM
)
# more `FetchContent_Declare`s (if any). They should all be declared
# before any calls to FetchContent_Make_available (see docs for why).
FetchContent_MakeAvailable(range-v3)
# more `FetchContent_MakeAvailable`s (if any).
FetchContent in some ways is designed to be a little bit "low level". It has a lot of machinery and customization points. If your project is super simple, you might find it useful to try out a CMake-external wrapper module called "CPM" (CMake Package Manager) that attempts to cater to sensible defaults for more common, simple use-cases.
If you use FetchContent or CPM, be aware that since they eventually just call add_subdirectory, you might need to take some steps to avoid naming conflicts of target names and CMake variable names between your CMake configs and the CMake configs of the dependencies you pull in. For more info, see How can I avoid clashes with targets "imported" with FetchContent_MakeAvailable?.
As others have mentioned, you can also look into using package managers like vcpkg or Conan. I don't know much about those so I can't comment.
I'm currently working on a small CMake project with visual studio and I'm wanting to distribute my current build but I have no clue how to, when testing the app on another system it would error saying what I'm guessing were runtime dll's that were missing. I can't find anything online about it, and the official CMake tutorial didn't work for me. Does anyone know a way to do this?
Just like you have been told in the comments, the proper way to distribute an application on Windows is to use VC redistributable package. But if one wants to create an independent "bundle" CMake can help with it. It has a module which helps to bring the necessary dlls wherever needed. You need to understand and plan how your resulting bundle will look like and adapt the paths accordingly. I will just show you how to bring the dlls to the current binary directory with the help of the said module:
set(CMAKE_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}")
include(InstallRequiredSystemLibraries)
These 2 lines will make sure that you have an install target by building which the dlls required will be copied (installed) to the ${CMAKE_INSTALL_PREFIX} path. That would be your start from which you can proceed further with any customization you need to make.
I was trying to install Mujoco as a library to be used in my own Cpp programs, but I don¡t know if I should build it, although it does not have any CMakeLists file. I do not have much experience using precompiled libraries, so there might be the problem. Don't have much experience with Mujoco, but since I wanted to create a ROS2 interface with Mujoco, I think that going to the Mujoco library and learn how to use it is a good starting point.
You are right in that we currently do not provide a CMakeLists.txt for MuJoCo. We are in the process of modifying our internal CMake setup to have it work for non-Google users. This will be made available as part of our open source release.
In the meantime, the CMakeLists.txt for our Python bindings should give you an idea for how to build a CMake project against the precompiled MuJoCo library.
In spite of many years of coding large-scale C++ applications, I do not understand how find_package is supposed to work in a medium-size CMake project, ASSUMING that I want to build the source to dependent packages myself and not simply rely on large systems like opencv, pcl or boost being installed somewhere in a system folder. I can't can't believe that I'm the only person in the world who has shipped multiple OpenCV and other open-source apps, has worked with meta-build systems like NAnt and SCons on major game projects, yet can't understand the most basic things about how CMake works or find a tutorial answering these questions.
In the past, I have essentially hacked around not understaning find_package by setting all the foo_DIR values by hand as CMake complains until I get a working folder.
I would like to run through a simple example which I'm working on right now, and dearly hope someone can explain what I'm doing so wrong.
Firstly, some assumptions:
I want to build everything for both MacOS and Windows, ideally via CMakeGUI. MacOS should build XCodeProjects and Windows should build Visual Studio Solutions.
Where there are dependencies, I want to compile them myself, so I have debug symbols and can modify the dependency source (or at least debug into it.)
No installation of pre-built binaries into system folders, i.e. no use of sudo port install opencv/pcl, etc on mac.
I have multiple projects, and prefer to keep a project and its dependencies in a single folder.
For the purposes of a concrete example, suppose I am building this project, although it's an arbitrary choice to illustrate the process and confusion I suffer:
https://github.com/krips89/opendetection
This lists dependencies, which I have intentionally reordered here so that I can take them in order, as follows:
find_package(OpenCV REQUIRED)
find_package(Eigen REQUIRED)
find_package(Boost 1.40 COMPONENTS program_options REQUIRED )
find_package(PCL REQUIRED)
find_package(VTK REQUIRED)
I would like to have all of these dependencies downloaded and configured in a single path (let's say c:\src on Windows, and ~\src on Mac for simplicity), NOT in a system path. Assume that the actual folder is a sub-folder for this project, and no a sub-folder for all projects. This should also allow for side-by-side installation of multiple projects on the same computer.
Taking this one step at a time:
(1) I clone openCV from https://github.com/opencv/opencv, sync to tag 3.1, configure into the folder opencv_build folder, build and install into opencv_install. I've done this so many times it's pretty straightforward.
(2) As above, but for eigen (although building for eigen doesn't actually do anything s it's a template library. I install to a folder eigen_install
Taking directory shows a series of folders for downloaded dependencies. I have assumed a convention where , and are source repos, and their following _build folders are the "WHere to build the binaries" folders in CMakeGui.
$ ls
boost_1_40_0 opencv opendetection_build
eigen opencv-build opendetection_data
eigen_build opencv_contrib pcl
eigen_install opendetection
All good so far, now let's try to configure opendetection and generate a solution into opendetection_build, and find pendetection's dependencies from within the ~/src folder, that is for the first two dependencies, I hope to find opencv and eigen in the opencv-build and eigen-build folders.
OpenCV immediately fails, as expected, saying:
Could not find a package configuration file provided by "OpenCV" with any of the following names:
OpenCVConfig.cmake
opencv-config.cmake
Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set "OpenCV_DIR" to a directory containing one of the above files. If "OpenCV" provides a separate development package or SDK, be sure it has been installed.
That's good, because I want to explicitly tell CMake to look for dependent packages under my ~/src folder. Question: Is the use of CMAKE_PREFIX_PATH=/users/foo/src the recommended way to accomplish what I want - looking for all sub-packages under a specific path?
Following this, CMake finds OpenCV (good), and sets OpenCV_DIR = /Users/foo/src/opencv-build.
Question: Given that I have made an "install" to opencv-install (using CMAKE_INSTALL_PREFIX and building the Install Target Of OpenCV, shouldn't it find OpenCV in the opencv-install folder not opencv-build?
Moving on to eigen, I have configured and built eigen, and installed it to ~/src/eigen-install, which since it is a subfolder of CMAKE_PREFIX_PATH (~/src) I might expect to be found. But it doesn't seem to be. Can somebody explain to me what I'm not understanding? Particularly given that Eigen in a template library, and that there are at least three folders (eigen, eigen_build and eigen_install) under CMAKE_PREFIX_PATH which I would have thought CMake would find something in, I assume I must be doing something wrong here. I KNOW from past experience, I can set EIGEN_INCLUDE_DIR by hand in CMakeGUI by hand, and continue hacking forth, but that just seems wrong.
I'm more than willing to write up a web page explaining this for future people as dumb as me if one does not already exist, although I can't understand how use of CMake for basic project configuration and generation is apparently so obvious to everyone but so opaque for me. I have actually been using CMake for some years, usually by just manually setting Boost_INCLUDE_Dir, Foo_INCLUDE_PATH etc manually, but clearly this is not the right solution. Generally, after spending a couple of days fighting through the various packages to generate a solution by manually setting INCLUDE PATHS, LIBRARY PATHS and other options, I just deal with the solution and don't touch CMake again. But I would love to understand what I'm missing about find_package for my (surely not uncommon) use case of wanting to control my project dependencies rather than just using sudo port install * and installing random versions of projects to my global system folders.
As error message says, CMAKE_PREFIX_PATH should be set to installation prefix of the package. E.g., if the package has been built using CMake, this is CMAKE_INSTALL_PREFIX variable's value, if the package has been build using Autotools, this is value of --prefix option used for configure it, and so on.
CMake doesn't search every directory under CMAKE_PREFIX_PATH. That is why specifying it as /users/foo/src is useless if you have the package installed at /users/foo/src/eigen-install.
Instead, you may install all 3d-party packages into /users/foo/src/install, and use that path as CMAKE_PREFIX_PATH in your main project.
I am working on a cross-platform project which uses a large number of third party libraries (currently 22 and counting, and I expect this number to increase significantly). My project is CMake-based, and keeps the ThirdParty/ directory organized like so:
ThirdParty/$libname/include/
ThirdParty/$libname/lib/$platform/$buildtype/
My CMakeLists.txt has logic to determine the appropriate values for $platform (mac-i386, mac-ia64, win32-i386, and so on) and $buildtype (debug/release).
The difficulty arises in keeping these libraries up-to-date for each platform. Currently I am doing this manually - when I update one library, I go and rebuild it for each platform. But this is a nightmare, and adding a new platform is a two day affair.
This would be easy to automate if the third party libraries were themselves CMake-based, but they use everything from CMake to autoconf to custom solutions to hand-rolled Makefiles. There is no consistency between them, and all require various hoops to be jumped through with regards to build platform (especially with regards to 32- vs. 64-bit builds).
Are there any tools (or CMake extensions) which would make this easier to manage? Is there even any reasonable common ground that can be reached between CMake and autoconf, for example?
The ideal solution would give me a single command to build everything that needs rebuilding for a given platform (cross-compilation is not necessary, as I have access to all necessary platforms), but anything that incrementally makes my life easier would be appreciated.
You can probably use ExternalProject for this.
Create custom targets to build projects in external trees.
The 'ExternalProject_Add' function creates a custom target to drive download, update/patch, configure, build, install and test steps of an external project.
If you already have the source in your project's file hierarchy, then you can use something like this (for zlib):
include(ExternalProject)
ExternalProject_Add(zlib URL ${CMAKE_CURRENT_SOURCE_DIR}/zlib-1.2.4/
CONFIGURE_COMMAND cd <SOURCE_DIR> && ./configure --prefix=${CMAKE_CURRENT_BINARY_DIR}/zlib-build
BUILD_IN_SOURCE 1
BUILD_COMMAND make)
That will copy the zlib source code from your source tree into the build tree, run the configure step (after cd'ing into the copied directory), then run make on the configured directory. Finally the built version of zlib is installed into the current build directory, in a sub-directory called zlib-build.
You can tweak the setup, configure, and build steps however you like - zlib 1.2.4 for example doesn't like to have "configure" run out-of-source.
For a custom setup, you can skip the CONFIGURE step and just run the build command (for example). This requires a recent version of CMake to work (2.8+).