In spite of many years of coding large-scale C++ applications, I do not understand how find_package is supposed to work in a medium-size CMake project, ASSUMING that I want to build the source to dependent packages myself and not simply rely on large systems like opencv, pcl or boost being installed somewhere in a system folder. I can't can't believe that I'm the only person in the world who has shipped multiple OpenCV and other open-source apps, has worked with meta-build systems like NAnt and SCons on major game projects, yet can't understand the most basic things about how CMake works or find a tutorial answering these questions.
In the past, I have essentially hacked around not understaning find_package by setting all the foo_DIR values by hand as CMake complains until I get a working folder.
I would like to run through a simple example which I'm working on right now, and dearly hope someone can explain what I'm doing so wrong.
Firstly, some assumptions:
I want to build everything for both MacOS and Windows, ideally via CMakeGUI. MacOS should build XCodeProjects and Windows should build Visual Studio Solutions.
Where there are dependencies, I want to compile them myself, so I have debug symbols and can modify the dependency source (or at least debug into it.)
No installation of pre-built binaries into system folders, i.e. no use of sudo port install opencv/pcl, etc on mac.
I have multiple projects, and prefer to keep a project and its dependencies in a single folder.
For the purposes of a concrete example, suppose I am building this project, although it's an arbitrary choice to illustrate the process and confusion I suffer:
https://github.com/krips89/opendetection
This lists dependencies, which I have intentionally reordered here so that I can take them in order, as follows:
find_package(OpenCV REQUIRED)
find_package(Eigen REQUIRED)
find_package(Boost 1.40 COMPONENTS program_options REQUIRED )
find_package(PCL REQUIRED)
find_package(VTK REQUIRED)
I would like to have all of these dependencies downloaded and configured in a single path (let's say c:\src on Windows, and ~\src on Mac for simplicity), NOT in a system path. Assume that the actual folder is a sub-folder for this project, and no a sub-folder for all projects. This should also allow for side-by-side installation of multiple projects on the same computer.
Taking this one step at a time:
(1) I clone openCV from https://github.com/opencv/opencv, sync to tag 3.1, configure into the folder opencv_build folder, build and install into opencv_install. I've done this so many times it's pretty straightforward.
(2) As above, but for eigen (although building for eigen doesn't actually do anything s it's a template library. I install to a folder eigen_install
Taking directory shows a series of folders for downloaded dependencies. I have assumed a convention where , and are source repos, and their following _build folders are the "WHere to build the binaries" folders in CMakeGui.
$ ls
boost_1_40_0 opencv opendetection_build
eigen opencv-build opendetection_data
eigen_build opencv_contrib pcl
eigen_install opendetection
All good so far, now let's try to configure opendetection and generate a solution into opendetection_build, and find pendetection's dependencies from within the ~/src folder, that is for the first two dependencies, I hope to find opencv and eigen in the opencv-build and eigen-build folders.
OpenCV immediately fails, as expected, saying:
Could not find a package configuration file provided by "OpenCV" with any of the following names:
OpenCVConfig.cmake
opencv-config.cmake
Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set "OpenCV_DIR" to a directory containing one of the above files. If "OpenCV" provides a separate development package or SDK, be sure it has been installed.
That's good, because I want to explicitly tell CMake to look for dependent packages under my ~/src folder. Question: Is the use of CMAKE_PREFIX_PATH=/users/foo/src the recommended way to accomplish what I want - looking for all sub-packages under a specific path?
Following this, CMake finds OpenCV (good), and sets OpenCV_DIR = /Users/foo/src/opencv-build.
Question: Given that I have made an "install" to opencv-install (using CMAKE_INSTALL_PREFIX and building the Install Target Of OpenCV, shouldn't it find OpenCV in the opencv-install folder not opencv-build?
Moving on to eigen, I have configured and built eigen, and installed it to ~/src/eigen-install, which since it is a subfolder of CMAKE_PREFIX_PATH (~/src) I might expect to be found. But it doesn't seem to be. Can somebody explain to me what I'm not understanding? Particularly given that Eigen in a template library, and that there are at least three folders (eigen, eigen_build and eigen_install) under CMAKE_PREFIX_PATH which I would have thought CMake would find something in, I assume I must be doing something wrong here. I KNOW from past experience, I can set EIGEN_INCLUDE_DIR by hand in CMakeGUI by hand, and continue hacking forth, but that just seems wrong.
I'm more than willing to write up a web page explaining this for future people as dumb as me if one does not already exist, although I can't understand how use of CMake for basic project configuration and generation is apparently so obvious to everyone but so opaque for me. I have actually been using CMake for some years, usually by just manually setting Boost_INCLUDE_Dir, Foo_INCLUDE_PATH etc manually, but clearly this is not the right solution. Generally, after spending a couple of days fighting through the various packages to generate a solution by manually setting INCLUDE PATHS, LIBRARY PATHS and other options, I just deal with the solution and don't touch CMake again. But I would love to understand what I'm missing about find_package for my (surely not uncommon) use case of wanting to control my project dependencies rather than just using sudo port install * and installing random versions of projects to my global system folders.
As error message says, CMAKE_PREFIX_PATH should be set to installation prefix of the package. E.g., if the package has been built using CMake, this is CMAKE_INSTALL_PREFIX variable's value, if the package has been build using Autotools, this is value of --prefix option used for configure it, and so on.
CMake doesn't search every directory under CMAKE_PREFIX_PATH. That is why specifying it as /users/foo/src is useless if you have the package installed at /users/foo/src/eigen-install.
Instead, you may install all 3d-party packages into /users/foo/src/install, and use that path as CMAKE_PREFIX_PATH in your main project.
Related
I have been trying to figure out the best way to handle dependencies in my C++ projects. For smaller libraries, like fmt, it is not too difficult to just add the following:
include(FetchContent)
FetchContent_Declare(FormatLib
GIT_REPOSITORY "https://github.com/..."
GIT_TAG "..."
)
FetchContent_MakeAvailable(FormatLib)
However, it is undesirable to have the project downloaded into the build directory every time with larger dependencies, for example, the Vulkan Loader which depends on other things.
What I would like would be the ability to do something like, before or at configure time of my main project, to download, configure, build and install a library system wide. I do not know if this is a feature possible with CMake but any help would be appreciated.
I also know that the ExternalProject module has existed in CMake for a long time now, and would be perfect for what I want, I think... if I were able to use it to install a library before I would then call find_package inside the same CMakeLists.txt
I purchased and read a considerable amount of the "Professional CMake 8th Edition" but still don't know if such a feature or technique exists in CMake.
I should clarify that I don't need the Vulkan loader, I was just giving an example. In the case of the Vulkan loader, if you have the SDK already installed, you can use the built in find_package command. My question is whether there's a way to populate such a library before hand. There are specific libraries that I haven't been able to find using package managers such as Conan and it seems like the best possible solution would be to actually just pull the source straight from an online git repo, build and install.
So in conclusion, what I would like to be able to do is the following: Write a CMakeLists.txt that details all the dependencies of a project, possibly searches for the dependencies using find_package, if they're not found, it installs them system wide so that from there on out, every build of said project or similar project on the system has access to the package installed.
Any advice, insight or suggestions on how I can mitigate this problem or even how my desired development strategy is flawed would be appreciated.
I have a C++ project, where I use opencv, VTK. I would like to build and install these dependent libraries (opencv,vtk) automatically, which are situated in my thirdparty project folder. I know that opencv and VTK use cmake build system to build their libraries. I even saw the function ExternalProject to add them to my CMakesLists.txt but I had problem of linking the target with opencv libs.
Searching on stack overflow I could not find a proper defined method to do it to implement this for version 3.0, the proposed solutions where Version < 3.0. I would like to know how you will structure your project to build opencv automatically as a thirdparty library, exports its targets and link it with my project target.
I think that the best way is to look for what's being done on larger project on github. I would advise to look at this repo in which they use opencv version 3.2, automatically download it and compile it. I believe that this is the correct way because you just have to change the version to try the compatibility with the newer versions if you want to update your code.
What you are looking for is probably what is called a superbuild. This post might help.
I would suggest to build and install opencv in a subfolder of your repo or your build directory. Then you could set the OpenCV_DIR environment variable to the <opencv_install_prefix>/lib/cmake/opencv4 folder (or at least point to the folder where OpenCVConfig.cmake is)
Then in your CMakeLists.txt you can directly use :
find_package(OpenCV 4 REQUIRED
COMPONENTS core imgproc ximgproc)
target_link_libraries(${PROJECT_NAME} PUBLIC ${OpenCV_LIBS})
The OpenCV_DIR variable tells CMake where to find the OpenCV configuration files needed for find_package function.
For the build and installation step, you could use ExternalProject or FetchContent (I prefer the later) but since OpenCV could be very long to build, you might want to keep the build artifact outside of your project's build folder. Then you can erase your build folder to rebuild your project without rebuilding the whole OpenCV library.
On my side, I'm using a shell script that build OpenCV if needed before building the project that needs it.
I am having no end of trouble trying to build gRPC on Windows 10 using Visual Studio 2015 (C++) and cmake.
I have downloaded and unzipped grpc from GitHub, along with all its .gitmodules (and their .gitmodules, etc.) and unzipped them to their specified locations. When I cmake grpc, it complains about no CMakeLists.txt in cares/cares. grpc's .gitmodules specifies cares-1_12_0, and that includes no CMakeLists.txt file. What to do? The master version of cares/cares includes a CMakeLists.txt file, so I copied it into the -1_12_0 tree. Now it finds CMakeLists.txt, but then complains of other files that it can't find.
If I just use the master version of cares instead of 1_12_0 (hoping whatever incompatibility has been fixed by now), I get no more cares complaints. Another way that I have found to get past the cares complaints is to unzip c-ares-master.zip into grpc/third_part/cares/cares and then unzip c-ares-cares-1_12_0.zip in the same place. I figure that that way c-ares-master.zip will provide any files that c-ares-cares-1_12_0.zip is missing, and c-ares-cares-1_12_0.zip will overwrite any files with the same names with the -1_12_0 version -- but is this a good practice (copying a specific branch on top of the master version when a specific version is specified)? (I am not using git to download because it is not available or approved for use here, so I must traverse the dependencies manually.)
The next complaint is from protobuf 3.0.x: repeated_field_reflection.h not found, but this is only a Warning.
Then there are Errors thrown from benchmark about can't find GTEST_LIBRARY, GTEST_INCLUDE_DIR, and GTEST_MAIN_LIBRARY.
Do I need to build/install all these submodules (from bottom up) before I try to build grpc? Differences between the different modules' build procedures (and resulting build directory structures) suggest to me that the answer to this question should be 'no', but I am not sure. I understand that cmake provides support for recursive builds down a source tree (through all third_party dependencies) starting from a single root CMakeLists.txt (i.e., a single execution of cmake should build everything), so it would make sense for this ability of cmake to be used and that dependents' build directory structures should be consistent.
I note that grpc's .gitmodules requires protobuf 3.0.x, but it also lists bloaty, and bloaty's .gitmodules requires protobuf (presumably, the master version). Will using different versions of protobuf in different parts of the src tree (and building two versions of protobuf) cause problems? If so, what should I do when different parts of the tree require different versions of the same module?
Googletest is required in at least 3 places (grpc, bloaty, protobuf-master (required by bloaty, but not required by protobuf-3.0.x which is required directly by grpc)). Where should I set my GTEST_ROOT to point to, and how will that work with a module that expects it to be somewhere under its own third_party branch? How does one install Googletest after it has been built?
cmake looks for what looks like all (or many of) the standard C #include files. Many are found, and many are not found (I am building from VS2015 x64 Native Tools Command Prompt, so the applicable LIB and INCLUDE paths should be available; I've looked at them, and they appear reasonable.). If it searches for all of these by default, then I am guessing that not finding some should not be a problem (if they are not used by grpc or its dependencies), so I haven't chased them down. However, one that is not found is pthread.h (and I understand at least one module uses pthreads, but the next line of cmake output is "Found Threads: TRUE"). Another disturbing finding is "-- Check size of off64_t - failed" (It seems that such a value could be important for defining protobuf structures.) CMake also runs many tests. Some succeed; others don't. Should I be concerned with the test failures (which ones)?
I've also noticed that many of these modules change [almost] daily, so it occurs to me that dependency on a master version in the GitHub repository could break at any time. Has anyone built a C++ gRPC for Windows recently?
Any help or suggestions would be appreciated.
Can't comment directly to roger as I'm missing the required karma points, but I was also able to encounter the same issue on Windows 10. This was working in CMAKE 3.14 and then got broken in 3.19 when I was forced to update my Jenkins server for another reason.
When I use the following (networked, not preferred for this answer but preferred for peace of mind), it works reliably; use this if you have network access on your build machine.
# Builds gRPC based on GIT checked-out sources
ExternalProject_Add(grpc
PREFIX grpc
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/path/to/grpc"
CMAKE_CACHE_ARGS
-DgRPC_INSTALL:BOOL=ON
-DgRPC_BUILD_TESTS:BOOL=OFF
-DgRPC_BUILD_GRPC_RUBY_PLUGIN:BOOL=OFF
-DgRPC_PROTOBUF_PROVIDER:STRING=module
-DgRPC_PROTOBUF_PACKAGE_TYPE:STRING=CONFIG
-DgRPC_ZLIB_PROVIDER:STRING=module
-DgRPC_CARES_PROVIDER:STRING=module
-DgRPC_SSL_PROVIDER:STRING=module
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/grpc
DEPENDS c-ares protobuf zlib
)
If you can't do that, I had to do things the difficulty way by building each component individually, tracking their install location, then adding them as arguments to the ExternalProjectAdd...
# Builds c-ares project from the git submodule.
ExternalProject_Add(c-ares
PREFIX c-ares
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/path/to/cares"
CMAKE_CACHE_ARGS
-DCARES_SHARED:BOOL=OFF
-DCARES_STATIC:BOOL=ON
-DCARES_STATIC_PIC:BOOL=ON
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/c-ares
)
# Builds protobuf project from the git submodule.
ExternalProject_Add(protobuf
PREFIX protobuf
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/path/to/protobuf/cmake"
CMAKE_CACHE_ARGS
-Dprotobuf_BUILD_TESTS:BOOL=OFF
-Dprotobuf_WITH_ZLIB:BOOL=OFF
-Dprotobuf_MSVC_STATIC_RUNTIME:BOOL=OFF
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/protobuf
)
# Builds zlib project from the git submodule.
ExternalProject_Add(zlib
PREFIX zlib
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/path/to/zlib"
CMAKE_CACHE_ARGS
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/zlib
)
# the location where protobuf-config.cmake will be installed varies by platform
set(_FINDPACKAGE_PROTOBUF_CONFIG_DIR "${CMAKE_CURRENT_BINARY_DIR}/protobuf/cmake")
# if OPENSSL_ROOT_DIR is set, propagate that hint path to the external projects with OpenSSL dependency.
set(_CMAKE_ARGS_OPENSSL_ROOT_DIR "-DOPENSSL_ROOT_DIR:PATH=${OPENSSL_ROOT_DIR}")
# Builds gRPC based on locally checked-out sources and set arguments so that all the dependencies
# are correctly located.
ExternalProject_Add(grpc
PREFIX grpc
SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/path/to/grpc"
CMAKE_CACHE_ARGS
-DgRPC_INSTALL:BOOL=ON
-DgRPC_BUILD_TESTS:BOOL=OFF
-DgRPC_BUILD_GRPC_RUBY_PLUGIN:BOOL=OFF
-DgRPC_PROTOBUF_PROVIDER:STRING=package
-DgRPC_PROTOBUF_PACKAGE_TYPE:STRING=CONFIG
-DProtobuf_DIR:PATH=${_FINDPACKAGE_PROTOBUF_CONFIG_DIR}
-DgRPC_ZLIB_PROVIDER:STRING=package
-DZLIB_ROOT:STRING=${CMAKE_CURRENT_BINARY_DIR}/zlib
-DgRPC_CARES_PROVIDER:STRING=module
-Dc-ares_DIR:PATH=${CMAKE_CURRENT_BINARY_DIR}/c-ares/lib/cmake/c-ares
-DgRPC_SSL_PROVIDER:STRING=package
${_CMAKE_ARGS_OPENSSL_ROOT_DIR}
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/grpc
DEPENDS c-ares protobuf zlib
)
I'm creating a very small project that depends on the following library: https://github.com/CopernicaMarketingSoftware/AMQP-CPP
I'm doing what i always do with third-party libraries: i add their git repo as a submodule, and build them along with my code:
option(COOL_LIBRARY_OPTION ON)
add_subdirectory(deps/cool-library)
include_directories(deps/cool-library/include)
target_link_libraries(${PROJECT_NAME} coollib)
This has worked perfectly for libraries like Bullet, GLFW and others. However, this AMQP library does quite an ugly hack. Their include directory is called include, but in their CMake install() command, they rename it to amqpcpp. And their main header, deps/cool-library/amqpcpp.h, is referencing all other headers using that "fake" directory.
What happens is: when CMake tries to compile my sources which depend on deps/cool-library/amqpcpp.h, it fails because it's not finding deps/cool-library/amqpcpp/*.h, only deps/cool-library/include.
Does anyone have any idea how i can fix this without having to bundle the library into my codebase?
This is not how CMake is supposed to work.
CMake usually builds an entire distributive package of a library once and then installs it to some prefix path. It is then accessible for every other build process on the system by saying "find_package()". This command finds the installed distibution, and all the libs, includes etc. automagically. Whatever weird stuff library implementers did, the resulting distros are more or less alike.
So, in this case you do a lot of unnecessary work by adding includes manually. As you see it can also be unreliable.
What you can do is:
to still have all the dependencies source distributions in submodules (usually people don't bother doing this though)
build and install each dependency package into another (.gitignored) folder within the project or outside by using their own CMakeLists.txt. Let's say with a custom build step in your CMakeLists.txt
use "find_package()" in your CMakeLists.txt when build your application
Two small addition to Drop's answer: If the library set up their install routines correctly, you can use find_package directly on the library's binary tree, skipping the install step. This is mostly useful when you make changes to both the library and the dependent project, as you don't have to run the INSTALL target everytime to make library changes available downstream.
Also, check out the ExternalProject module of CMake which is very convenient for having external dependencies being built automatically as part of your project. The general idea is that you still pull in the library's source as a submodule, but instead of using add_subdirectory to pull the source into your project, you use ExternalProject_Add to build it on its own and then just link against it from your project.
So basically I have the following setup:
A small test library called mylib with an according CMake file. The CMake file creates all the necessary information so that another project (which is going to be my binary) can use find_package to add the library. The library also has a install target.
A small binary called mybin again wih according CMake file which is depended on the mylib project. Using find_package I can add mylib either by specifying the location of the according myLibConfig.cmake (and myLibTargets.cmake etc.) files or by executing the install target of mylib and then letting CMake find the library itself.
Using CMake and XCode everything works like a charm. First, I configure and build the library. Second, I configure my binary project, specify the location of the library and then build it without any problems.
Now I want to do the same using CLion. My problem now is that CLion puts all the generated CMake file (which usually are placed in the build folder) in some cryptic location which can't be changed in the IDE. Now, when I want to build the mybin project I have to specify this cryptic location which seems kinda odd to me (especially because you have to find out first where CLion actually places those files).
So my question is: Is there a more convenient way to handle the described configuration? I guess this is more or less the standard use case which makes me wonder if I'm missing out on something. Being able to specify where CLion should put the generated CMake files would solve my problem I guess.
I know that i can install the library (using the install target) and then let CMake find it. The problem here is that CLion (to my understanding) doesn't support install targets and therefore I have to use (in my case) XCode to build and install the library first.
I was misunderstsanding the intention of find_package(as Tsyvarev pointed out). By using the solution proposed in this question I managed to add an install target to CLion which now allows me to conveniently build "mylib" and use it in the "mybin" project without having to configure anything manually.