CMake not building a library when added as a subdirectory - c++

I added the xgboost library as a git submodule of my project and I'm trying to add it to cmake as a subdirectory. Unfortunately it's not working. A simple hello world project with the following CMakeLists.txt replicates the error that I'm getting.
cmake_minimum_required(VERSION 3.2)
project(foo)
add_subdirectory(xgboost)
add_executable(${PROJECT_NAME} foo.cpp)
target_link_libraries(${PROJECT_NAME} xgboost)
After building the library there is nothing in the xgboost/lib directory so I get the following error.
clang: error: no such file or directory:
'/Users/.../myproject/xgboost/lib/libxgboost.dylib'
I think that the problem is generated in their CMakeLists file since they have two different targets. Maybe cmake is choosing the wrong target but I'm not familiar enough with cmake to figure it out. The following code is from xgboost's CMakeLists.
# Executable
add_executable(runxgboost $<TARGET_OBJECTS:objxgboost> src/cli_main.cc)
set_target_properties(runxgboost PROPERTIES
OUTPUT_NAME xgboost
)
set_output_directory(runxgboost ${PROJECT_SOURCE_DIR})
target_link_libraries(runxgboost ${LINK_LIBRARIES})
# Shared library
add_library(xgboost SHARED $<TARGET_OBJECTS:objxgboost>)
target_link_libraries(xgboost ${LINK_LIBRARIES})
set_output_directory(xgboost ${PROJECT_SOURCE_DIR}/lib)
#Ensure these two targets do not build simultaneously, as they produce outputs with conflicting names
add_dependencies(xgboost runxgboost)
My questions in order of importance are:
Is there any way to fix it without modifying xgboost's CMakeLists.txt file?
Is it reasonable to try to add xgboost to my project as a git submodule?
Is there any reason cmake is not instructing to build the library?
Note: There were several edits to this question since I tried to narrow down the problem and to provide more information.

(I would love to ask for few things beforehand in the comment section, but I have too low reputation to do so, so I will just give it a shot ;))
I have few suspects, and one of them is ${CMAKE_SOURCE_DIR} of the submodule's root CMakeLists.txt. Although the paths are set properly when you run that CMakeLists.txt alone, cmake gets confused the moment you add it as your subdirectory. Have you looked into another directories for your output binaries?
First I would suggest testing this hypothesis, and then I would suggest writing similar, but separate CMakeLists.txt file for xgboost library, and then substitute it in the project temporarily. Unfortunately the CMakeLists.txt filename is hardcoded and there is no possibility to have two files of that kind in one directory; so it seems that the answer to 1) is, that you rather have to change the file.
For the 2): as long as it does not require huge additional logic in your CMakeLists.txt, it makes sense. Other viable option is to create an install target, which you can use to install your xgboost library locally (using CMAKE_INSTALL_PREFIX(doc) variable), and then add the installation path to your CMAKE_LIBRARY_PATH(doc).

Related

Building tensorflow serving client with cmake

I searched for the best way to do this, but I was unable to find a clear answer.
Was anyone able to build a tensorflow-serving client using cmake?
I am having difficulties with generating CPP files from proto, since they are needed for prediction service. Those proto files also include proto files from tensorflow.
so far I have come up with this:
project(serving C CXX)
find_package(Protobuf REQUIRED)
file(GLOB_RECURSE proto_files RELATIVE ${serving_SOURCE_DIR}/tensorflow/
"${serving_SOURCE_DIR}/tensorflow/*.proto")
set(PROTOBUF_GENERATE_CPP_APPEND_PATH OFF)
include_directories(${PROTOBUF_INCLUDE_DIRS})
include_directories( ${CMAKE_CURRENT_BINARY_DIR} )
PROTOBUF_GENERATE_CPP(PROTO_SRCS PROTO_HDRS ${proto_files})
add_library(tf_protos ${PROTO_SRCS} ${PROTO_HDRS})
target_link_libraries(tf_protos PUBLIC ${PROTOBUF_LIBRARIES})
Cmake builds successfully but the make command gives me an error:
No rule to make target '../tensorflow/tools/proto_text/test.proto', needed by 'tensorflow/tools/proto_text/test.pb.cc'. Stop.
To overcome the problem of .proto includes not being found I used command
set(PROTOBUF_GENERATE_CPP_APPEND_PATH OFF)
which was explained here: https://groups.google.com/forum/#!topic/protobuf/eow2fNDUHvc
My current folder structure is
serving/
CmakeLists.txt
tensorflow/
tensorflow_serving/
apis/
Folder apis contains .proto files that are needed in the client implementation and they include .proto files from the folder tensorflow.
Is this even the right way to go?
Any help/advice would be much appreciated.
I was able to get it to work in the layout you have where the CMakeLists.txt file is placed in the same level as the serving repository here. You'll need to install Tensorflow too though (using tensorflow_cc).
However, you probably don't want to muck with a fork of the official tensorflow/serving repository so I went a step further and moved the CMakeLists.txt out so you can just submodule the official repository. I made an example here
The gist is that the protobuf CMake submodule expects proto files to be laid out in the same directory from which it's called. I made some modifications to the submodules to let us call it from a level above serving and to ensure it invokes the compiler with the include paths in the right order to support the nested structure of the proto files in serving/tensorflow_serving/apis/* (and placing it accordingly in the specified build directory)
Hopefully someone else with better knowhow can make this better!
These worked for me.
https://github.com/wardsng/inception_cmake
https://github.com/FloopCZ/tensorflow_cc
You can choose a private install directory instead of the default, e.g. /usr/local/...
cmake -DCMAKE_INSTALL_PREFIX= ..

Isolating gitsubmodule projects in CMake

I'm trying to manage my C++ project dependencies using CMake and gitsubmodules. I'm following the layout described here: http://foonathan.net/blog/2016/07/07/cmake-dependency-handling.html and it's worked really well for me on smaller projects. But I've started to use it on much larger projects and I'm hitting some issue with CMake.
My current setup
All my external build dependencies are in a contrib/ subfolder inside my main project. Each is a submodule and has its own separate directory.
/contrib
- /eigen
- /curl
- /leapserial
- /zlib
- /opencv
etc.
The contrib/CMakeListst.txt simply initializes the submodule and adds the subdirectory for each external dependency
# EIGEN
execute_process(COMMAND git submodule update --recursive --init -- eigen
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
# options..
add_subdirectory(eigen EXCLUDE_FROM_ALL)
# CURL
execute_process(COMMAND git submodule update --recursive --init -- curl
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
# Initialize cache with CMake build options..
add_subdirectory(curl EXCLUDE_FROM_ALL)
# LEAP SERIAL
execute_process(COMMAND git submodule update --recursive --init -- leapserial
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
# Initialize cache with CMake build options..
add_subdirectory(leapserial EXCLUDE_FROM_ALL)
# ZLIB
execute_process(COMMAND git submodule update --recursive --init -- zlib
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
# Initialize cache with CMake build options..
add_subdirectory(zlib EXCLUDE_FROM_ALL)
# OPENCV
execute_process(COMMAND git submodule update --recursive --init -- opencv
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
# Initialize cache with CMake build options..
add_subdirectory(opencv EXCLUDE_FROM_ALL)
This setup has worked fantastically for me:
It's system/packagemanager independent. You don't need to install any libraries to get started developing
I can maintain the exact versions of my dependencies by setting the submodule to a particular commit. There are no surprises with some external library breaking your build
Adding the libraries to my build in the root CMakeListst.txt is trivial. Since I have the target available I just have something like
add_executable(someProjectImWorkingOn
${CMAKE_CURRENT_SOURCE_DIR}/src/main.cpp )
target_link_libraries(someProjectImWorkingOn
opencv_world
eigen
zlib
etc.)
when you hook up an existing library target to your own target executable/library CMake will automatically (through the target interface) add include directories to your target and add any other necessary options the library target requires for it to be used
I can pick a toolchain/compiler-option/build-type in the root CMakeLists.txt and it'll propogate to all the subprojects (I need to build for multiple systems. So this is a big big deal)
Since it's all in one "mega-project" it makes it very easy to hook up to rtags/kdevelop/clion to navigate not on your own code, but also the library code
Some issues that I can't resolved:
1
Subdirectories will define targets with the same name. In the example I gave, both Eigen OpenCV as well as another library define an 'uninstall' target
I tried to update the
add_subdirectory(blah)
to
add_subdirectory(blah EXCLUDE_FROM_ALL)
but this doesn't fix the issue for some reason
Enabling the variable ALLOW_DUPLICATE_CUSTOM_TARGETS kinda works.. but this is a hack, only work with Make files, and the libraries are essentially still "mixing" so it's still an issue
2
The second issue came up in LeapSerial but illustrates a bigger issue. The project no longer knows it's own name. LeapSerial tried to determine the version of LeapSerial, but when it asks for the project version it's getting the root project version. Ie. when cmake code in a subproject asks for "what project am I in" it's getting the root project, and not the immediate project it's in.
So again, the parent "namespace"s are leaking everywhere. This is bound to create more and more issues down the line. I need to the submodules to be self-contained
Is there are a cleaner solution?
ExternalProjectAdd might solve some of these problems, but has a lot more issues of its own. It's a real non-starter b/c it doesn't do most of what I've listed. The central issue is that it doesn't expose the sub-project's targets - and just vomits back variables that you then have to juggle
As the asker said in the comments, they resolved their issue by using the Hunter package manager. The rest of this answer is about actually answering the question as posed.
Concerning your first issue with target name clashes when using add_subdirectory-based approaches to using dependencies, a very similar (or essentially the same?) question has also since been asked here: How to avoid namespace collision when using CMake FetchContent?. When the clashes are between targets from different project dependencies, there's nothing you can do right now except politely ask the project maintainers to consider modifying their non-import/export target names to be namespaced like.
For example, for a library target, that might look like:
add_library(projectname_targetnamepart)
add_library("importexportnamespacename::targetnamepart" ALIAS projectname_targetnamepart)
set_target_properties(projectname_targetnamepart PROPERTIES EXPORT_NAME targetnamepart)
set_target_properties(projectname_targetnamepart PROPERTIES OUTPUT_NAME targetnamepart)
install(TARGETS projectname_targetnamepart EXPORT projectname_targets ...)
install(EXPORT projectname_targets NAMESPACE "importexportnamespacename::" ...)
There's a Kitware issue ticket by Craig Scott proposing a CMake feature for Project-level namespaces. Here's an excerpt:
A common problem when pulling in subprojects using add_subdirectory() is that target names must be unique, but different subprojects might try to use the same target name, which results in a name clash. This issue proposes to introduce the concept of a project namespace to address this and related name uniqueness problems (this is the primary goal of this issue).
Sometimes the upstream maintainer will just decline to support the add_subdirectory / FetchContent use-case. That's the case with OpenCV, as shown in this issue ticket (#16896). As for eigen, there's an open ticket that hasn't had any activity in a while (#1892).
Concerning your second issue, there's not enough detail in your question post to confidently troubleshoot. What is LeapSerial? Are you referring to the leapmotion/leapserial GitHub repo? What version and what commit are you referring to? At the latest commit before the time of your question post, 41515db, it's not immediately obvious what's wrong.
Variables in CMake are scoped by directory, so even if a project is added by add_subdirectory and doesn't used the <PROJECT-NAME>_VERSION variable and instead uses the more general PROJECT_VERSION variable, it should be okay. It just shouldn't attempt to use the CMAKE_PROJECT_VERSION variable to get its own version, since that one is fixed to refer to the top-level project's version.

CMake and external dependency

I'd like to add an external dependency to my project. The one I'm trying to add is the Leptonica library as a submodule.
My project has the following directory structure:
|root
CMakeLists.txt
|-bin
|-build
|-buildsystem
|-executable
|-leptonica
|--CMakeLists.txt
|--cmake
|---Configure.cmake
|-production
In my root CMakeLists.txt file I added ADD_SUBDIRECTORY(${ROOT_DIR}/leptonica)
Unfortunately, CMake is not searching for Configure.cmake in the proper directory:
CMake Error at leptonica/CMakeLists.txt:107 (include):
include could not find load file:
Configure
CMake Error: File
<root>/cmake/templates/LeptonicaConfig-version.cmake.in does not exist.
CMake Error at leptonica/CMakeLists.txt:113 (configure_file):
configure_file Problem configuring file
When I build the project by myself, everything goes fine. In my opinion, the problem is with CMAKE_SOURCE_DIR. When using add_subdirectory it has the value of ROOT CMake instead ROOT/leptonica, so it's searching the wrong paths - as you can see in Leptonica CMake, it's used to determinate paths of its files.
What should be the proper way to fix this - should I set CMAKE_SOURCE_DIR to ROOT/leptonica just before calling add_subdirectory and set it back when it's finished, or does some other, more elegant solutions exist?
Not every CMake project is suitable for inclusion via add_subdirectory.
Among those are projects which uses CMAKE_SOURCE_DIR or CMAKE_BINARY_DIR variables.
However, inclusion via ExternalProject_Add (optionally wrapped with execute_process) always works.
Modifying variable CMAKE_SOURCE_DIR (and CMAKE_BINARY_DIR too) is a bad idea: this variable should be changed only by CMake itself. Otherwise you may get weird errors.
Instead, you may replace (automatically, with some script) all references to the variable with another variable, which is not used in the project. This new variable you may safely set before stepping into the subproject.
${CMAKE_SOURCE_DIR} and ${CMAKE_BINARY_DIR} are set relative to the top-level CMakeLists.txt. If you need something relative to your current CMakeLists.txt (leptonica), use ${CMAKE_CURRENT_SOURCE_DIR} and ${CMAKE_CURRENT_BINARY_DIR}.
If you're having trouble finding a cmake file like LeptonicaConfig-version.cmake.in, try appending the appropriate directory to ${CMAKE_MODULE_DIR}.
list(APPEND ${CMAKE_MODULE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/cmake/templates)
I prefer to use ${CMAKE_CURRENT_SOURCE_DIR} over ${CMAKE_SOURCE_DIR} any day because using the latter will break your build if you try to integrate it into a super-build later. If I need to pass my current top-level directory to subdirectories, then I do the following and use that later down the chain.
set( LEPTONICA_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR} )

How to do CMake modules

I try to learn c++ with cmake. I have done a project, and cmake runs, it compiles, it works, that is nice, ok. Now I started a new project in which I want to use some classes of the first one. I have tried to read some sourcecode and I have understand that I need to make a module that I can read from both the application. So this will be my repository:
/
/cmake
/modules/Network
/software/sw1
/software/sw2
both the projects sw1 and sw2 depends on the module Netowrk. In the folder cmake there has to be the FindNetwork.cmake file, and in sw1, sw2 and Network there has to be the CMakeList.txt .
But.. I messed up with include_directories and other cmake obscurities..
Can someone point me out to a nice overview how to easily organize a repository with softwares that depend on common modules?
This link will provide some examples for you.
In terms of how you are looking at the project/infrastructure then it's best to not confuse things too much. So here is a couple of points to get you started (I hope)
In c++ a module is a library (so refer to your Network module as a
library)
To include a library you need to link it and also make the
header files available.
In cmake this is two commands target_link_libraries and include_directories respectively.
With that in mind the project structure could be
/Network/include (api here)
/Network/src
/sw1/src
/sw2/src
with an example base CmakeLists.txt file for you: (place in root dir of project)
cmake_minimum_required(VERSION 2.7) // your choice
project(myproject) // change name
add_subdirectory(Network)
add_subdirectory(sw1)
add_subdirectory(sw2)
in the Network Directory you would have this
add_library(Network net1.cc net2.cc etc.)
In the sw1 dir
include_dirs(${MYPROJECT_SOURCE_DIR}/Network/include)
link_directories(${MYPROJECT_BINARY_DIR}/Network)
add_executable (sw1prog sw1.cc sw11.cc etc.)
target_link_libraries (sw1prog Network)
In the sw2 dir
include_dirs(${MYPROJECT_SOURCE_DIR}/Network/include)
link_directories(${MYPROJECT_BINARY_DIR}/Network)
add_executable (sw2prog sw2.cc sw21.cc etc.)
target_link_libraries (sw2prog Network)
This is a very simplified version of what you may require, it removes the need for a FindXXModule.cmake file to be created and refers to the library you create implicitly. I think this is the best mechanism for you, if you did want to create a FindXXModule.cmake then I would suggest it's when you actually install your libs to the machine and wish others to be able to find it, either that or have a mechanism for multiple projects to link to each other libraries.
I hope this is a little useful, please bear in mind the cmake site has some examples and cmake --help is your friend.

Handling header files dependencies with cmake

I am using CMake on a small C++ project and so far it works great... with one twist :x
When I change a header file, it typically requires recompiling a number of sources files (those which include it, directly or indirectly), however it seems that cmake only detects some of the source files to be recompiled, leading to a corrupted state. I can work around this by wiping out the project and rebuilding from scratch, but this circumvents the goal of using a make utility: only recompiling what is needed.
Therefore, I suppose I am doing something wrong.
My project is very simply organized:
a top directory where all resources sit, the main CMakeLists.txt sits there
a "include" directory where all public headers lies (in various subdirectories)
a "src" directory where all the subdirectories for sources files are, the src CMakeLists.txt sits there
a CMakeLists.txt per subdirectory in the "src" directory
The main directory has:
cmake_minimum_required(VERSION 2.8)
project(FOO)
set(EXECUTABLE_OUTPUT_PATH ${CMAKE_BINARY_DIR}/bin)
# Compiler Options
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -std=c++0x -Wall -Wextra -Werror")
include_directories($(FOO_SOURCE_DIR)/include)
add_subdirectory(src)
The "src" directory:
add_subdirectory(sub1)
add_subdirectory(sub2)
add_subdirectory(sub3)
add_subdirectory(sub4)
add_executable(foo main.cpp)
target_link_libraries(foo sub1 sub2 sub3 sub4)
Where sub4 depends on sub3 which depends on sub2 which depends on sub1
And an example of a subdirectory (sub3):
set(SUB3_SRCS
File1.cpp
File2.cpp
File3.cpp
File4.cpp
File5.cpp
File6.cpp
)
add_library(sub3 ${SUB3_SRCS})
target_link_libraries(sub3 sub1 sub2)
I'd be glad if anyone could point my mistake to me, searching here or on CMake didn't yield anything so I guess it's very easy or should work out of the box...
(for reference, I am using cmake version 2.8.2 on MSYS)
EDIT:
Thanks to Bill's suggestion I have checked the depend.make file generated by CMake, and it is indeed lacking (severely). Here is an example:
src/sub3/CMakeFiles/sub3.dir/File1.cpp.obj: ../src/sub3/File1.cpp
Yep, that's all, none of the includes were referenced at all :x
You should look at the depend.make files in your binary tree. It will be in CMakeFiles/target.dir/depend.make. Try to find one of those that is missing a .h file that you think it should have. Then create a bug report for cmake or email the cmake mailing list.
I just hit the same issue. After changing paths in include_directories() from absolute to relative it added appropriate dependencies.
Looks like CMake tries to guess which headers are system and which are project related. I suspect that directories that starts with / passed as -isystem /some/path and thus are not presented in generated dependencies.
If you can't replace ${FOO_SOURCE_DIR} with relative path you can try to calculate relative path using appropriate CMake functions. I.e.:
file(RELATIVE_PATH FOO_SOURCE_REL_DIR
${CMAKE_CURRENT_SOURCE_DIR}
${FOO_SOURCE_DIR}/.)
include_directories(${FOO_SOURCE_REL_DIR}/include)
Did you run cmake before or after adding the includes to your cpp files?
I ran into this problem and re-running cmake fixed it. I'd added the include post-cmake.
Apparently cmake removes system include paths from the dependency trees (thank you #ony for this hint). This probably makes sense most of the time, but sometimes cmake doesn't know what the compiler thinks is a system path or not. We are using a custom gcc build that ignores /usr/include, but cmake thinks it doesn't ignore it. To force cmake to make /usr/include a dependency that is NOT optimized away, try this trick: prepend /. to the path.
I am trying to make all of the library dependencies use the cmake dependency feature, including certain "3rd" party libraries that are not always installed by default on Linux or even available. For example, Z-lib compression.
The following interface target worked fine if the Z lib includes were not in /usr/include, but would break if they were.
find_package(ZLIB REQUIRED)
message(status "found zlib ${ZLIB_LIBRARIES}")
message(status "found zlib includes ${ZLIB_INCLUDE_DIRS}")
target_link_libraries(zlib_target INTERFACE ${ZLIB_LIBRARIES})
target_include_directories(zlib_target INTERFACE ${ZLIB_INCLUDE_DIRS})
I changed the last line to
target_include_directories(zlib_target INTERFACE /.${ZLIB_INCLUDE_DIRS})
and it worked. Now targets that depended on zlib_target would automatically get -I/./usr/include during compilation.
Make sure that the include statement for the missing header is placed before the first program instruction. In my case cmake depend.make was not including the header file, because it was following the first program instruction.