I want to reuse the source folder of my external libraries declared by FetchContent to avoid multiple downloads for multiple build configurations (debug/release/android/emscripten). To clarify, the build configurations have different binary directories e.g. /build/debug, build/release...
The ideal solution would be to have the source of the external depenendency in {CMAKE_SOURCE_DIR}/ext and the binary dir in {CMAKE_BINARY_DIR}/${name}-build
When using FETCHCONTENT_BASE_DIR (see example below) the source directory gets shared but also the binary directory, which triggers a complete rebuild whenever I switch between build configurations.
There is also the BINARY_DIR parameter for FetchContent_Populate, but it seems not do give the desired results.
set(FETCHCONTENT_BASE_DIR ${CMAKE_SOURCE_DIR}/depenendencies)
FetchContent_Declare(
${name}
GIT_REPOSITORY ${url}
GIT_TAG ${tag}
GIT_PROGRESS ON
)
FetchContent_GetProperties(${name})
if (NOT ${name}_POPULATED)
FetchContent_Populate(
${name}
)
add_subdirectory(${${name}_SOURCE_DIR} ${${name}_BINARY_DIR})
Related
I tried building the ORTools github package locally using cmake and it builds without any errors. However the environment which we are planning to ultimately use does not have an outbound network connection. The problem I see is that https://github.com/google/or-tools/blob/v9.4/cmake/dependencies/CMakeLists.txt performs a Git Clone to download the dependencies and add them. Since there is no outbound network access this step fails and I'm unable to build the dependency. To circumvent this we are planning to manually download the dependencies and add them to https://github.com/google/or-tools/blob/v9.4/cmake/dependencies/ folder.
I'm pretty new to CMake and I'm not sure what changes have to be made to accommodate this.
For example, what would the following snippet need to be changed to if I cloned Zlib v1.2.11 repository and added it to https://github.com/google/or-tools/blob/v9.4/cmake/dependencies/?
# ##############################################################################
# ZLIB
# ##############################################################################
if(BUILD_ZLIB)
message(CHECK_START "Fetching ZLIB")
list(APPEND CMAKE_MESSAGE_INDENT " ")
FetchContent_Declare(
zlib
GIT_REPOSITORY "https://github.com/madler/ZLIB.git"
GIT_TAG "v1.2.11"
PATCH_COMMAND git apply --ignore-whitespace "${CMAKE_CURRENT_LIST_DIR}/../../patches/ZLIB.patch")
FetchContent_MakeAvailable(zlib)
list(POP_BACK CMAKE_MESSAGE_INDENT)
message(CHECK_PASS "fetched")
endif()
Can FetchContent_Declare be used to point to a directory that already contains the source files? What's the alternative?
You could specify SOURCE_DIR parameter for FetchContent_Declare and omit download options:
FetchContent_Declare(
zlib
SOURCE_DIR <path/to/existing/directory>
PATCH_COMMAND git apply --ignore-whitespace "${CMAKE_CURRENT_LIST_DIR}/../../patches/ZLIB.patch"
)
This works in the same way as in ExternalProject_Add command, which options are accepted by FetchContent_Declare.
TLDR:
My problem is that CMake starts executing this function before
downloading the repository. I would like to declare a dependency for
that function on ExternalProject_Add so that CMake understands that it
should download, build and then run the function.
Context:
I have a cmake module SomeModule.cmake which is supposed to add flatbuffers as an external project from its repository and build it. The build would produce flatbuffers compiler executable which I intend to use in some/directory/CMakeLists.txt file to generate c++ header files from a flatbuffers schema. So in that same CMake module that I use ExternalProject_Add, I have declared a CMake function that generates header files from a given set of schema files and somewhere in some/directory/CMakeLists.txt I call that function.
My problem is that CMake starts executing this function before downloading the repository. I would like to declare a dependency for that function on ExternalProject_Add so that CMake understands that it should download, build and then run the function.
Enough talk. Here's relevant parts of the code:
SomeModule.cmake:
include(ExternalProject)
set(flatbuffers_CMAKE_ARGS
"-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}"
"-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
"-DFLATBUFFERS_BUILD_TESTS=OFF"
"-DFLATBUFFERS_BUILD_FLATC=ON"
"-DFLATBUFFERS_BUILD_FLATHASH=OFF"
"-DCMAKE_INSTALL_PREFIX=${OTS_DEPENDENCIES}"
)
ExternalProject_Add(
flatbuffers
GIT_REPOSITORY "https://github.com/google/flatbuffers.git"
GIT_TAG "v1.9.0"
SOURCE_DIR "${OTS_DEPDENDENCIES_DIR}/flatbuffers"
BINARY_DIR "${OTS_DEPDENDENCIES_DIR}/flatbuffers"
CMAKE_ARGS "${flatbuffers_CMAKE_ARGS}"
INSTALL_COMMAND ""
)
ExternalProject_Get_Property(flatbuffers SOURCE_DIR)
ExternalProject_Get_Property(flatbuffers BINARY_DIR)
set(flatbuffers_SOURCE_DIR ${SOURCE_DIR})
set(flatbuffers_BINARY_DIR ${BINARY_DIR})
set(flatbuffers_INCLUDE_DIR ${flatbuffers_SOURCE_DIR}/include)
set(flatbuffers_FLATC_EXECUTABLE ${flatbuffers_BINARY_DIR}/flatc)
# please assume that the variables above are all set to appropriate values
function(FlatbuffersGenerateCpp SCHEMA_FILES GENERATED_DIR GENERATED_CXX)
foreach(SCHEMA_FILE ${SCHEMA_FILES})
get_filename_component(NAME ${SCHEMA_FILE} NAME_WE)
set(GENERATED_HEADER_FILE_PATH ${GENERATED_DIR}/${NAME}_generated.h)
message(STATUS "attempting to generate: ${GENERATED_HEADER_FILE_PATH}")
add_custom_command(
DEPENDS ${flatbuffers_FLATC_EXECUTABLE}
OUTPUT ${GENERATED_HEADER_FILE_PATH}
COMMAND ${flatbuffers_FLATC_EXECUTABLE} -o ${GENERATED_DIR} -c ${SCHEMA_FILE}
COMMENT "generating flatbuffers c++ header file: ${GENERATED_HEADER_FILE_PATH}"
)
list(APPEND GENERATED_FILES ${GENERATED_HEADER_FILE_PATH})
endforeach()
message(STATUS "generated c++ header files: ${GENERATED_FILES}")
set(${GENERATED_CXX} ${GENERATED_FILES} PARENT_SCOPE)
endfunction()
And some/directory/CMakeLists.txt:
# cmake module path is properly set so the following works:
include(SomeModule)
set(flatbuffers_GENERATED_INCLUDES_DIR
${CMAKE_BINARY_DIR}/generated/config/flatbuffers
)
FlatbuffersGenerateCpp(
"${flatbuffers_SCHEMAS}"
"${flatbuffers_GENERATED_INCLUDES_DIR}"
flatbuffers_GENERATED_CXX
)
add_library(
my_framework
SHARED
${THE_PUBLIC_HEADER_FILES}
${THE_IMPL_SOURCE_FILES}
${THE_IMPL_HEADER_FILES}
${flatbuffers_GENERATED_CXX}
)
add_dependencies(my_framework flatbuffers ${flatbuffers_GENERATED_CXX})
target_include_directories(my_framework PRIVATE ${flatbuffers_INCLUDE_DIR})
target_include_directories(my_framework PRIVATE ${CMAKE_SOURCE_DIR})
target_include_directories(my_framework PRIVATE ${CMAKE_BINARY_DIR}/generated)
set_source_files_properties(${flatbuffers_GENERATED_CXX} PROPERTIES GENERATED TRUE)
I did start modifying my code based on comment posted by Tsyvarev:
CMake functions are executed at configure stage, so you need to build the external project at configure stage too.
While I trusted that his proposed solution would work, I was slightly uncomfortable and kept thinking that there has to be a more elegant solution. I consulted with a colleague and came up with a better solution which is as simple as the following diff (which removes ${flatbuffers_GENERATED_CXX}).
- add_dependencies(my_framework flatbuffers ${flatbuffers_GENERATED_CXX})
+ add_dependencies(my_framework flatbuffers)
we reviewed that the problem with the code in question is that as is, CMake reads add_dependencies(my_framework flatbuffers ${flatbuffers_GENERATED_CXX}) and understands that it needs ${flatbuffers_GENERATED_CXX} as target to build my_framework so it proceeds with running the function. But there is no way for it to understand that the function depends on the external project. Now if we remove the explicit dependency declaration of ${flatbuffers_GENERATED_CXX}, CMake defers running the function to after resolving other dependencies (flatbuffers target) which will effectively download and build the external project prior to running the project.
I'm building Google's FlatBuffers as a dependency for my own project and I need to compile a schema at build-time. I don't want to use BuildFlatBuffers.cmake or FindFlatBuffers.cmake because I'm using a specific version and I can't rely on it being locally installed.
This is a simplified version of my CMakeLists.txt:
ExternalProject_Add (
flatbuf
URL "https://github.com/google/flatbuffers/archive/v1.8.0.tar.gz"
CMAKE_ARGS -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
)
add_custom_target (
flatbuf_schema
PREFIX ${FLATBUF_PREFIX}
DEPENDS flatbuf
COMMAND ${FLATBUF_PREFIX}/src/flatbuf-build/flatc --cpp ${FLATBUF_SCHEMA}
)
It works fine for Make and Ninja but fails in Xcode, which builds flatc in the Debug directory.
I thought about these possible solutions:
use add_subdirectory instead of ExternalProject_Add so that I can use ${FLATBUFFERS_FLATC_EXECUTABLE} or $<TARGET_FILE:flatc>;
manually assign a RUNTIME_OUTPUT_DIRECTORY for flatbuf;
search for flatc in multiple paths (not portable; I also don't know how to make it happen at build-time).
I tried (2) and (3) but without success. As for (1), I'm not sure it's a good idea. How can I build schemas in a portable manner?
You can use ExternalProject_Get_Property, something like this...
note: I suppose you don't even need to install flatbuf, just build it and use it.
ExternalProject_Add (
flatbuf_project
URL "https://github.com/google/flatbuffers/archive/v1.8.0.tar.gz"
CMAKE_ARGS -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
INSTALL_COMMAND ""
)
ExternalProject_Get_Property(flatbuf_project source_dir)
ExternalProject_Get_Property(flatbuf_project binary_dir)
# Export flatbuf executable to consume schema file during build
add_executable(flatbuf::flatbuf IMPORTED)
set_target_properties(flatbuf::flatbuf PROPERTIES IMPORTED_LOCATION
"${binary_dir}/flatc")
add_dependencies(flatbuf::flatbuf flatbuf_project)
add_custom_target(flatbuf_schema
PREFIX ${FLATBUF_PREFIX}
COMMAND flatbuf::flatbuff --cpp ${FLATBUF_SCHEMA}
)
note2:
If COMMAND specifies an executable target name (created by the add_executable() command) it will automatically be replaced by the location of the executable created at build time. If set, the CROSSCOMPILING_EMULATOR executable target property will also be prepended to the command to allow the executable to run on the host. Additionally a target-level dependency will be added so that the executable target will be built before this custom target.
note3:
target ALIAS are not working on IMPORTED target unfortunately...
Apparently CMake provides a variable to solve this exact problem, i.e. CMAKE_CFG_INTDIR (docs). The path to the flatc executable should then be
ExternalProject_Get_Property(flatbuf BINARY_DIR)
set (FLATC "${BINARY_DIR}/${CMAKE_CFG_INTDIR}/flatc")
How do I specify an RPATH for a project that was built externally in CMake (using 3.0.0) via ExternalProject_Add() macro?
For reference, let's say my ExternalProject is SFML. My external project call looks like:
set(SFML_INSTALL_PATH "${CMAKE_CURRENT_BINARY_DIR}/ext-deps/ext-sfml-build")
set(SFML_RPATH "${SFML_INSTALL_PATH}/src/lib")
set(SFML_CMAKE_ARGS -DCMAKE_BUILD_TYPE:STRING=Release -DUSE_STATIC_LIBS:BOOL=true
-DCMAKE_INSTALL_PREFIX:PATH=${SFML_INSTALL_PATH}
-DCMAKE_INSTALL_RPATH:PATH=${SFML_RPATH})
ExternalProject_Add(ext-sfml
GIT_REPOSITORY "${SFML_REPO}"
GIT_TAG "${SFML_TAG}"
URL SFML_URL
URL_HASH 256=${SFML_SHA256}
CMAKE_ARGS "${SFML_CMAKE_ARGS}"
TMP_DIR "${CMAKE_CURRENT_BINARY_DIR}/${EXT_DEPS_PREFIX}/ext-sfml-tmp"
STAMP_DIR "${CMAKE_CURRENT_BINARY_DIR}/${EXT_DEPS_PREFIX}/ext-sfml-stamp"
INSTALL_DIR "${CMAKE_CURRENT_BINARY_DIR}/${EXT_DEPS_PREFIX}/ext-sfml-build"
SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/${EXT_DEPS_PREFIX}/ext-sfml"
BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/${EXT_DEPS_PREFIX}/ext-sfml-build"
TEST_COMMAND "")
The error command I get when I run the build step (configuration is fine) is
Make Error at src/cmake_install.cmake:45 (file):
file RPATH_CHANGE could not write new RPATH:
/home/hendrix/repo/project/build/ext_deps/ext_sfml-build/lib
to the file:
/home/hendrix/repo/project/build/ext_deps/ext_sfml-build/bin/exe
Finally, this "/home/hendrix/repo/project/build/ext_deps/ext_sfml-build/lib" is the line I'm trying to modify. It appears that it is the value of my ${CMAKE_INSTALL_RPATH} variable but containing the install prefix of my ext-dep project and not the top-level master project.
I figured it out (thanks to Tsyvarev's comments). The externally built project that I was using set it's own RPATH. So the string that I found in the error message was that RPATH, not any RPATH which I set in my "top-level" project.
It appears that setting the RPATH of an external project can not override that external project's internally set RPATH (if it is using CMAKE).
Let's say I have the following project setup with these dependencies:
MainProject
├─ Dependency_1
│ └─ Dependency_2
└─ Dependency_2
These dependencies are handled in MainProject and Dependency_1 with ExternalProject.
The problem is Dependency_2 will be downloaded twice: Dependency_1 will download a copy for itself, and MainProject will download a copy for itself.
This doesn't make for an efficient build process, is there a way where I can download Dependency_2 once for both projects?
Is has been suggested that this question is a duplicate of this one. That question slightly varies from mine, in that I cannot assume these libraries will be installed to the host system with ExternalProject. I would also like a CMake only solution, to which that question did not require.
From the main CMakeLists.txt, set an environment variable containing a common root path for downloading and building external projects, for example:
set (ENV EXTERNAL_PROJ_DOWNLOAD_DIR "${CMAKE_SOURCE_DIR}/externalProjects")
to be used as root folders for the download and the builds of the dependencies. You can set (and use) it in your main project, and read this value from within your first dependency (the one that also depends to your second dependency).
Seen it in practice applied to the project linked in your comments, you'll set EXTERNAL_PROJ_DOWNLOAD_DIR IN Khronos, and then to link to PortAudio in both Khronos and tritium projects you will have:
find_package(PortAudio)
if (${PORTAUDIO_FOUND})
include_directories(${PORTAUDIO_INCLUDE_DIRS})
else ()
ExternalProject_Add(
PortAudio
GIT_REPOSITORY "https://github.com/syb0rg/PortAudio2.git"
SOURCE_DIR "$ENV{EXTERNAL_PROJ_DOWNLOAD_DIR}/PortAudio"
UPDATE_COMMAND ""
INSTALL_COMMAND ""
BUILD_IN_SOURCE ON
LOG_DOWNLOAD ON
LOG_UPDATE ON
LOG_CONFIGURE ON
LOG_BUILD ON
LOG_TEST ON
LOG_INSTALL ON
)
ExternalProject_Get_Property(PortAudio SOURCE_DIR)
ExternalProject_Get_Property(PortAudio BINARY_DIR)
set(PORTAUDIO_SOURCE_DIR ${SOURCE_DIR})
set(PORTAUDIO_BINARY_DIR ${BINARY_DIR})
set(PORTAUDIO_LIBRARIES ${PORTAUDIO_SOURCE_DIR}/libportaudio_static.a)
set(DEPENDENCIES ${DEPENDENCIES} PortAudio)
include_directories(${PORTAUDIO_SOURCE_DIR}/include)
endif ()
SET(LIBS ${LIBS} ${PORTAUDIO_LIBRARIES})
You could also use set (ENV EXTERNAL_PROJ_BINARY_DIR "${CMAKE_BINARY_DIR}/externalProjects") if you wanted to activate the out of source build.
I suggest to use an environment variable because I don't know if a cache variable set from Khronos would be visible in tritium...
See documentation for set and env.