Avoid path problems at runtime when using cmake's ExternalProject_Add - c++

I'm pretty new to cmake, but I am stuck with an odd behaviour that I'd like to understand and overcome.
I'm creating an executable that uses the Paho MQTT C library. The content of this executable is irrelevant, let's say it just connects to a broker for now, nothing fancy.
To compile my program, I use cmake with a CMakeLists.txt file, which does a few things:
use an ExternalProject block to pull the MQTT C lib from Github
declare the executables and the linked libs
Here it is :
project(myproject)
include(ExternalProject)
########## CMAKE VERSION AND FLAGS ##########
cmake_minimum_required(VERSION 3.1 FATAL_ERROR)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/bin)
########## MQTT EXTERNAL LIB ##########
set(MQTT_BUILD_DIR ${CMAKE_CURRENT_BINARY_DIR}/mqtt)
ExternalProject_Add(
project_mqtt
GIT_REPOSITORY https://github.com/eclipse/paho.mqtt.c
GIT_TAG v1.2.0
PREFIX ${MQTT_BUILD_DIR}
CMAKE_ARGS "-DCMAKE_INSTALL_PREFIX=${CMAKE_RUNTIME_OUTPUT_DIRECTORY}"
)
add_library(mqtt STATIC IMPORTED)
set(MQTT_LIB ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/lib/libpaho-mqtt3a.1.2.0.dylib)
set_property(TARGET mqtt PROPERTY IMPORTED_LOCATION ${MQTT_LIB})
add_dependencies(mqtt project_mqtt)
include_directories(${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/include)
link_directories(${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/lib)
########## EXECUTABLE ##########
add_executable(myproject ../src/main.cpp)
target_link_libraries(myproject mqtt)
Note that in the file I especially compile for Mac and I hardcoded the .dylib suffix but this is not a problem here
My folder structure is as follows :
myproject/
-- CMakeLists.txt
-- bin/
-- build/
-- src/
- main.cpp
My compilation workflow is very standard : cd build && cmake ../ && make.
My sources are in /src, I build everything in /build and my executable is written in /bin. The *.dylib are explicitely built locally in the project folder (in bin/lib) since I don't want to install them in the system (I want my project to be self-contained in a way).
It works well (I mean, it compiles).
But when I run the program, I get a dyld: Library not loaded: libpaho-mqtt3a.1.dylib error.
.. which is quite "normal", because my lib has been installed in bin/lib and my executable searches for them in its own folder (which is bin/). So for instance if I run : cd bin/lib && ../myproject it works perfectly well.
So, now here are my questions :
[Philosophically,] Is it a good way of doing this ? I want to use this external lib but I don't want to include the Paho lib sources in my code obviously, so I though ExternalProject was a good way to go. It compiles the lib at compile time before compiling my project and I find the process quite handy. I looked for a "package manager" for C++ but there does not seem to be a strong consensus (I'm looking at something like *composer* or *npm* for web apps), so I did not go this route.
Why, oh why, does my executable look for the library file in its own directory ./ ? Can I change that ? Is it some cmake configuration that I missed ? I guess that I could just use a cmake custom command to copy the library file into the /bin folder, but this looks like a hack to me. Isn't it ?
To be honest, I'd like to have no "path dependencies" in my executable, it's to say that the executable, when compiled on a specific system, could be put anywhere on this system and work flawlessly afterwards. I understand that means that the compiled Paho MQTT lib should be statically in my executable, but I wonder how to do that (or even what that really means, in fact). Is it possible ?
I may miss a whole part of how the C/C++ compile and external lib workflow works but I think I'm close to something quite robust, it's just this lib path thing at runtime that I would like to properly understand and fix.
I'm really open to suggestions on this
Thanks a lot !

Related

Binaries missing after successful build?

I have a CMake project that I use to generate a Visual Studio solution, which I then try to compile. For some reason, the library file after compilation is nonexistent. I've searched my entire project folder, and cannot find any library files. Here's my CMake setup:
Project root:
cmake_minimum_required(VERSION 3.22)
project(ShadowContainers)
set(CMAKE_CXX_STANDARD 23)
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
include_directories(include/)
add_subdirectory(library)
...
library subdirectory:
add_library(libShadows shadows.tpp ../include/shadows.hpp)
set_target_properties(libShadows PROPERTIES LINKER_LANGUAGE CXX) # a tpp and an hpp aren't enough to make the system certain it's c++
target_compile_features(libShadows PRIVATE cxx_std_23)
The MSbuild output contains these lines:
Done Building Project "C:\Users\[..]\projects\Shadow Array\library\libShadows.vcxproj" (default targets).
Done Building Project "C:\Users\[..]\projects\Shadow Array\library\libShadows.vcxproj.metaproj" (default targets).
Even if libShadows is the only target in my project, it's nowhere in my project directory.
Has anybody else had this experience? Any help is appreciated. Thanks!
Edit:
To compile the project, I have tried:
My typical process (working directory = the project root):
cmake .
msbuild ShadowContainers.sln
An alternate process (working directory also project root):
mkdir build
cd build
cmake ..
cmake --build .
Both have produced the same result, albeit with VS and CMake files in different places. No library is outputted either way.
I was able to find the issue. CMake (and conventions for that matter) do not see a .tpp as a code file (similarly to how they treat header files). Thus, I was trying to compile a target with no legitimate code files. Template-only libraries are not libraries at all, so I should not be attempting to compile this standalone, and should instead put the .hpp and .tpp as the include in my projects!

Android Studio Compiles With Gcc, how to change that to G++ ? (Using CMake)

i have a Android studio project which uses Native C++ code with JNI, however when i try to Build my project, it throws error: undefined reference to 'std::basic_ostream<char...
errors.
I am using ubuntu and i believe the problem is caused by IDE using gcc compiler and tries to compile it as C code. When i try to run simple hello world program by running gcc -o foo foo.cpp in terminal, it gives same error. When i change gcc to g++, everything works fine.
But how do i change the compiler in IDE? I'm kinda new to using CMake and i think i need to use cmake's enable_language function but even i add enable_language(CPP) line to my CMakeLists.txt, it gives same errors.
Here is my CMakeLists.txt :
cmake_minimum_required(VERSION 3.4.1)
include_directories("/usr/include/x86_64-linux-gnu/c++/9")
include_directories("/usr/include/c++/9")
include_directories("/usr/include/x86_64-linux-gnu")
include_directories("/usr/include")
file(GLOB srcs *.cpp *.c)
file(GLOB hdrs *.hpp *.h)
include_directories(${OpenCV_DIR}/jni/include)
include_directories(${TESSERACT_INCLUDE_DIRS})
include_directories(${LEPTONICA_INCLUDE_DIRS})
add_library( lib_opencv SHARED IMPORTED )
set_target_properties(lib_opencv PROPERTIES IMPORTED_LOCATION ${OpenCV_DIR}/libs/${ANDROID_ABI}/libopencv_java4.so)
add_library(
native-lib
SHARED
#
native-lib.cpp )
find_library(
log-lib
log )
target_link_libraries(
native-lib
lib_opencv
${log-lib}
${TESSERACT_LIBRARIES}
${LEPTONICA_LIBRARIES})
Thanks!
ps: My G++ and Gcc versions are 9.3.0, build-essentials are already installed.
Your CMakeLists.txt is a whole lot of mess. Lets go through some potential problems, one by one, and see if it starts working.
The solution is probably in point 7, but I already wrote it all so I am leaving it here. Let me know if my answer helped you.
1. Does Android Studio really use this CMakeLists.txt file for building?
There are two usual build systems for NDK - older ndk-build and newer CMake. Check your Gradle build file to see if this CMakeLists.txt is really used. Here is documentation about how it should work: https://developer.android.com/studio/projects/gradle-external-native-builds.
And of course, you can make a deliberate syntax error in your CMakeLists.txt file to see if the build systems starts complaining about it.
And note that CMake should not by invoked separately when building for Android. It should be invoked from inside the Gradle script that is used by Android Studio for building the whole project.
2. Do all source files that use C++ libraries have .cpp extension?
CMake determines what compiler to use (C or C++) based on the extension of the file. If it is a .c file, then C compiler is used. If it is a .cpp file, then C++ compiler is used.
Also note that Android NDK now uses Clang (from LLVM project) instead of GCC, so GCC version or custom GCC invocations might not be that relevant.
You can also make CMake verbose - to print all the build commands, to see which commands are exactly executed. Add set(CMAKE_VERBOSE_MAKEFILE ON CACHE BOOL "ON") to your CMakeLists.txt file to make it verbose.
3. Is correct C++ library used?
Symbols you are missing (something from std::basic_ostream) are defined in C++ library (I think) that needs to be linked to your library. By default, libc++ should be linked and that is the right choice. But there is an option that can mess with that. So check your gradle build file for a ANDROID_STL variable and remove any assignments to that variable. The issue of C++ library in NDK is explained here: https://developer.android.com/ndk/guides/cpp-support#cmake
4. Missing source files.
When you call add_library, you are only adding one source file - native-lib.cpp. So only this one file will be compiled in the library (apart from lib_opencv that is linked to it). The line file(GLOB srcs *.cpp *.c) assigns all cpp and c files to the ${srcs} variable but that variable is then left unused. You could change add_library to use that variable: add_library( native-lib SHARED ${srcs} ).
If you want to also include files from subdirectories, use GLOB_RECURSE instead of GLOB in your file(GLOB srcs *.cpp *.c) command.
The line file(GLOB hdrs *.hpp *.h) is completely useless as far as I can tell. I don't see how a list of all header files could be useful in a CMake script.
5. Use new target-based CMake configuration.
Older CMake used file-based configuration, now target-based configuration is encouraged. File-based commands are for example link_libraries and include_directories, their target-based alternatives are target_link_libraries and target_include_directories. You are mixing those two styles. Basically, when a command has its target_ variant, you always want to use it.
You need to first create your target using add_library( native-lib SHARED ${srcs} ), the targets name is native-lib. Then add all include directories and link all libraries to it using target-based commands.
Also the way you add lib_opencv is missing a find_package command. Use find_package(OpenCV REQUIRED) and then target_link_libraries( native-lib PUBLIC ${OpenCV_LIBS} ) to link it.
The way you are adding logging library is OK, because the log-lib library is part of native API (https://developer.android.com/ndk/guides/stable_apis#logging) so it is more straightforward.
With CMake, you need to search internet or documentation to find out, which commands are needed to add each library to your target (unless you are compiling the library from sources). Tthough it is almost always some combination of find_package and target_link_libraries.
6. Missing project information.
I am not sure how important that is, but usually you start your main CMakeLists.txt with something like this:
cmake_minimum_required( VERSION 3.4.1 )
project( my_special_project )
set( CMAKE_CXX_STANDARD 14 )
set( CMAKE_CXX_STANDARD_REQUIRED True )
7. Don't manually include system directories.
Thinking about it, this is probably the problem. Toss out all of these lines:
include_directories("/usr/include/x86_64-linux-gnu/c++/9")
include_directories("/usr/include/c++/9")
include_directories("/usr/include/x86_64-linux-gnu")
include_directories("/usr/include")
Ndk uses its own toolchain, so never manualy add paths from your system toolchain to your NDK project.
I would discourage doing this also on any other project. It is not portable and things like that should be taken care of indirectly, by CMake itself.

How to link a library that was created from add_subdirectory in CMake?

I am trying to build a program using CMake that depends on a third party library. This third party library contains a CMakeLists.txt file so what I want to do is keep the source code of the third party library within my project directory, and build it using add_subdirectory(path/to/lib), and then link my target against the static library that the third party library generated.
my CMakeLists.txt:
cmake_minimum_version(VERSION 3.10)
project(my_project)
add_subdirectory("${CMAKE_SOURCE_DIR}/extern/somelib")
# my-code:
# somelib CMakeLists.txt file has a project name: SOMELIB
# which lets me access the directory where the files are built
# on windows it builds to /Release, on mac and linux it just builds
# to the binary dir
set(SOMELIB_LIBS "${SOMELIB_BINARY_DIR}/Release")
add_executable(my_program my_main.cpp)
target_link_libraries(my_program "${SOMELIB_LIBS}/SOMELIB.lib" "${SOMELIB_LIBS}/SOMELIBmain.lib")
I then make a build directory and from that directory I do:
cmake -G "Visual Studio 15 2017" ..
cmake --build .
The build command fails with a "LINK : fatal error LNK1181: cannot open input file 'extern/somelib/Release/SOMELIBmain.lib' ..."
My workaround for now has been to comment out the part that says "#my-code", build the somelib dependency first which generates the static libraries, and then uncomment out my-code and build again which then works correctly.
How can I tell CMake to build the subdirectory first, and then link against the static libraries that it generated?
Short answer: tell CMake that there is dependency between its targets.
target_link_libraries(my_program PRIVATE SOMELIB SOMELIBmain)
CMake will evaluate SOMELIBs locations for you and link my_program against SOMELIB and SOMELIBmain[1]. Works for both Debug and Release configurations and also for Linux.
You shouldn't have to worry where CMake places build files[2] and this is what so called "modern CMake" is trying to achieve. I will leave here just brief description but check link on the bottom of answer[3].
Anyway, the key concept is that whenever you create library/executable (target), you list its usage+build requirements and dependencies to other targets. You declare dependencies using target_link_libraries(<your_lib> KEYWORD <dependencies>). Such line will not only make <you_lib> link against listed dependencies, it will inherit their usage requirements (usually, public include directories) and order CMake to build dependent libraries before <your_lib>.
The beauty of it is that even if SOMELIB does not follow modern CMake ideas (does not declare build/usage requirements), you still should be able to do just with this single line.
[1] I assumed that CMake targets are named same as output library names, but it is not mandatory. For OP's case it turned out that static library targets are suffixed with -static, so he had to write SOMELIB-static SOMELIBmain-static
[2] Unfortunately, it is not so easy with shared libraries on Windows (DLLs)
[3] I'd start with this blog entry: https://pabloariasal.github.io/2018/02/19/its-time-to-do-cmake-right/

How to add libraries installed with ExternalProject_Add to target includes

I'm trying to include external libraries to a cmake project through ExternalProject_Add. To try out this feature I've created a minimal working example that involves adding pugixml to the project with ExternalProject_Add. However, I'm having problems finding a way to add the library's header files from the external project's local installation (i.e., pugixml's headers) in the project's include path.
The project tree of the minimal working example is organized as follows:
.
├── build
├── CMakeLists.txt
└── src
├── CMakeLists.txt
└── main.cpp
In this project tree, build refers to the build directory and the path where cmake is called to generate the build.
The contents of ./CMakeLists.txt are as follows:
cmake_minimum_required(VERSION 3.0)
include(ExternalProject)
ExternalProject_Add(pugixml
GIT_REPOSITORY https://github.com/zeux/pugixml.git
INSTALL_DIR ${PROJECT_BINARY_DIR}/extern_install/pugixml
CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
)
add_subdirectory(src)
In the above example I've added pugixml as an external project that is to be installed within the project's binary dir, whose file will be used by an executable stored in ./src. Thus, the contents of ./src/CMakeLists.txt are:
project(foo)
add_executable(foo main.cpp)
target_link_libraries(foo ${pugixml_LIBRARIES})
include_directories(${pugixml_INCLUDE_DIR}) # this part doesn't work
This is precisely the part I'm having problems. I've assumed that once the external project was added and installed that ExternalProject_Add would have defined some convenience libraries to help refer to library files and include directories. However, that doesn't work. Does anyone know what's the right way of using ExternalProject_Add to include external libraries?
Unfortunately this will not work at all. The build of the external project is done at build time not during CMake configure/generate. The hint is The ExternalProject_Add function creates a custom target to drive download, update/patch, configure, build, install and test steps of an external project: (from CMake)
Hence You will have to define all the variables yourself. See also: CMake - linking to library downloaded from ExternalProject_add() (I'd actually mark this as a duplicate of the linked question as the problem is the same)
BTW: I do also dislike the way this was done. This way one cannot simply use find_package etc.
The intended way to use this is a "super-build project": Define 1 CMakeLists where you only have ExternalProject_Adds for all dependencies AND your project. Then it will work with find_package.

How can I use CMake to both build wxwidgets on-demand and link with it

I have the following situation:
I'm working on an application that depends on a number of third party libs, among them wxwidgets
I build the application for multiple target configurations (x86, arm, Linux, Windows) using Linux as my build host system
Due to the above mentioned multiple target configurations, I have chosen to build those third-party libs from source, using CMake's ExternalProject_Add function.
The third-party libs are built 'on-demand' at a location separate from my application's CMAKE_BINARY_DIR so that I can wipe the build tree for my application without having to rebuild the third-party libs (takes a looooong time).
The location of the third-party libs is different depending on what target configuration I build them for (obviously).
I'm quite new to CMake and the problem I currently face is this:
The source files in my application can't find the wx include files and I need to set the correct linker flags to be able to link my application against wxwidgets.
This seems to be handled by a utility 'wx-config' that provides exactly that info as output when run with either the --cppflags or --libs flag. I can not however, figure out how to catch that output and append it to the include dirs and linked libraries I setup from my CMakeLists.txt files.
So basically what I want is.
Build wxwidgets (if it doesn't exist) for the current target configuration
Run wx-config --cppflags and --libs to find out the correct include dirs and linker flags for the current target configuration
Use the info from step 2 when building targets that are my own application
So far I've tried something like this:
# Set a target-configuration-specific location
set(wxwidgetsTop ${MYPROJECT_EXTERNAL_DIR}/wxwidgets/wxwidgets_${MYPROJECT_CURRENT_TARGET_CFG})
# Build the project
ExternalProject_Add( wxWidgetsExternal
PREFIX ${wxwidgetsTop}
URL ${MYPROJECT_EXTERNAL_DIR}/tarballs/wxWidgets-3.0.2.tar.bz2
SOURCE_DIR ${wxwidgetsTop}/src/wxwidgets
CONFIGURE_COMMAND ${configure_cmdline}
BUILD_COMMAND make -j${MYPROJECT_NCPU}
INSTALL_COMMAND make install
)
# Create a wxwidgets target to be used as a dependency from other code
add_library(wxWidgets IMPORTED STATIC GLOBAL)
add_dependencies(wxWidgets wxWidgetsExternal)
# (non-working) attempt to get the correct include dirs and linker
# flags for wxwidgets
add_custom_command(TARGET wxWidgetsExternal
POST_BUILD
COMMAND ${INSTALL_DIR}/bin/wx-config ARGS --cppflags
COMMENT "Running wx-config"
)
but the above does not provide a way to actually use the result from the custom command to append the cppflags and linker options when building the targets that make up my application.
What is a good way to achieve what I want?
I see three different ways of doing this:
Method 1: use find_package
Use wxWidgets as a standalone requirement for your project, and expect the devs to install it before building your project. In your CMakeLists.txt you will need to call find_package(wxWidgets), like this:
find_package(wxWidgets COMPONENTS net gl core base)
if(wxWidgets_FOUND)
include(${wxWidgets_USE_FILE})
# and for each of your dependent executable/library targets:
target_link_libraries(<YourTarget> ${wxWidgets_LIBRARIES})
endif()
This has the advantage of not rebuilding the lib if you rebuild your project, however it requires some work for your user (they need to handle the installation of wxWidgets by hand) and for you (you need to setup include paths / compile definitions / ... by hand).
Method 2: embed wxWidgets
The second option is to bundle wxWidgets in your repo (svn external or git submodule) and usually (re)write the CMakeLists.txt of this lib to be target-oriented. Then, in your top-most CMakeLists.txt, you can do the following:
# for example, if you just need core and net:
target_link_librairies(my_app PUBLIC wxWidgetsCore wxWidgetsNet)
# No need to manually setup include dirs, etc...
To make a CMakeLists.txt target-oriented, you define include directories and other compilation properties for a target, not a directory. Example:
# When defining wxWidgetsCore, for example
add_library(wxWidgetsCore ...)
target_include_directories(wxWidgetsCore PUBLIC someDir)
target_compile_definitions(wxWidgetsCore PUBLIC -pedantic)
target_link_libraries(wxWidgetsCore PUBLIC someLib)
The drawback of this approach is that rebuilding your project will trigger a rebuild of wxWidgets. However, it is possible to trick this by not using "rebuild" but "clean just my app, then build". Here is some insight on how to achieve this.
Method 3: some sort of hybrid
The big drawback of method 2 leads to the third approach: don't put wxWidgets in your project, but create a CMakeLists.txt that will "import" the lib. The idea: you ask your user for the directory where wxWidgets is installed, then this script will setup everything for your project. First, put the CMakeLists.txt here:
/your-project-root
/thirdparty
/wxWidgets
CMakeLists.txt
/dir-where-wxwidgets-is-installed
...
Now, you define an imported target:
# When defining wxWidgetsCore, for example
set(WX_INCLUDE_DIR ${USER_SPECIFIED_WX_ROOT}/include)
add_library(wxWidgetsCore IMPORTED GLOBAL)
set_property(TARGET wxWidgetsCore APPEND PROPERTY
INTERFACE_INCLUDE_DIRECTORIES ${WX_INCLUDE_DIR})
See INTERFACE_INCLUDE_DIRECTORIES and INTERFACE_LINK_LIBRARIES. You need your user to have build wxWidgets somewhere in his system, but from your point of view you just do target_link_libraries(your_app PUBLIC wxWidgets...), as in method 2. The advantage is that this approach is interchangeable with method 2 transparently, and you don't put the whole dependency in your project.
Setting cppflags and linker flags has to be done at CMake time, but you are trying to run wx-config at build time and you are not capturing its output anyway, so your add_custom_command() isn't doing anything useful other than printing things to the build tool's output.
Ideally, you would use the FindwxWidgets module CMake already provides. It requires wxWidgets to already be built (but see further below). Have a look at the CMake documentation for it and see if that at least sounds like what you are trying to achieve manually by using wx-config. If you can get FindwxWidgets to do the job for you, that would be a much cleaner approach.
Getting something to build at configure time so you can use it later on in your CMakeLists.txt file is a bit more tricky. ExternalProject_Add() downloads and builds things at build time, but you need wxWidgets to be built earlier at configure time. I wrote an article recently for how to do at least the downloading part at configure time and you should be able to adapt it to do the whole build at configure time instead. The article uses Google Test as its example and can be found here:
https://crascit.com/2015/07/25/cmake-gtest/
It would be trivial to make it put the wxWidgets build wherever you like, not just in the CMAKE_BINARY_DIR area. That would allow you to have different wxWidgets builds for each build configuration and to be able to wipe out your application's build tree independently of the wxWidgets builds.
Hope that points you in the right direction.
The solution I use checks for wxWidgets installation in the system using find_package, if it's not found, then the script downloads wxWidgets from github and links the program against downloaded library. The lib is installed in the build directory, so only the first build is slow - subsequent builds do not even check wxWidgets sources timestamps, so the process is as fast as building using preinstalled wxWidgets library.
Here's how my script does it:
It quietly checks for wxWidgets installation using find_package(wxWidgets QUIET),
If it's found, the script adds a dummy library wxWidgets_external,
If it's not, then it creates an ExternalProject named wxWidgets_external which downloads, builds and installs the library in the build dir, setting wxWidgets_ROOT_DIR to point to the wxWidgets installation dir,
Then we add another ExternalProject pointing to a folder with the main program's source files and CMakeLists.txt build script. This external projects depends on wxWidgets_external which is either a dummy library in case wxWidgets is preinstalled in the system, or an external project set up to download the library from github,
In the aforementioned CMakeLists.txt we again call find_package, this time with REQUIRED parameter and use the library the standard way (https://docs.wxwidgets.org/trunk/overview_cmake.html). Because we set up the dependencies and variables correctly, this call will use either preinstalled wxWidgets (if it's available) or the one downloaded from github.
There are more quirks to it, but that's the gist of it. The full sample code (tested on Linux, Windows and Mac) is available on github (https://github.com/lszl84/wx_cmake_template).
Also see full blog post which explains this in more detail: https://justdevtutorials.medium.com/wxwidgets-cmake-multiplatform-superbuild-4ea86c4e6eda