I'm working on multiple projects at the same time (some libraries, and some games that depend on them). They're all on GitHub, in separate repos.
For convenience, I pull every repo in a "workspace folder", like this:
/home/myWorkspace/Library1/
/home/myWorkspace/Library2/
/home/myWorkspace/Library3/
/home/myWorkspace/App1/
/home/myWorkspace/App2/
I have FindLibrary1, FindLibrary2 and FindLibrary3 .cmake files, which, in order, look for the library in ../ (which corresponds to the "myWorkspace" folder), then /usr/lib/.
While on Windows CMake finds the libraries in myWorkspace/, on Linux, no matter what, installed libraries are always found first.
Since I'd like to work in myWorkspace/ folder, then installing the libraries after I'm done, I'd prefer CMake to find and link everything within the myWorkspace/ folder.
I'd also like CMake to search for the libraries in /usr/lib/ and /usr/local/lib/ if there is no myWorkspace/ folder, but if myWorkspace/ exists, it should have the priority.
Examples of CMake files I'm using:
SSVUtils: library with no dependancies
SSVUtils CMakeLists: https://github.com/SuperV1234/SSVUtils/blob/master/CMakeLists.txt
FindSSVUtils.cmake: https://github.com/SuperV1234/SSVUtils/blob/master/cmake/modules/FindSSVUtils.cmake
SSVUtilsJson: library that depends on SSVUtils and SSVJsonCpp
SSVUtilsJson CMakeLists: https://github.com/SuperV1234/SSVUtilsJson/blob/master/CMakeLists.txt
FindSSVUtilsJson.cmake: https://github.com/SuperV1234/SSVUtilsJson/blob/master/cmake/modules/FindSSVUtilsJson.cmake
Any ideas how I can prioritize the myWorkspace/ folder while still having the possibility to find libs in file system paths?
By default, find_path (or find_library, etc.) first checks for files in the system standard locations, before searching in the values provided in PATH. That's why "installed" libraries are always found first on Linux (but not on Windows, that doesn't have standard locations for installed libraries).
You can disable that behavior by using the NO_CMAKE_SYSTEM_PATH option: it will skip detection of files in standard locations.
Now... if you still want to use the installed libraries as a fallback when local versions are not found, you can do it in a two step process:
find_path(... NO_CMAKE_SYSTEM_PATH)
if (nothing_found)
find_path(...)
endif()
Related
For my project, I am building specific versions of the dependency libraries in a separate folder, say, /home/ubuntu/libs. I will use real libraries as an example, however, the question is pretty generic.
I was able to build the freetype library and make installed the headers into /home/ubuntu/libs/include, the built library into /home/ubuntu/libs/lib and also added the freetype-config.cmake to /home/ubuntu/libs/lib/cmake.
Now, I am trying to build the freetype-gl library that depends on freetype and has a line
find_package(freetype REQUIRED) in its CMakeLists.txt.
Typically, when I install the freetype library to a common path like /usr/local/lib or /usr/lib, cmake picks up the *-config.cmake files from the corresponding ./cmake directory. However, when I call it with
cmake -DCMAKE_TOOLCHAIN_FILE=/my/custom/toolchain -DCMAKE_LIBRARY_PATH=/home/ubuntu/libs/lib -DCMAKE_INCLUDE_PATH=/home/ubuntu/libs/include /path/to/freetype-gl
it fails with the following error
CMake Error at CMakeLists.txt:102 (find_package):
By not providing "Findfreetype.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "freetype",
but CMake did not find one.
Could not find a package configuration file provided by "freetype" with any
of the following names:
freetypeConfig.cmake
freetype-config.cmake
Add the installation prefix of "freetype" to CMAKE_PREFIX_PATH or set
"freetype_DIR" to a directory containing one of the above files. If
"freetype" provides a separate development package or SDK, be sure it has
been installed.
What am I doing wrong? How to show the place of "freetype-config.cmake" to cmake.
as you state in your second paragraph:
I was able to build the freetype library and make installed the headers into /home/ubuntu/libs/include, the built library into /home/ubuntu/libs/lib and also added the freetype-config.cmake to /home/ubuntu/libs/lib/cmake.
find_package() in CMake works in a way that it checks the standard system paths on default. If you are using CMake version 3.17 you can (instead of reading the documentation) view these paths simply by adding this line to your CMakeLists.txt
SET(CMAKE_FIND_DEBUG_MODE TRUE)
In your case what you need to do is point CMake in the right direction of the cmake file you are looking for. In your case that would be the /home/ubuntu/libs/lib/cmake. So somewhere in the top of your CMakeLists.txt (before you call find_package()) add this line:
LIST(APPEND CMAKE_MODULE_PATH "/home/ubuntu/libs/lib/cmake")
Provided that this bash command:
ls /home/ubuntu/libs/lib/cmake | grep "*.cmake"
Returns an occurance of freetypeConfig.cmake (you get the drill :) )
You can read about CMAKE_MODULE_PATH variable here: https://cmake.org/cmake/help/latest/variable/CMAKE_MODULE_PATH.html
In short your CMake doesn't find the config file because it is not in the standard path where it expects to find it.
EDIT: You can ofcourse do the same via these variables as the error suggests - CMAKE_PREFIX_PATH or freetype_DIR
Presumably you also don't want the libraries you build to find their dependencies installed on our system, if any.
I've spent some time investigating how to isolate CMake builds, and my best recipe is below.
First, the terminology:
An installation "prefix" of a library is a directory where the binaries and headers are installed (normally in include and lib subdirectories). I preder distinct prefixes for each library.
A "dependency" of a library is any of other library it needs when compiling.
CMake path separator - the character used to separate path lists for CMake. It's : character on Windows and ; on Linux (when cross-compiling, the host system matters, not the target).
Yes, it's the opposite of what the documentation claims.
By Windows I mean MSYS2. If you want to build outside of it, check that : is still the right separator.
Environment variables for CMake:
PKG_CONFIG_PATH to empty string
PKG_CONFIG_LIBDIR to a :-separated list, for each dependency add <prefix>/lib/pkgconfig and <prefix>/share/pkgconfig.
(I didn't need to do it for any library I used, so this is theoretical. But in any case don't leave this variable unset, at least use an empty string. Otherwise some undesired dependencies from your system might leak in.)
CMake flags:
-DCMAKE_INSTALL_PREFIX=... - the installation prefix for this library.
-DCMAKE_PREFIX_PATH=... - a path list: the installation prefix, followed by the prefixes of all dependencies. Use the CMake path separator, as described above. All paths here must be absolute.
-DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=OFF
This prevents CMake from finding some system-wide dependencies. If I remember correctly, not doing this makes CMake look for dependencies in directories listed in PATH and their parent directories, which is annoyting.
This has an undesired effect of disabling some ..._SYSTEM_... CMake variables, so we can't use those, even though some of them would be more appropriate.
-DCMAKE_STAGING_PREFIX=/. On Windows hosts replace / with the current drive name, e.g. C:.
This is only useful when cross-compiling, to disable the effects of CMAKE_FIND_ROOT_PATH in the toolchain file, which otherwise limits dependency search to that path.
This also messes up the installation path that would otherwise be taken from CMAKE_INSTALL_PREFIX, the fix is explained below.
Only on Windows hosts:
-DCMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH=OFF - otherwise CMake tends to look for dependencies in the PATH and parent directories, which is annoying. This is hardcoded to only happen on Windows hosts, which is nonsense and was reported here.
We don't want to set this unconditionally, because it has a side effect of requiring all your toolchain executables to be in the same directory, which is annoying in general, but IMO tolerable on Windows.
It also prevents CMake from searching for executables in PATH, so we also need...
-DCMAKE_PROGRAM_PATH= - set this to the contents of PATH, with the original separator replaced with the CMake path separator. At least on MSYS2 the separators are the same, so no modifications are needed.
-DCMAKE_MAKE_PROGRAM=ninja -GNinja - here CMAKE_MAKE_PROGRAM is strictly necessary on Windows hosts because of CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH=OFF, and doesn't hurt on other platforms. -G... is to make sure CMake doesn't pick a different generator that doesn't match CMAKE_MAKE_PROGRAM. You can use any other generator+program if you wish.
Then build with cmake --build <build_dir> -j<num_threads> as usual.
Install with cmake --install <build_dir> --prefix <prefix>. We need to explicitly set prefix because of CMAKE_STAGING_PREFIX=/.
I have downloaded a ROS2 demo from the examples repository.
Specifically, I have used minimal_subscriber and minimal_publisher.
I have ROS2 installed in /opt/ros2 and when I build these two examples with colcon build, it generates an install/ directory with lib/, shared/ and the other usual directory structure.
I can execute and run these demos perfectly fine with my current setup in my machine, but these executables link to libraries present in /opt/ros2, so when I want to execute them in another machine without ROS2 installed or I move my ROS2 installation in my machine, the executables cannot find the shared objects.
I've added a really simple script that adds all dependencies to install/lib when building but the executables don't seem to care, they aren't looking for shared libraries in the generated lib directory, they keep trying to search in /opt/ros2.
I believe this is something I should solve in CMake and it's not ROS2 specific, so, is there any way I can tell my generated executables to search in a diferent directory? (../lib or ./lib in my case)
If you are building them yourself (assumed since you mention CMake), you can set the RPATH in CMake (docs here). Specifically set the CMAKE_INSTALL_RPATH something like:
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
If you can't rebuild them, you can set LD_LIBRARY_PATH in your environment to where the libraries are located, or you can patch the executables themselves with an updated RPATH by using patchelf.
In order to get a relative RPATH rather than an absolute RPATH, use the $ORIGIN variable in your rpath spec. See "Recommendations" the the link above for more details.
I have a question related to how to install a built executable program with cmake when it relies on some external libraries. Suppose my executable is abc, and it relies on two external libraries: lib1.so and lib2.so. The structure of the codes are as follows:
-.........
|----bin (lib1.so lib2.so)
|----include(lib1.h lib2.h)
|----src(main.cpp)
When the executable program is installed using the following cmake commands:
INSTALL(TARGETS ${Exe_Name}
RUNTIME DESTINATION Path to bin
LIBRARY DESTINATION Path to bin)
I expect that the executable program will be in the same directory with lib1.so and lib2.so. However, when I execute the built program in the installation folder, I met the following error:
error while loading shared libraries: lib1 can not open shared object file No such file or directory
If I use ldd to check the executable, I found lib1.so and lib2.so not found. After searching for possible solutions, I found if I call the executable in this way, then it worked:
LD_LIBRARY_PATH=./ ./my_program_run
Then my question is how I can let my executable program knows the locations of the shared libraries with cmake when it is installed? Thanks.
This is best solved this with the RPATH of the final executable. RPATH is a hardcoded search path for the executable itself, and allows the use of the string $ORIGIN, which expands to the location of the executable at runtime. See this reference: http://man7.org/linux/man-pages/man8/ld.so.8.html
CMake strips the rpath of a binary at installation time, to avoid the binary picking up libraries littered around your development tree. But it also provides a simple way to modify the installation rpath for exactly this reason. Here's the short answer:
IF(UNIX)
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:\$ORIGIN/../bin:\$ORIGIN")
ENDIF()
This particular example appends to the existing rpath, and adds . and ../bin to the search path, all relative to the location of the binary.
Some developers claim that adjusting the RPATH of the binary is not a good idea. In the ideal world, all the libraries would live in the system library directories. But if you take this to the extreme, you end up with Windows (at least the older ones), where c:\windows\system32 is full of junk that came from who knows where, and may or may not conflict with other software, etc. Using rpath and installing everything in one place seems like a great solution.
If the application is to be cleanly installed to a standard linux distribution, then you should either install the supporting shared libraries into a standard location (/usr/lib), or you should add the libraries location to the ld.so config, by create an /etc/ld.so.conf.d/myprogram.conf file containing the name of the directory the libraries are in.
If the installation is temporary or more ad-hoc, then a script to set the LD_LIBRARY_PATH is suitable.
The libraries are searched in the predefined locations which includes standard library paths configured with ld.so.conf and LD_LIBRARY_PATH. You can also try to compile your app with -rpath, but it is not recommended by some developers. I suggest you to create a wrapper script which will set LD_LIBRARY_PATH and run the real application like that:
"theapp" script:
#!/bin/sh
dir="`dirname \"$0\"`"
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}"$dir"
exec "$dir/theapp.real" # your real application
The application, script and libraries should be in the same location (under bin/).
Usually, open source packages will have cmake to check if some headers or libraries exists or not, I wish my own project has this same functionality.
So I wish to know if cmake provides some command to check if some ".so"/".a"/".h" files exists in the current linux system or not, if not cmake will give me some hint to install them?
How does cmake support this?
Usually one would use the find_package(ABC REQUIRED) and then refer to it in your project. This will ensure that the dependent library is installed and cmake will fail if it is not.
You can find lot's of example on how this works in your cmake installation, for example C:\Program Files\CMake\share\cmake-3.13\Modules\FindZLIB.cmake will search for the zlib library by looking in the file system for normal places where this library would be installed and if it finds it will set these variables accordingly:
# ZLIB_INCLUDE_DIRS - where to find zlib.h, etc.
# ZLIB_LIBRARIES - List of libraries when using zlib.
# ZLIB_FOUND - True if zlib found.
To achieve this header files are found using the cmake command find_path, and libraries (static and shared) are found using find_library.
To search for arbitrary library you can use find_library() command. Same task for headers is accomplished by find_file(). You can also search for executables with find_program().
As #Damian said in his naswer, many libraries provide "config" files, like FindBoost.cmake. Such libraries can be found by calling find_package(Boost) command. This command would locate and load config file and set corresponding variables.
You need to provide -config.cmake files to tell other projects where your libraries and headers are. Here, you will found what are you looking for.
I'm trying setup a subset of boost and get it properly compiled using bjam, however I'm not getting the result I'm looking for. I'm working on windows using boost 1.37.0.
Let's say I want the libraries smart_ptr and filesystem built/installed. I intentionally chose a header only library and one library needing to compile a library. I want them to be built into a directory structure similar to the one I would get if I built a complete boost installation with bjam (i.e not specifying any --with-libraryX) but of course without all the libraries I'm not interested in.
My first approach was to use the --with-filesystem --with-smart_ptr in bjam, however it seemed like it didn't recognize smart_ptr (I tried smartptr and smart-ptr without success). I guess this is because it's a header only library.
When I removed it and only had --with-filesystem it seemed to copy ALL libraries header files to the install path and only build the libraries for filesystem. The library building behavior was the one I hoped for, but I got tons of header files for libraries I'm not interested in installed in my boost directory.
My second approach was to use bcp to copy the relevant projects. This did work out in the sense that I only got the projects I cared about (and their dependencies). However they didn't provide any make files for building the libraries that was copied. This means I would need to setup projects files for all the libraries that are not header only and manually build them.
So my question is basically, is there a way of selectively building boost that only copies the headers for the libraries I'm interested in and only building library files for the non header only libraries I'm interested in (and the boost libraries they are dependent on course)?
There are probably tons of manual/scripting based solutions for this, but if I could get something running only using bjam would be way more useful for me since adding new libraries or upgrading to a new boost version would be very simple.
EDIT:
Added the complete command line for the first approach:
bjam install --prefix=c:\temp\boostsmall
--build-dir=C:\temp\boostsmalltemp --layout=system
--with-filesystem variant=debug link=static threading=multi
runtime-link=static
Changed scoped_ptr to smart_ptr
Solved it.
The bcp solution had make files for the projects, however I needed to copy the tools directory and the root of the boost directory to the place I copied all my libs to get things up running.
Great question! This is an issue I have recently managed to figure out, I think.
I already had the full Boost libraries installed, including the ones requiring separate compilation.
I managed to create a subset of the Boost libraries (regex) for a particular application I was working on using the following steps:
First ensure the bcp executable is installed in the first place. In your Boost root folder, navigate to the tools/bcp folder and run the bjam.exe in here. For me this created the bcp executable in [Boost path]\bin.v2\tools\bcp\msvc-10.0\release\link-static directory:
Then create the folder where you would like the Boost subset to reside. This might already be a folder for a Visual Studio project you are working on.
Run the bcp exectubale you created, making sure to include the libraries you wish to include, as well as the location of the boost root directory and the destination folder. Observe how the required Boost components (eg for regex) have been included in your destination folder.
bcp.exe regex --boost="C:\Program Files\boost_1_55_0\boost_1_55_0" C:\Projects\RegexProject\BoostLite
In your Visual Studio project set the Additional Include Directories and Libraries to point to the 'Boost lite' folder that was created.
One final thing - which I think is described previously - I needed to go to my regular Boost folder setup and copy the required stage/lib folder containing all the lib files needed for the subset version and copy them here, just a few regex-related lib files in my case.
I found that this worked for me. Any feedback from other persons experiences would be appreciated.
A blog posting describing the same thing can be found here.