I'm currently trying to package a project as a Conan package. This project manages its dependencies with Conan itself already and as such has a conanfile.txt which lists its dependencies. I'm doing it in-repo following how the documentation suggests doing it.
Now I would like to package this project but I can't figure out what the proper way to simply let that conanfile.txt be the reference for the project's dependencies, and if it's possible to leverage the fact that it's already there instead of re-listing all the dependencies in the conanfile.py. I could skip it by just removing the conanfile.txt and using the conanfile.py as a consumer, but that forces a more complex handling of the actual build() step in the recipe which I would like to avoid. I would also like to let the people that develop the library be able to build it the way they would prefer, not necessarily through Conan except for the dependency management.
Is there a way to fetch a project dependencies from within the Conan recipe instead of listing twice the dependencies, both in the conanfile.txt and in the requires variable of the Conan recipe? If possible I would like to avoid reading the conanfile.txt and manually feeding line by line the dependencies into the requires variable.
I'm aware that I might be looking for a way that doesn't fit Conan's design and/or how packaging should work in general.
You can use conanfile.txt, but I strongly recommend using conanfile.py instead.
You can run conan install . as well, and install all dependencies listed in the conanfile.py. As the command install will not run your build() section, I can't see the problem.
Anyway, you can create a base class in conanfile.py which loads the conanfile.txt content and filter the requirements. Oblivious, it sounds be more complex than using only conanfile.py.
In terms of feature, there is no way, Conan doesn't load both files. By default it will conanfile.py and ignores conanfile.txt. You can't force both by argument or variables.
Related
Let's say I have two projects. Lib and App.
App has Lib in it's conanfile.txt. Normally when conan install of App's dependencies is performed conan downloads and compiles Lib to ~/.conan/data/.
Is it possible to link App to Lib that is currently being worked on instead (e.g. /home/path/to/code/Lib/cmake-build-release/lib/ )?
The reason I want to do this is to debug a Lib's bug whose only known way to reproduce is by using App. I want to be able to quickly add std::cout to certain places and incrementally recompile. Rebuilding the whole conan package and doing conan install each time is too long. I was thinking about some hack that would change include path and linking path.
Yes, it is possible, Conan uses the same approach as Python pip: the "editable" packages. You can read more about it in this section of the docs: Editable packages. The basic idea is:
You cd libfolder and define it as a package in edition: conan editable add lib/1.0
You can build the lib there, like conan install . + cmake .. or conan build .
You can go to the App folder cd appfolder and do a conan install .. You should see that lib/1.0 is marked as "editable" and not from the Conan cache. You can build App, and it will be linking your dependency to lib from the local libfolder.
Every change that you do to libfolder and build there locally (incremental builds), will be directly available to App without needing to create or export-pkg to the Conan cache.
The editable feature relies on the correct definition in lib/1.0 of the project layout, in its layout() method. There are built-in layouts like cmake_layout(self).
If the different packages lib and app generate compatible projects, like Visual Studio, it is possible to join those projects in the IDE and have a convenient development and debugging experience of both packages together. There is a live demo in this C++ Italian Meetup presentation
I would like to know how to add external libraries into my project. Is there a standard way of doing so?
The way I do it and that I don't like is:
Have a folder called vendors where I add submodules e.g. boost, openssl...
I build the external libraries (as they come with a cmake to build in general).
I add a premake (I could have used a cmake) to each external library and I configure so I can see the project in VS as well as the cpp and the hpp files.
I don't like this because I do copy the binaries of the external libraries manually, hence if I delete the bin folder I can't build my solution just by clicking build but I have to build the external libraries first using there cmake and then I copy the binaries manually to the bin/ folder of my solution.
Could you please give me a "standard" way I can do this? I feel like there could be better ways by just using to the max the CMake that comes with the external library. Also, I don't like changing the external libs too much, I just want to be able to update them anytime and everything works without me touching stuff.
How can this be done?
One approach is to use CMake's FetchContent functionality.
The FetchContent module allows specifying Git repositories of CMake projects to fetch at configure time. The default setting assumes that that repository has a CMakeLists.txt file at the repo's root directory. It clones the repo to a default (but configurable) location, and then just calls add_subdirectory() on the cloned directory.
You can read about how to use it in the reference documentation, and you can read about how that approach compares with some other CMake-supported approaches for using dependencies in the official Using Dependencies Guide. Do brace yourself when reading the reference docs, though. They're not designed to be like a beginner-friendly tutorial, and since FetchContent is built upon another module called ExternalProject, some of the docs for FetchContent just point you to go read sections from the ExternalProject docs. Be prepared to do a bit of digging.
Here's a basic example I used in a project at one point.
include(FetchContent)
FetchContent_Declare(
range-v3
GIT_REPOSITORY git#github.com:ericniebler/range-v3.git
GIT_TAG "0.12.0" # https://github.com/ericniebler/range-v3/releases
GIT_SHALLOW TRUE
GIT_PROGRESS ON
SYSTEM
)
# more `FetchContent_Declare`s (if any). They should all be declared
# before any calls to FetchContent_Make_available (see docs for why).
FetchContent_MakeAvailable(range-v3)
# more `FetchContent_MakeAvailable`s (if any).
FetchContent in some ways is designed to be a little bit "low level". It has a lot of machinery and customization points. If your project is super simple, you might find it useful to try out a CMake-external wrapper module called "CPM" (CMake Package Manager) that attempts to cater to sensible defaults for more common, simple use-cases.
If you use FetchContent or CPM, be aware that since they eventually just call add_subdirectory, you might need to take some steps to avoid naming conflicts of target names and CMake variable names between your CMake configs and the CMake configs of the dependencies you pull in. For more info, see How can I avoid clashes with targets "imported" with FetchContent_MakeAvailable?.
As others have mentioned, you can also look into using package managers like vcpkg or Conan. I don't know much about those so I can't comment.
I am making a CMake library around some installable SDK. So the dependency tree looks like:
Application --> MyLibrary --> OfficialSDK
This SDK is installed by some setup.exe and does not have a CMake module.
So instead I include a custom find script inside MyLibrary: MyLibrary/cmake/FindOfficialSDK.cmake. Then inside the CMakeLists.txt of MyLibrary I can use find_package(OfficialSDK).
This works well for MyLibrary. I can build it and install it, together with the CMake export. So then Application can run find_package(MyLibrary) out of the box, since MyLibrary was installed properly using CMake.
However, when configuring Application I get an error:
Target "Application" links to target "OfficialSDK" but the target was not found.
Okay, so MyLibrary remembers it needs OfficialSDK, but it cannot find it in this CMake project.
I could solve this by including cmake/FindOfficialSDK.cmake in Application, but I would rather not make my users to copy the find script in case I need to update it in the future.
Is there some way of including the imported target OfficialSDK and install it together with MyLibrary, so Application doesn't need to search for it?
I found a solution, largely based on https://discourse.cmake.org/t/install-findpackage-script/5307.
Another example I found inside Pagmo2, for a custom FindBoost script: https://github.com/esa/pagmo2/blob/master/pagmo-config.cmake.in#L10
In a nutshell, I've added the following:
Added an install for my custom find script: install(FILE ${CMAKE_CURRENT_DIR}/cmake/FindOfficialSDK ...)
Added a custom *-config.cmake.in script that will be installed, aside from the more automatic *-targets.cmake export
Added an explicit find_dependency(OfficialSDK) (not find_package) inside this config script, but inside a clause that adds the MyLibrary cmake install directory to the CMake module path, so the custom find script is used.
A complete example can be seen this PR: https://github.com/ET-BE/AdsClient/pull/1/files (as well as a bunch of other stuff). The OfficialSDK is TwinCAT's ADS library.
CMake has a find_package() backed by a bunch of FindXYZ scripts (which you can also add to).
What mechanism, if any, is available to me to tell cmake: "Find this package, and if you haven't found it, download it and trigger its build" - with the downloading and building part also backed by per-package scripts or settings (so that downloading could be with wget or git clone, building could be with cmake or maven or a package-specific command, etc.) ?
Yeah, I was bitten by that Friday.
So, CMake has an ExternalProject directive, meant for exactly that, get/update if necessary, configure, build and install this and that external project. Awesome!
Sadly, CMake isn't that awesome.
You can't use the target defined by ExternalProject as a library in target_link_libraries. I've really tried to.
The basic problem is that the updating, building and installation of the external project happens at build time, whereas CMake insists on only using libraries that it found during pre-build (i.e. during the CMake run); you can't re-detect stuff while running make/ninja/msvc… .
You can define a custom target, tell it where the .so you'd want to link against later will be, and try to coerce CMake into believing you without checking at pre-build. Sadly, at least in the CMake versions I had, that broke dependency tracking, so that it simply didn't build the external library, because nothing needed it.
From the error messages you get when trying to use an external project in target_link_library, it seems CMake assumes you'd only want to install tools you need at build time that way, not libraries. A bummer.
You can roll your own version of download-on-demand using execute_process() (which runs on the CMake configure step) with ${CMAKE_COMMAND} as the command invoked on a CMakeLists.txt containing ExternalProject_Add().
You could even either configure_file() the CMakeLists.txt to fill out custom variables or dynamically create the CMakeLists.txt file.
I'm creating a very small project that depends on the following library: https://github.com/CopernicaMarketingSoftware/AMQP-CPP
I'm doing what i always do with third-party libraries: i add their git repo as a submodule, and build them along with my code:
option(COOL_LIBRARY_OPTION ON)
add_subdirectory(deps/cool-library)
include_directories(deps/cool-library/include)
target_link_libraries(${PROJECT_NAME} coollib)
This has worked perfectly for libraries like Bullet, GLFW and others. However, this AMQP library does quite an ugly hack. Their include directory is called include, but in their CMake install() command, they rename it to amqpcpp. And their main header, deps/cool-library/amqpcpp.h, is referencing all other headers using that "fake" directory.
What happens is: when CMake tries to compile my sources which depend on deps/cool-library/amqpcpp.h, it fails because it's not finding deps/cool-library/amqpcpp/*.h, only deps/cool-library/include.
Does anyone have any idea how i can fix this without having to bundle the library into my codebase?
This is not how CMake is supposed to work.
CMake usually builds an entire distributive package of a library once and then installs it to some prefix path. It is then accessible for every other build process on the system by saying "find_package()". This command finds the installed distibution, and all the libs, includes etc. automagically. Whatever weird stuff library implementers did, the resulting distros are more or less alike.
So, in this case you do a lot of unnecessary work by adding includes manually. As you see it can also be unreliable.
What you can do is:
to still have all the dependencies source distributions in submodules (usually people don't bother doing this though)
build and install each dependency package into another (.gitignored) folder within the project or outside by using their own CMakeLists.txt. Let's say with a custom build step in your CMakeLists.txt
use "find_package()" in your CMakeLists.txt when build your application
Two small addition to Drop's answer: If the library set up their install routines correctly, you can use find_package directly on the library's binary tree, skipping the install step. This is mostly useful when you make changes to both the library and the dependent project, as you don't have to run the INSTALL target everytime to make library changes available downstream.
Also, check out the ExternalProject module of CMake which is very convenient for having external dependencies being built automatically as part of your project. The general idea is that you still pull in the library's source as a submodule, but instead of using add_subdirectory to pull the source into your project, you use ExternalProject_Add to build it on its own and then just link against it from your project.