I'm relatively new to Conan. I'm trying to use packages provided by conan in a very natural cmake way...i.e. I don't want anything conan specific in the consuming library's CMakeLists.txt. I just want to find_package my dependency, target_link_libraries to it, and move on just like I could pre-conan. If the dependency did their cmake correctly, all transitive dependencies are handled automagically. Per this blog article: https://blog.conan.io/2018/06/11/Transparent-CMake-Integration.html it seems the way to do this is using the cmake_paths generator. I can make and consume packages with that generator no problem.
I'm now trying to comsume a number of third party libraries, namely grpc, yaml-cpp, and Catch2, however none of those packages work with the cmake_paths generator because as part of their package recipe they explicitly delete the cmake package configuration files.
See
https://github.com/conan-io/conan-center-index/blob/ce2f6b89606cc582ccabbb5420f18a29e705dae3/recipes/grpc/all/conanfile.py#L171
https://github.com/conan-io/conan-center-index/blob/ce2f6b89606cc582ccabbb5420f18a29e705dae3/recipes/catch2/2.x.x/conanfile.py#L97
https://github.com/conan-io/conan-center-index/blob/ce2f6b89606cc582ccabbb5420f18a29e705dae3/recipes/yaml-cpp/all/conanfile.py#L95-L96
I obviously haven't done an exhaustive search to see how many packages do this, I just find it hard to believe it's just a coincidence that the three libraries I want to pull in first are the only three that do this.
Is there a reason this is done? or is this a hold-over from times before the cmake_paths generator was a thing and should now be considered a bug?
In the blog article about transparent cmake integration, it states that one of the downsides of the cmake_paths generator is that transitive dependency information is not propagated, but the only reason I can see that would be is because the CMake config modules are deleted as shown above--a major feature of what cmake does (especially modern CMake) is to manage those transitive dependencies. Why does conan seem to want to throw that information away?
The reasons why ConanCenter (not Conan, this is only a requirement of public packages in ConanCenter) is removing the packaged findxxx.cmake or xxxx-config.cmake files from packages are:
Packages from ConanCenter should work from any other build system, not only CMake. Even if CMake is now used by a majority of devs (some surveys shows around 50-55%), there are still a lot of people using other build systems, MSBuild, Meson, Autotools, ec. Conan defines the information for consumers in its package_info() method, which is an abstraction that will work for any consumer build system. When this was not mandatory in ConanCenter, it resulted in many packages that only worked for CMake.
It happens that some of the packaged cmake files can be problematic, and depending on how they are generated, they will not handle transitive dependencies as expected, and they can find transitive dependencies in the system instead of other Conan packages. This happens sometimes when an open source library doesn't have a modern and correct CMakeLists.txt and locates some dependencies in the system directly. Unfortunately the quality of CMakeLists.txt files out there is varying and they not always follow best practices.
Conan handles binary configuration separately, to scale to support many different binary configurations (for example ConanCenter builds around 130 different binaries for each package version), so for example the Debug and Release packages are in separate locations. The native find_package() CMake files cannot handle this situation, so all users expecting to have multi-config setup (like Visual Studio and Xcode) will not manage to achieve this without the Conan generated .cmake files.
Again, this only applies to ConanCenter packages, because ConanCenter is trying to be as universal as possible (to support fairly all build systems) and to allow multi-configuration setups, while being as robust as possible given the complexities of the diverse ecosystem.
In any case, the modern CMakeDeps and CMakeToolchain experimental generators will achieve a transparent integration, they are already released in latest Conan 1.X and as those will be the ones that will survive in next 2.0, it is recommended to start trying them soon.
Related
I am very much confused between these two terms Build tools and Package Manager. According to my current knowledge, Package managers are the ones use to install dependencies required for the code to execute while Build tools are used to Package the code plus dependencies into single file i.e. building the code. Building our application will enable to make it production ready.
Am I right???
Short answer
Build systems/tools manage your compilation requirements.
Package managers/tools manager your library requirements.
A build tool may have integrated package management.
For example, in both C++ and Java you can directly call the compiler and provide all the include, source and library paths manually or you can use a build system (make/cmake... for c++, maven/gradle/ant... for java).
When you link external libraries with your build system it will do its best to find them in its search path, and will link with the first version that meets its requirements or tell you that it couldn't find it. Adding libraries manually is fairly easy, but sometimes each library you add will require another library with it.
A package manager would make sure that your libraries are downloaded, are the right version, and all the libraries they depend on are downloaded. some examples are maven and gradle which have integrated package managment for java, and conan is a fairly popular option to combine with cmake.
So ideally you would use both, but it can be more work setting them up than you save not doing things manually. It depends on your programming language, if you need multiple versions of something, and your OS.
I'm a DevOps engineer creating CI processes for projects. I was wondering what is the best way to deal with the following scenario: Let's say I have a C++ project (using CLion + CMake) with several developers working on it. Now in order to be built, the project has some libraries it depends on. That automatically reflects on the CMakeLists.txt file that should know where to look for those libraries.
Basically the problem is that we need to take care that every developer has these libraries in the correct paths on his machine, which is a big hassle.
One approach to handle this would be to keep those dependencies in the repository. That's great since all the developer has to do is to clone the repo and he got everything he needs in order to run compilation. But as we know, keeping binaries in SCM is not a good practice.
The question is, is there a good method to handle project dependencies in a C++ project?
I know that with C# for example, we could use NuGet packages to handle these kind of scenarios. So we'd have a NuGet repository in Artifactory that would host the dependency packages, and then in our project we'd keep a reference to the required packages, and in build time we would just download the dependecies and build the project.
Is there something alike in C++ (Running on Linux I mean)?
Hope the question is clear enough lol, had a hard time wording it..
It depends on how those dependencies are delivered and packaged. If whoever maintains your dependencies took CMake into account you can probably use find_package. If they didn't account for this, but they support pkg-config you can use FindPkgConfig. Now all you need to do is let the developers know what dependencies they need to install. This should work regardless of the OS used for development.
Other solutions may involve pulling and building the dependency code when you build your project (for example, by using git submodules if possible, or FetchContent, but this can become a nightmare if you have a lot of dependencies).
Additionally, you can try using a package manager like vcpkg, or conan (if all your dependencies are available there), or CPM.
I have to deal with a cross-platform application which needs an additional 3rd party library on Windows but not on Linux (and that library doesn't support Linux anyway). I have packed the library into a Conan package, supporting only `os=Windows complier="Visual Studio". Now if I put this library as
[requires]
Library/1.2.3#foo/bar
in my conanfile.txt, conan install will logically fail with the Error Missing prebuilt package on Linux.
So is there a way to specify required packages conditionally in conanfile.txt? Something like Library/1.2.3#foo/bar [os="Windows"]. I read through the Conan docs but found nothing.
Or to tell conan install ignore the error?
Or my only luck is to use two different conanfile.txt on two platforms?
I can not use a conanfile.py, since the build process is not managed by Conan, only the dependencies.
I would say the best option in your case is conanfile.py, but as you are not able to use it, you will need to keep two conanfile.txt, one per platform.
It's not possible adding condition in conanfile.txt, because it would cost a big effort of development, and conafile.py is able to deal with conditions since is a python script.
#nada solution is good, you can use CMake to call conan according your OS, but also you can try cmake-conan, which sounds better for your specific scenario.
A little background, we have a fairly large code base, which builds in to a set of libraries - which are then distributed for internal use in various binaries. At the moment, the build process for this is haphazard and everything is built off the trunk.
We would like to explore whether there is a build system which will allow us to manage releases and automatically pull in dependencies. Such a tool exists for java, Maven. I like it's package, repository and dependency mechanism, and I know that with either the maven-native or maven-nar plugin we could get this. However the problem is that we cannot fix the source trees to the "maven way" - and unfortunately (at least the maven-nar) plugins don't seem to like code that is not structured this way...
So my question is, is there a tool which satisfies the following for C++
build
package (for example libraries with all headers, something like the .nar)
upload package to a "repository"
automatically pull in the required dependencies from said repository, extract headers and include in build, extract libraries and link. The depedencies would be described in the "release" for that binary - so if we were to use CI server to build that "release", the build script has the necessary dependencies listed (like the pom.xml files).
I could roll my own by modifying either make+shell scripts or waf/scons with extra python modules for the packaging and dependency management - however I would have thought that this is a common problem and someone somewhere has a tool for this? Or does everyone roll their own? Or have I missed a significant feature of waf/scons or CMake?
EDIT: I should add, OS is preferred, and non-MS...
Most of the linux distributions, for example, contain dependency tracking for their packages. Of all the things that I've tried to cobble together myself to take on your problem, in the end they all are "not quite perfect". The best thing to do, IMHO, is to create a local yum/deb repository or something (continuing my linux example) and then pull stuff from there as needed.
Many of the source-packages also quickly tell you the minimum components that must be installed to do a self-build (as opposed to installing a binary pre-compiled package).
Unfortunately, these methods are that much easier, though it's better than trying to do it yourself. In the end, to be cross-platform supporting, you need one of these systems per OS as well. Fun!
I am not sure if I understand correctly what you want to du, but I will tell you what we use and hope it helps.
We use cmake for our build. It hat to be noted that cmake is quite powerful. Among other things, you can "make install" in custom directories to collect headers and binaries there to build your release. We combine this with some python scripting to build our releases. YMMV, but some things might just be too specific for a generic tool and a custom script may be the simpler solution.
Our build tool builds releases directly from a svn reposity (checkout, build, ...) which I can really recommend to avoid some local state polluting the release in some unforseen way. It also enforces reproducability.
It depends a lot on the platforms you're targeting. I can only really speak for Linux, but there it also depends on the distributions you're targeting, packages being a distribution-level concept. To make things a bit simpler, there are families of distributions using similar packaging mechanisms and package names, meaning that the same recipe for making a Debian package will probably make an Ubuntu package too.
I'd definitely say that if you're willing to target a subset of all known Linux distros using a manageable set of packaging mechanisms, you will benefit in the long run from not rolling your own and building packages the way the distribution creators intended. These systems allow you to specify run- and build-time dependencies, and automatic CI environments also exist (like OBS for rpm-based distros).
I'm in the middle of setting up an build environment for a c++ game project. Our main requirement is the ability to build not just our game code, but also its dependencies (Ogre3D, Cegui, boost, etc.). Furthermore we would like to be able build on Linux as well as on Windows as our development team consists of members using different operating systems.
Ogre3D uses CMake as its build tool. This is why we based our project on CMake too so far. We can compile perfectly fine once all dependencies are set up manually on each team members system as CMake is able to find the libraries.
The Question is if there is an feasible way to get the dependencies set up automatically. As a Java developer I know of Maven, but what tools do exist in the world of c++?
Update: Thanks for the nice answers and links. Over the next few days I will be trying out some of the tools to see what meets our requirements, starting with CMake. I've indeed had my share with autotools so far and as much as I like the documentation (the autobook is a very good read), I fear autotools are not meant to be used on Windows natively.
Some of you suggested to let some IDE handle the dependency management. We consist of individuals using all possible technologies to code from pure Vim to fully blown Eclipse CDT or Visual Studio. This is where CMake allows use some flexibility with its ability to generate native project files.
In the latest CMake 2.8 version there is the new ExternalProject module.
This allows to download/checkout code, configure and build it as part of your main build tree.
It should also allow to set dependencies.
At my work (medical image processing group) we use CMake to build all our own libraries and applications. We have an in-house tool to track all the dependencies between projects (defined in a XML database). Most of the third party libraries (like Boost, Qt, VTK, ITK etc..) are build once for each system we support (MSWin32, MSWin64, Linux32 etc..) and are commited as zip-files in the version control system. CMake will then extract and configure the correct zip file depending on which system the developer is working on.
I have been using GNU Autotools (Autoconf, Automake, Libtool) for the past couple of months in several projects that I have been involved in and I think it works beautifully. Truth be told it does take a little bit to get used to the syntax, but I have used it successfully on a project that requires the distribution of python scripts, C libraries, and a C++ application. I'll give you some links that helped me out when I first asked a similar question on here.
The GNU Autotools Page provides the best documentation on the system as a whole but it is quite verbose.
Wikipedia has a page which explains how everything works. Autoconf configures the project based upon the platform that you are about to compile on, Automake builds the Makefiles for your project, and Libtool handles libraries.
A Makefile.am example and a configure.ac example should help you get started.
Some more links:
http://www.lrde.epita.fr/~adl/autotools.html
http://www.developingprogrammers.com/index.php/2006/01/05/autotools-tutorial/
http://sources.redhat.com/autobook/
One thing that I am not certain on is any type of Windows wrapper for GNU Autotools. I know you are able to use it inside of Cygwin, but as for actually distributing files and dependencies on Windows platforms you are probably better off using a Windows MSI installer (or something that can package your project inside of Visual Studio).
If you want to distribute dependencies you can set them up under a different subdirectory, for example, libzip, with a specific Makefile.am entry which will build that library. When you perform a make install the library will be installed to the lib folder that the configure script determined it should use.
Good luck!
There are several interesting make replacements that automatically track implicit dependencies (from header files), are cross-platform and can cope with generated files (e.g. shader definitions). Two examples I used to work with are SCons and Jam/BJam.
I don't know of a cross-platform way of getting *make to automatically track dependencies.
The best you can do is use some script that scans source files (or has C++ compiler do that) and finds #includes (conditional compilation makes this tricky) and generates part of makefile.
But you'd need to call this script whenever something might have changed.
The Question is if there is an feasible way to get the dependencies set up automatically.
What do you mean set up?
As you said, CMake will compile everything once the dependencies are on the machines. Are you just looking for a way to package up the dependency source? Once all the source is there, CMake and a build tool (gcc, nmake, MSVS, etc.) is all you need.
Edit: Side note, CMake has the file command which can be used to download files if they are needed: file(DOWNLOAD url file [TIMEOUT timeout] [STATUS status] [LOG log])
Edit 2: CPack is another tool by the CMake guys that can be used to package up files and such for distribution on various platforms. It can create NSIS for Windows and .deb or .tgz files for *nix.
At my place of work (we build embedded systems for power protection) we used CMake to solve the problem. Our setup allows cmake to be run from various locations.
/
CMakeLists.txt "install precompiled dependencies and build project"
project/
CMakeLists.txt "build the project managing dependencies of subsystems"
subsystem1/
CMakeLists.txt "build subsystem 1 assume dependecies are already met"
subsystem2/
CMakeLists.txt "build subsystem 2 assume dependecies are already met"
The trick is to make sure that each CMakeLists.txt file can be called in isolation but that the top level file can still build everything correctly. Technically we don't need the sub CMakeLists.txt files but it makes the developers happy. It would be an absolute pain if we all had to edit one monolithic build file at the root of the project.
I did not set up the system (I helped but it is not my baby). The author said that the boost cmake build system had some really good stuff in it, that help him get the whole thing building smoothly.
On many *nix systems, some kind of package manager or build system is used for this. The most common one for source stuff is GNU Autotools, which I've heard is a source of extreme grief. However, with a few scripts and an online depository for your deps you can set up something similar like so:
In your project Makefile, create a target (optionally with subtargets) that covers your dependencies.
Within the target for each dependency, first check to see if the dep source is in the project (on *nix you can use touch for this, but you could be more thorough)
If the dep is not there, you can use curl, etc to download the dep
In all cases, have the dep targets make a recursive make call (make; make install; make clean; etc) to the Makefile (or other configure script/build file) of the dependency. If the dep is already built and installed, make will return fairly promptly.
There are going to be lots of corner cases that will cause this to break though, depending on the installers for each dep (perhaps the installer is interactive?), but this approach should cover the general idea.
Right now I'm working on a tool able to automatically install all dependencies of a C/C++ app with exact version requirement :
compiler
libs
tools (cmake, autotools)
Right now it works, for my app. (Installing UnitTest++, Boost, Wt, sqlite, cmake all in correct order)
The tool, named «C++ Version Manager» (inspired by the excellent ruby version manager), is coded in bash and hosted on github : https://github.com/Offirmo/cvm
Any advices and suggestions are welcomed.