I am interested in building a cross-platform C++ Library and distributing it in source form. I want the consumers of this library to be able to acquire it, build it and consume it inside their software very easily on whatever platform they are working on and for whatever platform they are targeting. At the same time while building my library, I also want to be able to consume other popular OSS libraries through a similar mechanism.
I see that CMake and Ryppl were created with these intentions in mind and to some extent they do solve some of these problems, especially the build problem. But I don't quite know how exactly to go about achieving the above mentioned goals. Is it OK to settle on CMake as the build solution? How do I solve the library acquisition and distribution problem? Simply host the sources somewhere and let people discover, download and build them? Or is there a better way?
At the time of writing there is no accepted solution that handles everything you want. CMake gives you cross-platform builds and git (with submodules) gives you a way to manage source-level dependencies if all other projects are using CMake. But, in practice, many common dependencies you project wil need don't use CMake, or even Git.
Ryppl is meant to solve this, but progress is slow because the challenge is such a hard one.
Arguably, the most successful solution so far is to write header-only libraries. This is common practice and means your users just include your files, and their build system of choice takes care of everthing. There are no binary dependencies to manage.
TheHouse's answer is still essentially true. Also there don't seem to have been any updates to ryppl itself for a while (3 years) and the ryppl.org domain has expired.
There are some new projects aiming to solve the packaging issue.
Both build2 and wrap from mesonbuild have that goal in mind.
A proposal was made recently to add packages to the c++ standard which may open up the debate (reddit discussion here).
Wrap looks promising as meson's author has learned from cmake.
There is a good video when its author discussing this here.
build2 seems more oblivious (and therefore condemned to reinvent). However both suffer from trying to solve the external project dependencies issue simultaneously with providing a complete build system.
conan.io is another recent attempt which doesn't try to provide the build system as well. Time will tell if any of these gain any traction.
The accepted standard for packaging C and C++ projects on Unix was always a source tarball + a configure script (autotools) + make.
cmake is now beginning to replace autotools as your first choice.
It is able create RPMs and tarballs for distribution purposes.
Its also worth considering the package managers built into the various flavours of Linux. The easiest to build and install projects are those where most of the dependencies can be pulled in via yum or apt. This won't help you on windows of course. While there is a high barrier to entry getting your own projects added to the main Linux repositories (e.g. RedHat, Debian) there is nothing to stop you adding your maintaining your own satellite repo.
The difference between that and just hosting your project on github or similar is you can provide pre-built binaries for a number of popular systems.
You might also consider that configure times checks (e.g. from cmake findLibrary()) and your own documentation will tell people what needs to be installed as a prerequisite and providing you don't make it too onerous that might be enough.
Related
I'm a DevOps engineer creating CI processes for projects. I was wondering what is the best way to deal with the following scenario: Let's say I have a C++ project (using CLion + CMake) with several developers working on it. Now in order to be built, the project has some libraries it depends on. That automatically reflects on the CMakeLists.txt file that should know where to look for those libraries.
Basically the problem is that we need to take care that every developer has these libraries in the correct paths on his machine, which is a big hassle.
One approach to handle this would be to keep those dependencies in the repository. That's great since all the developer has to do is to clone the repo and he got everything he needs in order to run compilation. But as we know, keeping binaries in SCM is not a good practice.
The question is, is there a good method to handle project dependencies in a C++ project?
I know that with C# for example, we could use NuGet packages to handle these kind of scenarios. So we'd have a NuGet repository in Artifactory that would host the dependency packages, and then in our project we'd keep a reference to the required packages, and in build time we would just download the dependecies and build the project.
Is there something alike in C++ (Running on Linux I mean)?
Hope the question is clear enough lol, had a hard time wording it..
It depends on how those dependencies are delivered and packaged. If whoever maintains your dependencies took CMake into account you can probably use find_package. If they didn't account for this, but they support pkg-config you can use FindPkgConfig. Now all you need to do is let the developers know what dependencies they need to install. This should work regardless of the OS used for development.
Other solutions may involve pulling and building the dependency code when you build your project (for example, by using git submodules if possible, or FetchContent, but this can become a nightmare if you have a lot of dependencies).
Additionally, you can try using a package manager like vcpkg, or conan (if all your dependencies are available there), or CPM.
A little background, we have a fairly large code base, which builds in to a set of libraries - which are then distributed for internal use in various binaries. At the moment, the build process for this is haphazard and everything is built off the trunk.
We would like to explore whether there is a build system which will allow us to manage releases and automatically pull in dependencies. Such a tool exists for java, Maven. I like it's package, repository and dependency mechanism, and I know that with either the maven-native or maven-nar plugin we could get this. However the problem is that we cannot fix the source trees to the "maven way" - and unfortunately (at least the maven-nar) plugins don't seem to like code that is not structured this way...
So my question is, is there a tool which satisfies the following for C++
build
package (for example libraries with all headers, something like the .nar)
upload package to a "repository"
automatically pull in the required dependencies from said repository, extract headers and include in build, extract libraries and link. The depedencies would be described in the "release" for that binary - so if we were to use CI server to build that "release", the build script has the necessary dependencies listed (like the pom.xml files).
I could roll my own by modifying either make+shell scripts or waf/scons with extra python modules for the packaging and dependency management - however I would have thought that this is a common problem and someone somewhere has a tool for this? Or does everyone roll their own? Or have I missed a significant feature of waf/scons or CMake?
EDIT: I should add, OS is preferred, and non-MS...
Most of the linux distributions, for example, contain dependency tracking for their packages. Of all the things that I've tried to cobble together myself to take on your problem, in the end they all are "not quite perfect". The best thing to do, IMHO, is to create a local yum/deb repository or something (continuing my linux example) and then pull stuff from there as needed.
Many of the source-packages also quickly tell you the minimum components that must be installed to do a self-build (as opposed to installing a binary pre-compiled package).
Unfortunately, these methods are that much easier, though it's better than trying to do it yourself. In the end, to be cross-platform supporting, you need one of these systems per OS as well. Fun!
I am not sure if I understand correctly what you want to du, but I will tell you what we use and hope it helps.
We use cmake for our build. It hat to be noted that cmake is quite powerful. Among other things, you can "make install" in custom directories to collect headers and binaries there to build your release. We combine this with some python scripting to build our releases. YMMV, but some things might just be too specific for a generic tool and a custom script may be the simpler solution.
Our build tool builds releases directly from a svn reposity (checkout, build, ...) which I can really recommend to avoid some local state polluting the release in some unforseen way. It also enforces reproducability.
It depends a lot on the platforms you're targeting. I can only really speak for Linux, but there it also depends on the distributions you're targeting, packages being a distribution-level concept. To make things a bit simpler, there are families of distributions using similar packaging mechanisms and package names, meaning that the same recipe for making a Debian package will probably make an Ubuntu package too.
I'd definitely say that if you're willing to target a subset of all known Linux distros using a manageable set of packaging mechanisms, you will benefit in the long run from not rolling your own and building packages the way the distribution creators intended. These systems allow you to specify run- and build-time dependencies, and automatic CI environments also exist (like OBS for rpm-based distros).
There are already some questions about dependency managers here, but it seems to me that they are mostly about build systems, while I am looking for something targeted purely at making dependency tracking and resolution simpler (and I'm not necessarily interested in learning a new build system).
So, typically we have a project and some common code with another project. This common code is organized as a library, so when I want to get the latest code version for a project, I should also go get all the libraries from the source control. To do this, I need a list of dependencies. Then, to build the project I can reuse this list too.
I've looked at Maven and Ivy, but I'm not sure if they would be appropriate for C++, as they look quite heavily java-targeted (even though there might be plugins for C++, I haven't found people recommending them).
I see it as a GUI tool producing some standardized dependency list which can then be parsed by different scripts etc. It would be nice if it could integrate with source control (tag, get a tagged version with dependencies etc), but that's optional.
Would you have any suggestions? Maybe I'm just missing something, and usually it's done some other way with no need for such a tool? Thanks.
You can use Maven in relationship with C++ in two ways. First you can use it for dependency management of components between each other. Second you can use Maven-nar-plugin for creating shared libraries and unit tests in relationship with boost library (my experience). In the end you can create RPM's (maven-rpm-plugin) out of it to have adequate installation medium. Furthermore i have created the installation for CI environment via Maven (RPM's for Hudson, Nexus installation in RPM's).
I'm not sure if you would see an version control system (VCS) as build tool but Mercurial and Git support sub-repositories. In your case a sub-repository would be your dependencies:
Join multiple subrepos into one and preserve history in Mercurial
Multiple git repo in one project
Use your VCS to archive the build results -- needed anyway for maintenance -- and refer to the libs and header files in your build environment.
If you are looking for a reference take a look at https://android.googlesource.com/platform/manifest.
I've done a lot of projects and a lot of different build systems and CI tools. Most recently, I've been exposed to the occasionally challenging task of adding to an autotools based environment for a reasonably sized C++ application. While I love the ease of use for the end user, I'm not so fond of dealing with m4 and all of the auto* tools from the developer side.
I'm working on a reasonably large side project in my spare time and have decided that I'd like to take CMake for a test drive. Since I'm just starting out, I'm obviously planning on digging through the documentation, FAQ, wikis, etc. and learning by doing. BTW, I'd fork over money for the "Mastering CMake" book, but the comments that I found on Amazon were enough to make me decide that it probably isn't worth the money. All that being said, in anything new there are often "gotchas" that a newcomer will often stumble over that the old pros have long since learned to avoid. I'm wondering what those are based on people's experience with CMake, and I'm hoping to cut my learning pains down a bit by asking here.
I should point out that I'm planning on building on Linux primarily, and other UN*X variants. Windows isn't really a concern from my POV. This is a large server-side application, with a Web and CLI interface for operators, a northbound REST interface for automation/integration with OSS tooling, and a SOAP southbound interface for CPEs. I'm going to need a lot of third-party libraries and applications to get this all to work unless I want to take the next 10 years to build this all by hand. :)
First of all, I think CMake is an excellent build-tool. It has in my opinion by far the best support for multi-platform-builds and has a powerful mechanism for finding 3rd-party libraries. Combined with CPack it even provides reasonable options for packaging and installation. Some hints and possible problems:
Always aim for out-of-source builds: An obvious one: It can be difficult for a project designed for in-source builds to get it built out-of-source. So, if you can design it from the start, aim for out-of-source builds.
Syntax: The syntax can be bizarre with a lot of strange quirks, although this is getting better since the 2.6 and 2.8 versions.
Caching of variables. CMake keeps track of a cache of variables and settings and sometimes this can be a problem when rebuilding something. Try to remove the CMakeCache.txt (or clean your out-of-source build-directory) and rebuild. This one is also mentioned by DarenW.
Finding and configuring 3rd-party libraries: If you have a large project depending on several libraries (your own or 3rd-party), this will likely be the most troublesome.
I seriously recommend diving into the find_package command. It is very powerful, but fully depends on the quality of the FindXXX.cmake files. Especially when building on multiple platforms (mostly Windows and Mac), these files may require some tweaking or you may need to write your own.
Problems with linking libraries like debugging "undefined references" or mismatches between debug|release or static|shared can be difficult to debug. Especially with 3rd-party libraries these problems may be caused by incorrect 3rd-party FindXXX.cmake files...
One thing that took me a while to figure out: if anything goes haywire with a build because I've changed a compiler option, added/removed a .c file, had confusion with support library versions, etc. - it's best to delete the build tree and run CMake from scratch. Otherwise, even though CMake and make seem to do their job, the resulting executable crashes.
It could be we're "not doing it right", but that's becoming a common problem with many tools, languages, frameworks, etc. It's not possible for everyone to be expert at all the software they depend on, and "not doing it right" is the only way a lot of things get done even among professionals. CMake is pretty smart about a lot of things, but not everything. Nuking the build tree remains a commonly used technique on our project.
What are the differences between Autotools, Cmake and Scons?
In truth, Autotools' only real 'saving grace' is that it is what all the GNU projects are largely using.
Issues with Autotools:
Truly ARCANE m4 macro syntax combined with verbose, twisted shell scripting for tests for "compatibility", etc.
If you're not paying attention, you will mess up cross-compilation ability (It
should clearly be noted that Nokia came up with Scratchbox/Scratchbox2 to side-step highly broken Autotools build setups for Maemo/Meego.) If you, for any reason, have fixed, static paths in your tests, you're going to break cross-compile support because it won't honor your sysroot specification and it'll pull stuff from out of your host system. If you break cross-compile support, it renders your code unusable for things like
OpenEmbedded and makes it "fun" for distributions trying to build their releases on a cross-compiler instead of on target.
Does a HUGE amount of testing for problems with ancient, broken compilers that NOBODY currently uses with pretty much anything production in this day and age. Unless you're building something like glibc, libstdc++, or GCC on a truly ancient version of Solaris, AIX, or the like, the tests are a waste of time and are a source for many, many potential breakages of things like mentioned above.
It is pretty much a painful experience to get an Autotools setup to build usable code for a Windows system. (While I've little use for Windows, it is a serious concern if you're developing purportedly cross-platform code.)
When it breaks, you're going to spend HOURS chasing your tail trying to sort out the things that whomever wrote the scripting got wrong to sort out your build (In fact, this is what I'm trying to do (or, rather, rip out Autotools completely- I doubt there's enough time in the rest of this month to sort the mess out...) for work right now as I'm typing this. Apache Thrift has one of those BROKEN build systems that won't cross-compile.)
The "normal" users are actually NOT going to just do "./configure; make"- for many things, they're going to be pulling a package provided by someone, like out of a PPA, or their distribution vendor. "Normal" users aren't devs and aren't grabbing tarballs in many cases. That's snobbery on everyone's part for presuming that is going to be the case there. The typical users for tarballs are devs doing things, so they're going to get slammed with the brokenness if it's there.
It works...most of the time...is all you can say about Autotools. It's a system that solves several problems that only really concerns the GNU project...for their base, core toolchain code. (Edit (05/24/2014): It should be noted that this type of concern is a potentially BAD thing to be worrying about- Heartbleed partially stemmed from this thinking and with correct, modern systems, you really don't have any business dealing with much of what Autotools corrects for. GNU probably needs to do a cruft removal of the codebase, in light of what happened with Heartbleed) You can use it to do your project and it might work nicely for a smallish project that you don't expect to work anywhere except Linux or where the GNU toolchain is clearly working correctly on. The statement that it "integrates nicely with Linux" is quite the bold statement and quite incorrect. It integrates with the GNU toolsuite reasonably well and solves problems that IT has with it's goals.
This is not to say that there's no problems with the other options discussed in the thread here.
SCons is more of a replacement for Make/GMake/etc. and looks pretty nice, all things considered However...
It is still really more of a POSIX only tool. You could probably more easily get MinGW to build Windows stuff with this than with Autotools, but it's still really more geared to doing POSIX stuff and you'd need to install Python and SCons to use it.
It has issues doing cross-compilation unless you're using something like Scratchbox2.
Admittedly slower and less stable than CMake from their own comparison. They come up with half-hearted (the POSIX side needs make/gmake to build...) negatives for CMake compared to SCons. (As an aside, if you're needing THAT much extensibility over other solutions, you should be asking yourself whether your project's too complicated...)
The examples given for CMake in this thread are a bit bogus.
However...
You will need to learn a new language.
There's counter-intuitive things if you're used to Make, SCons, or Autotools.
You'll need to install CMake on the system you're building for.
You'll need a solid C++ compiler if you don't have pre-built binaries for it.
In truth, your goals should dictate what you choose here.
Do you need to deal with a LOT of broken toolchains to produce a valid working binary? If yes, you may want to consider Autotools, being aware of the drawbacks I mentioned above. CMake can cope with a lot of this, but it worries less with it than Autotools does. SCons can be extended to worry about it, but it's not an out-of-box answer there.
Do you have a need to worry about Windows targets? If so, Autotools should be quite literally out of the running. If so, SCons may/may not be a good choice. If so, CMake's a solid choice.
Do you have a need to worry about cross-compilation (Universal apps/libraries, things like Google Protobufs, Apache Thrift, etc. SHOULD care about this...)? If so, Autotools might work for you so long as you don't need to worry about Windows, but you're going to spend lots of time maintaining your configuration system as things change on you. SCons is almost a no-go right at the moment unless you're using Scratchbox2- it really doesn't have a handle on cross-compilation and you're going to need to use that extensibility and maintain it much in the same manner as you will with Automake. If so, you may want to consider CMake since it supports cross-compilation without as many of the worries about leaking out of the sandbox and will work with/without something like Scratchbox2 and integrates nicely with things like OpenEmbedded.
There is a reason many, many projects are ditching qmake, Autotools, etc. and moving over to CMake. So far, I can cleanly expect a CMake based project to either drop into a cross-compile situation or onto a VisualStudio setup or only need a small amount of clean up because the project didn't account for Windows-only or OSX-only parts to the codebase. I can't really expect that out of an SCons based project- and I fully expect 1/3rd or more Autotools projects to have gotten SOMETHING wrong that precludes it building right on any context except the host building one or a Scratchbox2 one.
An important distinction must be made between who uses the tools. Cmake is a tool that must be used by the user when building the software. The autotools are used to generate a distribution tarball that can be used to build the software using only the standard tools available on any SuS compliant system. In other words, if you are installing software from a tarball that was built using the autotools, you are not using the autotools. On the other hand, if you are installing software that uses Cmake, then you are using Cmake and must have it installed to build the software.
The great majority of users do not need to have the autotools installed on their box. Historically, much confusion has been caused because many developers distribute malformed tarballs that force the user to run autoconf to regenerate the configure script, and this is a packaging error. More confusion has been caused by the fact that most major linux distributions install multiple versions of the autotools, when they should not be installing any of them by default. Even more confusion is caused by developers attempting to use a version control system (eg cvs, git, svn) to distribute their software rather than building tarballs.
It's not about GNU coding standards.
The current benefits of autotools — specifically when used with automake — is that they integrate very well with building Linux distribution.
With cmake for example, it's always "was it -DCMAKE_CFLAGS or -DCMAKE_C_FLAGS that I need?" No, it's neither, it's "-DCMAKE_C_FLAGS_RELEASE". Or -DCMAKE_C_FLAGS_DEBUG. It's confusing - in autoconf, it's just ./configure CFLAGS="-O0 -ggdb3" and you have it.
In integration with build infrastructures, scons has the problem that you cannot use make %{?_smp_mflags}, _smp_mflags in this case being an RPM macro that roughly expands to (admin may set it) system power. People put things like -jNCPUS here through their environment. With scons that's not working, so the packages using scons may only get serialed built in distros.
What is important to know about the Autotools is that they are not a general build system - they implement the GNU coding standards and nothing else. If you want to make a package that follows all the GNU standards, then Autotools are an excellent tool for the job. If you don't, then you should use Scons or CMake. (For example, see this question.) This common misunderstanding is where most of the frustration with Autotools comes from.
While from a developers point of view, cmake is currently the most easy to use, from a user perspective autotools have one big advantage
autotools generate a single file configure script and all files to generate it are shipped with the distribution. it is easy to understand and fix with help of grep/sed/awk/vi. Compare this to Cmake where a lot of files are found in /usr/share/cmak*/Modules, which can't be fixed by the user unless he has admin access.
So, if something does not quite work, it can usually easily be "fixed" by using Standard Unix tools (grep/sed/awk/vi etc.) in a sledgehammer way without having to understand the buildsystem.
Have you ever digged through your cmake build directory to find out what is wrong? Compared to the simple shellscript which can be read from top to bottom, following the generated Cmake files to find out what is going on is quite difficult. ALso, with CMake, adapting the FindFoo.cmake files requires not only knowledge of the CMake language, but also might require superuser privileges.