How to organize large project in several GIT repositories? [closed] - c++

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have a C++ project cat that depends on libzzz library. The libzzz has its own git repository and now I am going to create a repository for the cat project.
How to organize a CMake build scripts for the cat?
option 1: CMake scripts of cat consider libzzz built and installed in the system and cat project provide FindLibZZZ.cmake script that search libzzz in /usr/include/libzzz + /usr/lib/libzzz. But how to deal with non-linux platforms? I presonally don't like this option.
option 2: add some kind of link or dependency in GIT repository cat that would automatically checkout libzzz sources from its origin into some cat subdirectory. So cat's CMakeLists.txt consider libzzz is placed in some cat's subdirectory. How to do that?

If libzzz is also a CMake project, then I'd suggest an alternative approach (let's call it option 3). Rather than relying on libzzz being already available on your system (which can make integrating into CI systems, etc. harder), you may want to consider building it as part of your overall project as well. Instead of trying to add it to cat via git submodules (valid, but has its own downsides, e.g. see here), you can make a top level build (aka superbuild) control the building of both libzzz and cat.
Bringing in external projects into a superbuild is often done using CMake's ExternalProject module. This allows you to download and build a project in one operation. Unfortunately, it does so at build time and provides no CMake targets for you to link against, so you end up having to manually work out the name and location of the library you want to link against from it, etc. Yes, you can robustly predict these for your build, but you have to manually handle all the platform differences. This is what CMake is supposed to take care of for us!
Instead of using ExternalProject in the normal way, you can, however, coax it to perform the download at CMake time. This then makes the external project's source code available immediately and you can call add_subdirectory to bring it directly into the main build. This in turn means any CMake targets defined by that no-longer-external build will also be visible, so you can simply link against those targets. In your particular example, the libzzz build would presumably define a zzz CMake target or similar and in your cat build, you would just add a call like:
target_link_libraries(cat PUBLIC zzz)
No need to manually work out any library names or locations, since CMake now does it for you. The trick to all this is getting ExternalProject to execute at CMake time rather than build time. The method is fully explained here, using GoogleTest as the example. The technique essentially involves using external_process to invoke a small script via CMake's script mode to call ExternalProject_Add immediately. That article includes a link to a fully generalised implementation you should be able to use to perform the downloading of libzzz in your particular case.

I believe that Option 1 is best if you ever plan to distribute the project to others (which it sounds like you do, if you are considering non-linux platforms). If you don’t plan to distribute cat to the wider world, then by all means go with Option 2, which is likely to be slightly easier to implement. However, suppose I want to use cat, and you go with option 2. In that case, there could be several problems:
If I already have libzzz because it’s also used by the dog project, then you are downloading and building something I already have. This is, at the very least, a waste of time.
If libzzz releases a critical security update, then I need to re-download your project also. If I’m lucky, you updated the cat repository to use the new version of libzzz. If I’m unlucky, then you have not noticed the new security update yet, you are in the middle of maintaining a different part of the repository, or you are dead and cat is completely unmaintained. If I’m really unlucky, I don’t notice that cat now needs to update because libzzz released a new version, and I am subject to the security problems.
Your cat project becomes popular. Now package maintainers everywhere need to find out how to compile cat with the existing libzzz from their repository.
I know I’ve seen an article from somebody who actually was a package maintainer arguing for Option 1, but I can’t seem to find it again. If I do, I’ll update this answer with the link.

Related

Handling external C++ dependencies [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 months ago.
Improve this question
The project structure we used to use was that code + prebuild external dependencies were source controlled in SVN. This was cumbersome because the external libraries were large and didn't need to be source controlled since they were prebuilt binaries.
Now we have source in git and the prebuilt binaries are in the cloud. The dev has to download this lib folder from the cloud after cloning the repo. Problem here is that if you make changes to lib then things will not build correctly until the developer goes and redownloads this lib folder.
Our projects are generally developed for Windows (MSVC) but we just added Linux (GCC+Docker) compatibility and likely in the future Linux will be the main version. So now our libraries each have a Windows and a Linux build folder. Our dev environments are Windows + VSCode/WSL2/Docker for Linux.
What is the best, common practice here for handling external dependencies. I can think of 2 ways.
Version the lib folder in the cloud and check that during building. If Developer A adds/changes this and updates the CMakeLists file then when Developer B updates his git repo and tries to build then CMake can see that the version of the libs folder he has is out of date and will be told to go update that. This is little effort and changes almost nothing of our process. Cons are that Developer A has to remember to update the version in both the cloud and in the cmake check.
Build all external libraries locally. Use git submodules and have cmake build all dependencies while building the main project. I assume there's a way to cache it so it doesn't rebuild constantly (some of these libraries are large and take a long time to build). More work, but less maintenance and needing of extra developer steps. Also probably easier to link and include against.
Problem here is that if you make changes to lib then things will not build correctly until the developer goes and redownloads this lib folder.
clear indication that the exact version you want to use is part of what you should track alongside your source code.
I assume there's a way to cache it so it doesn't rebuild constantly (some of these libraries are large and take a long time to build)
As long as files don't change, nothing needs to be rebuilt.
So, yeah, if your project depends on external libraries in specific, git submodules do sound kind of attractive.
Also note that other build systems (like meson) have a neater understanding of in-tree dependency projects, can check your system for an installed version of the dependency, and if not there in appropriate version, download, and if necessary, build themselves.
So, the second option is probably the easiest to maintain solution, as you said.
I, however, come from an free and open source background, and my users and their platforms are diverse, and Linux distros have strict guidelines about not packaging N copies of the same dependency. So, that would make it harder to upstream such packages to debian, Ubuntu, Fedora, Arch… . So, for me the situation is this: if there's a library that we want to use in a project, we define very clearly what the oldest version of that library is that would work. Within a release cycle, we cannot bump the required version.
So, say, we've released 2.0.0 of some software. The CMake files define which version of a library we support. "Releasing" software means that we guarantee to devs as well as to users that the next bugfix/feature extension versions in our 2.a.b series still build on the same systems – and that includes the same libraries that might be installed there. So, if 2.0.0 built on your computer, so will 2.0.1 and 2.9.0. Development that requires a new version of an external dependency can only happen on a git branch that's not meant for further 2.a.b releases, but targetting an eventual 3.0.0. When picking minimum dependency versions for that 3.0.0 release, we look what is commonly available on the operating systems we support. For example, if my timeline was that 3.0.0 be released within 2022 or 2023, that version would be the one available in Ubuntu 22.04LTS (because that will be an important system for our users for a long time, and also relatively conservative), also looking at the debian version most likely to be the current unstable (or stable, depending on what your target audience is), the RHEL version, the next Fedora, and what is currently available in our condaforge and macports repos.
Everything not available in tolerable versions through these standard packaging channels needs to be built locally anyway. Turns out that if you're not crazily progressive and don't try to support 5 year old Linuxes, the number of projects that you need to build locally is quite small.
On windows, you're basically handicapped by Microsoft's inability to provide a really sensible way of downloading packages of binary shared libraries that are actually shared between different application software. So, on Windows systems, you're down to either doing all your builds locally, or using a third-party way of distributing platform-dependent packages, like Conan.
No matter how you do it, you'd let your build fail as early as possible, with a clear indication that the library version found is not sufficiently new. CMake makes this easy; its find_package command takes a minimum version as argument.

What is Cmake? what is its role in the building process? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I really don't understand what "Cmake" is, I know the building process which is consist of:
Preprocessor
Compiler
Assembler
Linker
but where is "Cmake" in this process? what is its role and why do we need it?
I read that, it helps in compiler dependencies, but I don't get it.
please help me to know more about it and thanks in advance.
Cmake is a build system configuration generator. To understand what that is, you must first learn what a build system is (such as make, msbuild).
It's challenging to combine the tools that you've listed for a large project to produce multiple variations of multiple executables and or libraries. The complete build process may consist of a large number of differing commands that must be invoked in the correct order, and may each have a large number of options that must be passed to the tool. This is what a build system automates. It invokes those build tools with appropriate options as described in the configuration that you write for it. They also provide convenient features such as incremental re-compilation (checking whether a translation unit has already been compiled previously, avoiding re-compilation when the source hasn't been changed).
But there is a problem: There are many build systems each of which have their own configuration format, and those build systems are often specific to particular system. In order to compile a project on multiple differing systems (such as windows, linux, osx), you would have to write and maintain configuration for each. This is what a build system generator solves: You write a single configuration, and cmake configures your system specific build system, and you invoke that build system through cmake.
CMake is a cross-platform build tool. The idea is to let you define how to build a C++ project at a higher level of abstraction than however you build on one particular platform.
The issue is that although C++, the language, is independent of platform, how you build on any particular platform is not. Say you are implementing an application and would like to support Windows, Linux, and MacOS. If you do not use CMake or something like it this means that you will have to separately maintain a Visual Studio solution file on Windows, a MAKE file on Linux (say), and an Xcode project file on the Mac -- and this assumes that your application has no third party dependencies. If it does have third party dependencies then handling those in a cross-platform manner is a whole other can of worms. You might use per-platform scripts to manage dependencies, for example.
The idea of CMake and similar tools is to solve this problem. You define how to build your project and find its dependencies once and use the CMake definition across platforms.
Like many build tools, it creates output objects based on input artifacts (e.g. by running a compiler).
It only rebuilds an output object if...
if one of the input has been modified since last time the output object was built
if the build recipe for the output object has changed
This can be done with a classical Makefile also, but writing a Makefile that lists all dependencies for building an output object (including header files for instance) is not trivial.

Linux folders structure from developer perspective [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm rather new to Linux filesystems, so could you help me, please? I have to write a sample C++ project (tests) using Ubuntu.
Could you clarify me with a file/folder structure from a developer's perspective? Here are some questions I would like answered:
Where is typical location of the projects (sources, object files, etc.)?
Where is a typical place for dev environment (Eclipse, QT Creator, etc.)?
Where is a typical place for the libraries? Are there different places for binaries and for header-only libraries?
Where is a typical place for a various tools for the development (code analyser, git client, etc.)?
Answers and links would be appreciated. Thank you.
Where is typical location of the projects (sources, object files, etc.)?
I store my projects in $HOME/dev but its entirely up to you.
Where is a typical place for dev environment (Eclipse, QT Creator, etc.)?
I use eclipse and set its workspace pointing to $HOME/dev.
Where is a typical place for the libraries? Are there different places for binaries and for header-only libraries?
Typically libraries are installed to /usr/lib and headers are installed to /usr/include.
Where is a typical place for a various tools for the development (code analyser, git client, etc.)?
Typically these are installed into /usr/bin. I also put tools in $HOME/bin, especially if I made them.
But its more complicated than that. What if you want to develop/test an application with a version of a library that is different to the one that comes with your Linux distribution? Sometimes I will install different library versions in my $HOME/dev folder and compile against those rather than the system version.
Also I run Fedora 21 that comes with GCC 4.9.2, however I have installed GCC 5.1.0 into /opt/gcc-5.1.0 and use that for some things.
A typical project structure for me would be something like:
$HOME/
/dev/
/my-prog/
/src/
/include/
/my-prog.h
/my-prog.cpp
/build-debug/
/src/
/my-prog
/build-release/
/src/
/my-prog
/Makefile
The implementation varies somewhat between Linux distributions, but all seek to implement the Linux Filesystem Hierarchy Standard for the most part. Some distributions more so than others. The latest version is 2.3 and can be viewed via the links in the URL above or directly via FHS 2.3 PDF
Take time to view the document and compare it with the implementation Ubuntu uses. It will provide you with a good background to where and why files are located where they are in Linux, and also give you a quick lesson on how this is more of a goal for distributions than a hard requirement, as distributions are pretty much free to put things anywhere they want.
As for your project source tree layout, a common setup looks like this:
src/ this is were your .cpp files go
include/ for header that should get installed when you are writing a library
data/ is where graphics, icons, sounds, etc. go
external/ for git submodules or external libraries your project requires and that are obscure enough that you don't want users to manually install them
a build file in the top level directory (autoconf/automake was used historically, but these days CMake seems to be more popular)
a README or README.md explaining what the project does and which dependency it needs
Naming and organisation can of course vary between projects, there is no real standard, some use Source/ instead of src/. Some might have the headers within the src/ directory and so on. It's really up to your projects needs. If a library is commonly shipped with most distributions there is no need to include it in the source tree. Same with most programming tools, git, gcc and Co. are provided by the Linux distribution and you don't have to worry about where they go.
If you are writing a library you might also want to looking into pkg-config and the .so naming rules.
As for installation directories, the FHS explains that in detail, short summary:
/usr/bin for executables
/usr/lib for public libraries
/usr/lib/${APPNAME} for private libraries
/usr/include for public header files
/usr/share/${APPNAME} for data files
/opt/${APPNAME} is used by some commercial products instead of spreading the data over the hierachy
/usr itself is reserved for distribution installed packages, manually compiled stuff should go to /usr/local instead. Most build systems will have a --prefix=PREFIX option to allow the user to change that.
If you need to compile a library yourself, I would generally recommend to avoid installing it into /usr/local, as uninstalling software from there is hard and it also makes it impossible to keep different versions of the same software around at the same time.
Instead install software it into a PREFIX of it's own, something like ~/run/somelibrary-0.0.1. Installing software in such a way has however the disadvantage that tools won't find it. You can fix that by setting a few environment variables, I use a simple bash function for that purpose:
function activateprefix {
PREFIX="$1"; \
export PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig/"; \
export LD_LIBRARY_PATH="${PREFIX}/lib/"; \
export LD_RUN_PATH="${PREFIX}/lib/"; \
export LIBRARY_PATH="${PREFIX}/lib/"; \
export CPLUS_INCLUDE_PATH="${PREFIX}/include/"; \
export C_INCLUDE_PATH="${PREFIX}/include/"; \
}

Cleanest way for automated dependency management in c++ [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Related question: C++ Build Systems - What to use?
I am struggling to find a nice build / dependency management tool for my c++ project, with the following desired features:
able to specify a dependency by name, version
dependencies' "include" directories are automatically included during compilation of my application
the dependencies are automatically downloaded, built, and linked to my application
transitive dependencies also have the above two behaviours
ability to specify test-scope dependencies
tests are automatically built and run, potentially with a memory leak check tool (e.g. valgrind)
potentially run a coverage tool, e.g. gcov
cross platform supported
I have used Maven, with the [nar-maven-plugin], and sometimes the [cmake-maven plugin]. However, this means I have to craft a [pom.xml per dependency]. This method isn't particularly nice, as at times, a [nasty pom.xml] has to be crafted to get things to work. Also, it doesn't support running valgrind (no support built in yet).
I have attempted using CMake as I've seen many projects use it, but I find that a lot of time is spent on "writing a build/dependency management system" instead of "using it". Yes it's true I can write many functions that go:
function(RequireSomeLib artifact)
# ExternalProject_Add(SomeLib ... etc.)
# find SomeLib package
# add include dirs(artifact SomeLib_INCLUDE_DIRS)
# if SomeLib is not just a header library, also link its built library to the artifact
# for each of SomeLib's dependencies, do this same "call" (transitive dependencies' libraries must also be linked when building an executable)
endfunction()
for each dependency. Tedious, but currently the cleanest way I see going forward.
With the premise that CMake is used by libraries that my project depends on, is there a better method to solving this problem?
I have not seen or tried SCons, AutoTools, or QMake (yet).
In Java, the "retrieve dependencies, build, test, and publish" issue is much simpler ._.
All build systems for C++ will need you to code in dependencies and package detection. Every one of them has been created out of frustration with previous technologies, with the stated intent to remove the need for boilerplate code, and to create a complete, cross-platform, automated, easy to use solution, but at the end of the day, you will end up writing code besides your code for your package to be built.
If you look hard enough, you will find debates among advocates of each build system. I found one of such a couple years ago. Their arguments were so weak that I ended up abandoning the search.
I am a user of CMake for one simple reason: it was the first I could find some years ago that allowed me to spawn different build directories. I'm sure all modern build systems have implemented this idea already, but I stuck with CMake just because I became used to it. Honestly I have found very few sensible advantages over bare bones Makefile. I had to write boilerplate code for my CMakeFiles.txt, even though it was a C++ project without dependencies.
Some time later I decided to try out some different IDEs; I had the good fortune that Qt Creator runs on CMake projects; there is the other reason I stayed on CMake.
My advice to you is to visit each build system's webpage; check out their currently implemented features (not TODOs), comparisons to other systems, IDE support, and the complexity of the code you have to write for them. I am pretty sure you will not find a build system satisfying all of the requirements you have, so you will have to test them thoroughly to see which one works best for you.

C++ Libraries ecosystem using CMake and Ryppl

I am interested in building a cross-platform C++ Library and distributing it in source form. I want the consumers of this library to be able to acquire it, build it and consume it inside their software very easily on whatever platform they are working on and for whatever platform they are targeting. At the same time while building my library, I also want to be able to consume other popular OSS libraries through a similar mechanism.
I see that CMake and Ryppl were created with these intentions in mind and to some extent they do solve some of these problems, especially the build problem. But I don't quite know how exactly to go about achieving the above mentioned goals. Is it OK to settle on CMake as the build solution? How do I solve the library acquisition and distribution problem? Simply host the sources somewhere and let people discover, download and build them? Or is there a better way?
At the time of writing there is no accepted solution that handles everything you want. CMake gives you cross-platform builds and git (with submodules) gives you a way to manage source-level dependencies if all other projects are using CMake. But, in practice, many common dependencies you project wil need don't use CMake, or even Git.
Ryppl is meant to solve this, but progress is slow because the challenge is such a hard one.
Arguably, the most successful solution so far is to write header-only libraries. This is common practice and means your users just include your files, and their build system of choice takes care of everthing. There are no binary dependencies to manage.
TheHouse's answer is still essentially true. Also there don't seem to have been any updates to ryppl itself for a while (3 years) and the ryppl.org domain has expired.
There are some new projects aiming to solve the packaging issue.
Both build2 and wrap from mesonbuild have that goal in mind.
A proposal was made recently to add packages to the c++ standard which may open up the debate (reddit discussion here).
Wrap looks promising as meson's author has learned from cmake.
There is a good video when its author discussing this here.
build2 seems more oblivious (and therefore condemned to reinvent). However both suffer from trying to solve the external project dependencies issue simultaneously with providing a complete build system.
conan.io is another recent attempt which doesn't try to provide the build system as well. Time will tell if any of these gain any traction.
The accepted standard for packaging C and C++ projects on Unix was always a source tarball + a configure script (autotools) + make.
cmake is now beginning to replace autotools as your first choice.
It is able create RPMs and tarballs for distribution purposes.
Its also worth considering the package managers built into the various flavours of Linux. The easiest to build and install projects are those where most of the dependencies can be pulled in via yum or apt. This won't help you on windows of course. While there is a high barrier to entry getting your own projects added to the main Linux repositories (e.g. RedHat, Debian) there is nothing to stop you adding your maintaining your own satellite repo.
The difference between that and just hosting your project on github or similar is you can provide pre-built binaries for a number of popular systems.
You might also consider that configure times checks (e.g. from cmake findLibrary()) and your own documentation will tell people what needs to be installed as a prerequisite and providing you don't make it too onerous that might be enough.