Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 months ago.
Improve this question
The project structure we used to use was that code + prebuild external dependencies were source controlled in SVN. This was cumbersome because the external libraries were large and didn't need to be source controlled since they were prebuilt binaries.
Now we have source in git and the prebuilt binaries are in the cloud. The dev has to download this lib folder from the cloud after cloning the repo. Problem here is that if you make changes to lib then things will not build correctly until the developer goes and redownloads this lib folder.
Our projects are generally developed for Windows (MSVC) but we just added Linux (GCC+Docker) compatibility and likely in the future Linux will be the main version. So now our libraries each have a Windows and a Linux build folder. Our dev environments are Windows + VSCode/WSL2/Docker for Linux.
What is the best, common practice here for handling external dependencies. I can think of 2 ways.
Version the lib folder in the cloud and check that during building. If Developer A adds/changes this and updates the CMakeLists file then when Developer B updates his git repo and tries to build then CMake can see that the version of the libs folder he has is out of date and will be told to go update that. This is little effort and changes almost nothing of our process. Cons are that Developer A has to remember to update the version in both the cloud and in the cmake check.
Build all external libraries locally. Use git submodules and have cmake build all dependencies while building the main project. I assume there's a way to cache it so it doesn't rebuild constantly (some of these libraries are large and take a long time to build). More work, but less maintenance and needing of extra developer steps. Also probably easier to link and include against.
Problem here is that if you make changes to lib then things will not build correctly until the developer goes and redownloads this lib folder.
clear indication that the exact version you want to use is part of what you should track alongside your source code.
I assume there's a way to cache it so it doesn't rebuild constantly (some of these libraries are large and take a long time to build)
As long as files don't change, nothing needs to be rebuilt.
So, yeah, if your project depends on external libraries in specific, git submodules do sound kind of attractive.
Also note that other build systems (like meson) have a neater understanding of in-tree dependency projects, can check your system for an installed version of the dependency, and if not there in appropriate version, download, and if necessary, build themselves.
So, the second option is probably the easiest to maintain solution, as you said.
I, however, come from an free and open source background, and my users and their platforms are diverse, and Linux distros have strict guidelines about not packaging N copies of the same dependency. So, that would make it harder to upstream such packages to debian, Ubuntu, Fedora, Arch… . So, for me the situation is this: if there's a library that we want to use in a project, we define very clearly what the oldest version of that library is that would work. Within a release cycle, we cannot bump the required version.
So, say, we've released 2.0.0 of some software. The CMake files define which version of a library we support. "Releasing" software means that we guarantee to devs as well as to users that the next bugfix/feature extension versions in our 2.a.b series still build on the same systems – and that includes the same libraries that might be installed there. So, if 2.0.0 built on your computer, so will 2.0.1 and 2.9.0. Development that requires a new version of an external dependency can only happen on a git branch that's not meant for further 2.a.b releases, but targetting an eventual 3.0.0. When picking minimum dependency versions for that 3.0.0 release, we look what is commonly available on the operating systems we support. For example, if my timeline was that 3.0.0 be released within 2022 or 2023, that version would be the one available in Ubuntu 22.04LTS (because that will be an important system for our users for a long time, and also relatively conservative), also looking at the debian version most likely to be the current unstable (or stable, depending on what your target audience is), the RHEL version, the next Fedora, and what is currently available in our condaforge and macports repos.
Everything not available in tolerable versions through these standard packaging channels needs to be built locally anyway. Turns out that if you're not crazily progressive and don't try to support 5 year old Linuxes, the number of projects that you need to build locally is quite small.
On windows, you're basically handicapped by Microsoft's inability to provide a really sensible way of downloading packages of binary shared libraries that are actually shared between different application software. So, on Windows systems, you're down to either doing all your builds locally, or using a third-party way of distributing platform-dependent packages, like Conan.
No matter how you do it, you'd let your build fail as early as possible, with a clear indication that the library version found is not sufficiently new. CMake makes this easy; its find_package command takes a minimum version as argument.
Related
I am making a modified C++ compiler and I have it built and tested locally. However, I would like to be able to package my build for Windows, Linux (Debian), and Mac OSX.
All of the instructions I can find online deal with building gcc but have no regard for making something distributable (or perhaps I am missing something?). I know for Windows I will need to bundle MinGW somehow, but that only further confuses me - and I have no idea how well Mac works with GCC these days..
Can anyone layout a set of discrete high-level steps I could try on each system so I can allow people to install my modified compiler easily?
First make sure your project installs well including executables, headers, runtime dependencies. If you're using something like cmake, this is a matter of installing things to CMAKE_INSTALL_PREFIX while possibly appending GnuInstallDirs. If you are using make, then you need to ensure that make install --prefix=... works well.
From here, you can target each platform independently. Treat the packaging independently from your project. Like Chipster mentioned, making rpm files isn't so tough. deb files for Debian-based OSs, tar.xz files for Arch-based OSs are similar. The rules for creating these packages can use your install rules to create the package. You mentioned mingw. If you're targeting an msys distribution of mingw for Windows deployment, then the Arch-based packaging of pacman will work on msys as well. You can slowly work on supporting one-platform at a time with almost no changes to your actual project.
Typically in the open-source world, people will release a tar.gz file supporting ./configure && make && make install or similar. Then someone associated with the platform (like a Debian-developer) will find your project, make some packaging rules for it, and release it into their distribution. That means your project can be totally agnostic to where it's being release. It also means you don't really need to worry about MacOS yet, you can wait until you have someone who wants it there, or some hardware to test it on.
If you really want to be in control of how things are packaged for each platform from inside of your project, and you are already using cmake, cpack is a great tool which helps out. After writing cpack rules for your project, you can simply type cpack to generate many types of deployable archives. You won't get the resulting *.deb file into Debian or Ubuntu official archives, but at least people can using those formats can install your package.
Also, consider releasing one package with the runtime libraries, and one with the development content (headers, compiler, static libraries). This way, if someone uses your compiler, they can re-distribute the runtime libraries which is probably going to be a much simpler install.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm a Python developer with some background in another language such as Ruby.
In both language, dependency is managed by package manager automatically, such as pip or gem. Anyone could install such dependency by calling pip install -r requirements.txt, and it will install the necessary dependency via Python Package Index. Although, there has been an option to build the dependency manually from the source and install into the project, it is not a recommended process, and I have not done it.
I notice that C++ unfortunately have different nature in how dependency is being resolved for some reason. (e.g. different compiler flavor, compiler parameters, platforms, etc...)
At the moment, I am learning C++ using VS2015. and I have been stumbled again and again upon these library dependencies matter. With VS2015, there is a dependency package manager like python, and it is NuGet. However, not every library is available in NuGet, in fact, there are a lot of library developed independent from its IDE.
First I'm trying to use Boost. There is a manual on how to build the project, but I'm not sure what I need. Do I need to build from source? or Perhaps I just need a library that has been readily available?
Same reason for another library that I found. (e.g. QT, yaml-cpp, googletest, etc..)They only have a document how to build, instead of how to install as dependency.
And Ultimately, I will need to use lots of 3rd party library to be more productive. So, here's some of my questions that are very related.
How do C++ developer normally include 3rd party library into their project (the flow of installation 3rd party library)?
Do I have to build from source everytime I want to include? or perhaps you just need the header file which you could just copy and paste into your project directory?
I'm working in team (git), do each of my team need to build the dependency manually? Can it be automate such that the process of including new library is transparent for everyone?
Or perhaps, I don't really understand what specific question do I need
to ask. But why it is so painful to reuse library in C++?
Do I miss some fundamental understanding of C++ environement?
I'm not sure how much relevance it is, but CMake as a build tool that most library uses to build their project. Do I really need to build these library project?
More Questions:
After building some libraries, some of them generate static library .lib or dynamic library .dll to be included into the project. So is it correct to copy these generated library in our project? Should this be committed into the source version control? Some libraries are very large, and we don't want to maintain it. Yet we need the entire team to get the library transparently.
I understand you situation quite well. You cannot see the forest because too many trees are standing in your way...
Let me get one thing clear before I start to address your specific questions:
Generally speaking, dependencies in C++ are not more complicated than in Python.
The command pip install -r requirements.txt will establish an internet connection and download the necessary libraries and files from a repository server to fulfill the requirements. Under the Linux operating system (Ubuntu) the command: sudo apt-get install libboost-all-dev installs all required dependencies for boost. This is possible because there is a whole environment with servers that hold source-code as well as libraries and binaries that work together with the client programs (apt-get) that use it. This is exactly the same thing that the authors of pip have done for microsoft windows. microsoft themselves have never done this at the operating system level. They always left that to the programmer. NuGet is microsofts attempt to make-up for past mistakes.
Having this out of the way, let me address your questions:
It depends on the size of the 3rd party library. Small libraries like pugixml can be included as source in the source tree of your project. Bigger libraries like boost are better included as binary object code (library objects). Not all libraries do have binaries available to download (boost has), so you might be required to build from source. Bear in mind that all binaries are required to be built with exact the same compiler that you use in your project. The general steps to include it in your VS-Project:
Get the distribution files (either build from source or download and install binaries)
Add include paths to your Project:
Project > "projectname" properties > Configuration Properties > C/C++ > General > Additional Include Directories
Add paths to libraries:
Project > "projectname" properties > Configuration Properties > Linker > Input > Additional Dependencies.
No. You normally just use the header file. But it's better to add the path of the library into your project instead of copying the header file, because some projects (boost) have a huge hierarchy of header files.
It is a good idea that each member of your team has the same development environment with the same set of libraries installed. There are tools for this task: Chocolatey builds on top of NuGet and is therefore windows-affine. Vagrant deals with virtual boxes ands thus offers cross-platform development environments.
But more important is a decent source-control-management system. If you don't already use one - start using one Today!. This is the main collaboration-tool. It can really save your neck if you loose a developer machine.
There is another dependency problem: We've only addressed the development dependencies above. There is the problem of deployment dependencies:
your customers will need the libraries (*.dll files) that you have used for the development. You will need to package them as well into your deployment package (Installer). This is another issue which is probably already answered on SO.
Qt: if you start using Qt, I'd suggest that you start using their development environment Qt-Creator. This will automatically handle all dependencies. It will automatically detect the Visual Studio Compiler that you have installed, and use it. The IDE is quite close to Visual Studio.
CMake: No, it is not always required to use CMake to build a library project, some also include Makefiles. Others use CMake to produce Makefiles. "Follow the instructions" is the best advice I can give here.
Update 2015-10-24: paragraph point three reworked
How do C++ developer normally include 3rd party library into their
project (the flow of installation 3rd party library)?
It depends... There are a lot of ways, how to redistribute C++ libraries.
Do I have to build from source everytime I want to include? or perhaps
you just need the header file which you could just copy and paste into
your project directory?
For now, most C++ libraries contains two parts: binaries + header files. But often, there are a lot of problems, if compiler version of library is different with your compiler.
I'm working in team, do each of my team need to build the dependency
manually? Can it be automate such that the process of including new
library is transparent for everyone?
It depents on your team guidelines. You can choose what you want.
Or perhaps, I don't really understand what specific question do I need
to ask. But why it is so painful to reuse library in C++?
Because of some legacy of C. And because C++ is low-level language in compare with python/java/c#. C++ is supported by a lot of different platforms, including embedded. And ofter, it is not possible to install complex runtime on this platforms. So there are no mechanism to transparently link a "modules" in runtime.
Hopefully, there will be a normal support of modules in C++17 standard. And Microsoft will provide a technology preview of modules in C++ in MSVC 2015 update 1.
Do I miss some fundamental understanding of C++ environement?
Yes, I propose you to read about compiling and linking in C/C++. This two things are often come together, but they are different.
First, that you should mind: code in C/C++ is splitted in two parts: declaration (.h files) and implementation (.cpp files). .CPP files are compiled into binaries. .H files just declares an interfaces.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm rather new to Linux filesystems, so could you help me, please? I have to write a sample C++ project (tests) using Ubuntu.
Could you clarify me with a file/folder structure from a developer's perspective? Here are some questions I would like answered:
Where is typical location of the projects (sources, object files, etc.)?
Where is a typical place for dev environment (Eclipse, QT Creator, etc.)?
Where is a typical place for the libraries? Are there different places for binaries and for header-only libraries?
Where is a typical place for a various tools for the development (code analyser, git client, etc.)?
Answers and links would be appreciated. Thank you.
Where is typical location of the projects (sources, object files, etc.)?
I store my projects in $HOME/dev but its entirely up to you.
Where is a typical place for dev environment (Eclipse, QT Creator, etc.)?
I use eclipse and set its workspace pointing to $HOME/dev.
Where is a typical place for the libraries? Are there different places for binaries and for header-only libraries?
Typically libraries are installed to /usr/lib and headers are installed to /usr/include.
Where is a typical place for a various tools for the development (code analyser, git client, etc.)?
Typically these are installed into /usr/bin. I also put tools in $HOME/bin, especially if I made them.
But its more complicated than that. What if you want to develop/test an application with a version of a library that is different to the one that comes with your Linux distribution? Sometimes I will install different library versions in my $HOME/dev folder and compile against those rather than the system version.
Also I run Fedora 21 that comes with GCC 4.9.2, however I have installed GCC 5.1.0 into /opt/gcc-5.1.0 and use that for some things.
A typical project structure for me would be something like:
$HOME/
/dev/
/my-prog/
/src/
/include/
/my-prog.h
/my-prog.cpp
/build-debug/
/src/
/my-prog
/build-release/
/src/
/my-prog
/Makefile
The implementation varies somewhat between Linux distributions, but all seek to implement the Linux Filesystem Hierarchy Standard for the most part. Some distributions more so than others. The latest version is 2.3 and can be viewed via the links in the URL above or directly via FHS 2.3 PDF
Take time to view the document and compare it with the implementation Ubuntu uses. It will provide you with a good background to where and why files are located where they are in Linux, and also give you a quick lesson on how this is more of a goal for distributions than a hard requirement, as distributions are pretty much free to put things anywhere they want.
As for your project source tree layout, a common setup looks like this:
src/ this is were your .cpp files go
include/ for header that should get installed when you are writing a library
data/ is where graphics, icons, sounds, etc. go
external/ for git submodules or external libraries your project requires and that are obscure enough that you don't want users to manually install them
a build file in the top level directory (autoconf/automake was used historically, but these days CMake seems to be more popular)
a README or README.md explaining what the project does and which dependency it needs
Naming and organisation can of course vary between projects, there is no real standard, some use Source/ instead of src/. Some might have the headers within the src/ directory and so on. It's really up to your projects needs. If a library is commonly shipped with most distributions there is no need to include it in the source tree. Same with most programming tools, git, gcc and Co. are provided by the Linux distribution and you don't have to worry about where they go.
If you are writing a library you might also want to looking into pkg-config and the .so naming rules.
As for installation directories, the FHS explains that in detail, short summary:
/usr/bin for executables
/usr/lib for public libraries
/usr/lib/${APPNAME} for private libraries
/usr/include for public header files
/usr/share/${APPNAME} for data files
/opt/${APPNAME} is used by some commercial products instead of spreading the data over the hierachy
/usr itself is reserved for distribution installed packages, manually compiled stuff should go to /usr/local instead. Most build systems will have a --prefix=PREFIX option to allow the user to change that.
If you need to compile a library yourself, I would generally recommend to avoid installing it into /usr/local, as uninstalling software from there is hard and it also makes it impossible to keep different versions of the same software around at the same time.
Instead install software it into a PREFIX of it's own, something like ~/run/somelibrary-0.0.1. Installing software in such a way has however the disadvantage that tools won't find it. You can fix that by setting a few environment variables, I use a simple bash function for that purpose:
function activateprefix {
PREFIX="$1"; \
export PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig/"; \
export LD_LIBRARY_PATH="${PREFIX}/lib/"; \
export LD_RUN_PATH="${PREFIX}/lib/"; \
export LIBRARY_PATH="${PREFIX}/lib/"; \
export CPLUS_INCLUDE_PATH="${PREFIX}/include/"; \
export C_INCLUDE_PATH="${PREFIX}/include/"; \
}
I'm responsible for developing a set of C++ libraries and programs. Currently building on Linux and MacOS, but Windows support is also a requirement. We will need to support VS2010 and VS2012, and in the future will also include VS2013 and maybe also MinGW. We're using cmake for building, so our code should build on all the platforms without issues; my problem is how to manage all the dependencies on Windows in order to be able to build in the first place, and keeping it up-to-date over time. At the moment, we have one virtual machine per visual studio version as a jenkins slave, so parallel builds of all the variants is fairly easy, but managing it is not.
The problem is the number of variants this requires building. If we consider only VS2010 and VS2012, with debug/release and i386/x64 builds, that's already 8 copies of each library; 16 if we include the other compilers. We will need all the libraries our code depends on, which will include at a minimum boost, qt, xerces+xalan, zlib, icu, libpng/tiff/jpeg, hdf5 and more, plus python, and all their dependencies. And as new upstream releases are made, we'll need to keep the entire collection up-to-date and consistent for all the build/arch/compiler variants.
I don't want to do this by hand, since this really needs automating. However, I'm unaware of any good solution for doing this on Windows. The Windows building guides I've seen for other projects often involve hand-building all the dependencies, and only build for a single variant. On Linux, it's already packaged, you don't need separate debug builds, and the arch variants can be catered for with chroots; on MacOS there's homebrew, macports etc., and it's also fairly simple to automate stuff there as well. Is there any equivalent for Windows? I've looked at stuff like chocolatey, but it's entirely unsuited to handling libraries, and is pretty poor as a package manager.
This seems like it should be a common problem for anyone doing C++ development on Windows? Are there any common solutions, tools or methodologies for managing a complex set of libraries and tools for development? How do other developers manage this?
NB. Just for the record, we are not using the visual studio application; we're doing all builds non-interactively via scripts driving the compilers directly with cmake and/or msbuild.
Many thanks,
Roger
I worked on large windows C++ project that delivers X86 Release, x86 Debug, x64 Release and x64 debug. Very similarly I used build system that does parallel builds for all target platforms using custom script.
We manage all third party dependency libraries in organized folders.
For example x86\release\Zlib.dll x86\Debug\zlib.dll x64\release\zlib.dll x64\zlib.dll
Custom script is made to pick all these libraries and project source code from configuration management tool. This allows to automatically build the relevant target binaries as needed.
any third party libraries change is updated in configuration management tool and then later picked up by the script for the next build.
For your question on VS2010 and 2012 support I don't understand importance. Is not one version of VS enough to support for the project?
you may take a look at Link, and their build system https://github.com/gisinternals/buildsystem
It's basicly a set of batch and make files calling each others. You still need keep track of lib update manually.
I have installed the latest ncurses library which my project is using. Now, I want to check in the ncurses static libraries into svn so that I can checkout the project on a different machine and compile it without having to install ncurses on the system again.
So the question is what is the difference between libncurses.a, libncurses++.a and libncurses_g.a files? And do I need all of them for my C++ project?
Thanks!
libncurses.a - This is the C compatible library.
libncurses++.a - This is the C++ compatible library.
libncurses_g.a - This is the debug library.
libncurses_p.a - This is the profiling library.
If you want to find out if you can get by without using libncurses.a, you can rename the library and run a build of your application.
My answer comes a little late [ :-) ] since you posted your question more than 4 years ago. But:
Archiving the pre-compiled library in your SVN means that your built application may fail if the target machine differs under some critical aspect.
Yes, you can safely run the application on other machines which are configured entirely in the same way (e.g., on a fully homogeneous computation cluster). However, if the machines differ (e.g., because one machine had a system upgrade and the other not), it may stop working. This is not very likely, so the risk may be acceptable for what you'd like to do.
I would suggest another solution: Commit a recent, stable version of the libncurses sources (tarball) to your SVN repo and add a little script (or make target) that runs the libncurses build and installs the built library to some project directory (not the system directory but next to your applciation build directories, without committing to SVN). This build step only needs to be repeated if the libary shall be upgraded or if you would like to build/run on another machine.
This does not apply to the ncurses library in special but to any library.
Depending on your project target, consider further reading about
package management
cross compile