I am working on a C++ library. Ultimately, I would like to make it publicly available for multiple platforms (Linux and Windows at least), along with some examples and Python bindings. Work is progressing nicely, but at the moment the project is quite messy, built solely in and for Visual C++ and not multi-platform at all.
Therefore, I feel a cleanup is in order. The first thing I'd like to improve is the project's directory structure. I'd like to create a structure that is suitable for the Automake tools to allow easy compilation on multiple platforms, but I've never used these before. Since I'll still be doing (most of the) coding in Visual Studio, I'll need somewhere to keep my Visual Studio project and solution files as well.
I tried to google for terms like "C++ library directory structure", but nothing useful seems to come up. I found some very basic guidelines, but no crystal clear solutions.
While looking at some open source libraries, I came up with the following:
\mylib
\mylib <source files, read somewhere to avoid 'src' directory>
\include? or just mix .cpp and .h
\bin <compiled examples, where to put the sources?>
\python <Python bindings stuff>
\lib <compiled library>
\projects <VC++ project files, .sln goes in project root?>
\include?
README
AUTHORS
...
I have no/little previous experience with multi-platform development/open source projects and am quite amazed that I cannot find any good guidelines on how to structure such a project.
How should one generally structure such a library project? What ca be recommended to read? Are there some good examples?
One thing that's very common among Unix libraries is that they are organized such that:
./ Makefile and configure scripts.
./src General sources
./include Header files that expose the public interface and are to be installed
./lib Library build directory
./bin Tools build directory
./tools Tools sources
./test Test suites that should be run during a `make test`
It somewhat reflects the traditional Unix filesystem under /usr where:
/usr/src Sometimes contains sources for installed programs
/usr/include Default include directory
/usr/lib Standard library install path
/usr/share/projectname Contains files specific to the project.
Of course, these may end up in /usr/local (which is the default install prefix for GNU autoconf), and they may not adhere to this structure at all.
There's no hard-and-fast rule. I personally don't organize things this way. (I avoid using a ./src/ directory at all except for the largest projects, for example. I also don't use autotools, preferring instead CMake.)
My suggestion to you is that you should choose a directory layout that makes sense for you (and your team). Do whatever is most sensible for your chosen development environment, build tools, and source control.
There is this awesome convention that I recently came across that might be helpful: The Pitchfork Layout (also on GitHub).
To sum up, subsection 1.3 states that:
PFL prescribes several directories that should appear at the root of the project tree. Not all of the directories are required, but they have an assigned purpose, and no other directory in the filesystem may assume the role of one of these directories. That is, these directories must be the ones used if their purpose is required.
Other directories should not appear at the root.
build/: A special directory that should not be considered part of the source of the project. Used for storing ephemeral build results. must not be checked into source control. If using source control, must be ignored using source control ignore-lists.
src/: Main compilable source location. Must be present for projects with compiled components that do not use submodules.
In the presence of include/, also contains private headers.
include/: Directory for public headers. May be present. May be omitted for projects that do not distinguish between private/public headers. May be omitted for projects that use submodules.
tests/: Directory for tests.
examples/: Directory for samples and examples.
external/: Directory for packages/projects to be used by the project, but not edited as part of the project.
extras/: Directory containing extra/optional submodules for the project.
data/: Directory containing non-source code aspects of the project. This might include graphics and markup files.
tools/: Directory containing development utilities, such as build and refactoring scripts
docs/: Directory for project documentation.
libs/: Directory for main project submodules.
Additionally, I think the extras/ directory is where your Python bindings should go.
I don't think there's actually any good guidelines for this. Most of it is just personal preference. Certain IDE's will determine a basic structure for you, though. Visual Studio, for example, will create a separate bin folder which is divided in a Debug and Release subfolders. In VS, this makes sense when you're compiling your code using different targets. (Debug mode, Release mode.)
As greyfade says, use a layout that makes sense to you. If someone else doesn't like it, they will just have to restructure it themselves. Fortunately, most users will be happy with the structure you've chosen. (Unless it's real messy.)
I find wxWidgets library (open source) to be a good example. They support many different platforms (Win32, Mac OS X, Linux, FreeBSD, Solaris, WinCE...) and compilers (MSVC, GCC, CodeWarrior, Watcom, etc.). You can see the tree layout here:
https://svn.wxwidgets.org/svn/wx/wxWidgets/trunk/
I can realy recommend you using CMake... it's for cross platform development and it's much more flexible that automake, use CMake and you will be able to write cross platform code with your own direcory structure on all systems.
Related
I am trying to use gnuplot++, but this is really a more general question about downloaded source code. I have downloaded the gnuplot++ source code and it consists of multiple .h and .cc files. I would like to use it in othercopy projects in the future so I am reluctant to add all the files into my project directory.
From what I understand gcc will look in /usr/local/include for header files, so I have put the code there for now. But what is the best way to compile and link the code?
Should I use the makefile to include the directory of the source code?
Should I keep it somewhere easy to find like /usr/local/include?
How do I know the best way to compile the code in gnuplot++?
Typically, if the project itself doesn't come with install instructions, I usually add it somewhere "public", e.g. /usr/local/project/{lib,include,src,...} where "project" in this case would be gnuplot++.
In this case, there doesn't appear to be any support for building this into a library, which makes it a little more awkward, as you need the sources included in your project itself. I'd still keep those sources separate, but you may prefer to just put them into a separate directory within the project [or spend an hour or three making a library of it].
For general practice, yes, keep the source for gnuplot++ (or any other similar 3rd-party project) separate from your own application source code. This makes it much easier to manage updates to the 3rd party projects, etc.
Yes, I would use the makefile for your application to also include the path to the headers for gnuplot++ and I would not copy those files directly into /usr/local/include. Instead, I would consider a couple options: do nothing and point your include path in your makefile to the gnuplot++ directory, or put symbolic links in /usr/local/include to point to the gnuplot++ files.
As for the best way to compile gnuplot++, I would have to look at gnuplot++ myself and see what it has to say, perhaps in a README file or similar.
In general, when using third-party libraries, you build and install those libraries according to the installation description that comes with the downloaded source.
If there is no installation guideline, it is typically a set of steps like
./configure
make
make install
Then it is the responsibility of the library to ensure the relevant headers and library files are easily locatable for use in your project.
gnuplot++ is an exception here, because it does not seem to come with its own build structure.
The best advice in cases such as this is to put the source from gnuplot++ in a directory within your project (possibly parallel to your own sources) and include the files in your own build setup.
Imagine an overall project with several components:
basic
io
web
app-a
app-b
app-c
Now, let's say web depends on io which depends on basic, and all those things are in one repo and have a CMakeLists.txt to build them as shared libraries.
How should I set things up so that I can build the three apps, if each of them is optional and may not be present at build time?
One idea is to have an empty "apps" directory in the main repo and we can clone whichever app repos we want into that. Our main CMakeLists.txt file can use GLOB to find all the app directories and build them (not knowing in advance how many there will be). Issues with this approach include:
Apparently CMake doesn't re-glob when you just say make, so if you add a new app you must run cmake again.
It imposes a specific structure on the person doing the build.
It's not obvious how one could make two clones of a single app and build them both separately against the same library build.
The general concept is like a traditional recursive CMake project, but where the lower-level modules don't necessarily know in advance which higher-level ones will be using them. Yet, I don't want to require the user to install the lower-level libraries in a fixed location (e.g. /usr/local/lib). I do however want a single invocation of make to notice changed dependencies across the entire project, so that if I'm building an app but have changed one of the low-level libraries, everything will recompile appropriately.
My first thought was to use the CMake import/export target feature.
Have a CMakeLists.txt for basic, io and web and one CMakeLists.txt that references those. You could then use the CMake export feature to export those targets and the application projects could then import the CMake targets.
When you build the library project first the application projects should be able to find the compiled libraries automatically (without the libraries having to be installed to /usr/local/lib) otherwise one can always set up the proper CMake variable to indicate the correct directory.
When doing it this way a make in the application project won't do a make in the library project, you would have to take care of this yourself.
Have multiple CMakeLists.txt.
Many open-source projects take this appraoch (LibOpenJPEG, LibPNG, poppler &etc). Take a look at their CMakeLists.txt to find out how they've done this.
Basically allowing you to just toggle features as required.
I see two additional approaches. One is to simply have basic, io, and web be submodules of each app. Yes, there is duplication of code and wasted disk space, but it is very simple to implement and guarantees that different compiler settings for each app will not interfere with each other across the shared libraries. I suppose this makes the libraries not be shared anymore, but maybe that doesn't need to be a big deal in 2011. RAM and disk have gotten cheaper, but engineering time has not, and sharing of source is arguably more portable than sharing of binaries.
Another approach is to have the layout specified in the question, and have CMakeLists.txt files in each subdirectory. The CMakeLists.txt files in basic, io, and web generate standalone shared libraries. The CMakeLists.txt files in each app directory pull in each shared library with the add_subdirectory() command. You could then pull down all the library directories and whichever app(s) you wanted and initiate the build from within each app directory.
You can use ADD_SUBDIRECTORY for this!
https://cmake.org/cmake/help/v3.11/command/add_subdirectory.html
I ended up doing what I outlined in my question, which is to check in an empty directory (containing a .gitignore file which ignores everything) and tell CMake to GLOB any directories (which are put in there by the user). Then I can just say cmake myrootdir and it does find all the various components. This works more or less OK. It does have some side drawbacks though, such as that some third-party tools like BuildBot expect a more traditional project structure which makes integrating other tools with this sort of arrangement a little more work.
The CMake BASIS tool provides utilities where you can create independent modules of a project and selectively enable and disable them using the ccmake command.
Full disclosure: I'm a developer for the project.
Is there some way to externalize the paths of libraries that are used in the compilation process on Visual Studio 2008? Like, *.properties files?
My goal is to define "variables" referencing locations to headers files and libraries, like *.properties files are used in the Ant build system for Java.
I think you're looking for .vsprops files. They're comparable to the *.properties files.
Environment Variables?
All the $(xyz) replacements allowed in the propertier are, and you are allowed to "bring your own".
They are normally inherited from the parent process, so you can set them
for the machine / user in the system settings (usually inherited via explorer)
in a batch file that sets them before running devenv.exe
in an addin like SolutionBuildEnvironment to read them from a project file
I don't know how Ant works, but for your static libraries and headers you can edit the .vcproj file. These are XML files in fact. Libraries go in the VCLinkerTool tool tag, in AdditionalDependencies
<Tool
Name="VCLinkerTool"
AdditionalOptions=" /subsystem:windowsce,5.01"
AdditionalDependencies="iphlpapi.lib commctrl.lib coredll.lib"
/>
Additional header paths are defined in the VCCLCompilerTool tool tag, in AdditionalIncludeDirectories
<Tool
Name="VCCLCompilerTool"
Optimization="0"
AdditionalIncludeDirectories="dev\mydir"
PreprocessorDefinitions="WIN32;_DEBUG;_CONSOLE"
/>
Be careful, there is one such section for each build configuration.
Is this what you are looking for?
Edit : the .vsprops suggested by MSalters are more powerful; you can define additional dependencies and libraries in them an make your projects inherit these properties. Well I learned something useful today!
If you're referring to influencing the location of #includes, Project properties|Configuration Properties|C/C++/Additional Include Directories is the ticket. There is also project properties|Common Properties|Additional reference search paths.
If your question is how do I parameterize stuff in a VCProj file like I would in Ant, the answer is that in VS2010 VC projects are[/can?] be MSBuild-based whereas VS2008 vcproj files are a proprietary XML based format [but as the other answers say, they have an analogous properties capability].
In the absence of more info, I'm pretty sure the standard approach for what you're doing is to add your search paths a la the first or second paragraph.
You can use a build system like CMake. You give CMake a high-level description of your project, and it spits out the necessary files to get your project to build correctly via another tool (e.g. Visual Studio's IDE, or a Unix-style makefile).
Paths: You can use CMake's INCLUDE_DIRECTORIES() and LINK_DIRECTORIES() commands in the CMakeList.txt configuration file to specify these paths. CMake has variables which describe both aspects of your environment (many of which can be autodiscovered, e.g. CMAKE_C_COMPILER which is the command to run your C compiler) plus any options you wish to allow the user to specify directly. All variables are stored in a separate plain text configuration file, CMakeCache.txt, that can be edited in a text editor or using a special GUI configuration tool.
CMake has many other features, like the ability to autodiscover the locations of many useful libraries, and to produce customised source/header files from "template" files containing CMake directives using the CONFIGURE_FILE() command.
Advantages:
Highly portable across common environments (e.g. it can produce solution files for several versions of MS Visual C++, as well as makefiles for Unix (e.g. Linux) systems).
Used by several large multiplatform projects (e.g. KDE)
Very simple to set up simple projects
I've found the dependency checking system to be rock-solid -- e.g. it knows to rebuild if you change compiler options (unlike naive use of make for example)
Disadvantages:
Ugly, primitive syntax
Documentation quality varies (e.g. it's sometimes hard to tell exactly what properties affect any given object)
Some time investment involved
I'm in the middle of setting up an build environment for a c++ game project. Our main requirement is the ability to build not just our game code, but also its dependencies (Ogre3D, Cegui, boost, etc.). Furthermore we would like to be able build on Linux as well as on Windows as our development team consists of members using different operating systems.
Ogre3D uses CMake as its build tool. This is why we based our project on CMake too so far. We can compile perfectly fine once all dependencies are set up manually on each team members system as CMake is able to find the libraries.
The Question is if there is an feasible way to get the dependencies set up automatically. As a Java developer I know of Maven, but what tools do exist in the world of c++?
Update: Thanks for the nice answers and links. Over the next few days I will be trying out some of the tools to see what meets our requirements, starting with CMake. I've indeed had my share with autotools so far and as much as I like the documentation (the autobook is a very good read), I fear autotools are not meant to be used on Windows natively.
Some of you suggested to let some IDE handle the dependency management. We consist of individuals using all possible technologies to code from pure Vim to fully blown Eclipse CDT or Visual Studio. This is where CMake allows use some flexibility with its ability to generate native project files.
In the latest CMake 2.8 version there is the new ExternalProject module.
This allows to download/checkout code, configure and build it as part of your main build tree.
It should also allow to set dependencies.
At my work (medical image processing group) we use CMake to build all our own libraries and applications. We have an in-house tool to track all the dependencies between projects (defined in a XML database). Most of the third party libraries (like Boost, Qt, VTK, ITK etc..) are build once for each system we support (MSWin32, MSWin64, Linux32 etc..) and are commited as zip-files in the version control system. CMake will then extract and configure the correct zip file depending on which system the developer is working on.
I have been using GNU Autotools (Autoconf, Automake, Libtool) for the past couple of months in several projects that I have been involved in and I think it works beautifully. Truth be told it does take a little bit to get used to the syntax, but I have used it successfully on a project that requires the distribution of python scripts, C libraries, and a C++ application. I'll give you some links that helped me out when I first asked a similar question on here.
The GNU Autotools Page provides the best documentation on the system as a whole but it is quite verbose.
Wikipedia has a page which explains how everything works. Autoconf configures the project based upon the platform that you are about to compile on, Automake builds the Makefiles for your project, and Libtool handles libraries.
A Makefile.am example and a configure.ac example should help you get started.
Some more links:
http://www.lrde.epita.fr/~adl/autotools.html
http://www.developingprogrammers.com/index.php/2006/01/05/autotools-tutorial/
http://sources.redhat.com/autobook/
One thing that I am not certain on is any type of Windows wrapper for GNU Autotools. I know you are able to use it inside of Cygwin, but as for actually distributing files and dependencies on Windows platforms you are probably better off using a Windows MSI installer (or something that can package your project inside of Visual Studio).
If you want to distribute dependencies you can set them up under a different subdirectory, for example, libzip, with a specific Makefile.am entry which will build that library. When you perform a make install the library will be installed to the lib folder that the configure script determined it should use.
Good luck!
There are several interesting make replacements that automatically track implicit dependencies (from header files), are cross-platform and can cope with generated files (e.g. shader definitions). Two examples I used to work with are SCons and Jam/BJam.
I don't know of a cross-platform way of getting *make to automatically track dependencies.
The best you can do is use some script that scans source files (or has C++ compiler do that) and finds #includes (conditional compilation makes this tricky) and generates part of makefile.
But you'd need to call this script whenever something might have changed.
The Question is if there is an feasible way to get the dependencies set up automatically.
What do you mean set up?
As you said, CMake will compile everything once the dependencies are on the machines. Are you just looking for a way to package up the dependency source? Once all the source is there, CMake and a build tool (gcc, nmake, MSVS, etc.) is all you need.
Edit: Side note, CMake has the file command which can be used to download files if they are needed: file(DOWNLOAD url file [TIMEOUT timeout] [STATUS status] [LOG log])
Edit 2: CPack is another tool by the CMake guys that can be used to package up files and such for distribution on various platforms. It can create NSIS for Windows and .deb or .tgz files for *nix.
At my place of work (we build embedded systems for power protection) we used CMake to solve the problem. Our setup allows cmake to be run from various locations.
/
CMakeLists.txt "install precompiled dependencies and build project"
project/
CMakeLists.txt "build the project managing dependencies of subsystems"
subsystem1/
CMakeLists.txt "build subsystem 1 assume dependecies are already met"
subsystem2/
CMakeLists.txt "build subsystem 2 assume dependecies are already met"
The trick is to make sure that each CMakeLists.txt file can be called in isolation but that the top level file can still build everything correctly. Technically we don't need the sub CMakeLists.txt files but it makes the developers happy. It would be an absolute pain if we all had to edit one monolithic build file at the root of the project.
I did not set up the system (I helped but it is not my baby). The author said that the boost cmake build system had some really good stuff in it, that help him get the whole thing building smoothly.
On many *nix systems, some kind of package manager or build system is used for this. The most common one for source stuff is GNU Autotools, which I've heard is a source of extreme grief. However, with a few scripts and an online depository for your deps you can set up something similar like so:
In your project Makefile, create a target (optionally with subtargets) that covers your dependencies.
Within the target for each dependency, first check to see if the dep source is in the project (on *nix you can use touch for this, but you could be more thorough)
If the dep is not there, you can use curl, etc to download the dep
In all cases, have the dep targets make a recursive make call (make; make install; make clean; etc) to the Makefile (or other configure script/build file) of the dependency. If the dep is already built and installed, make will return fairly promptly.
There are going to be lots of corner cases that will cause this to break though, depending on the installers for each dep (perhaps the installer is interactive?), but this approach should cover the general idea.
Right now I'm working on a tool able to automatically install all dependencies of a C/C++ app with exact version requirement :
compiler
libs
tools (cmake, autotools)
Right now it works, for my app. (Installing UnitTest++, Boost, Wt, sqlite, cmake all in correct order)
The tool, named «C++ Version Manager» (inspired by the excellent ruby version manager), is coded in bash and hosted on github : https://github.com/Offirmo/cvm
Any advices and suggestions are welcomed.
The other day I set up an Ubuntu installation in a VM and went to gather the tools and libraries I figured I would need for programming mostly in C++.
I had a problem though, where to put things such as 3rd party source libraries, etc. From what I can gather, a lot of source distributions assume that a lot of their dependencies are already installed in a certain location and assume that a lot of tools are also installed in particular locations.
To give an example of what I currently do on Windows, is I have a directory where I keep all source code. C:\code. In this directory, I have a directory for all 3rd party libraries, c:\code\thirdparty\libs. This way I can easily set up relative paths for all of the dependencies of any projects I write or come across and wish to compile. The reason I am interested in setting up a linux programming environment is that it seems that both the tool and library dependency problems have been solved efficiently making it easy for example to build OpenSSH from source.
So what I was looking for was a decent convention I can use when I am trying to organize my projects and libraries on linux that is easy to maintain and easy to use.
Short answer: don't do a "heaps of code in local dir" thing.
Long answer: don't do a "heaps of code in local dir" thing, because it will be nightmare to keep up-to-date, and if you decide to distribute your code, it will be nightmare to package it for any decent distribution.
Whenever possible, stick to the libraries shipped in the distribution (ubuntu has 20000+ packets, it ought to have most of what you'll need prepackaged). When there is not package, you caninstall by hand to /usr/local (but see above about upgrades and DONT do that).
Better, use "stow" or "installwatch" (or both) to install to per-library dirs (/usr/local/stow/libA-ver123) and then symlink files from there to /usr/local or /usr/ (stow does the simlinking part). Or just package the lib for your distribution.
For libraries/includes...
/usr/local/lib
/usr/local/include
Where possible code against the system/distro provided libraries. This makes it easiest to ship a product on that distro.
However, if you are building a commercial application, because there are so many flavors of Linux distros that can mean you have to maintain a plethora of different application builds for each distro. Which isn't necessarily a bad thing as it means you can more cleanly integrate with the distro's package management system.
But in the case where you can't do that it should be fairly easy to download the source of each 3rd party dependency you have and integrate the building of that dependency into a static lib that is linked to your executable. That way you know exactly what you're linking against but has the downside of bloating out your executable size. This can also be required if you need a specific library (or version) not provided by the distro.
If you want your code to build on as broad a variety of different Unix systems then you're probably wise looking into GNU autoconf and automake. These help you construct a configure script and makefile for your project so that it will build on practically any Unix system.
Also look into pkg-config which is used quite a bit now on Linux distributions for helping you include and link to the right libraries (for libs that support pkg-config).
If you're using subversion to manage your source there is a "convention" that most subversion repositories use to manage their own code and "vendor" code.
Most svn repositories have a "vendor" tree (that goes along with the trunk, branches & tags trees). That is the top for all 3rd party vendor code. In that directory you have directories for each library you use. Eg:
branches/
tags/
trunk/
vendor/somelib
vendor/anotherlib
Beneath each of these libs is a directory for each library version and a "current" directory for the most up-to-date version in your repository.
vendor/somelib/1.0
vendor/somelib/1.1
vendor/somelib/current
Then your project's tree should be laid out something like this:
trunk/source # all your code in here
trunk/libs # all vendor code in here
The libs directory should be empty but it will have svn:externals meta data associated with it, via:
svn propedit svn:externals trunk/libs
The contents of this property would be something along the lines of (assumes subversion 1.5):
^/vendor/somelib/current somelib
^/vendor/anotherlib/1.0 anotherlib
This means that when you checkout your code subversion also checks out your vendor libraries into your trunk/libs directory. So that when checked out it looks like this:
trunk/source
trunk/libs/somelib
trunk/libs/anotherlib
This is described (probably a whole lot better) in the Subversion Book. Particularly the section on handling vendor branches and externals.
Ubuntu = Debian = apt-get goodness
Start with Linux Utilities:
%> sudo apt-get install util-linux