I'm working at an organization with a product suite based on several hundred Visual Studio solutions (mostly C++). Some of these solutions generate libraries that are used by other solutions and there's also a common "include" folder containing headers that shared by multiple modules.
The issue is that the dependencies are not explicitly stated anywhere, and the build system resolves dependencies by specifying a linear build order that makes sure the dependent modules get built at the right time. This works well for the build system but leaves developers at a disadvantage when trying to work on components with many direct and indirect external dependencies. For example, I might want to edit one of the library projects or shared headers and then build all the affected modules without necessarily knowing ahead of time which ones are affected. Another use case involves building a module after doing a fresh pull from TFS and having the modules it depends on built first without having to build the entire system.
I am wondering if there is/are any tool(s) available that can automate dependency generation for building large projects. I have considered creating a few really big solutions that encapsulate the other solutions but that seems really awkward and clumsy. Also, I don't like the idea of having developers manually specify dependencies as it can error prone, especially with such a large code base. I worked with scons a few years ago and really liked the way it could parse source files and automatically discover all the dependencies dependencies. Is there anything available today that can do the same thing with Visual Studio solutions?
This is not a duplicate of Visual Studio: how to handle project dependencies right?
I need to emphasize the magnitude of the problem I am trying to solve. This is a very large existing code base. In the main directory there are several hundred sub-folders, each one containing one of more VS solutions (not projects). Each solution, in turn, contains one or more projects. As I said before, I'm not trying to establish dependencies among a few projects in a solution. The problem is much bigger than that. I'm trying to find a way to establish dependencies among the solutions themselves (several hundred of them). For example, one solution may contain some projects that generate libraries for security, others for communications, etc. There may be, for example, dozens of solutions that use the communications libraries. So essentially I'm trying to create a directed a cyclic graph with hundreds of nodes and potentially tens of thousands of edges.
You could use cmake (https://cmake.org/). With it, you can specify several libraries and apps to be built. Once configured, you can modify a project and the build will just update the dependent projects. Cmake also provides a visual studio generator, so that you can continue using that IDE.
A possible disavantage to you is that, to configure, you must explictly specify, for each project (library or executable), with what projects it must be linked and what folders it must include. There are ways to define some global includes and links, but the use will depends on your problem.
VS does track dependencies (by parsing source files). It doesn't make sense that something could automatically set dependencies of your VS projects, in any other build tools you'd still have to specify in some way that for linking project A.exe you need to use B.lib.
If you use newer VS versions you should simply add references to lib to your exe/dll projects. If you manually added project dependencies, most likely you should remove them all, especially make sure you don't make static lib projects dependent on each other. VS allows you to do that (for example, if build of one library generates some source files that another static lib uses), but in general these shouldn't have any dependencies and this allows VS to optimize builds by building them in parallel.
For example, commonly you could have some kind of Base.lib, then System.lib and Graphics.lib. All of these are user by your App.exe. System.lib uses code from Base.lib, Graphics.lib uses code from System.lib and Base.lib. So, naturally the dependency chain is clear and you go and set them in VS, and that's a mistake! In cases like this in VS you should make these 4 libs independent and only App.exe should be dependent on all these libs (e.g. it should have references to all of these). VS will figure out what is the the correct dependency of these projects.
Regarding Cmake case: it simply generates VS projects and solutions, if you use VS then cmake cannot do more than VS itself can.
Related
When creating new projects I've always stumbled upon the issue that my final executable or dll does not properly run because it is missing dependencies I was using.
In my live as a developer I've seen several approaches to handle that (of which I don't like any)
Set environment path to all those dependencies so that the OS can find them (bad, since very unportable, requires environment, won't work out of the box)
Copy every dependency of a potentially large package into the binary output directory (e.g. dozens of dlls regardless of need). For dependencies like OSG, Qt, etc this is quite odd as you typically won't link all dlls provided by larger packages and you may end up copying much more data than necessary.
Hand-Pick single dependencies (and optionally their pdbs) with a fine-grained xcopy/robocopy or whatever task. (Don't like that as it needs attention, I add a dependency in Visual Studio and then I need to adjust some script). There are tools like dependency walker which help this, but still, this might still be unportable because your include path in VS may have a version in it, if you change that, you need to change it also in the script, which is too much Department of Redundancy Department
In larger projects or companies I would expect this issue to be quite common. Is their nothing in Visual Studio or in Windows that aids that? I need to pick my include paths and import lib paths properly in order to compile a binary but then I need some other mechanism to actually make things work.
I am thinking about solving this problem in a more general way for more projects but really wonder if I am missing something.
Visual studio doesn't support native projects as it supports .NET projects. In the sense that when for example creating a static(.lib) library. Including of the static library, and the directory containing the headers, has to be done manually.
For one project this isn't some much of a problem. But if you're like me managing several projects. A lot of which are somewhat depenendend. It becomes a huge hassle to manage all of it.
I was wondering if there is any official 'microsoft approved' approach to this. And if not, what is the best way to deal with this situation. Supposing the following conditions occur:
several static libraries(.lib) projects. Which are included in several solutions
several dynamic libraries (.dll) projects. Which are included in several solutions
multiple applications using the same libraries(both dynamic and static), in one solution
My personal solution to the problem is as follows.
Every project generating a binary builds to:
$(SolutionDir)build\$(Configuration)\`
Every project generating a static library builds to:
$(SolutionDir)build\$(Configuration)\Libraries\
The intermediate directory for all projects is:
$(SolutionDir)build\$(ProjectName)\$(Configuration)\
And runs the following pre-build command:
Copy /Y "$(ProjectDir)*.h" + "$(ProjectDir)*.hpp" "$(SolutionDir)build\$(Configuration)\Libraries\"
Advantages of this system include:
All the project directories stay free of builds (useful when using source control). And all the binaries are in one place.
Setting additional include directories is never required when using outputs from other projects in the same solution. A dynamic library doesn't have to be added at all. And all that is required to include a static library is adding it to the Additional Dependencies field under:
Configuration Properties->Linker->Input
Drawbacks of this system include:
Since all the header files are copied, the risk exists of accidentally editing those. Which results in loss of work, when the copying occurs again.
Since the settings are per project, they have to be set for ever project
The libraries are built separately for every solution
Let's say you have several bespoke C++ projects in separate repositories or top-level directories in the same repository. Maybe 10 are library projects for stuff like graphics, database, maths, etc and 2 are actual applications using those libraries.
What's the best way to organise those 2 application projects to have the .libs they need?
Each lib project builds the .lib in its own directory, developers have to copy these across to the application area manually and make sure to get the right version
Application projects expect lib projects to be in particular paths and look for .libs inside those locations
A common /libs directory is used by all projects
Something else
This is focused on C++, but I think it's pretty similar with other languages, for instance organising JARs in a Java project.
I'd suggest this approach:
Organise your code in a root folder. Let's call it code.
Now put your projects and libraries as subfolders (e.g. Projects and Libraries).
Build your libraries as normal and add a post-build step that copies the resulting headers and .lib files into a set of shared folders. For example, Libraries\include and Libraries\lib. It's a good idea to use subfolders or a naming convention (myLib.lib, myLib_d.lib) to differentiate different builds (e.g. debug and release) so that any lib reference explicitly targets a single file that can never be mixed up. It sucks when you accidentally link against the wrong variant of a lib!
You can also copy third-party libraries that you use into these folders as well.
Note: To keep them organised, include your files with #include "Math\Utils.h" rather than just "Utils.h". And put the headers for the whole Math library into include\Math, rather than dropping them all in the root of the include folder. This way you can have many libraries without name clashes. It also lets you have different versions of libraries (e.g. Photoshop 7, Photoshop 8) which allows you to multi-target your code at different runtime environments.
Then set up your projects to reference the libraries in one of two ways:
1) Tell your IDE/compiler where the libs are using its global lib/include paths. This means you set up the IDE once on each PC and never have to specify where the libs are for any projects.
2) Or, set each project to reference the libs with its own lib/include paths. This gives you more flexibility and avoids the need to set up every PC, but means you have to set the same paths in every new project.
(Which is best depends on the number of projects versus the number of developer PCs)
And the most important part: When you reference the includes/libs, use relative paths. e.g. from Projects\WebApp\WebApp.proj, use "..\..\Libraries\include" rather than "C:\Code\Libraries\Include". This will allow other developers and your buildserver to have the source code elsewhere (D:\MyWork instead of C:\Code) for convenience. If you don't do this, it'll bite you one day when you find a developer without enough disk space on C:\ or if you want to branch your source control.
I have a number of native C++ libraries (Win32, without MFC) compiling under Visual Studio 2005, and used in a number of solutions.
I'd like to be able to choose to compile and link them as either static libraries or DLLs, depending on the needs of the particular solution in which I'm using them.
What's the best way to do this? I've considered these approaches:
1. Multiple project files
Example: "foo_static.vcproj" vs "foo_dll.vcproj"
Pro: easy to generate for new libraries, not too much manual vcproj munging.
Con: settings, file lists, etc. in two places get out of sync too easily.
2. Single project file, multiple configurations
Example: "Debug | Win32" vs "Debug DLL | Win32", etc.
Pro: file lists are easier to keep in sync; compilation options are somewhat easier to keep in sync
Con: I build for both Win32 and Smart Device targets, so I already have multiple configurations; I don't want to make my combinatorial explosion worse ("Static library for FooPhone | WinMobile 6", "Dynamic library for FooPhone | WinMobile 6", "Static library for BarPda | WinMobile 6", etc.
Worse Con: VS 2005 has a bad habit of assuming that if you have a configuration defined for platform "Foo", then you really need it for all other platforms in your solution, and haphazardly inserts all permutations of configuration/platform configurations all over the affected vcproj files, whether valid or not. (Bug filed with MS; closed as WONTFIX.)
3. Single project file, selecting static or dynamic via vsprops files
Example: store the appropriate vcproj fragments in property sheet files, then apply the "FooApp Static Library" property sheet to config/platform combinations when you want static libs, and apply the "FooApp DLL" property sheet when you want DLLs.
Pros: This is what I really want to do!
Cons: It doesn't seem possible. It seems that the .vcproj attribute that switches between static and dynamic libraries (the ConfigurationType attribute of the Configuration element) isn't overrideable by the .vsprops file. Microsoft's published schema for these files lists only <Tool> and <UserMacro> elements.
EDIT: In case someone suggests it, I've also tried a more "clever" version of #3, in which I define a .vsprops containing a UserMacro called "ModuleConfigurationType" with a value of either "2" (DLL) or "4" (static library), and changed the configuration in the .vcproj to have ConfigurationType="$(ModuleConfigurationType)". Visual Studio silently and without warning removes the attribute and replaces it with ConfigurationType="1". So helpful!
Am I missing a better solution?
I may have missed something, but why can't you define the DLL project with no files, and just have it link the lib created by the other project?
And, with respect to settings, you can factor them out in vsprop files...
There is an easy way to create both static and dll lib versions in one project.
Create your dll project. Then do the following to it:
Simply create an nmake makefile or .bat file that runs the lib tool.
Basically, this is just this:
lib /NOLOGO /OUT:<your_lib_pathname> #<<
<list_all_of_your_obj_paths_here>
<<
Then, in your project, add a Post Build Event where the command just runs the .bat file (or nmake or perl). Then, you will always get both a dll and a static lib.
I'll refrain from denigrating visual studio for not allowing the tool for this to exist in a project just before Linker (in the tool flow).
I think the typical way this is done is choice 2 above. It is what I use and what I have seen done by a number of libraries and companies.
If you find it does not work for you then by all means use something else.
Good luck.
I prefer 2 configurations way.
Setup all common settings via 'All configurations' item in a project properties windows. After it separated settings. And it's done. Let's go coding.
Also there is very good feature named 'Batch build', which builds specified configurations by turn.
Multiple projects are the best way to go - this is the configuration i have most widely seen in umpteen no of projects that i have come across.
That said, it might be also possible to implement the third option by modifying your vcproj files on the fly from external tools(like a custom vbscript), that you could invoke from a make file. You can use shell variables to control the behavior of the tool.
Note that you should still use use visual studio to make the build, the makefile should only launch your external tool if required to make the mods and then follow that by the actual build command
I use Visual Studio 6.0 (Still) due to issues that are preventing us from Migrating to VS2005 or newer. Rebuilding causes severe issues (everything breaks)... so many of us are considering lobbying a migration to GnuC++ moving forward in a structured way to eventually get us off of licensed Visual Studio products and onto Eclipse and Linux.
In Unix/Linux it is easy to build for all configurations.. so I can't believe what a time and productivity sink it is to try and accomplish the same task in Visual Studio. For VS6.0 I have so far found that only having two separate projects seems to be workable. I haven't yet tried the multiple configuration technique, but will see if it works in the older VS6.0.
Why not go for version 1 and generate the second set of project files from the first using a script or something. That way you know that the differences are JUST the pieces required to build a dll or static lib.
I have a few things that I cannot find a good way to perform in Visual Studio:
Pre-build step invokes a code generator that generates some source files which are later compiled. This can be solved to a limited extent by adding blank files to the project (which are later replaced with real generated files), but it does not work if I don't know names and/or the number of auto-generated source files. I can easily solve it in GNU make using $(wildcard generated/*.c). How can I do something similar with Visual Studio?
Can I prevent pre-build/post-build event running if the files do not need to be modified ("make" behaviour)? The current workaround is to write a wrapper script that will check timestamps for me, which works, but is a bit clunky.
What is a good way to locate external libraries and headers installed outside of VS? In *nix case, they would normally be installed in the system paths, or located with autoconf. I suppose I can specify paths with user-defined macros in project settings, but where is a good place to put these macros so they can be easily found and adjusted?
Just to be clear, I am aware that better Windows build systems exist (CMake, SCons), but they usually generate VS project files themselves, and I need to integrate this project into existing VS build system, so it is desirable that I have just plain VS project files, not generated ones.
If you need make behavior and are used to it, you can create visual studio makefile projects and include them in your project.
If you want less clunky, you can write visual studio macros and custom build events and tie them to specific build callbacks / hooks.
You can try something like workspacewhiz which will let you setup environment variables for your project, in a file format that can be checked in. Then users can alter them locally.
I've gone through this exact problem and I did get it working using Custom Build Rules.
But it was always a pain and worked poorly. I abandoned visual studio and went with a Makefile system using cygwin. Much better now.
cl.exe is the name of the VS compiler.
Update: I recently switched to using cmake, which comes with its own problems, and cmake can generate a visual studio solution. This seems to work well.
Specifically for #3, I use property pages to designate 3rd party library location settings (include paths, link paths, etc.). You can use User Macros from a parent or higher level property sheet to designate the starting point for the libraries themselves (if they are in a common root location), and then define individual sheets for each library using the base path macro. It's not automatic, but it is easy to maintain, and every developer can have a different root directory if necessary (it is in our environment).
One downside of this approach is that the include paths constructed this way are not included in the search paths for Visual Studio (unless you duplicate the definitions in the Projects and Directories settings for VS). I spoke to some MS people at PDC08 about getting this fixed for VS2010, and improving the interface in general, but no solid promises from them.
(1). I don't know a simple answer to this, but there are workarounds:
1a. If content of generated files does not clash (i.e. there is no common static identifiers etc.), you can add to the project a single file, such as AllGeneratedFiles.c, and modify your generator to append a #include "generated/file.c" to this file when it produces generated/file.c.
1b. Or you can create a separate makefile-based project for generated files and build them using nmake.
(2). Use a custom build rule instead of post-build event. You can add a custom build rule by right-clicking on the project name in the Solution Explorer and selecting Custom Build Rules.
(3). There is no standard way of doing this; it has to be defined on a per-project basis. One approach is to use environment variables to locate external dependencies. You can then use those environment variables in project properties. Add a readme.txt describing required tools and libraries and corresponding environment variables which the user has to set, and it should be easy enough for anyone to set up.
Depending on exactly what you are trying to do, you can sometimes have some luck with using a custom build step and setting your dependencies properly. It may be helpful to put all the generated code into its own project and then have your main project depend on it.