After looking through software files, I often came across a folder named "Common".
I suspect this is only used for convention and can be used however the developers desire, but what data is generally stored within this file? When would and wouldn't I use a folder named "Common"?
The directory C:\Program Files\Common Files on Windows contains files which are used among several different applications, is the "Common" file I'm referring to used in this manner?
A typical use might be part of a modular application, where the user can choose to install one or more optional modules (or plug-ins). These then get loaded at runtime by the application.
Any files provided by this application for use by the modules (APIs, libraries, frameworks, shared resources etc) would then be found in a 'Common' folder. This is the case with the C:\Program Files\Common Files folder.
This is useful to module authors, as they then know a reliable location to find their dependencies without having to include them in the module, unnecessarily increasing module size.
This idea can also be extended to similar architectures, e.g. multiple versions of an application with files shared by all versions in a 'Common' folder.
This brings disadvantages too; versioning becomes more complicated as upgrades to the application could also lead to the shared files being updated and becoming incompatible with the installed modules. Old file versions must then be retained in case an old module is still installed.
The meaning of 'Common' here is 'shared' ("something in common") rather than 'frequent' ("commonly seen").
Related
I have a lot of basic questions about DLLs and how to bundle third-party libraries with your software, but most everything I find when searching is specific to C# (such as this question) and I'm looking for more general information about how DLLs work and what their purpose is.
I have, for example, a CMake-managed C++ application that links to one or more third-party DLLs.
Their DLLs are installed to C:/Program Files/Intel/SomeLibrary/bin/somelibrary.dll. I can build my app fine by having an environment variable take care of finding their headers.
Obviously, I can't actually run my executables out of the box, because somelibrary.dll can't be found. I have to either copy it into the same folder as the executable, or add Intel/SomeLibrary/bin to my Path.
Questions
How do other programs link to their required DLLs? Do they just package copies of all the third-party DLLs they use in with their own software?
Is it a different process for some third-party libraries -- like maybe Intel libraries you'd bundle with your software, but for Microsoft's VS Redistributables you'd require the user to install them? It looks like your installer (WiX example) handles installing them if they don't exist (and then does it install a new copy for every piece of software that needs them?).
Could you potentially have many copies on your computer of the same DLL because a bunch of separate programs needed it but had no way of knowing if you had it somewhere already?
Put another way, could two programs, written by different people, ever link to the same DLL file? How do they find that DLL, or install it if it doesn't exist?
I apologize in advance for whatever fundamental misunderstandings I've made in asking these -- I'm pretty new to the idea of packaging already-built software.
The short answer is that the dll loader will look for dlls in the local application folder and then searching a bunch of system locations. There are complexities with manifests, the SafeDllSearchMode feature (affects directory search order) and developers loading dlls from custom locations by override (LoadLibraryEx) and several others (see below).
Dependencty Types: There are many types of dll dependencies: Module / dll dependencies mind map.
Dynamic-Link Library Search Order: Here is an extract from Microsoft Docs on the topic: "A system can contain multiple versions of the same dynamic-link library (DLL). Applications can control the location from which a DLL is loaded by specifying a full path or using another mechanism such as a manifest."
In other words you can 1) load a dll from a specific location or 2) you can let the OS find it for you via standard mechanisms. The "SafeDllSearchMode" registry setting affects when the users current directory is searched. It now defaults to on, but can be disabled. See the link above for the authoritative details. Beware of OS changes and security fixes - all can affect this order in the future.
Q1: Always use the runtime provided for your dependent dll if available. Don't just include a few dlls if you are not supposed to.
Q2: It is correct that runtimes are often very different. Sometimes they are just a single or a few dlls that you can just install in your local application folder. Other times you install a runtime setup which installs files to appropriate folders for shared use. Some runtimes come as "merge modules" that can merge files into your own MSI installer, and some come as separate MSI installers or setup.exe installers.
Q3: The whole idea of using dlls is essentially to be able to load the same binary component into many processes in order to be able to update a single dll to fix problems in all "consumers" of the dll and also to share them in memory (important back in the day with little memory). Dlls can also be "delay loaded" - which means they won't be loaded until they are necessary - avoiding memory bloat. That is the idea anyway.
Windows OS Version: The search order (where to search on disk) for the right dlls to load is a pretty complicated topic. How the OS loader mechanism locates DLLs can be affected by a number of things, for example 1) the OS version, 2) manifests, 3) hard coded altered load paths (LoadLibraryEx), or 4) dll redirects, and 5) previously loaded dlls in memory (here be dragons), etc...
UWP & .NET: There are also new development with regards to UWP applications. For .NET assemblies the resolution is different again (which you state), please see this article: How the Runtime Locates Assemblies.
WinSxS: The manifest is a newer concept in Windows - in the olden days there would be no such thing and the search order would be simpler, but less flexible. With the advent of manifests and side-by-side Win32 assemblies (several versions of the same dll installed in different versions) the manifest is embedded in the executable (or it is added externally) and it tells the binary where to look for the runtime dlls - in the WinSxS folder hierarchy of Windows (Normally: C:\Windows\WinSxS) where multiple versions of the same dll can reside (in different sub folders).
Most Microsoft runtimes - basically all modern ones - reside here and hence require a manifest to instruct the binary what version to look for and load. There are ways to redirect such manifests too... "Publisher Configuration Files" and 2. Sigh... This can be used to enforce a certain new runtime version when the old one is undesired for use. Unused runtimes can also be "garbage collected" via ways to shrink the WinSxS folders (See this answer - section "Scavenging / Putting WinSxS on a Diet")... Hmmm... Never hard code paths to WinSxS.
Dll Search Order: The old and outdated tool Dependency Walker has an outdated, but very good description of the DLL Search Order in its help file.
Tip: The updated documentation for this is: Microsoft Docs: Module Search Order for Desktop Applications. However, to get a rough idea of search order, you can just launch Dependency Walker and select: Options => Configure Module Search Order.... Then make sure the Expand button is deselected. What you see - outdated as it might be - can help to recollect how module search order works:
Links:
DLL search on windows (old Stackoverflow answer on the same topic)
The path-searching algorithm is not a backtracking algorithm
The case of the DLL that refuses to load
I was wondering whether there is a convention for storing config files in c++. At the moment I am storing the config files in the same directory as the source code that uses them. When building I make sure via the CMakeLists to copy them to the correct location so I can just access them in a relative way (e.g. "config.cfg" iso "/foo/bar/config.cfg) for convenience.
The practice for config files is non portable and operating system dependent. You have also to ask yourself if your configuration is defined per installation/system or per user.
In general, it is a very bad idea to store config in the same directory as your execuable, at least once the developpement is finished. In general executables may be shared between several users and should therefore for security reasons be located in directories that are write-protected for everybody but the system administrator.
For unix/linux, you could for example consider:
/etc or a subfolder thereof, if your configuration is per installed system,
~/ if it's user defined configuration. The usual practice would be to start the filename with a dot. This article will tell you more.
For windows systems, you should consider:
the usual approach now, goes to the registry. Of course, this uses the windows api and is fully non portable.
a subfolder of C:\ProgramData or C:\Users\All users\AppData\Local if your configuration is per installed system,
a subfolder of C:\Users\%USERNAME%\AppData\Local for the users's own configuration.
This SO questions shows how to find the right folders.
Currently, I program in Java and use Maven quite a bit. As so I've become accustom to the naming schemes and folder structures that I've used over the past 4 or 5 years.
As I have recently started to learn C++, I'm realizing that I have no idea where to put all my files. Should I keep everything broken down by namespace, or by what tier it is in? Where, for example, would I keep a series of files devoted to UI, as apposed to files meant to help store data?
Are there any standards for this sort of thing?
Clearly, there is no definitive answer to this question. I'm simply looking for a good guide. I do not want to start learning C++ by spending too much time worrying about how my files are laid out. I'd rather have some good models, and just get to the coding.
The following is fairly typical...
third-party library
release
obj
debug
obj
include
src
sublib 1
sublib 2
mylibrary
release
obj
debug
obj
include
src
sublib 1
sublib 2
myapp
release
obj
debug
obj
subapp 1
subapp 2
mylittleapp
release
obj
debug
obj
Basically, subfolders for subprojects is common for larger projects, but mostly a particular project has folders for src, include etc. A folder for each build configuration is common, and keeping the obj files and other intermediates in a subfolder of that is a good idea. It may be tempting to put subproject folders in obj folders, but usually that's unnecessary - the obj folders don't need to be well organised, so the only concern is a filename conflict, and the best fix for that is to have unique source filenames within (at least) each project.
The "include" folders should IMO only contain headers that will be #included by other projects - internal headers belong in the "src" folder.
Putting UI stuff in a separate folder isn't a bad idea, if it's big enough. I've seen UI stuff done as a separate static-linked top-level project, and I do mean app-specific here, not (e.g.) wxWidgets. Usually, though, that level of division is sub-project if it's worth separating at all. How you divide subprojects is more a matter of application-specific blocks in general, so it depends on whether UI stuff is best handled as a separate block or as separate chunks mixed in with task-specific logic.
Namespaces aren't the most used language feature, possibly because a lot of people use "using" so much they don't make much difference. A namespace for a main library project makes sense, but associating subfolders to namespaces 1:1 isn't something I've seen. I personally have a namespace that encompasses most of my library code, with a couple of sub-namespaces for things rarely used in general, but used a lot in a few places (e.g. a "bitwise" namespaces). The sub-namespaces are limited to single source/header pairs, so no need for subfolders. Most of the library-specific selection is done by including the right header - except that I usually include the lot through a main-project top-level header anyway.
Basically, namespaces are a way of avoiding naming conflicts. They don't necessarily associate with abstractions or functional blocks or anything. Within a particular project, you're probably better off just making sure the names don't conflict. As with the "std" namespace, it's fine to put a lot of stuff in one namespace.
As you say, though, this isn't a definitive answer - there are of course minor variations and quite different approaches.
On small projects my team groups all the files together by a link unit ie library, DLL, EXE. If the unit is very large we will sometimes breakup the files by functional unit or subsystem so that if you need to edit a component they are generally in the same place.
I break my projects by theme, one directory for theme:
menu_planner
src
recipes
debug -- contains debug object files and libraries
release -- contains release object files and libraries
obsolete -- contains obsolete source files
ingredients
debug -- contains debug object files and libraries
release -- contains release object files and libraries
obsolete -- contains obsolete source files
references
debug -- contains debug object files and libraries
release -- contains release object files and libraries
obsolete -- contains obsolete source files
meals
debug -- contains debug object files and libraries
release -- contains release object files and libraries
obsolete -- contains obsolete source files
menus
debug -- contains debug object files and libraries
release -- contains release object files and libraries
obsolete -- contains obsolete source files
docs
designs
My experience with C and C++ has shown to me that header and source files should be in the same directory. Often, finding a header file is more difficult when it is not in the same directory as the source file.
One directory (folder) per concept is a good idea. Any concept that is complex or compound should be split into multiple folders or concepts.
I've also learned to make libraries. I use libraries to contain code that doesn't change much. The linking step performs faster with libraries than directories of object files.
However, work places (a.k.a. shops) may have different style rules that must be followed.
It is not necessary to have your header files and cpp files in the same folder. I have done this many times. You can have the in different folders and use another file to fetch/include both files on file, which you will use as your include.
Let's say you have several bespoke C++ projects in separate repositories or top-level directories in the same repository. Maybe 10 are library projects for stuff like graphics, database, maths, etc and 2 are actual applications using those libraries.
What's the best way to organise those 2 application projects to have the .libs they need?
Each lib project builds the .lib in its own directory, developers have to copy these across to the application area manually and make sure to get the right version
Application projects expect lib projects to be in particular paths and look for .libs inside those locations
A common /libs directory is used by all projects
Something else
This is focused on C++, but I think it's pretty similar with other languages, for instance organising JARs in a Java project.
I'd suggest this approach:
Organise your code in a root folder. Let's call it code.
Now put your projects and libraries as subfolders (e.g. Projects and Libraries).
Build your libraries as normal and add a post-build step that copies the resulting headers and .lib files into a set of shared folders. For example, Libraries\include and Libraries\lib. It's a good idea to use subfolders or a naming convention (myLib.lib, myLib_d.lib) to differentiate different builds (e.g. debug and release) so that any lib reference explicitly targets a single file that can never be mixed up. It sucks when you accidentally link against the wrong variant of a lib!
You can also copy third-party libraries that you use into these folders as well.
Note: To keep them organised, include your files with #include "Math\Utils.h" rather than just "Utils.h". And put the headers for the whole Math library into include\Math, rather than dropping them all in the root of the include folder. This way you can have many libraries without name clashes. It also lets you have different versions of libraries (e.g. Photoshop 7, Photoshop 8) which allows you to multi-target your code at different runtime environments.
Then set up your projects to reference the libraries in one of two ways:
1) Tell your IDE/compiler where the libs are using its global lib/include paths. This means you set up the IDE once on each PC and never have to specify where the libs are for any projects.
2) Or, set each project to reference the libs with its own lib/include paths. This gives you more flexibility and avoids the need to set up every PC, but means you have to set the same paths in every new project.
(Which is best depends on the number of projects versus the number of developer PCs)
And the most important part: When you reference the includes/libs, use relative paths. e.g. from Projects\WebApp\WebApp.proj, use "..\..\Libraries\include" rather than "C:\Code\Libraries\Include". This will allow other developers and your buildserver to have the source code elsewhere (D:\MyWork instead of C:\Code) for convenience. If you don't do this, it'll bite you one day when you find a developer without enough disk space on C:\ or if you want to branch your source control.
Much related to this question, we have a scenario on my team where we need to copy the contents of a folder for a suite of libraries and configuration files for said libraries to our folder where our test code is running from, as part of the test's deployment step.
Due to the installation size, and other factors, checking in this install folder into source control for sharing between team members just isn't viable.
The install path for the folder is either /Program Files/InternalTool/ or /Program Files (x86)/InternalTool/ depending on the installed environment. I want to setup my .testrunconfig file such that when a person gets the latest version of the solution, they don't have to worry about fixups for the path to the shared internal library suite.
Is there a way to make this seamless for all members involved, and if so, how could one accomplish this?
Restrictions are as follows:
can't check in shared suite
shared suite has no override for installation path
Is this possible, or am I asking for too much?
We handle this sort of issue (our issues are not the same but are similar) by having different config files with different names and copying the correct one over when it is needed.
In some cases we automate this within the batch job that gets the latest version.
This was actually way, way easier than I expected.
While the UI doesn't support many things with the local test run config file, I was able to set the path using the standard %ProgramFiles%.
On x86 systems, this resolves, on most systems, to C:\Program Files\.
On x64 systems, this resolves, on most systems, to C:\Program Files\.
But! If the caller is 32-bit, and not 64-bit or set to MSIL, %ProgramFiles% will resolve to C:\Program Files(x86)\. Since there is no 64-bit mstest, the resolution should happen seamlessly. For an example, this is ripped from my LocalTestRun.testrunconfig file, and then properly sanitized:
<Deployment>
<DeploymentItem filename="%ProgramFiles%\InternalSuite\" />
</Deployment>
While I haven't had the chance to fully test this yet, this should resolve our issue just fine. I have tested this on my 32-bit system, and have found that it resolves right as rain.
Hope this helps someone else!