Follow up of this question:
When I do include <iostream>.
It happens that it includes many files from /usr/include .A grep "\usr\include" over g++ -E prog.cpp counted to about 1260 entries ;).
Is their a way to control including various files?
Platform: Linux
G++ version: 4.2.4
No, <iostream> includes them because it depends on them directly, or it's dependancies depend on them.
Ain't nothing you can do about it.
You can (depending on your compiler) limit the effect this has on compilation times by using Precompiled Headers
My suggestion is not to worry about how many files the compiler is including. Focus more on correctness, robustness, and schedule. If build time is a concern, get a faster machine, build overnight, go on a walk, or divide the code into smaller translation units. Translation units should be small enough to contain code that doesn't change often. Changes are EVIL.
The foundation of the build system is to compile only the few files that have changed. If your development process is functioning correctly, the build time will reside more and more in the linking phase as the project grows.
If the compile time is still lengthy, see if your compiler supports precompiled headers. Generally, this is method for the compiler to store all the declarations and definitions in a more efficient form.
You #include <iostream> when you need to use streams. That should define some things you need. How much activity it needs to do this is a quality of implementation issue. You could remove files from /usr/include, but that would break things.
I really doubt it's including 1260 files. Almost certainly most of those are duplicate entries that don't load anything but aren't pruned from the -E output.
Related
I have shared code given to me that compiles on one linux system but not a newer system. The error is uint32_t does not name a type. I realize that this is often fixed by including the <cstdint> or stdint.h. The source code has neither of these includes and I am trying to seek an option that doesn't require modifying due to internal business practices that I can't control. Since it compiles as is on one machine they don't want changes to the source code.
I am not sure if it matters but the older system uses gcc 4.1 while the newer one uses gcc 4.4. I could install different versions of gcc if needed, or add/install library/include files on the newer machine, I have full control of what is on that machine.
What are my options for trying to compile this code on my machine without modifying the source? I can provide other details if needed.
I am not sure if it matters but the older system uses gcc 4.1 while the newer one uses gcc 4.4
GCC stopped including <stdint.h> some time ago. You now have to include something to get it...
I realize that this is often fixed by including the <cstdint> or stdint.h. The source code has neither of these includes and I am trying to seek an option that doesn't require modifying due to internal business practices that I can't control...
I hope I am not splitting hairs... If you can't modify the source files, then are you allowed to modify the build system or configuration files; or the environment? If so, you can use a force include to insert the file. See Include header files using command line option?
You can modify Makefile to force include stdint.h. If the build system honors CFLAGS or CXXFLAGS, then you can force include it in the flags. You last choice is probably to do something like export CC="gcc -include stdint.h".
The reason I am splitting hairs is OpenSSL and FIPS. The OpenSSL source files for the FIPS Object Module are sequestered and cannot be modified. We have to fallback to modifying supporting scripts and the environment to get some things working as expected.
If you really don't want to amend the file you could wrap it. Suppose it's called src.c create a new file src1.c:
#include <stdint.h>
#include "src.c"
And then compile src1.c.
PS: The problem may arise because compilers include other headers in their header files. This can mean some symbols 'officially' defined in other headers are quietly defined when you include a header that isn't specified as including it.
It's an error to write a program relying on a symbol for which the appropriate header hasn't been included - but it's easy to do and difficult to spot.
A changing compiler or version sometimes reveals these quiet issues.
Unfortunately, you can't force your code to work on a newer compiler without modifying something.
If you are allowed to modify the build script and add source files to the project, you might be able to add another source file to the project which, in turn, includes your affected file and headers it really needs. Remove the affected source files from the build, add the new ones, and rebuild.
If your shared source is using macro magic (e.g. an #include SOME_MACRO, where SOME_MACRO can be defined on the command line), you might be able to get away with modifying build options (to define that macro for every compilation of each file). Apart from relying on modifying the build process, it also relies on a possible-but-less-than-usual usage of macros in your project.
It may be possible to modify the standard headers in your compiler/library installation - assuming you have sufficient access (administrative) to do so. The problem with this is that the problem will almost certainly re-emerge whenever an update/patch to the compiler/library is installed. Over time, this approach will lock the code into relying on an older and older compiler/library that has been superseded - no ability to benefit from compiler bug fixes, evolution of standards, etc. This also severely limits your ability to share the code, and ability of others to use it - anyone who receives the code needs to modify their compiler/library installation.
The basic fact, however, is that your shared code relies on a particular implementation (compiler/library) that exhibits non-standard behaviour. Hence it has failed with an update of that implementation - which removed those non-standard occurrences - it is likely to fail with other implementations (porting to different compilers in future, etc). The real technical solution is to modify the source, and #include needed headers correctly. The real business solution is to make a business case justifying the need for such modifications, citing inefficiency - which will grow over time - in terms of cost and effort needed to maintain the shared code whenever it needs to be ported, or whenever a compiler is updated.
look at the second last line of code above your error, you'll find everything above that terminates with a , and only use a ; on the last entery
The usual approach is to have one precompiled header in a project that contains the most common includes.
The problem is, that it is either too small or two big. When it is too small, it doesn't cover all the used headers so these have to be processed over and over in every module. When it is too large, it slows down the compilation too much for two reasons:
The project needs to be recompiled too often when you change something in header contained in the precompiled header.
The precompiled header is too large, so including it in every file actually slows down compilation.
What if I made all of the header files in a project precompiled. This would add some additional compiler work to precompile them, but then it would work very nicely, as no header would have to be processed twice (even preparing the precompiled header would use precompiled headers recursively), no extra stuff would have to be put into modules and only modules that are actually needed to be recompiled would be recompiled. In other words, for extra work O(N) complexity I would (theoretically) optimise O(n^2) comlexity of C++ includes. The precosseor to O(N), the processing of precompiled data would still be O(N^2), but at least minimised.
Did anyone tried this? Can it boost compile times in real life scenarios?
With GCC, the reliable way to use precompiled headers is to have one single (big) header (which #include-s many standard headers ...), and perhaps include some small header after the precompiled one.
See this answer for a more detailed explanation (for GCC specifically).
My own experience with GCC and Clang with precompiled headers is that you only can give a single pre-compiled header per compilation. See also the GCC documentation, I quote:
A precompiled header file can be used only when these conditions apply:
Only one precompiled header can be used in a particular compilation.
...
In practice, it's possible to compile every header to a precompiled header. (Recommended if you want to verify if everything is included, not recommended if you want to speed up compilation)
Based on your code, you can decide to use a different precompiled header based on the code that needs to be compiled. However, in general, it's a balancing act between compile time of the headers, compile-time of the CPP files and maintenance.
Adding a simple precompiled header that already contains several standard headers like string, vector, map, utility ... can already speed up your compilation with a remarkable percentage. (A long time ago, I've noticed a 15-20% on a small project)
The main gain you get from precompiled headers is that it:
only have to read 1 file instead of more, which improves on disk access
reads a binary format that's optimized for reading instead of plain text
it doesn't need to do all of the error checking as this was already done on creation
Even if you add a few headers that you don't use everywhere, it can still be much faster.
Lately, I also found the Clang build analyzer, it ain't ideal for big projects (see issue on github), though, it can give you some insights on where the time is being spent and what it can improve. (Or what you can improve in the codebase)
In all fairness, I don't use precompiled headers at this point in time. However, I do want to see it enabled on the project I'm working on.
Some other interesting reads:
https://medium.com/#unicorn_dev/speeding-up-the-build-of-c-and-c-projects-453ce85dd0e1
https://llunak.blogspot.com/2019/05/why-precompiled-headers-do-not-improve.html
https://www.bitsnbites.eu/faster-c-builds/
My single C++ file includes many header files from various libraries such as MKL and template libraries. Compiling this single file every time takes a long time due to the many include statements. I tried separating the include statements in a single file, making an object file out of it, and linking it during the compilation of the main file, but it seems the compiler doesn't recognize the definitions of the objects and functions that are defined in the header files.
So, how can I save the compilation time of these header files?
Precompiled headers:
What you are doing does sound like it would benefit heavily from precompiled headers (pch's). The intel compiler does support pch's, as you can see here:
https://software.intel.com/en-us/node/522754.
Installing tools without root permissions
You can still experiment with ccache to see if that will help your situation -- it can make a huge difference in cases where units do not need to be recompiled, which happens surprisingly often. The path to doing this is to install ccache locally. Generally this can be done by downloading sources to a directory where you have write access (I keep an install folder in my home directory) , and then following the build directions for that project. Once the executable are built, you'll have to add the path for the executable to your path -- by doing
export PATH=$PATH:the-path-to-your-compiled-executables.
in BASH. At that point ccache will be available to your user.
A better explanation is available here: https://unix.stackexchange.com/questions/42567/how-to-install-program-locally-without-sudo-privileges
CMAKE, cotire, etc
One final point -- I haven't read the docu for the intel compiler's pch support, but pch support generally involves some manual code manipulation to instruct the compile which header to precompile and so on. I recommend looking in to managing your build with CMAKE, and using the cotire plugin to experiment with pch.
While I have gripes about CMAKE from a software engineering perspective, it does make managing your build, and especially experimenting with various ways to speed up your build, much much easier. After experimenting with pch's, and ccache, you could also try out ninja instead of make and see if that gets you any improvement.
Good luck.
There are basically two options for tracking dependencies which are -M and -MM. The difference is that -MM omits system headers and headers included by them.
My question: Why would anyone want to use -M? It inflates the generated .d files drastically, since a system header usually includes a large pack of other system headers. In addition, system headers cannot be built by make, so having them as a depencies yields no benefit. The only little benefit I could see is that - if a required system header is missing - make reports the missing header instead of gcc reporting it. But what is the benefit of this?
To sum things up, I see no reason why -M would be useful at all. Am I missing something? Which scenarios are there that require one to use -M over -MM.
Most header files can't be "built" by make. They're listed as prerequisites so that if they change, then the source code that relies on them is rebuilt. For example, if you install security fix packages on your system and they modify one of the system headers you use, you may want to be sure all your code is rebuilt. These days the backward-compatibility of most base libraries is such that this is not really needed most of the time, I agree.
Also, if you're cross-compiling then your "system" header files are provided to you from the cross-target; these headers might be for an embedded system or similar, and may change (in non-backward-compatible ways) more often than a standard system.
Why would anyone want to use -M?
If the system headers change you want a rebuild to operate accordingly. That is, if your code uses a header, and that header changes, your code should rebuild even if your code didn't change.
Listing headers as dependencies is rarely about 'building' those headers. System headers are no different.
There could be a number of reasons.
Rebuild only necessary parts after system libraries update. Rare thing, but someone may need it.
Get full dependencies list, for some reasons - maybe just drawing dependency graph?
Generate ctags for all potentially used header files.
...
Personally I use it for ctags, so it is not all hypothetical examples.
Pre-compiled headers seem like they can save a lot of time in large projects, but also seem to be a pain-in-the-ass that have some gotchas.
What are the pros & cons of using pre-compiled headers, and specifically as it pertains to using them in a Gnu/gcc/Linux environment?
The only potential benefit to precompiled headers is that if your builds are too slow, precompiled headers might speed them up. Potential cons:
More Makefile dependencies to get right; if they are wrong, you build the wrong thing fast. Not good.
In principle, not every header can be precompiled. (Think about putting some #define's before a #include.) So which cases does gcc actually get right? How much do you want to trust this bleeding edge feature.
If your builds are fast enough, there is no reason to use precompiled headers. If your builds are too slow, I'd consider
Buying faster hardware, which is cheap compared to salaries
Using a tool like AT&T nmake or like ccache (Dirk is right on), both of which use trustworthy techniques to avoid recompilations.
I can't talk to GNU/gcc/linux, but I've dealt with pre-compiled headers in vs2005:
Pros:
Saves compile time when you have large headers that lots of modules
include.
Works well on headers (say from a third party) that change very
infrequently.
Cons:
If you use them for headers that change a lot,
it can increase compile time.
Can be fiddly to set up and maintain.
There are cases where changes to headers are apparently ignored
if you don't force the pre-compiled header to compile.
The ccache caching frontend to gcc, g++, gfortran, ... works great for me. As its website says
ccache is a compiler cache. It acts as
a caching pre-processor to C/C++
compilers, using the -E compiler
switch and a hash to detect when a
compilation can be satisfied from
cache. This often results in a 5 to 10
times speedup in common compilations.
On Debian / Ubuntu, just do 'apt-get install ccache' and create soft-links in, say, /usr/local/bin with names gcc, g++, gfortran, c++, ... that point to /usr/bin/ccache.
[EDIT] To make this more explicit in response to some early comments: This provides essentially pre-compiled headers and sources by caching a larger chunk of the compilation step. So it uses an idea that is similar to pre-compiled headers, and carries it further. The speedups can be dramatic -- a factor of 5 to 10 as the website says.
For plain C, I would avoid precompiled headers. As you say, they can potentially cause problems, and preprocessing time is really small compared to the regular compilation.
For C++, precompiled headers can potentially save a lot of time, as C++ headers often contain large template code whose compilation is expensive. I have no practical experience with them, so I recommend you measure how much savings in compilation you get in your project. To so so, compile the entire project with precompiled headers once, then delete a single object file, and measure how long it takes to recompile that file.
The GNU gcc documentation discusses possible pitfalls with pre-compiled headers.
I am using PCH in a Qt project, which uses cmake as build system, and it saves a lot of time. I grabbed some PCH cmake scripts, which needed some tweaking, since they were quite old but it generally was easier to set up than I expected. I have to add, I am not much of a cmake expert.
I am including now a big part of Qt (QtCore, QtGui, QtOpenGL) and a few stable headers at once.
Pros:
For Qt classes,no forward declarations are needed, and of course no includes.
Fast.
Easy to setup.
Cons:
You can't include the PCH include in headers. This isn't much of a problem, exept you use Qt and let the build system translate the moc files seperatly, which happens to be exactly my configuration. In this case, you need to #include the qt headers in your headers, because the mocs are genreted from headers. Solution was to put additional include guards around the #include in the header.