I have a cross Java-clojure project that uses maven as the build tool. AOT compiling of some of the namespaces are required.
The problem is, the compilation takes like forever to complete. Inspecting the maven process gives the following observations:
When compiling, the CPU wait time is very high (single core 70-90%).
Sampling that high wait thread gives a stack where the innermost calls are:
at java.io.FileDescriptor.sync(Native Method)
at clojure.lang.Compiler.writeClassFile(Compiler.java:7269)
Checking the target/classes/ folder, a great number of .class files are generated, mostly for the clojure-based libraries on which my project depends.
What I have tried:
Instructed the clojure-maven-plugin to compile only given namespaces. But the propagation along the dependency line is not inhibited.
Explicitly included given namespaces, and excluded the namespace of dependency. The dependency library is still recompiled by the AOT process.
Well, just before I pressed the "POST" button, I found this issue. It seems back in 5 years we've had such discussion, and the problem is not specific to clojure-maven-plugin. So what's new about this issue? It takes about 4 minutes to mvn clean compile once on my machine for my not-so-huge project, and I can literally grab a coffee, sit back and relax..
If the issue is not to be resolved in any way in the near future, what might be suggested to mitigate my pain? Any thought are welcome.
As it turns out, there might not be a good solution to this problem yet. I now however resorts to this walk-around:
Instead of running the clean goal, I now use my own clean.sh script to clean only class files in my own namespaces. By doing this, my 4-minute compilation time drops to a little bit more than 1 minute.
I'll leave the question open and perhaps some day we'll see a better solution to it.
Related
Compiling my project takes ages and I thought I would like to improve the compile time of it. The first thing I am trying to do is to break down the compile time into the individual files.
So that the compiler tells me for example:
boost/variant.hpp: took 100ms in total
myproject/foo.hpp: took 25ms in total
myproject/bar.cpp: took 125ms in total
I could then specifically try to improve the compile time of the files taking up the most time, by introducing forward declaration and/or reordering things so I can omit include files.
Is there something for this task? I am using GCC and ICC (intel c++)
I use Scons as my build system.
The important metric is not how long it takes to process (whatever that means) a header file, but how often the header file changes and forces the build system to reinvoke the compiler on all dependent units.
The time the compiler spends parsing useless code is really small compared to all the other steps of the compilation process. Even if you include entire unneeded files, they're likely hot in disk cache. And precompiled headers make this even better.
The goal is to avoid recompiling units due to unrelated changes in header files. That's where techniques such as pimpl and other compile firewalls come in.
And link-time-code-generation aka whole-program-optimization makes matters worse, by undoing compile-time firewalls and reprocessing the entire program anyway.
Anyway, information on how unstable a header file is should be attainable from build logs, commit logs, even last modified date in the filesystem.
You have an unusual, quirky definition of the time spent processing header files that doesn't match what anyone else uses. So you can probably make this happen, but you'll have to do it yourself. Probably the best way is to run gcc under strace -tt. You can then see when it opens, reads, and closes each file, allowing you to tell how long it processes them.
Have you tried instrumenting the build as a whole yet? Like any performance problem, it's likely that what you think is the problem is not actually the problem. Electric Make is a GNU-make-compatible implementation of make that can produce an XML-annotated build log, which can in turn be used for analysis of build performance issues with ElectricInsight. For example, the "Job Time by Type" report in ElectricInsight can tell you broadly what activities consume the most time in your build, and specifically which jobs in the build are longest. That will help you to direct your efforts to the places where they will be most fruitful.
For example:
Disclaimer: I am the chief architect of Electric Make and ElectricInsight.
I'm compiling a very small Win32 command-line application in VS2010 Release-Mode, with all speed optimizations turned on (not memory optimizations).
This application is designed to serve a single purpose - to perform a single pre-defined complex mathematical operation to find a complex solution to a specific problem. The algorithm is completely functional (confirmed) and it compiles and runs fine in Debug-Mode. However, when I compile in Release-Mode (the algorithm is large enough to make use of the optimizations), Link.exe appears to run endlessly, and the code never finishes linking. It sits at 100% CPU usage, with no changes in memory usage (43,232 K).
My application contains only two classes, both of which are pretty short code files. However, the algorithm comprises 20 or so nested loops with inline function calls from within each layer. Is the linker attempting to run though every possible path through these loops? And if so, why is the Debug-Mode linker not having any problems?
This is a tiny command-line application (2KB exe file), and compiling shouldn't take more than a couple minutes. I've waited 30 minutes so far, with no change. I'm thinking about letting it link overnight, but if it really is trying to run through all possible code paths in the algorithm it could end up linking for decades without a SuperComputer handy.
What do I need to do to get the linker out of this endless cycle? Is it possible for such code to create an infinite link-loop without getting a compiler-error prior to the link cycle?
EDIT:
Jerry Coffin pointed out that I should kill the linker and attempt again. I forgot to mention this in the original post, but I have aborted the build, closed and re-opened VS, and attempted to build multiple times. The issue is consistent, but I haven't changed any linker options as of yet.
EDIT2:
I also neglected to mention the fact that I deleted the "Debug" and "Release" folders and re-built from scratch. Same results.
EDIT3:
I just confirmed that turning off function inlining causes the linker to function normally. The problem is I need function inlining, as this is a very performance-sensitive operataion with a minimal memory footprint. This leads me to ask, why would inlining cause such a problem to occur?
EDIT4:
The output that displays during the unending link cycle:
Link:
Generating code
EDIT5:
I confirmed that placing all the code into a single CPP file did not resolve the issue.
Nested loops only affect the linker in terms of Link Time Code Generation. There's tons of options that determine how this works in detail.
For a start I suggest disabling LTCG alltogether to see if there's some other unusual problem.
If it links fine in Release with LTCG disabled you can experiment with inlining limits, intrinsics and optimization level.
Have you done a rebuild? If one of the object files got corrupted, or a .ilk file (though release mode probably isn't using one, I can't be sure about your project settings), then the cleaning pass which rebuild does should cure it.
Closing and re-opening Visual Studio is only helpful in cases when the debugging (or one of the graphical designers) is keeping an open handle to the build product.
Which leads to another suggestion -- check Task Manager and make sure that you don't have a copy of your application still running.
If the linker seems to be the bottle neck and both classes are pretty short, why not put all the code in a single file? Also, there is a multi-processor compilation option in visual studio under C++ general tab in the project settings dialog box. Both of these might help.
Some selected answers:
Debug builds don't inline. Not at all. This makes it possible to put breakpoints in all functions. It wou
It doesn't really matter whether the linker truly runs an infinite loop, or whether it is evaluating 2^20 different combinations of inline/don't inline. The latter still could take 2^20 seconds. (We know the linker is inlining from the "Generating code" message).
Why are you using LTCG anyway? For such a small project, it's entirely feasible to put everything in one .cpp file. Currently, the compiler is just parsing C++, and not generating any code. That also explains why it doesn't have any problem: there's only a single loop in the compiler, over all lines of source code.
Here in my company we discovered something similar.
As far as I know it is no endless loop, just a very long operation.
This means you have to options:
turn off the optimizations
wait until the linker has finished
You have to decide if you really need these optimizations. Here the program is just a little helper, where it does not matter if the program is running 23.4 seconds or 26.8. The gain of program speed compared to the build speed are making the optimizations nearly useless. At least in our special scenario.
I encountered exactly this problem trying to build OSMesa x64 Release mode. My contribution to this discussion is that after letting the linker run overnight, it did complete properly.
I am using SUSE10 (64 bit)/AIX (5.1) and HP I64 (11.3) to compile my application. Just to give some background, my application has around 200KLOC (2Lacs) lines of code (without templates). It is purely C++ code. From measurements, I see that compile time ranges from 45 minutes(SUSE) to around 75 minutes(AIX).
Question 1 : Is this time normal (acceptable)?
Question 2 : I want to re-engineer the code arrangement and reduce the compile time. Is there any GNU tool which can help me to do this?
PS :
a. Most of the question in stackoverflow was related to Visual Studio, so I had to post a separate question.
b. I use gcc 4.1.2 version.
c. Another info (which might be useful) is code is spread across around 130 .cpp files but code distribution varies from 1KLOC to 8 KLOCK in a file.
Thanks in advance for you help!!!
Edit 1 (after comments)
#PaulR "Are you using makefiles for this ? Do you always do a full (clean) build or just build incrementally ?"
Yes we are using make files for project building.
Sometimes we are forced to do the full build (like over-night build/run or automated run or refresh complete code since many members have changed many files). So I have posted in general sense.
Excessive (or seemingly excessive) compilation times are often caused by an overly complicated include file hierarchy.
While not exactly a tool for this purpose, doxygen could be quite helpful: among other charts it can display the include file hierarchy for every source file in the project. I have found many interesting and convoluted include dependencies in my projects.
Read John Lakos's Large-Scale C++ Design for some very good methods of analysing and re-organising the structure of the project in order to minimise dependencies. Ultimately the time taken to build a large project increases as the amount of code increases, but also as the dependencies increase (or at least the impact of changes to header files increases as the dependencies increase). So minimising those dependencies is one thing to aim for. Lakos's concept of Levelization is very helpful in working out how to split several large monolothic inter-dependent libraries into something with a much better structure.
I can't address your specific questions but I use ccache to help with compile times, which caches object files and will use the same ones if source files do not change. If you are using SuSE, it should come with your distribution.
In addition to the already mentioned ccache, have a look at distcc. Throwing more hardware at such a scalable problem is cheap and simple.
Long compile times in large C++ projects are almost always caused by inappropriate use of header files. Section 9.3.2 of The C++ Programming Language provide some useful points this. Precompiling header files can considerably reduce compile time of large projects. See the GNU documentation on Precompiled Headers for more information.
Make sure that your main make targets can be executed in parallel (make -j <CPU_COUNT+1>) and of course try to use ccache. In addition we experimented with ccache and RAM disks, if you export CCACHE_DIR and point it to a RAM disk this will speed up your compilation process as well.
I was asked of this question when mentor an entry-level programmer, I was thinking of this compile + link process so official and usual that I never think about why.
One thing I could think of is to improve the development productivity, but should there be any other more compiler-related reasons?
Efficiency.
When you compile a program you create an object file for each source file, if you change a source file you only need to recompile that module and then relink (relinking is cheap).
If the compiler did everything in one pass it would have to recompile everything for every change.
It also fits with the unix philosophy of small programs that do one thing, so you have a pre-processor, a compiler, a linker, a library creator. These steps might now be different modes of the same tool.
However there are reasons why you want the compiler to link in one step, there are some optimizations you can do if you allow the compiler to change object files at link time - most modern compilers allow this but it requires them to put extra info into the object files at compile time.
It would be better if the compiler could store the entire project in a single database, rather than the mess of sources, resources, browse info files, object files etc - but developers are very conservative!
Part of this is historical. Back in the dark ages, computers had little memory. It was easy to to create a program with enough source code that all of it couldn't be processed at one time. So the processing had to be done in stages: preprocessing source code, compile source to assembly (one by one), assembly to object code, all object files linked into the final executable. Each of these steps had one or more stand alone tools to do its task. Over the years the tools were improved incrementally, but no major redesign of the process has ever become mainstream.
It's important that the build time, even for a very large project, be under 24 hours. And being able to build overnight is better. Separate compilation, which is to say dividing a program into "compilation units" and compiling them independently, is the way to reduce build time:
If a compilation unit hasn't been changed, and if nothing it depends on has changed, you can reuse the result of an old compilation.
You can often compile multiple units in parallel, or even distributed over a network of workstations. The lowly Make will compile in parallel, and other tools like ccdist exist to distribute the work of compilation.
Linking provides few benefits in and of itself but is necessary to use the results of separate compilation.
What an excellent time to teach your protégé about the Single Responsibility Principle!
Compiling the file changes the code into binary that the computer can read. Linking the file tells the computer how to complete a command. So its impossible to generate it all at once, without the two steps.
Want to improve this post? Provide detailed answers to this question, including citations and an explanation of why your answer is correct. Answers without enough detail may be edited or deleted.
I recently had cause to work with some Visual Studio C++ projects with the usual Debug and Release configurations, but also 'Release All' and 'Debug All', which I had never seen before.
It turns out the author of the projects has a single ALL.cpp which #includes all other .cpp files. The *All configurations just build this one ALL.cpp file. It is of course excluded from the regular configurations, and regular configurations don't build ALL.cpp
I just wondered if this was a common practice? What benefits does it bring? (My first reaction was that it smelled bad.)
What kinds of pitfalls are you likely to encounter with this? One I can think of is if you have anonymous namespaces in your .cpps, they're no longer 'private' to that cpp but now visible in other cpps as well?
All the projects build DLLs, so having data in anonymous namespaces wouldn't be a good idea, right? But functions would be OK?
It's referred to by some (and google-able) as a "Unity Build". It links insanely fast and compiles reasonably quickly as well. It's great for builds you don't need to iterate on, like a release build from a central server, but it isn't necessarily for incremental building.
And it's a PITA to maintain.
EDIT: here's the first google link for more info: http://buffered.io/posts/the-magic-of-unity-builds/
The thing that makes it fast is that the compiler only needs to read in everything once, compile out, then link, rather than doing that for every .cpp file.
Bruce Dawson has a much better write up about this on his blog: http://randomascii.wordpress.com/2014/03/22/make-vc-compiles-fast-through-parallel-compilation/
Unity builds improved build speeds for three main reasons. The first reason is that all of the shared header files only need to be parsed once. Many C++ projects have a lot of header files that are included by most or all CPP files and the redundant parsing of these is the main cost of compilation, especially if you have many short source files. Precompiled header files can help with this cost, but usually there are a lot of header files which are not precompiled.
The next main reason that unity builds improve build speeds is because the compiler is invoked fewer times. There is some startup cost with invoking the compiler.
Finally, the reduction in redundant header parsing means a reduction in redundant code-gen for inlined functions, so the total size of object files is smaller, which makes linking faster.
Unity builds can also give better code-gen.
Unity builds are NOT faster because of reduced disk I/O. I have profiled many builds with xperf and I know what I'm talking about. If you have sufficient memory then the OS disk cache will avoid the redundant I/O - subsequent reads of a header will come from the OS disk cache. If you don't have enough memory then unity builds could even make build times worse by causing the compiler's memory footprint to exceed available memory and get paged out.
Disk I/O is expensive, which is why all operating systems aggressively cache data in order to avoid redundant disk I/O.
I wonder if that ALL.cpp is attempting to put the entire project within a single compilation unit, to improve the ability for the compiler to optimize the program for size?
Normally some optimizations are only performed within distinct compilation units, such as removal of duplicate code and inlining.
That said, I seem to remember that recent compilers (Microsoft's, Intel's, but I don't think this includes GCC) can do this optimization across multiple compilation units, so I suspect that this 'trick' is unneccessary.
That said, it would be curious to see if there is indeed any difference.
I agree with Bruce; from my experience I had tried implementing the Unity Build for one of my .dll projects which had a ton of header includes and lots of .cpps; to bring down the overall Compilation time on the VS2010(had already exhausted the Incremental Build options) but rather than cutting down the Compilation time, I ran out of memory and the Build not even able to finish the Compilation.
However to add; I did find enabling the Multi-Processor Compilation option in Visual Studio quite helps in cutting down the Compilation time; I am not sure if such an option is available across other platform compilers.
In addition to Bruce Dawson's excellent answer the following link brings more insights onto the pros and cons of unity builds - https://github.com/onqtam/ucm#unity-builds
We have a multi platform project for MacOS and Windows. We have lots of large templates and many headers to include. We halfed build time with Visual Studio 2017 msvc using precompiled headers. For MacOS we tried several days to reduce build time but precompiled headers only reduced build time to 85%. Using a single .cpp-File was the breakthrough. We simply create this ALL.cpp with a script. Our ALL.cpp contains a list of includes of all .cpp-Files of our project. We reduce build time to 30% with XCode 9.0. The only drawback was we had to give names to all private unamed compilation unit namespaces in .cpp-Files. Otherwise the local names will clash.