I have been learning Chapel with small programs and they are working great. But as a program becomes longer, the compilation time also becomes longer. So I looked for the way to compile multiple files one by one, but not with success yet. By searching the internet, I found this and this pages, and the latter says
All of these incremental compilation features are enabled with the new --incremental flag in the Chapel compiler, which will be made available in Chapel 1.14.0 release.
Although the Chapel compiler on my computer accepts this option, it does not seem to generate any *.o (or *.a?) when compiling a file containing only a procedure (i.e. no main()). Is this because the above project is experimental...? In that case, can we expect this feature to be included in some future version of Chapel?
(Or, the word "incremental compilation" above is not what I'm expecting for usual compilers like GCC?)
My environment: Chapel-1.14.0 installed via homebrew on Mac OSX 10.11.6.
The Chapel implementation only fully compiles code that is used through the execution of the main() routine. As a starting foray, the incremental compilation project tried to minimize the executable difference between code compiled through normal compilation and code compiled with the --incremental flag. This was to ensure that the user would not encounter a different set of errors when developing in one mode than they would the other. As a consequence, a file containing only a procedure would not be compiled until a compilation attempt when that file/procedure was used.
The project you reference was an excellent first start but exposed many considerations to the team which we had not previously considered (including the one you have raised). We're still discussing the future direction of this feature, so it isn't entirely clear what that would entail. One possible expansion is "separate compilation", where code could be compiled into a .o or .a which could be linked to other programs. Again, though, this is still under discussion.
If you have thoughts on how this feature should develop, we would love to hear them via an issue on our Github page, or via our developers or users mailing lists.
Related
PROBLEM SUMMARY
We took a crack at using cotire, the Compile-Time Reducer, as our precompiled header system due to the extremely long compile times caused by usage of the Boost C++ template library. We are getting poor to dangerous results-- the precompiled headers seem to be constantly rebuilt, and are occasionally masking build problems. Specifically, back-to-back builds result in differing hashes of the actual cotire precompiled headers themselves, and, when attempting incremental builds, the precompiled headers are rebuilt every time, even when no header material has changed.
BACKGROUND
The project is a Linux CMake build using g++ 6.3.1, which produces, as its artifacts, an installable shared-library and several executables. Possibly of note, we have no interest in using unity builds, due to constant requirements for rapid-iteration development. However, the project is very large, hence the interest in cotire.
It is worth noting that, to avoid unexpected interactions while troubleshooting this issue, we have disabled ccache, the compiler cache. (Our intent is to eventually enable ccache if and only if we get consistent, expected results from cotire, or after we give up and remove cotire.)
The project uses C++03 (though building with 11 did not change our results for purposes of this question) and relies extensively on the Boost C++ library. Specifically, we make constant use of Boost Signal-Slot, boost::bind, boost::function, and quite a few of the iteration capabilities. Boost implements all of these via extensive template meta-programming and variant self-inclusion.
It is possible, though not entirely straightforward, to "blacklist" header files from cotire, excluding them from the generated precompiled headers. We attempted to blacklist various subdirectories within Boost itself (Boost includes some 900 different header files, so we mostly restricted ourselves to its immediate subdirectories). This seemed to produce improved results-- blacklisting certain parts of Boost resulted in back-to-back clean builds that produced matching precompiled header hashes.
Unfortunately, after a more significant delay-- perhaps ten minutes-- a third attempt to do a clean build resulted in a different hash for the precompiled header.
At this point, our working theory was that various special preprocessor symbols such as __TIME__ and __DATE__ were somehow getting mangled into cotire's input. Searching for these in Boost's headers does indeed produce a number of hits in wave, spirit, etc. Presumably this causes cotire to believe that the headers have changed in some way, because it attempts to rebuild the entire precompiled header with every build (though not with every compilation unit; we tentatively believe we have it integrated into the build process correctly).
QUESTION
Has anyone successfully used cotire with gcc/g++ and Boost? Were any unusual steps required, as opposed to using cotire in a project without Boost?
We are interested in cotire specifically; results from, for example, Visual Studio's precompiled header system might be interesting but are unlikely to be helpful. However, if you are troubleshooting similar issues and wish to provide your observations here, please feel free to do so.
I have shared code given to me that compiles on one linux system but not a newer system. The error is uint32_t does not name a type. I realize that this is often fixed by including the <cstdint> or stdint.h. The source code has neither of these includes and I am trying to seek an option that doesn't require modifying due to internal business practices that I can't control. Since it compiles as is on one machine they don't want changes to the source code.
I am not sure if it matters but the older system uses gcc 4.1 while the newer one uses gcc 4.4. I could install different versions of gcc if needed, or add/install library/include files on the newer machine, I have full control of what is on that machine.
What are my options for trying to compile this code on my machine without modifying the source? I can provide other details if needed.
I am not sure if it matters but the older system uses gcc 4.1 while the newer one uses gcc 4.4
GCC stopped including <stdint.h> some time ago. You now have to include something to get it...
I realize that this is often fixed by including the <cstdint> or stdint.h. The source code has neither of these includes and I am trying to seek an option that doesn't require modifying due to internal business practices that I can't control...
I hope I am not splitting hairs... If you can't modify the source files, then are you allowed to modify the build system or configuration files; or the environment? If so, you can use a force include to insert the file. See Include header files using command line option?
You can modify Makefile to force include stdint.h. If the build system honors CFLAGS or CXXFLAGS, then you can force include it in the flags. You last choice is probably to do something like export CC="gcc -include stdint.h".
The reason I am splitting hairs is OpenSSL and FIPS. The OpenSSL source files for the FIPS Object Module are sequestered and cannot be modified. We have to fallback to modifying supporting scripts and the environment to get some things working as expected.
If you really don't want to amend the file you could wrap it. Suppose it's called src.c create a new file src1.c:
#include <stdint.h>
#include "src.c"
And then compile src1.c.
PS: The problem may arise because compilers include other headers in their header files. This can mean some symbols 'officially' defined in other headers are quietly defined when you include a header that isn't specified as including it.
It's an error to write a program relying on a symbol for which the appropriate header hasn't been included - but it's easy to do and difficult to spot.
A changing compiler or version sometimes reveals these quiet issues.
Unfortunately, you can't force your code to work on a newer compiler without modifying something.
If you are allowed to modify the build script and add source files to the project, you might be able to add another source file to the project which, in turn, includes your affected file and headers it really needs. Remove the affected source files from the build, add the new ones, and rebuild.
If your shared source is using macro magic (e.g. an #include SOME_MACRO, where SOME_MACRO can be defined on the command line), you might be able to get away with modifying build options (to define that macro for every compilation of each file). Apart from relying on modifying the build process, it also relies on a possible-but-less-than-usual usage of macros in your project.
It may be possible to modify the standard headers in your compiler/library installation - assuming you have sufficient access (administrative) to do so. The problem with this is that the problem will almost certainly re-emerge whenever an update/patch to the compiler/library is installed. Over time, this approach will lock the code into relying on an older and older compiler/library that has been superseded - no ability to benefit from compiler bug fixes, evolution of standards, etc. This also severely limits your ability to share the code, and ability of others to use it - anyone who receives the code needs to modify their compiler/library installation.
The basic fact, however, is that your shared code relies on a particular implementation (compiler/library) that exhibits non-standard behaviour. Hence it has failed with an update of that implementation - which removed those non-standard occurrences - it is likely to fail with other implementations (porting to different compilers in future, etc). The real technical solution is to modify the source, and #include needed headers correctly. The real business solution is to make a business case justifying the need for such modifications, citing inefficiency - which will grow over time - in terms of cost and effort needed to maintain the shared code whenever it needs to be ported, or whenever a compiler is updated.
look at the second last line of code above your error, you'll find everything above that terminates with a , and only use a ; on the last entery
I have a rather complex SCons script that compiles a big C++ project.
This gcc manual page says:
The compiler performs optimization based on the knowledge it has of the program. Compiling multiple files at once to a single output file mode allows the compiler to use information gained from all of the files when compiling each of them.
So it's better to give all my files to a single g++ invocation and let it drive the compilation however it pleases.
But SCons does not do this. it calls g++ separately for every single C++ file in the project and then links them using ld
Is there a way to make SCons do this?
The main reason to have a build system with the ability to express dependencies is to support some kind of conditional/incremental build. Otherwise you might as well just use a script with the one command you need.
That being said, the result of having gcc/g++ optimize as the manual describe is substantial. In particular if you have C++ templates you use often. Good for run-time performance, bad for recompile performance.
I suggest you try and make your own builder doing what you need. Here is another question with an inspirational answer: SCons custom builder - build with multiple files and output one file
Currently the answer is no.
Logic similar to this was developed for MSVC only.
You can see this in the man page (http://scons.org/doc/production/HTML/scons-man.html) as follows:
MSVC_BATCH When set to any true value, specifies that SCons should
batch compilation of object files when calling the Microsoft Visual
C/C++ compiler. All compilations of source files from the same source
directory that generate target files in a same output directory and
were configured in SCons using the same construction environment will
be built in a single call to the compiler. Only source files that have
changed since their object files were built will be passed to each
compiler invocation (via the $CHANGED_SOURCES construction variable).
Any compilations where the object (target) file base name (minus the
.obj) does not match the source file base name will be compiled
separately.
As always patches are welcome to add this in a more general fashion.
In general this should be left up to the program developer. Trying to compile all together in an amalgamation may introduce unintended behaviour to the program if it even compiles in the first place. Your best bet if you want this kind of optimisation without editing the source yourself is to use a compiler with inter-process optimisation like icc -ipo.
Example where an amalgamation of two .c files would not compile is for example if they use two identical static symbols with different functionality.
I am currently developing a C++ command line utility to be distributed as an open-source utility on Github. However, I want people who download the program to be able to easily compile and run the program on any platform (specifically Mac, Linux, and Windows) in as few steps as possible. Assuming only small changes have to be made to the code to make it compatible with the various platform-independent C++ compilers (g++ and win32), how can I do this? Are makefiles relevant?
My advice is, do not use make files, maintaining the files for big enougth projects is tedious and errors happen sometimes which you don't catch immediatly (because the *.o file is still there).
See this question here
Makefiles are indeed highly relevant. You may find that you need (at least) two different makefiles to compensate for the fact that you have different compilers.
It's hard to be specific about how you solve this, since it depends on how complex the project is. It may be easiest to write a script/batchfile, and just document "Use the command build.sh on Linux/Unix, and build.bat on Windows") - and then let the respective files deal with for example setting up the name of the compiler and flags, etc.
Or you can have an include into the makefile, which is determined by the architecture. Or different makefiles.
If the project is REALLY simple, it may be just enough to provide a basic makefile - but it's unlikely, as a compile of x.cpp on Linux/MacOS makes an object file is called x.o, on windows the object file is called x.obj. Libraries have different names, dll's have differnet names, and on Linux/MacOS, the final executable has no extension (typically) so it's called "myprog", where the executable under windows is called "myprog.exe".
These sorts of differences mean that the makefile needs to be different.
Now almost an every user have a 2 or 4 cores on desktop (and on high number of notebooks). Power users have 6-12 cores with amd or i7.
Which x86/x86_64 C/C++ compilers can use several threads to do the compilation?
There is already a 'make -j N'-like solutions, but sometimes (for -fwhole-program or -ipo) there is the last big and slow step, which started sequentially.
Does any of these can: GCC, Intel C++ Compiler, Borland C++ compiler, Open64, LLVM/GCC, LLVM/Clang, Sun compiler, MSVC, OpenWatcom, Pathscale, PGI, TenDRA, Digital Mars ?
Is there some higher limit of thread number for compilers, which are multithreaded?
Thanks!
Gcc has -flto=n or -flto=jobserver to make the linking step (which with LTO does optimization and code generation) parallel. According to the documentation, these have been available since version 4.6, although I am not sure how good it was in those early versions.
Some build systems can compile independent modules in parallel, but the compilers themselves are still single-threaded. I'm not sure there is anything to gain by making the compiler multi-threaded. The most time-consuming compilation phase is processing all the #include dependencies and they have to be processed sequentially because of potential dependencies between the various headers. The other compilation phases are heavily dependent on the output of previous phases so there's little benefit to be gained from parallelism there.
Newer Visual Studio versions can compile distinct translation units in parallel. It helps if your project uses many implementation files (such as .c, .cc, .cpp).
MSDN Page
It is not really possible to multi-process the link stage. There amy be some degree of multi-threading possible but it is unlikely to give much of a performance boost. As such many build systems will simply fire off a seperate process for seperate files. Once they are all compiled then it will, as you note, perform a long single threaded link. Alas, as I say, there is precious little you can do about this :(
Multithreaded compilation is not really useful as build systems (Make, Ninja) will start multiple compilation units at once.
And as Ferrucio stated, concurrent compilation is really difficult to implement.
Multithreaded linking can though be useful (concurrent .o/.a reading and symbol resolution.) as this will most likely be the last build step.
Gnu Gold linker can be multithreaded, with the LLVM ThinLTO implementation:
https://clang.llvm.org/docs/ThinLTO.html
Go 1.9 compiler claims to have:
Parallel Compilation
The Go compiler now supports compiling a package's functions in parallel, taking advantage of multiple cores. This is in addition to the go command's existing support for parallel compilation of separate packages.
but of course, it compiles Go, not C++
I can't name any C++ compiler doing likewise, even in October 2017. But I guess that the multi-threaded Go compiler shows that multi-threaded C or C++ compilers are (in principle) possible. But they are few of them, and making new ones is a huge work, and you'll practically need to start such an effort from scratch.
For Visual C++, I am not aware whether it does any parallel compilation (I don't think so). For versions later than Visual Studio 2005 (i.e. Visual C++ 8), the projects within a solution are built in parallel as far as is allowed by the solution dependency graph.