I am working with a large C++ project (100K lines), and every time I change a line(s) I need to re-build it. Due to its size, it takes quite a while and that's hitting my performance. So, my question is: if there is any way to speed up the build time in Visual Studio in order to selectively re-compile only those files which actually changed. And thus re-using all the other unchanged compiled files (object files). And then, obviously linking.
I know I should be able to do so directly from the command line as Visual Studio is actually doing it "behind the scenes". Though the "Property Pages" display the Command Lines for compile and link, I think it would get quite troublesome as these commands is so long that I will probably make edit mistakes. Thanks in advance for sharing your experience on how to deal with this kind of situation.
Related
I've got a problem for which I would very much like a solution please:
All I'm trying to do is use wxWidgets in Visual Studio 2012 (in C++), and make it accessible to my code project. That shouldn't be so hard, should it? But as usual, even though there are programming tutorials for it, wxWidgets is evidently one of those things that has no proper documentation just to simply get it running, or if it does then I sure can't find it.
I downloaded the installer for Windows and installed it. When it finished, it gave me a message that said:
wxWidgets was installed successfully!
Please note: before using wxWidgets in your application,
you must build the library using the same compiler you
use for building your program.
Please see docs/msw/install.md file in the installation
directory for the detailed instructions about how to build
wxWidgets and use it from your application.
It should be noted that I think I remember something on the website or somewhere saying that I can just include a header file into my project and forego the DLL altogether (which is what I'd prefer to do, if possible), though I may possibly be misremembering that or confusing it with something else.
Anyway, when I opened the docs/msw/install.md file, I found (amongst other things) these instructions:
--- From the IDE
Ready to use project files are provided for VC++ versions 8, 9,
10, 11, 12, 14, 15, 16 and 17 (also known as MSVS 2005, 2008, 2010, 2012,
2013, 2015, 2017, 2019 and 2022 respectively).
Simply open wx_vcN.sln (for N=8, 9, 10, 11, 12, 14, 15, 16 or 17) file,
select the appropriate configuration (Debug or Release, static or DLL)
and build the solution. Notice that when building a DLL configuration,
you may need to perform the build several times because the projects
are not always built in the correct order, and this may result in link
errors. Simply do the build again, up to 3 times, to fix this.
Of course, it doesn't even say WHERE the "wx_vcN.sln" files are, but I ran a file search and found them in C:\wxWidgets-3.1.6\build\msw and I opened the one called wx_vc11.sln, because that seems to be the one that corresponds to VS2012, right?
Anyway, it took a LONG time to load, because it had to keep scanning through thousands of files and so forth, but when it was finally finished, I did the "Rebuild All" command thrice in a row, like it says to do (which also took a while).
But I'm not sure where it even put the actual DLL file. It says that it's in C:\wxWidgets-3.1.6\build\msw....\lib\vc_lib\wxmsw31ud_xrc.lib, but where the hell is "...."? It doesn't even bother to specify the entire path! What kind of nonsense is that?! I just want the file so that I can put it into my project! Or better yet, just a .h file that takes care of everything would be great. Supposedly there's one to include called "wx/wx.h", which then includes everything else, I think, but I don't know if that one indirectly also requires a DLL.
I always have problems with DLLs, anyway, though I'll use one if I absolutely have to. You'd think it would be the easiest thing to import a DLL into a project, but for some reason, I've never been able to find any sort of tutorial that explains a specific, step-by-step way to do it which is guaranteed to work, so on the rare occasion that I actually need to use a DLL (99% of the time I can just rely on my own code and the .h files included with Visual Studio), I always end up muddling through it until it FINALLY works by sheer accident (if I'm lucky). Mostly, it just reports errors for one reason or another, that it's not properly included or compatible, or that the header files that I'm using which reference it keep showing the same errors, as though I hadn't even imported the DLL file at all! I've been programming for about 30 years, and I work on things like AI and compression and cryptographic algorithms. I don't say this to brag, but to point out that if someone as experienced as me has trouble getting a DLL to attach properly to a roject, then there's something severely wrong with the whole process, or the documentation for it! I HATE using DLLS, because it's always a HUGE chore, but I'll do it if there's absolutely no other way (though I'd appreciate step-by-step instructions, if you don't mind).
But enough about that. In any case, I have an example project in which wxWidgets is being used in conjunction with Vulkan, and though it has compilation errors (because Vulkan didn't properly install on my machine (when I try to run the sample programs it says they're missing DLLs - big surprise) and as it turns out, I don't think it's compatible with my video card, anyway (which might possibly be why it didn't install properly), so I think I'll have to resort to OpenGL instead - PLEASE tell me that's compatible with wxWidgets!), but even in that example project, it has errors where it's trying to include and use wxWidgets stuff (like "wx/wx.h"), even though that's supposed to be a complete and already working project! I can't find any DLL files or the header files in question anywhere in the project directories!
So anyway, there's got to be an easier way to get this all set up, just to the point that I can make a simple test program using wxWidgets, and compile it. I'd REALLY like to be able to get at least that much working sometime tomorrow morning, so I'd greatly appreciate any help you can give me! Thank you.
#ThisIsMe,
First of all you need to educate yourself on properly using C++.
When you acquire the C++ code it contains of 2 portion - header file and source code.
Header file is what you call a declaration you declare you classes there - class name, member variable and method.
In the source files You define what you declared and you put an actual code there.
Now with that in mind:
wxWidgets contains both header and source files. When you downloaded the installer and did the installation it just copied the "text file" (headers and sources) into the folder of you choice.
Now as you can imagine text files are not executable - you need to compile them into the binary files And that is exactly what you did. (I don't know why you did it 3 times though - I guess you got an errors after building 1st and 2nd time).
Now, you said you have a project that (either uses wxWidgets or you want to incorporate wxWidgets into - which one is it, BTW?). The answer to this question will help direct you to a proper solution.
Now, you said that you tried the Vulkan and it fails to run the demo. How did it fails? Did it produce an error? What was it? Did it even run?
Now wxWidgets does support OpenGL as can be seen by looking at the documentation and checking the different samples that comes with the library, but I'm just curious if what you are seeing from Vulkan is really an incompatibility with the Video Card.
So, we need you to answer the questions I put and so we can help you further. Just keep in mind - C++ is not for the beginner programmers and if you are struggling with understanding the difference between compilation, installation , headers and sources - it will be very hard for you.
Maybe you should start with something simpler - like wxPython and forget about C++ for now.
I have a console application written in C/C++. Usually it takes, 5-10 minutes to get compiled on non-windows platforms even though optimization flag is set to -o3. But it takes approximate 1-2 hours to get compiled on Windows platform when optimization flag is set to Full Optimization (/Ox) and Inline Function expansion is set to Any Suitable (/Ob2) in visual studio. This happens in release/debug mode both.
I understand compiler is trying to optimize the code hence it is bound to take more time but isn't it too much time compare to time taken by other compilers(mainly g++) on non-windows platforms.
So far I tried..
Removed unnecessary headers from source and header file, introduced forward declarations wherever possible but no respite.
I analyzed the all header files. Templates are used hardly in 2-3 header files out of ~50 header files in project. These headers are also not widely included in the source files.
I've two observations from this behaviour -
There is nothing terribly wrong in the source code otherwise compilers on non-windows platforms would not be able to finish so quick.
Seems VS compiler genuinely taking more time(1-2 hours) which other compilers are able to do in(10 minutes) but VS compiler can't be that bad. Therefore, I must be missing to change some configuration (apart from optimization).
Does anyone has idea how to find out what is going wrong here ? May be starting point will be to identify compilation time taken by each file. How do I find compilation time of each file ?
Could there be possibility if I can still improve/try something ?
Here are additional details about hardware, source code etc as requested in some of comments
RAM - 8.0 GB RAM
OS - Windows 7 64 bit
Processor - Intel Core i5 2.6 GHz
Visual Studio - 2013 Ultimate
Note - If I disable optimization (set /Od and /Ob0 flags in VS) then program compiles in less than 5 minutes on the same machine.
Source files - approx 55, header and source files each and 80KLOC code.
Does anyone has idea where to start to find out what is going wrong here ?
Not without more details.
Could there be possibility if I can still improve/try something ?
Yes. In particular, consider:
removing any templated code from header files (both included and defined in the .h file), and accessing that code through pimpl (because templates are reevaluated on each pass).
optimizing your usage of precompiled headers
splitting your console application into separate modules (so your build system will only update the dirty binaries on build)
Based on suggestions received in comments, I started by finding out compilation time taken by each file:
Clean project to ensure all *.obj files are removed
Build the project again
Noticed the time stamp for each file and I found a file which was
taking almost two hours to compile.
When I open the source file I see something terribly wrong in the code in contrast to my observation. It is a huge monster file of 27KLOC (Opps!, of course I didn't write this file).
There are 739 instances of a class created dynamically and assigned to an array. Each instance in turn dynamically creates some of its members as well. In shorts thousands of objects are being created in this file.
To ensure that this file is the culprit and VS studio is taking way too much optimizing this file. I disabled the optimization
in this file as proposed by #Predelnik in comments. Voilà! program compiles within couple of minutes now. This source code needs a serious re-factoring.
If someone is facing such problems, I would go as following -
Enable Build-And-Run option and /MP flag. As discussed Here. If there is some problem with the code the parallel projects and file compilations would not help.
Find out if any source file is the culprit as above. I believe the link I found Here is a way to calculate the build time not compile time of each file.
I use Visual C++ 2012 with a project that makes a heavy use of precompiled headers. So heavy that the infamous /Zm switch is in use.
When I cancel a build in progress, I sometimes get this error on the next build:
error C1852: 'foo.pch' is not a valid precompiled header file
Nine times out of ten, things will go smoothly, but when this happens I have to find the .pch and delete it manually before restarting the build.
That annoys me a bit. Is there a way to prevent this from happening? A patch from Microsoft? Or a way to force Visual to delete the .pch and restart the build automatically when the issue occurs? Or some other solution I didn't think about?
EDIT: Here's the version of Visual I'm running:
Microsoft Visual Studio Professional 2012
Version 11.0.61030.00 Update 4
This is a pure conjecture, as I did not run into this issue.
Try to find out how Visual detect a .pch file is corrupted (i.e. empty file, file not correctly ended, ...). If it follow a clear pattern, write a pre-build script that parse all .pch and delete corrupted ones.
I would create a script that would attempt to recompile the stdafx.cpp file, but this time using the PCH instead of generating it. I.e. the expected outcome is the successful compilation of an empty file. If this fails, delete the PCH. Now run this script as a pre-build step.
It sounds fairly expensive, but it's very reliable. Any problem loading the PCH causes its regeneration, even compiler upgrades. Also, your PCH is now in file cache, which means the actual use is slightly cheaper.
This might be implemented as an NMAKE build script with somewhat unusual rules.
I followed rockeye's suggestion of trying to find a pattern in these corrupted files. Turns out it's very simple: valid files start with a VCPCH0 header, corrupted files don't.
A simple C# program run as a Pre-Build Event of the failing project(s) and deleting the corrupted files solves the issue. If anyone's interested, the source is right here.
It been happening for a while now, that every time I make changes to my code and compile, the behavior of program won't reflect the changes I made. What i have to do to get a change is to clean my project and compile all the files again. Visual studio shouldn't be doing this and its starting get in the way of actually debugging my code to begin with as the program is large and it takes a while to compile.
How can I fix this so I don't have to go through this ritual of cleaning then compiling just to see changes.
Aren't your sources on some network/mounted drive? That may confuse the build.
My experience is that it can miss a dependency with some 1% chance on random files, but no more than that.
If they are not, in options/build&run you can set diag level of build to higher and see what steps are considered for build. Also you can look the discovered depencencies in the intermediate files folder, the .tlog files. You may also try to look at debug output as described here though most of the latter belongs to discovering excess compilations...
Why when I watch the build output from a VC++ project in VS do I see:
1>Compiling...
1>a.cpp
1>b.cpp
1>c.cpp
1>d.cpp
1>e.cpp
[etc...]
1>Generating code...
1>x.cpp
1>y.cpp
[etc...]
The output looks as though several compilation units are being handled before any code is generated. Is this really going on? I'm trying to improve build times, and by using pre-compiled headers, I've gotten great speedups for each ".cpp" file, but there is a relatively long pause during the "Generating Code..." message. I do not have "Whole Program Optimization" nor "Link Time Code Generation" turned on. If this is the case, then why? Why doesn't VC++ compile each ".cpp" individually (which would include the code generation phase)? If this isn't just an illusion of the output, is there cross-compilation-unit optimization potentially going on here? There don't appear to be any compiler options to control that behavior (I know about WPO and LTCG, as mentioned above).
EDIT:
The build log just shows the ".obj" files in the output directory, one per line. There is no indication of "Compiling..." vs. "Generating code..." steps.
EDIT:
I have confirmed that this behavior has nothing to do with the "maximum number of parallel project builds" setting in Tools -> Options -> Projects and Solutions -> Build and Run. Nor is it related to the MSBuild project build output verbosity setting. Indeed if I cancel the build before the "Generating code..." step, none of the ".obj" files will exist for the most recent set of "compiled" files. This implies that the compiler truly is handling multiple translation units together. Why is this?
Compiler architecture
The compiler is not generating code from the source directly, it first compiles it into an intermediate form (see compiler front-end) and then generates the code from the intermediate form, including any optimizations (see compiler back-end).
Visual Studio compiler process spawning
In a Visual Studio build compiler process (cl.exe) is executed to compile multiple source files sharing the same command line options in one command. The compiler first performs "compilation" sequentially for each file (this is most likely front-end), but "Generating code" (probably back-end) is done together for all files once compilation is done with them.
You can confirm this by watching cl.exe with Process Explorer.
Why code generation for multiple files at once
My guess is Code generation being done for multiple files at once is done to make the build process faster, as it includes some things which can be done only once for multiple sources, like instantiating templates - it has no use to instantiate them multiple times, as all instances but one would be discarded anyway.
Whole program optimization
In theory it would be possible to perform some cross-compilation-unit optimization as well at this point, but it is not done - no such optimizations are ever done unless enabled with /LTCG, and with LTCG the whole Code generation is done for the whole program at once (hence the Whole Program Optimization name).
Note: it seems as if WPO is done by linker, as it produces exe from obj files, but this a kind of illusion - the obj files are not real object files, they contain the intermediate representation, and the "linker" is not a real linker, as it is not only linking the existing code, it is generating and optimizing the code as well.
It is neither parallelization nor code optimization.
The long "Generating Code..." phase for multiple source files goes back to VC6. It occurs independent of optimizations settings or available CPUs, even in debug builds with optimizations disabled.
I haven't analyzed in detail, but my observations are: They occur when switching between units with different compile options, or when certain amounts of code has passed the "file-by-file" part. It's also the stage where most compiler crashes occured in VC6 .
Speculation: I've always assumed that it's the "hard part" that is improved by processing multiple items at once, maybe just the code and data loaded in cache. Another possibility is that the single step phase eats memory like crazy and "Generating code" releases that.
To improve build performance:
Buy the best machine you can afford
It is the fastest, cheapest improvement you can make. (unless you already have one).
Move to Windows 7 x64, buy loads of RAM, and an i7 860 or similar. (Moving from a Core2 dual core gave me a factor of 6..8, building on all CPUs.)
(Don't go cheap on the disks, too.)
Split into separate projects for parallel builds
This is where 8 CPUS (even if 4 physical + HT) with loads of RAM come to play. You can enable per-project parallelization with /MP option, but this is incompatible with many other features.
At one time compilation meant parse the source and generate code. Now though, compilation means parse the source and build up a symbolic database representing the code. The database can then be transformed to resolved references between symbols. Later on, the database is used as the source to generate code.
You haven't got optimizations switched on. That will stop the build process from optimizing the generated code (or at least hint that optimizations shouldn't be done... I wouldn't like to guarantee no optimizations are performed). However, the build process is still optimized. So, multiple .cpp files are being batched together to do this.
I'm not sure how the decision is made as to how many .cpp files get batched together. Maybe the compiler starts processing files until it decides the memory size of the database is large enough such that if it grows any more the system will have to start doing excessive paging of data in and out to disk and the performance gains of batching any more .cpp files would be negated.
Anyway, I don't work for the VC compiler team, so can't answer conclusively, but I always assumed it was doing it for this reason.
There's a new write-up on the Visual C++ Blog that details some undocumented switches that can be used to time/profile various stages of the build process (I'm not sure how much, if any, of the write-up applies to versions of MSVC prior to VS2010). Interesting stuff which should provide at least a little insight into what's going on behind the scenes:
http://blogs.msdn.com/vcblog/archive/2010/04/01/vc-tip-get-detailed-build-throughput-diagnostics-using-msbuild-compiler-and-linker.aspx
If nothing else, it lets you know what processes, dlls, and at least some of the phases of translation/processing correspond to which messages you see in normal build output.
It parallelizes the build (or at least the compile) if you have a multicore CPU
edit: I am pretty sure it parallelizes in the same was as "make -j", it compiles multiple cpp files at the same time (since cpp are generally independent) - but obviously links them once.
On my core-2 machine it is showing 2 devenv jobs while compiling a single project.