I have a cpp cmake project that takes a bit to compile the core components.
Let's say
Component A (Takes 2 hours to compile)
Component B (Takes 1 hour to compile)
Component C needs to statically link in Component A and B (takes 5 minutes)
We mostly are only changing Component C.
But we want our PR Build gates to rebuild A and B if they have to, but not necessarily if there wasn't any changes.
I'd like to build component A and B once per night.
Then during the day, allow our PR gates to download the intermediates from them, and then do an incremental build, only rebuilding the parts that have changed.
However, cmake seems to embed a lot of folder path information into the intermediates/ cache stamp files and stuff. And our build machines have slightly different paths each time. (e.g. c:\repo###\sourceDir\
and the ### changes each time.
Is there any easy way to do this? Or do I have to 'modify all the relevant .txt, .tlog, cache text, etc.. files' and also manipulate the timestamps back to their originals after modifying the files path to match the current machine.
I tried just copying over the /intermediates/ folder with all the .obj and other things, but I think cmake is picky about the folderpaths.
And I don't want to consume the prebuilt .lib or .dll from component A or B, because we might have made a change to them, so it should do incremental in that case.
Edit/answers
I haven't looked into ccache yet, hadn't heard of it.
we evoke it via cmake.exe (configure/build/install) cmakelists.txt. Each component does have their own cmakelists.txt in a toplevelfolder
src/componentA/cmakelists.txt (controls everything in itself, has no ledge of B or C)
src/componentA/cmakelists.txt (controls everything in itself, has no ledge of A or C)
src/componentC/cmakelists.txt (controls everything in itself + include from A and B)
Related
As far as I know, CMake checks the time stamp of a source file to detect if it is outdated and needs to be rebuild (and with it, all files including it). When switching branches in a large git repository, this can causes problems.
Let's say I have one source folder and two build directories (build1 and build2), which correspond to two different branches (branch1 and branch2)
project
+-- src
+-- branch1_build
+-- branch2_build
Say my two branches have few differences, in few files; mostly, they only differ for some configuration option, all encapsulated in a config.h file, generated by the CONFIGURE_FILE command in cmake. The source files for the two config.h files (the config.h.in, as it is often called) are different. For instance, one branch introduces a new subfolder, which can be activated with a config-time option, which gets put in config.h.in with something like #cmakedefine HAVE_NEW_FEATURE_FOLDER. In such a scenario, when switching branches in the source folder, this happens: cmake recognizes that something changed in the config.h.in file, so it runs again; by running again, it generates a new config.h file; since config.h has a new time stamp, all files that includes it (directly or indirectly) end up being recompiled.
Now, if I alternatively switch between branch1 and branch2 in the source folder (cause I'm working on both branches every day), two consecutive make commands issued in the same build folder (either branch1_build or branch2_build) will trigger a full recompilation, since, although config.h has not changed in content, its time stamp has changed, so cmake flags it has changed.
My question is: what options do I have to avoid this? Or, better phrased, how can I avoid recompiling a source-build tree pair that is in fact unchanged since the last build, while also minimizing the changes to the source code?
The only solution I can think of is to execute CONFIGURE_FILE on config.h.in, with output config.h.tmp; compare config.h.tmp with config.h, and, only if different, copy config.h.tmp to config.h. However, this seems clumsy, and overcomplicated. I hoped cmake already had a mechanism for this, perhaps hidden under some options/variations of CONFIGURE_FILE...
Assuming this is not yet possible, I was wondering how complicated it would be for cmake to check the sha (rather than the timestamp) of a particular file when deciding whether it is outdated or not, and comparing it with the sha of a previous build (yes, the word outdated has date in it, but let's not get into enlish vocabulary discussions here). I imagine this is more expensive, so I would think that, if possible at all, this behavior should not be the default, and the user should use sparingly this feature, by explicitly tagging a file as check_sha_not_time kind of file. In the example above, the user would tag config.h as check_sha_not_time, and avoid recompilation of pretty much the whole library.
Note 0: I know little of how cmake internally works, so my suggestion of using sha rather than timestamp could be completely crazy and/or impossible given cmake implementation. I apologize for that. But that's why one asks things here, cause he/she doesn't know, right?
Note 1: I also tried using ccache, but unsuccessfully. Perhaps I need to use some particular flag or configuration option in ccache to trigger this capability.
Note 2: I want to avoid duplicating the source folder.
The title of this question does not begin to capture my years of exasperation with the RPM system. There is a vast gulf between a development system (./configure; make; make install;) and a rpm system (tar files, patch files, spec files, arcane build scripts, environments and tools) which I cannot bridge.
All I want to do is to change a few lines of code in a bigger program.
The problems which I run into:
Getting the source code of the system as-installed (e.g. SRPM from EPEL, original tarball, something else). What source should I use?
Getting that source code into a ready-to-edit form - something that I can edit with my favourite editor. How can I know that I'm editing the code as-deployed, bugs and all? (rpm -ivh x.src.rpm gives me tar files and squabs of patch files littered about in the SOURCES directory ... how can I get it right?)
Editing the code to implement some amazing hack (this part I can actually do).
Compiling the amazing code as edited - just compiling it in-place. Usually I can get this right, but it would be nice to have a hand sometimes, e.g. with ./configure set to use something other than the default /usr/local and /lost+found/opt/etc/opt or whatever crazy default autoconf decides to use.
Transforming my edits into a patch against the previous source and building new RPMs to test on some remote system (this is the great promise of RPM - pristine sources and hacky patches). If I do a diff of the original and the edited directories, the resulting patch contains all sorts of rubbish that I don't want to delete because I'm still developing (e.g. object code). (Actually, I don't have an 'original' at this point to do a diff against ... because I was only looking at the code casually when I realised I could "improve" it ...) Should I use some revision control system to track the changes I am making?
This should be simple stuff, but somehow all I can do is edit the code. After I have edited the code, it can never get over the hump, even though it is an already-solved problem. I have a GREAT fix for an open source project, but every single time that I finish developing my amazing hack, having delved into the code and made it compile (and possibly work), I am completely stumped. Nothing at all can turn my modified and now amazing source tree into a RPM. I end up deploying source code (into /usr/local), because that at least works.
How do people who do (say) security fixes actually go about the extract-edit-compile-test loop?
The SRPM is (relatively) self-contained: there are often some assumptions about build requirements not reflected in the spec file.
I would start by taking the SRPM, and rebuilding it to address the issue of build-requirements (adding whatever is needed to get it to build).
Then, extract the spec-file and sources from the SRPM, putting the patches and tar-file(s) into ~/rpmbuild/SOURCES, and building from the spec-file
Next, modify the spec-file to add my own patch file (or scripting changes),
Finally there's a new SRPM with my changes.
For extracting, I use an unrpm script (essentially a wrapper around cpio) which can be found on the network.
Making your own patch file is discussed here:
HowTo Create A Patch File For A RPM
RPM - Creating Patches
Patches for .spec file
Based on several kits I am using, the Qt Creator build directory is derived from
project name
Qt version
compiler
build configuration: debug / release
Example: build-client-Qt_5_3_0_fooProject_MINGW_32bit_qmake_MinGW22-Debug
My problem is, if I switch my git branch, I always have to recompile from scratch (time consuming). I wonder, if it might be possible to include the branch name, so I can keep different builds at the same time
=> Possible build dirs:
build-client-Qt_5_3_0_fooProject_MINGW_32bit_qmake_MinGW22-Debug_master
build-client-Qt_5_3_0_fooProject_MINGW_32bit_qmake_MinGW22-Debug_branchbar
Or is there a chance to add a third build configuration, e.g. Debug_others, resulting in a build directory such as
build-client-Qt_5_3_0_fooProject_MINGW_32bit_qmake_MinGW22-Debug_others
You can manage builds in Project tab on the left. If you want to have build automaticaly switching dirs depending on the branch you are on, on each branch you would have to edit *.pro.user file. Find line:
<value type="QString" key="ProjectExplorer.BuildConfiguration.BuildDirectory">
/home/etc/etc/etc
</value>
And change the directory to the one you want it to be (ex. edit the ending part of the value). On a different branch have it point somewhere else. Thus changing branches will change the shadow build directory.
P.S. If you have several builds (like debug, release) you need to check if the build directory you're editing is the one you want. Check activeBuildConfiguration number and edit it's directory only (the number of build is usually a line above the one I showed), or change activeBuildConfiguration to 0 and the first build directory.
It's a bit knotty but I think manageable.
I use QMake to gen *.vcproj (from *.pro) to create DLLs that expose C++ classes to QML (Qt5.0.1/MSVC2010). The DLLs work.
However, I have two problems (they might be related):
When I "build", it ALWAYS recompiles everything in the DLL. (It's never "up-to-date".)
If I get a "link-fail" (for example, some input lib doesn't have a needed-function defined), MSVC will delete all the *.obj and other intermediate files, requiring the next build to (unnecessarily) rebuild those intermediates. (The "link-success" will LEAVE the intermediate *.obj files, but the next build will force them to be regenerated anyway.) This means I can never do a "Project Only => Link Only" when I fix an "input-static-lib".
Both of these seem weird. And, they make me sad.
Note that since I "regenerate" the .vcproj each time, there are no "files-in-the-project" that do not exist on disk (which others observe as a cause to always-trigger-rebuild).
I've already tried the "silly-stuff" like re-installing service packs, clean builds, manually removing all intermediates/artifacts, reboot, etc.
From web searches (lots of reading for weeks to get around this), this might be related to:
Each DLL has its own directory for its "intermedate-OBJs", but they all write to the same (shared-debug)-"output" folder for their .dll/.lib. (That should be fine, right?)
Failing a "proper-fix" (where MSVC realizes that things are up-to-date), is there a work-around where I can suppress the delete-intermediate-files so I can later force a mere "re-link", when I know that things should be fine? At present, I can't "Project Only => Link Only" because the intermediates get deleted. (How to stop MSVC from deleting the .obj files?)
I have been trying to set up an EDE project for C++ (emacs24 + builtin CEDET) and I'm starting to get desperate because I can't seem to find the way I want the makefiles to be generated. I'm relatively new to Emacs.
I'll try to describe what I'm doing:
I have a toy project set like so:
main.cpp
other/
Utils.cpp
Utils.h
CGrabBuffer.cpp
CGrabBuffer.h
main.cpp includes both .h's inside the "other/" directory. These are the steps I follow to set up an EDE project with this simple directory setup:
Open main.cpp in emacs and do M-x ede-new ; type: Make ; name: main-proj.
Open one of the files in the "other" directory and do M-x ede-new ; type: Make ; name: aux-proj.
Now it's time to create the targets (which I believe are three in this case):
On the main.cpp buffer: M-x ede-new-target ; name: main ; type: program. When prompted, I add the main.cpp to this target.
I repeat the same for the other two targets (Utils which has Utils.cpp and Utils.h and CGrabBuffer which has CGrabBuffer.cpp and CGrabBuffer.h). Here I find the first problem. What type do these two targets have to be? I only want them to generate .o files.
Once this is done, I type M-x ede-customize-current-target to all three targets and I add some include paths, some libraries, etc.
After this, if I call M-x ede-compile-project it doesn't compile because:
It tries to compile main.cpp first; I have no idea how to specify (using EDE) that both Utils.o and CGrabBuffer.o are needed before attempting to build main.cpp.
If I manually change the order (editing the Makefile), it's not able to link main.cpp because it can't find Utils.o and CGrabBuffer.o.
As you can see, I am in the middle of a great mess. Maybe I'm not even understanding what "target" means in EDE. I have also read about the existence of ede-cpp-root-project which has to be specified inside the .emacs file. I haven't tried it because what I think it does is just help with the semantics. It doesn't generate Makefiles, does it? Can I have (or do I need) an EDE project built with Project.el's and the same thing using ede-cpp-root-project for the semantics? Or is it redundant?
Sorry If I misunderstood a lot of things but I'm very confused and being new to emacs makes things worse. Thanks for your patience!
EDIT: with some tinkering and the responses I received I have been able to figure out a lot of stuff, so thanks a lot. What I still don't understand is the use of the ede-cpp-root-project which has to be specified inside the .emacs file. Is it just for c++ semantics? Is it redundant to have the project with Project.el's AND also the elisp lines in .emacs?
EDE is designed to handle many different kinds of projects, usually of a type where the build system was written outside of Emacs in some other tool.
The EDE project type that creates Makefiles for you can do quite a few things, but you need to have some basic understanding of build systems for it to be helpful, and you really do need to customize the projects to get anything of any complexity working.
I've recently added a section to the EDE manual to help with basic project setups that autogenerate Automake files. You can check out the tutorial here:
http://www.randomsample.de/cedetdocs/ede/ede/Quick-Start.html
The same steps will apply for projects that just use Make instead, but Make based projects often have trouble with shared libraries due to the extra complexity.
Mike's answer is quite good, but I think it is ok to just add .h files to the same target as your .cpp sources. It will keep track of them separately.
Another useful trick is to use the whole project compile keystroke (C-c . C) which uses a capital C whenever you change something big. That will regenerate the Makefiles, rerun any needed Automake features, and start at the top.
EDIT: You only need one EDE project for a give project area. The ede-cpp-root project is useful when no other automatic project type works. That's when you create that in your .emacs file so that the other tools that need a project definition, like semantic's smart completion, and tag lookup, will work.
Well, I think I actually have it figured out this time, but it's ugly. Utils.cpp and CGrabBuffer.cpp should not get their own individual targets, because there doesn't seem to be an appropriate target type. Instead, you'll need to create an archive or library, which will automatically compile Utils.cpp and CGrabBuffer.cpp for you. Below, I'll assume you want static, but it's easy to change.
[For anyone to whom archives or libraries are not familiar, they basically just gather up .o files into a separate unit. It doesn't actually make the compilation harder. Read more here.]
1) Follow the first two and a half steps above (including making the main target, but not the other targets).
2) Switch to Utils.cpp and do M-x ede-new-target ; name: aux ; type: archive. When prompted, add Utils.cpp to this target.
3) Switch to CGrabBuffer.cpp and do C-c . a ; Target: aux .
4) Regenerate the Makefile with M-x ede-proj-regenerate. At this point, if you run make in the other subdirectory, you should get the archive libaux.a.
5) Switch back to main.cpp and do M-x ede-customize-current-target. This brings up an interactive emacs customization buffer, which allows you to edit details of the ede configuration. Under the Ldflags section, click [INS]. This pops out a new line that says Link Flag: and has some different-colored box for you to type in (mine is grey). Type -Lother -laux, so that other/libaux.a is included when compiling main. Then, at the top of the buffer, press [Accept], which should save that change and switch back to main.cpp.
6) Regenerate the Makefile with M-x ede-proj-regenerate.
Now, unfortunately, the Makefile makes the main target first, then descends into the other directory and makes that. Unfortunately, this means that a make from the top-level directory will not work on a clean tree. I don't know why this is, because it seems like that would never be what you want in any project that is ever made with EDE. I can't find any way to change that, except for this hack:
7) Do M-x customize-project; under Inference-Rules click [INS]. Then enter Target: all ; Dependencies: aux main ; Rules: [INS] ; String #: . (This last one is just to prevent an error on an empty rule with a tab; presumably an EDE bug.) Click [Accept], and regenerate the Makefiles.
So now, in your top directory, you can just run make, and main should be a working executable.
I'm quickly becoming convinced that EDE is not yet ready to be used by people other than its authors. Despite its size and the amount of effort they've clearly put into it, it is too buggy, too counterintuitive, and just not smart enough. That's a shame. Emacs needs something like this.