I have a project consisting of a set of makefiles that CANNOT be run with make --jobs=N because the dependencies are not specified tightly enough for make to correctly execute the recipes in correct order (ie I get race conditions).
I am currently using Huddle, by Electric-Cloud.com, and it does exactly what I need: it parses the makefile and then executes the jobs in parallel and accounts for the unspecified dependencies.
Question: is there a free or free-er thing that does this?
Yes I know I could re-write the makefiles but project management says "no way".
UPDATE #1
I understand that I'll have to do some work to get functionality similar to Electric-Cloud's functionality.
I know that Electric-Cloud parses the makefile(s) to find the dependencies so wouldn't the same thing be accomplished using makedepend?
I'm thinking:
Run makedepend on existing makefiles
Feed in the output using include <makedepend.output>
make all --jobs=64
UPDATE 2
Turns out makedepend is specific to C/C++: it merely runs the pre-processor on source files and parses any #include statements; not what I need.
I need what this guy is asking for:
Build a makefile dependency / inheritance tree
UPDATE 3
The makefile "dependency graph generator" actually already exists
http://plindenbaum.blogspot.com/2012/11/visualizing-dependencies-of-makefile.html?m=1
but that's not going to help me.
Many of my recipes create directories which are used by other targets' recipes, effectively making them implicit prerequisites.
The graph dependency tool at above URL works by parsing the build log's statements but those statements don't indicate the implicit dependencies.
Even if I try to run my makefile with --dry-run, the build fails because some of the recipes that aren't executed - cause it's a dry run - create directories that other invocations of make need simply to 'pretend execute' a recipe.
UPDATE 4
Electric-Cloud has made Huddle - 4 local cores, non-clustered - free for anyone forever.
Furthermore, they output an .xml file that lists each job's dependencies so I can use it to fix my makefiles compatible so they're compatible with the --jobs option.
I am currently using Huddle, by Electric-Cloud.com, and it does exactly what I need: it parses the makefile and then executes the jobs in parallel and accounts for the unspecified dependencies.
I actually don't know about these tools, but can't you provide them with a super makefile under your control, that clarifies the inner dependencies of the various targets?
You probably just have to add some indirection level for these (imported?) projects directory structure and another Makefile.
Related
We’re currently upgrading our archaic build system from a bunch of batch scripts to a makefile system using NMake. It’s challenging as we use a custom intermediate language that ends up getting translated to C++ where some of our translators can generate 10’s of files what have a common parts in the file names. The other challenging thing is we use a bunch of CSV files to configure our interfaces and these files get passed through to our configuration tools which generate more source code files. Right now I am focusing on creating the simple rules for our configuration files but can’t seem to figure out a way associate a dependency with a rule if the dependency exists. I tried to use $(wildcard xxx.csv) but found out that this command doesn’t exist for NMake like it does for GNU Make.
So how can I create my rule so that it executes and runs my commands if I have two dependency csv files that will always exists and a third csv file that will exist only when my project calls for it?
[..] will exist only when my project calls for it?
This is a bit unclear. Assuming that there is a command that - depending on some external circumstances - might generate that third csv file, you could use a "stamp file" (I think they call it "pseudo target" in NMAKE):
stamp:
command_that_might_generate_csv3
touch stamp # updates timestamp of "stamp" (or creates it)
target: csv1 csv2 stamp
command_using_all_of csv1 csv2 csv3
As of yet, at least to my knowledge, there is no standard way in CMake to specify the addition of a precompiled header (PCH) to a project in a cross-platform manner because the way PCHs are handled by C++ compilers is very different among vendors. For G++, this is usually this is worked around by simply adding a custom command which takes care of invoking the compiler with the appropriate input and has it generate the PCH.
My current problem is that CMake will not parse the dependencies of the dependencies you specify for the custom command. For instance, assume the following structure:
pch.h
|- dependA.h
|- dependB.h
...
Only providing pch.h as a dependency will lead to the generation of the appropriate target in the corresponding makefile, which tracks changes to pch.h. However, CMake does not parse the includes inside pch.h and will therefore not recognize changes to dependA.h and dependB.h. This extends furhter if there are dependencies for dependsA.h and so on.
Note: I'm aware that the fact that PCH dependencies can and do change regularly puts the whole process in question. However, this is just the way it is and I can't really do anything about it.
Since the task isn't too hard, there are a couple of obvious ideas that come to mind:
Solution A:
Enter all the dependencies by hand. Obviously this works, but is tedious as hell and doesn't scale at all.
Solution B:
If possible, write a CMake function that automates the process and parse the includes "manually".
Solution C:
Do something similar using a different language, for instance Python, and just provide CMake a list of dependencies to add to the custom command.
Solution D:
Use gcc/g++'s feature to parse and print out the dependency tree of the PCH and parse the output to extract the list of dependencies.
My question is: does anyone know a more convenient and faster way to get this done?
The IMPLICIT_DEPENDS option of the add_custom_command might do the trick:
add_custom_command(
OUTPUT outFile
COMMAND ...
IMPLICIT_DEPENDS CXX "pch.h")
The IMPLICIT_DEPENDS option makes the generated build system scan the implicit dependencies of the given input file at build time. It is only supported for Makefile generators, though.
I'm using SCons to build a very large project, with many buildable sub-projects. I can easily use keyword commands like scons group=ai to build the AI sub-projects with if statements (choosing the right SConscripts based on the keyword command), but I want to make it as easy as possible for others to use scons. Ideally, I'd like to use it like so: scons ai to build the AI components. However, the only single-word command functionality I've found in SCons so far is aliasing, and all the examples are about changing the target. This is not what I want. Since I have a very large project with multiple sub-SConscript files to build the subprojects, I want to call the SConscripts selectively. I've tried code like so:
env.Alias("ai", SConscript("ai/SConscript", 'env'))
but this calls the AI SConscript every time, regardless of whether I use the "ai" alias or a different one. Does anyone know if it is possible to use aliasing this way to selectively call SConscripts based on the alias?
As you mentioned, the Alias() function is only used for targets. I can think of 2 ways to solve this
Alias() can be called multiple times for the same alias with different targets, so you could call it for all targets in each SConscript, then you could build everything in a SConscript. Here's an example of what I mean:
ai/SConscript:
# targets, etc
env.Alias("ai", target1)
env.Alias("ai", target2)
...
env.Alias("ai", targetn)
Another option would be to put some logic in your root SConstruct so it only calls sub-project SConscript's based on a command line argument. This option would require you to use a command line argument of this form: group=ai
I have been trying to set up an EDE project for C++ (emacs24 + builtin CEDET) and I'm starting to get desperate because I can't seem to find the way I want the makefiles to be generated. I'm relatively new to Emacs.
I'll try to describe what I'm doing:
I have a toy project set like so:
main.cpp
other/
Utils.cpp
Utils.h
CGrabBuffer.cpp
CGrabBuffer.h
main.cpp includes both .h's inside the "other/" directory. These are the steps I follow to set up an EDE project with this simple directory setup:
Open main.cpp in emacs and do M-x ede-new ; type: Make ; name: main-proj.
Open one of the files in the "other" directory and do M-x ede-new ; type: Make ; name: aux-proj.
Now it's time to create the targets (which I believe are three in this case):
On the main.cpp buffer: M-x ede-new-target ; name: main ; type: program. When prompted, I add the main.cpp to this target.
I repeat the same for the other two targets (Utils which has Utils.cpp and Utils.h and CGrabBuffer which has CGrabBuffer.cpp and CGrabBuffer.h). Here I find the first problem. What type do these two targets have to be? I only want them to generate .o files.
Once this is done, I type M-x ede-customize-current-target to all three targets and I add some include paths, some libraries, etc.
After this, if I call M-x ede-compile-project it doesn't compile because:
It tries to compile main.cpp first; I have no idea how to specify (using EDE) that both Utils.o and CGrabBuffer.o are needed before attempting to build main.cpp.
If I manually change the order (editing the Makefile), it's not able to link main.cpp because it can't find Utils.o and CGrabBuffer.o.
As you can see, I am in the middle of a great mess. Maybe I'm not even understanding what "target" means in EDE. I have also read about the existence of ede-cpp-root-project which has to be specified inside the .emacs file. I haven't tried it because what I think it does is just help with the semantics. It doesn't generate Makefiles, does it? Can I have (or do I need) an EDE project built with Project.el's and the same thing using ede-cpp-root-project for the semantics? Or is it redundant?
Sorry If I misunderstood a lot of things but I'm very confused and being new to emacs makes things worse. Thanks for your patience!
EDIT: with some tinkering and the responses I received I have been able to figure out a lot of stuff, so thanks a lot. What I still don't understand is the use of the ede-cpp-root-project which has to be specified inside the .emacs file. Is it just for c++ semantics? Is it redundant to have the project with Project.el's AND also the elisp lines in .emacs?
EDE is designed to handle many different kinds of projects, usually of a type where the build system was written outside of Emacs in some other tool.
The EDE project type that creates Makefiles for you can do quite a few things, but you need to have some basic understanding of build systems for it to be helpful, and you really do need to customize the projects to get anything of any complexity working.
I've recently added a section to the EDE manual to help with basic project setups that autogenerate Automake files. You can check out the tutorial here:
http://www.randomsample.de/cedetdocs/ede/ede/Quick-Start.html
The same steps will apply for projects that just use Make instead, but Make based projects often have trouble with shared libraries due to the extra complexity.
Mike's answer is quite good, but I think it is ok to just add .h files to the same target as your .cpp sources. It will keep track of them separately.
Another useful trick is to use the whole project compile keystroke (C-c . C) which uses a capital C whenever you change something big. That will regenerate the Makefiles, rerun any needed Automake features, and start at the top.
EDIT: You only need one EDE project for a give project area. The ede-cpp-root project is useful when no other automatic project type works. That's when you create that in your .emacs file so that the other tools that need a project definition, like semantic's smart completion, and tag lookup, will work.
Well, I think I actually have it figured out this time, but it's ugly. Utils.cpp and CGrabBuffer.cpp should not get their own individual targets, because there doesn't seem to be an appropriate target type. Instead, you'll need to create an archive or library, which will automatically compile Utils.cpp and CGrabBuffer.cpp for you. Below, I'll assume you want static, but it's easy to change.
[For anyone to whom archives or libraries are not familiar, they basically just gather up .o files into a separate unit. It doesn't actually make the compilation harder. Read more here.]
1) Follow the first two and a half steps above (including making the main target, but not the other targets).
2) Switch to Utils.cpp and do M-x ede-new-target ; name: aux ; type: archive. When prompted, add Utils.cpp to this target.
3) Switch to CGrabBuffer.cpp and do C-c . a ; Target: aux .
4) Regenerate the Makefile with M-x ede-proj-regenerate. At this point, if you run make in the other subdirectory, you should get the archive libaux.a.
5) Switch back to main.cpp and do M-x ede-customize-current-target. This brings up an interactive emacs customization buffer, which allows you to edit details of the ede configuration. Under the Ldflags section, click [INS]. This pops out a new line that says Link Flag: and has some different-colored box for you to type in (mine is grey). Type -Lother -laux, so that other/libaux.a is included when compiling main. Then, at the top of the buffer, press [Accept], which should save that change and switch back to main.cpp.
6) Regenerate the Makefile with M-x ede-proj-regenerate.
Now, unfortunately, the Makefile makes the main target first, then descends into the other directory and makes that. Unfortunately, this means that a make from the top-level directory will not work on a clean tree. I don't know why this is, because it seems like that would never be what you want in any project that is ever made with EDE. I can't find any way to change that, except for this hack:
7) Do M-x customize-project; under Inference-Rules click [INS]. Then enter Target: all ; Dependencies: aux main ; Rules: [INS] ; String #: . (This last one is just to prevent an error on an empty rule with a tab; presumably an EDE bug.) Click [Accept], and regenerate the Makefiles.
So now, in your top directory, you can just run make, and main should be a working executable.
I'm quickly becoming convinced that EDE is not yet ready to be used by people other than its authors. Despite its size and the amount of effort they've clearly put into it, it is too buggy, too counterintuitive, and just not smart enough. That's a shame. Emacs needs something like this.
I'm using Autoconf to build my c++ project. It uses third party code which is also built with the help of Autoconf/Automake. So in my configure.ac I've got the following line:
AC_CONFIG_SUBDIRS([subdirectoryname])
Everything works fine, but I also use the feature to let tests be automatically made if make check is executed - which is done by the third party code as well. Because these tests take a while it's annoying to execute them each time I want to test my own code. So is there any way to avoid that the check option is passed to the subdirectory's Makefile?
Update: Overriding check-recursive does not seem to be an option, as my top-level Makefile.am looks (more or less) like this:
SUBDIRS=library src
So disabling checking on this level would also disable the checking inside my src folder. And that's not what I want to achieve. I just want to disable the checking in the library directory.
Overriding check-recursive in your Makefile.am should work:
check-recursive:
#true
or, if you only wanted to check in a specific directory:
check-recursive:
$(MAKE) -C src check
according to the autoconf manual, it will execute a configure.gnu script in the subdirectory if it finds one. Theoretically that could be a script which adds a --disable-tests or similar option to a call to ./configure
That said, I've yet to get this to work on a project of my own. :-/