I have recently begun writing unit tests (using GoogleTest) for a C++ project. Building the main project is fairly simple: I use GCC's -MM and -MD flags to automatically generate dependencies for my object files, then I link all of the object files together for the output executable. No surpirses.
But as I'm writing unit tests, is there a way to have make or GCC figure out which object files are needed to compile each test? Right now, I have a fairly naive solution (if you can call it that) which compiles ALL available object files together for EVERY unit test, which is obviously wasteful (in terms of both time and space). Is there a way (with make, gcc, sed, or whatever else) to divine which object files are needed for a given unit test in a fashion similar to how dependencies are generated for the original source files?
It sounds like you have two groups of source files: one that actually implements your program, and another that's all the unit tests. I assume each unit test has its own main function, and unit tests never need to call each other.
If all that's true, you can put all the files from the first group in a static library, and link each of the unit tests against that library. The linker will automatically pull from the library only the object files that are needed.
In concrete Makefile terms:
LIBRARY_OBJECTS = a.o b.o c.o d.o # etc
UNIT_TESTS = u1 u2 u3 u4 # etc
UNIT_TEST_OBJECTS = $(UNIT_TESTS:=.o)
libprogram.a: $(LIBRARY_OBJECTS)
ar cr $# $?
$(UNIT_TESTS): %: %.o libprogram.a
$(CC) $(CFLAGS) -o $# $< -lprogram
You should look to a higher abstraction of project management, like Cmake or GNU Automake.
In your Makefile
SOURCES.cpp = a.cpp b.cpp ...
OBJECTS = $(SOURCES.cpp:%.cpp=%.o)
all: program
program: $(OBJECTS)
$(LINK) -o $# $(OBJECTS)
Maybe, depending on how orderly your test system is.
If there's a nice one-to-one relationship between header and source files, then you can use some text-converting functions (or a call to sed) to convert the machine-generated rule you already have:
foo.o: foo.cc foo.h bar.h gaz.h
into a rule for the corresponding test:
unit_test_foo: unit_test_foo.o foo.o stub_bar.o stub_gaz.o
Or if you use a lot of stubs without corresponding headers (which is a warning sign) you can link with every stub except stub_foo.o. These object files are small and don't change often, so it's cheap.
Related
So far, I have a Makefile that looks like this:
# Program list:
# a.cpp, a.h,
# b.cpp, b.h,
# c.cpp. c.h,
# d.cpp, d.h,
# commonA.cpp, commonA.h,
# commonB.cpp, commonB.h,
# mainA.cpp, mainB.cpp,
# mainC.cpp, mainD.cpp.
CXX=g++
CXXFLAGS = -std=c++11 -g -Wall
prog1: mainA.cpp a.cpp
$(CXX) $(CXXFLAGS) programOne mainA.cpp a.cpp
prog2: mainB.cpp b.cpp
$(CXX) $(CXXFLAGS) programTwo mainB.cpp b.cpp
prog3: mainC.cpp c.cpp commonA.cpp
$(CXX) $(CXXFLAGS) programThree mainC.cpp c.cpp commonA.cpp
prog4: mainD.cpp d.cpp commonA.cpp commonB.cpp
$(CXX) $(CXXFLAGS) programFour mainD.cpp d.cpp commonA.cpp commonB.cpp
# etc...
I've had a look at the make GNU documentation, and it's fairly daunting - I've tried to understand as much as I can from it. I was wondering given my example how would I be able to shorten this even more than what it currently is? Is there a good rule to use that could make all the object files for each of the files in the program list above, and then I include the particular objects for each part? Any help is much appreciated!
Here's a tip: make has built-in rules that know how to create all kinds of different targets. Taking advantage of them will give you the shortest possible makefile.
Here's another tip: make uses string comparison to match up targets, prerequisites, and rules. So choosing the names of your files and programs wisely will give the best chance of taking advantage of make's built-in rules.
For example, you know already that a source file foo.cpp will be compiled into an object file foo.o. Make has a built-in rule that will handle that for you. If you also choose to name one of your source files so that it maps to the name of the program you want to create, you can take advantage of a built-in rule that will link your program as well.
As mentioned in the comments above, your rules are broken because the makefile target is named one thing (e.g., prog1) but the link line generates a different thing: programOne (here I'm assuming you simply forgot the -o option in your link line when you posted your code... please remember that when asking for help on SO--or really anywhere--it's best to create a SSCE). That should never happen for a non-special target; every rule should create a file with the identical name as the target.
Typically for C/C++ programs, the source file containing main() is named the same as the program you want to create. So for example if you want to create a program programOne then the source file containing main() should be named programOne.cpp.
If you follow these recommendations, then your entire makefile can be just:
CXX = g++
CXXFLAGS = -std=c++11 -g -Wall
all: programOne
programOne: programOne.o a.o
and that's it. Make knows how to build those targets for you so you don't have to tell it. If you want more programs, just add them as well:
all: programOne programTwo
programOne: programOne.o a.o
programTwo: programTwo.o b.o
The one issue left is header files prerequisites. If you want you can declare them yourself like this:
programOne.o: programOne.h a.h
etc. That's very simple but tedious to maintain. If you want to get make to configure them for you you can, but it's not simple. See this discussion for some ideas.
Why this rule cannot override the default implicit rule ?
When make is invoked like: make myapp (suppose myapp.c is there). The make runs the default command to build and link the program instead the commands defined in this implicit rule:
#... omitted code
LCUS=$(LIBS)/libcus.a
#... omitted code
% : %.o $(LCUS)
echo !!! Custom build !!!
$(MY_CMD) $< -o $# $(LCUS)
Taken from the GNU online make manual:
You can override a built-in implicit rule (or one you have defined
yourself) by defining a new pattern rule with the same target and
prerequisites, but different commands.
So I would assume it is because the prerequisites are not the same as the implicit rule.
Also take from the make manual:
Linking a single object file n is made automatically from n.o by
running the linker (usually called ld) via the C compiler. The
precise command used is $(CC) $(LDFLAGS) n.o $(LOADLIBES) $(LDLIBS).
This rule does the right thing for a simple program with only one
source file. It will also do the right thing if there are multiple
object files (presumably coming from various other source files), one
of which has a name matching that of the executable file. Thus,
x: y.o z.o
when x.c, y.c and z.c all exist will execute:
cc -c x.c -o x.o
cc -c y.c -o y.o
cc -c z.c -o z.o
cc x.o y.o z.o -o x
rm -f x.o
rm -f y.o
rm -f z.o
So basically make understands implicit rules for program files that are generated from .o files but when you throw in your static library it doesn't understand. A simple way to test would be to remove $(LCUS) from your dependencies (as a temporary measure) to see if it then uses your rule instead of the built in one. If it does then you know that is your problem. If simply adding myapp in replace of % is a problem because you want the rule to build multiple targets you can try the following:
$(APPS): % : %.o $(LCUS)
where $(APPS) is a variable containing all the applications you wish to build. This will allow one rule to build multiple targets. You could also skip the use of the variable altogether and place a space separated list. This is an example of a static pattern rule, more information can be found here. The difference between static pattern and implicit rules can be found here.
Your rule isn't the same as the built-in implicit rule so it doesn't cancel it.
Additionally, make always prefers a rule that doesn't require intermediate files to be built to one that does. If you pre-create the .a file make might use your rule instead (but it might not even still).
If you cancel the built-in rule and leave your rule in place it should work correctly I believe.
I am working on my first open source C++ project: https://github.com/jehugaleahsa/spider-cpp.
I am managing my own Makefile and I have "best practices" question regarding how to manage dependencies. Right now, I make each .o file dependent on each of the included header files in the .cpp file. So:
code.o: code.cpp code.hpp dep1.hpp de2.hpp
g++ -c code.cpp
First of all, I am pretty sure Make supports a shorthand for creating object files. If someone would show an example of this, I'd appreciate it.
Next, is there a way to avoid listing every included header as a dependency? I want to make sure if I change a dependency, that the changes are still compatible. Listing the included headers is tedious and easy to mess up.
OP:
First of all, I am pretty sure Make supports a shorthand for creating object files. If someone would show an example of this, I'd appreciate it.
From here:
OBJS := foo.o bar.o
#Your program should have the objects as dependencies, and link them
proggie: $(OBJS)
gcc $(OBJS) -o proggie
# compile
%.o: %.c
gcc -c $(CFLAGS) $*.c -o $*.o
OP:
Next, is there a way to avoid listing every included header as a dependency
Lower down on the same page, see these lines:
# pull in dependency info for *existing* .o files
-include $(OBJS:.o=.d)
# compile and generate dependency info
%.o: %.c
gcc -c $(CFLAGS) $*.c -o $*.o
gcc -MM $(CFLAGS) $*.c > $*.d
Basically what this does is use gcc's -MM option to obtain a list of header files, and now we can depend on them. Thus we output a file with a list of such header files to a .d file, and then next time, we add the list of files as a dependency, which is what the -include command does. The "-" avoids error if the dependency .d files don't exist yet.
Note, you should modify the above to account for .cpp files
Yes, make supports shorthand for creating object files. This is called rules. And yes, there is a way to avoid listing every included header as a dependency. g++/gcc has -MM option which will generate the full list of dependencies.
Unfortunately, there is no simple explanation on how to do it which I could put here.. You have to read docs and play with make utility. I found this book very helpful: "Managing Projects with GNU Make". There is a doc on GNU site but I found it a bit harder to read. www.gnu.org/software/make/manual/make.html
I have a c++ program which has many many functions and I have different .cpp files for each of the function. From the main program, I only supply a few parameters and just call the functions. However, the compilation of the full thing takes a lot of time. For each compilation I only change a few parameters in the main program and leave all the functions as it is.
Is there anyway to speed up the compilation.?
You are recompiling unnecessary code. Usually IDEs handle this automatically. Otherwise, it depends on how you compile your code. For example lines like this:
g++ *.cpp
or
g++ -o program a.cpp b.cpp c.cpp
are terribly slow, because on every compilation, you recompile everything.
If you are writing Makefiles, you should carefully write it to avoid recompilation. For example:
.PHONY: all
all: program
program: a.o b.o c.o
g++ -o $# $^ $(LDFLAGS)
%.o: %.cpp
g++ $(CXXFLAGS) -o $# $<
# other dependencies:
a.o: a.h
b.o: b.h a.h
c.o: c.h
In the above example, changing c.cpp causes compilation of c.cpp and linking of the program. Changing a.h causes compilation of a.o and b.o and linking of the program. That is, on each build, you compile the minimum number of files possible to make the program up-to-date.
Side note: be careful when writing Makefiles. If you miss a dependency, you will may not compile enough files and you may end up getting hard-to-spot segmentation faults (at best). Take a look also at the manual of gcc for -M* options where you can use gcc itself to generate the dependencies and then include the generated output in the Makefile.
Try to minimize the code impacted by your parameter changes, ideally only change one source file no one depens on (main.cpp).
Check your includes: do you really need it all? Use forward declaration where possible (e.g. #include instead of ), for your own classes, forward declare what you can.
Try using the clang (llvm.org) compiler. It sometimes compiles faster than gcc (assuming you're on linux/unix) and gives more readable errors.
Edit: I was assuming you were only recompiling what's needed. As others suggested, use a buildsystem (Makefile, IDE, CMake...) to run a minimal number of compiles.
Maybe this will or won't help much, but I run code through ssh and I know that it takes forever to run/compile. If you are reading from data files, instead of running entire sets of data, run only over a file or two to see your intended result. This will be a sample of your final result, but should still be accurate (just with less statistics). Once you've tweaked your code to work to your satisfaction, then run everything. usually you will have no problems that way, and your compile time is much quicker comparatively speaking.
I know I am doing it wrong, but I can't figure out how to organize this makefile. I define my util source files, and use some functions to define the .o files from them here:
UTIL_SRC = utils/src/foo.cpp utils/src/bar.cpp utils/src/baz.cpp
UTIL_OBJS = $(patsubst utils/src/%.cpp,utils/obj/%.o,$(UTIL_SRC))
This is the target that I use these files for:
lib : lib/libutils.a
lib/libutils.a : $(UTIL_OBJS)
rm -f lib/libutils.a
ar -c -q lib/libutils.a $(UTIL_OBJS)
Then, when I get to the rule to compile these babies, I would love to just have one command that would iterate through each UTIL_OBJS file and each UTIL_SRC file. Instead I have resorted to this monstrosity, which defeats the purpose of storing them in variables.
$(UTIL_OBJS) : $(UTIL_SRC)
g++ $(UTIL_FLAGS) utils/src/foo.cpp -o utils/obj/foo.o
g++ $(UTIL_FLAGS) utils/src/bar.cpp -o utils/obj/bar.o
g++ $(UTIL_FLAGS) utils/src/baz.cpp -o utils/obj/baz.o
Can I condense this down to one line? How? Thanks, great ones!
It's usually easier to work with implicit rules. There are a lot of predefined ones, where you'll only need to specify variables.
CXX=g++
CXXFLAGS=$(UTIL_FLAGS)
Then you need to define an executable, like this
myutil: $(UTIL_OBJS)
Since you're not storing your objects in the same directory, you'll need to specify a new implicit rule as well though (otherwise, we'd be done now).
utils/obj/%.o: utils/obj/%.cpp
% is a pattern-match, it'll match the same text on both left and right side, so this rule will make foo.o out of foo.cpp.
Try if that'll work without the command (it might have grabbed that from another rule, I'm not sure), otherwise let it say:
utils/obj/%.o: utils/obj/%.cpp
$(CXX) $(CXXFLAGS) -o $# $^
$# is the target of the rule (e.g. foo.o), and $^ is all files on the right hand side.
I'm writing this off the top of my head, without the possibility to test it, so please let me know how it turned out.. :)
To make it even more elegant, you can include a dependency file
include .depend
If you're running GNU make, it'll try to make the .depend file if it can't find it (with old school make, you need to create it yourself first, it can be just a dummy though, if you'd like to manage it through the makefile)
.depend: $(UTIL_SRC)
$(CXX) -MM -o $# $^
The dependency file will contain lines for each .cpp file, telling make what header files it needs, which will allow make to recompile the necessary files when you change something. This doesn't help with your original question though, just thought it might come in handy.
EDIT:
As a response to your edit. You could probably drop the commands for creating the .a-file as well, that too is already available as an implicit rule. Not sure exactly how it works though, haven't used it much. I do know that there are a bunch of quirks in make for dealing with .a(rchive?)-files.
I think you could use this:
$(UTIL_OBJS) : $(UTIL_SRC)
g++ $(UTIL_FLAGS) $(# : .o = .cpp) -o $#
again, I'm not quite sure... especialy about the $(# : .cpp = .o) part