I'm using a couple boost libraries and using the rule below to generate automatic dependencies. I think boost headers really slow down the compilation because without the dependency includes in the Makefile, it is abour 10 times faster to compile the project. Is there a way to increase the speed with generated dependencies?
%.o: %.cc
$(CXX) $(CFLAGS) $(INCLUDES) -MD -c $< -o $#
#mv $*.d .deps/
#cp .deps/$*.d .deps/$*.tmp
#sed -e 's;#.*;;' -e 's;^[^:]*: *;;' -e 's; *\\$$;;' \
-e '/^$$/d' -e 's;$$; :;' < .deps/$*.tmp >> .deps/$*.d
#rm .deps/$*.tmp
As you are likely not going to change the boost headers, there is no reason, that you add them as dependencies. By using -MMD to generate the dependency files, system headers should be ignored and thus your dependency files should get smaller, this means make have to scan less of your hard drive to see if files are still up to date etc. Of course you have to include the headers with <> not "".
Related
I have couple of cpp and hpp files in directory ./src. I compile all cpp files in one binary file, say ./bin/run. I want to re-compile only if I need i.e it or one of its header was changed.
I, probably, can create Makefile where file will be recompiled if and only if it was changed, but it's quite uncomfortable because big part of my code is in the headers. (It's not going to be changed, because the product is header itself and cpp files are tests).
I want to store temporary .o files in ./build
I know about g++ -MM function but I'm not sure how to use it.
I'll glad to see solutions that use not necessary make but any other system possible if they are easy enough.
UPD
I'll try to clarify, what's the problem is:
New cpp's maybe created, includes may be added or gone, etc. I don't want to edit my makefile each time.
To solve the problem I mentioned (-include is not a good solution), I use something like this:
build/%.o: %.cpp
#$(CC) -MD -c -Wall -o $# $<
#cp build/$*.d build/$*.P
#sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < build/$*.P >> build/$*.d
#rm build/$*.P
-include build/*.d
No `%.d rule is needed.
EDIT:
#JackKelly has [*cough*, *choke*] shown me a better way to get effectively the same dependency file:
build/%.o: %.cpp
#$(CC) -MD -c -Wall -o $# $<
#$(CC) -MM -MP -Wall -o $*.d $<
-include build/*.d
Ye, you can have multiple rules for the same target, as long as only one of them has commands; the prerequisites accumulate. The idea is to get a file like this:
file.o: file.cpp headerfile.h
headerfile.h:
The second line (headerfile.h:) is a rule for headerfile.h that has no prerequisites or commands. It does nothing, but it's a rule, so if headerfile.h is missing, Make is satisfied.
You mention g++ -MM, which can do what you're trying to do:
include $(ALLOBJ:%.o=%.d)
%.d: %.cxx
#echo making dependencies for $<
#g++ -MM -MP $(CXXFLAGS) $< -o $#
#sed -i 's,$*\.o,& $# ,g' $#
Basically this defines a rule that creates .d files from .cxx files. The .d files are, in turn, required by the include statement, which requires one for each .o file in ALLOBJ.
The last line in the dependency rule is the 'sed magic' that makes the dependency files regenerate themselves. If you think regular expressions are hacks at best, and evil more often than not, you can use the -MT flag.
The outline of a solution is as follows:
Use auxiliary dependency files for each source file (that is, create foo.dep for foo.c, bar.dep for bar.c etc)
Use gcc -MM to create the dependency files
In order to force make to do this automatically, use foo.c as a prerequisite for foo.dep and foo.o; this requires some minor sed magic on the output of gcc -MM
Include all the dependency files in your main makefile; this is a key step that makes this approach possible.
The last step is written as follows:
-include $(dependency_files)
This is very tricky but possible; see the GNU make manual for more information.
You can do this with Make, you just need to specify the headers in your rule's sensitivity list. For example
myfile.o: myfile.cpp
gcc -c myfile.o myfile.cpp ${LDFLAGS} # This is optional. make can infer this line.
Turns into
myfile.o: myfile.cpp myfile.h
gcc -c myfile.o myfile.cpp ${LDFLAGS} # Again, optional.
Now, whenever myfile.h changes, myfile.cpp will be rebuild. More headers can be chained in a similar way.
I'm currently trying to build a proper Makefile.
What I want is full control of what's happening, so I don't want any third party software.
My current attempt seems logic to me, but since the dependency generation is not valid, I'm kind of stuck.
For better readabilty, the full Makefile is broken into little pieces. I would appreciate any comment on any section if there's something to improve.
First of all, I have the following static definitions
CXX = g++
CXXFLAGS = -Wall \
-Wextra \
-Wuninitialized \
-Wmissing-declarations \
-pedantic \
-O3 \
-p -g -pg
LDFLAGS = -p -g -pg
DEPFLAGS = -MM
Afaik this should be fine. It would be perfect to make the profiling flags optional but that's not important.
SRC_DIR = ./src
OBJ_DIR = ./obj
SRC_EXT = .cpp
OBJ_EXT = .o
TARGET = ./bin/my_target
SRCS = $(wildcard $(SRC_DIR)/*$(SRC_EXT))
OBJS = $(subst $(SRC_DIR), $(OBJ_DIR), $(SRCS:$(SRC_EXT)=$(OBJ_EXT)))
DEP = depend.main
Basically, this should just extract all the *.cpp files out of the subfolder src and additionally replace ./src with ./obj and .cpp with .o as names of the objects.
.PHONY: clean all depend
all: $(TARGET)
$(TARGET): $(OBJS)
#echo "-> linking $#"
#$(CXX) $^ $(LDFLAGS) -o $#
$(OBJ_DIR)/%.$(EXT_OBJ):
#echo "-> compiling $#"
#$(CXX) $(CXXFLAGS) -c $< -o $#
Afaik, this block - provided a valid dependency file exists - should do all necessary compiling and linking.
clean:
#echo "removing objects and main file"
#rm -f $(OBJS) $(TARGET)
Should be self-explanatory and correct, or am I missing something here?
$(SRC_DIR)/%.$(SRC_EXT):
$(CXX) $(DEPFLAGS) -MT \
"$(subst $(SRC_DIR),$(OBJ_DIR),$(subst $(SRC_EXT),$(OBJ_EXT),$#))" \
$(addprefix ,$#) >> $(DEP);
clear_dependencies:
#echo "-> (re-)building dependencies";
#$(RM) $(DEP)
depend: clear_dependencies $(SRCS)
This is the non-functional part. What I intend to do is using the g++ Compiler flag -MM to auto-create dependencies and using -MT to use a different path than the default one.
The resulting dependency should look like
./obj/main.o: ./src/main.cpp ./src/some_header_file.h
Unfortunately, this will never be called and I lack the knowledge why this is the case.
In a similar question, user Beta gladly provided a temporary solution by adding a .Phony but this has the side effect on rebuilding every object without any change.
Finally, there is just the one line
-include $(DEP)
to include the dependency file, once created.
Any answer providing some hints about any part are very welcome.
So my question is: What can I do better or maybe "cleaner" and why doesn't the dependency generation work?
Here goes.
Assign simply expanded variables where possible:
SRCS := $(wildcard $(SRC_DIR)/*$(SRC_EXT))
From GNU Make manual:
Another disadvantage [of recursively expanded variables] is that any functions referenced in the definition will be executed every time the variable is expanded. This makes make run slower; worse, it causes the wildcard and shell functions to give unpredictable results because you cannot easily control when they are called, or even how many times.
Use substitution references or patsubst function to convert sources into objects:
OBJS := $(SRCS:$(SRC_DIR)/%$(SRC_EXT)=$(OBJ_DIR)/%$(OBJ_EXT))
Specify proper prerequisites in compilation pattern rule. This is mandatory to get Make keeping your object files up to date and updating them on source changes.
$(OBJ_DIR)/%$(OBJ_EXT) : $(SRC_DIR)/%$(SRC_EXT)
#echo "-> compiling $#"
#$(CXX) $(CXXFLAGS) -o $# -c $<
Compile sources and generate dependency files for them at the same time. Use -MMD -MP flags to get things work (just append them to CXXFLAGS).
CXXFLAGS += -MMD -MP
-include $(OBJS:$(OBJ_EXT)=.d)
From GCC manual:
-MD
-MD is equivalent to -M -MF file, except that -E is not implied. The driver determines file based on whether an -o option is given. If it is, the driver uses its argument but with a suffix of .d, otherwise it takes the name of the input file, removes any directory components and suffix, and applies a .d suffix.
-MMD
Like -MD except mention only user header files, not system header files.
-MP
This option instructs CPP to add a phony target for each dependency other than the main file, causing each to depend on nothing. These dummy rules work around errors make gives if you remove header files without updating the Makefile to match.
Also consider studying this article of Paul Smith (he is a maintainer of GNU Make). It gives a rather good overview of different autodep-generation approaches.
I have a project with 50+ .h files and 50+ .cpp files. I'm using make to build a project, which looks something like this (it's just a piece of an entire file):
HEADERS := $(shell find $(INCLUDE) -name "*.h")
%.obj: %.cpp $(HEADERS)
$(CPP) $(CPPFLAGS) -fPIC -o $# -g -c $<
When I'm making changes to one .h file, the whole project has to be re-compiled. It's annoying and time-consuming. But I don't want to hard-code file dependencies inside Makefile, since it's even more time-consuming. I would like to have some make-like tool, which will find dependencies right inside my .cpp/.h files, automatically. Any suggestions? Thanks in advance!
There are many options, but nothing that works exactly like make. Many build systems will generate the Makefiles with dependencies for you though. CMake and Automake are two commonly used systems that work like that. You define your project file in a meta-language and it will generate the Makefiles.
Scons is good in finding the right sources that need to be recompiled.
It scans the files and the headers and builds up an internal dependency tree.
www.scons.org
If you are using gcc then you can use the -M option. It is designed to do exactly what you want i.e. generate a Makefile rule describing the dependencies.
Simplest way:
depend:
g++ -M *.cpp >.depends
-include .depends
Better way:
SRC=foo.cpp bar.cpp ...
OBJ=$(patsubst %.cpp,%.o,$(SRC))
DEPS=$(patsubst %.o,.deps/%.o.dep,$(OBJ))
all: .deps
.deps:
mkdir -p .deps
.cpp.o:
$(CXX) $(CXXFLAGS) -MD -MF .deps/$#.dep -c -o $# $<
-include $(DEPS)
So compiler will generate all dependencies for each file during build automatically.
Or even better: use Autotools, CMake or other build system that does this job for you.
Jam.
Boost.Build.
The latter is based on a (massively) enhanced version of the former.
What approach do C++ programmers on Unix platform use to create and manage Makefiles?
I was using hand made Makefiles for my projects but they don't handle header file changes and other dependencies. I googled around and found a good solution here.
But I ran into a problem here in the sed command -
sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < $*.d >> $*.P; \
The problem is with the 3rd expression "-e 's/ *\$$//'.
It doesn't work. Its supposed to remove trailing backslashes. I understand that there has to be double dollar there since this is part of a Makefile. Can someone tell me what wrong here?
Here's the complete Makefile -
CC=g++
CFLAGS=-g -Wall
LIBS=-lpthread
OBJS=file1.o file2.o
TARGET=testProg
$(TARGET) : $(OBJS)
$(CC) -o $# $^ $(CFLAGS) $(LIBS)
%.o : %.cpp
$(CC) -MMD -c -o $# $< $(CFLAGS)
#cp $*.d $*.P; \
sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < $*.d >> $*.P; \
rm -f $*.d
-include $(OBJS:%.o=%.P)
clean :
rm -f $(TARGET) $(OBJS)
all : $(TARGET)
Apart from the solution to this problem, I would also like some tips/pointers to my 1st question.
gcc/g++ can generate dependencies for you with the -M family of options.
The following works by specifying how to generate .depends files given a source file. By doing -include $(DEPS) $(DEPS) is recognized as a target and will be built/rebuilt when the source files change.
CXX = g++
CXXFLAGS = -Wall -O3
LDFLAGS =
TARGET = testcpp
SRCS = main.cc x.cc foo.cc
OBJS = $(SRCS:.cc=.o)
DEPS = $(SRCS:.cc=.depends)
.PHONY: clean all
all: $(TARGET)
$(TARGET): $(OBJS)
$(CXX) $(CXXFLAGS) $(LDFLAGS) $(OBJS) -o $(TARGET)
.cc.o:
$(CXX) $(CXXFLAGS) -c $< -o $#
%.depends: %.cc
$(CXX) -M $(CXXFLAGS) $< > $#
clean:
rm -f $(OBJS) $(DEPS) $(TARGET)
-include $(DEPS)
I use that approach too and can't praise it highly enough. And I write my makefiles by hand and reuse them a lot on new projects.
.The expression "s/ *\\$//" will work outside the context of Make. Within a makefile it doesn't work because Make tries to interpret "$/" before handing the result to the shell. So you must use "s/ *\\$$//" (note the extra $) within the makefile, but this won't work outside the context of Make (so testing it is a slight pain).
EDIT:
I've tried your makefile, and that sed statement seems to remove trailing backslashes just fine. Try something simpler, like this:
backslash:
#echo " \\" > $#
test: backslash
#echo without sed:
#cat backslash
#echo with sed:
#sed -e 's/ *\\$$//' < backslash
EDIT:
All right, now I'm hooked. Could you try these experiments and tell us the results?
Change the last character to 'z' : s/.$/z/
Change a trailing backslash to 'z' : s/\\$/z/
Change a trailing backslash to 'z' : sm\\$mzm
Delete a trailing backslash : s/\\$//
Delete spaces and a trailing backslash: s/ *\\$//
Try all of these inside and outside of Make, with '$' and '$$'.
In a make file anything you list on the dependency line is a dependency header files or other files included.
BSD tutorial on make Note: you can auto generate header dependency info with the -MM switch of GCC.
I must be missing something. Why doesn't generating dependency files work for you?
I prefer to use CMake, even though it's not strictly the solution to your problem.
It's a project description language that'll generate your Makefiles, Visual Studio Project, Eclipse Project, KDevelop, etc for you. All the dependencies are done for you:
CMakeLists.txt
add_executable(my_exe file1.c file2.c)
target_link_libraries(my_exe my_library)
add_subdirectory(lib)
In lib/CMakeLists.txt
add_library(my_library file3.c file4.c)
This creates a my_exe from file1.c file2.c linked against my_library. I find this much simpler. It also has things like package discovery:
find_package(Qt4)
The makedepend utility is installed on many systems and can be quite useful for generating dependency information.
Here is an example Makefile that uses the include directive (plus a little Perl magic) to incorporate the output from makedepend:
# the name of the executable that we'll build
TARGET = foo_prog
# our .cc source files
SRCS = foo.cc main.cc
# the .o versions of our source files
OBJS := $(patsubst %.cc, %.o, $(filter %.cc, $(SRCS)))
# some flags for compiling
CXXFLAGS = -Wall -Werror
# In order to build $(TARGET), we first build each of the $(OBJS).
# Then we use the given command to link those $(OBJS) into our
# $(TARGET) executable. $^ is a shortcut for $(OBJS). $# is a
# shortcut for $(TARGET).
#
# The default compile rule will compile each of the $(OBJS) for us.
$(TARGET): $(OBJS)
$(CXX) $(CXXFLAGS) $^ -o $#
# Use "make clean" to remove all of the support files.
clean:
rm -f $(OBJS) $(TARGET) Makefile.depend *~
# This automatically uses the 'makedepend' utility to add any
# dependencies that our source files have, namely .h files. This way,
# if the .h files change, the code will be re-compiled.
include Makefile.depend
Makefile.depend: $(SRCS)
makedepend -f- -Y $(SRCS) 2> /dev/null | \
perl -p -e "s/(^.*?:)/Makefile.depend \1/" > Makefile.depend
If both foo.cc and main.cc depend on foo.h, then the contents of Makefile.depend would be:
Makefile.depend foo.o: foo.h
Makefile.depend main.o: foo.h
The end result is that the dependency information from makedepend is injected into the Makefile as a series of rules. It's similar to the approach of using a .d file for each .cc file, but keeps the dependency information in one file instead of scattered all over the place.
In Mozilla's build system, we use GCC's -MD switch to generate the dependency files:
http://mxr.mozilla.org/mozilla-central/source/configure.in#7134
and then we use a script called mddepend.pl to check for removed header files, such that
removing a header simply causes a rebuild, not an error:
http://mxr.mozilla.org/mozilla-central/source/config/rules.mk#2066
http://mxr.mozilla.org/mozilla-central/source/build/unix/mddepend.pl
That script generates an .all.pp file containing all the dependencies, with extra foo.o: FORCE dependencies stuck in for missing header files. We then simply -include the .all.pp file in rules.mk right below there.
You can use qmake to generate Makefiles for a project even if that project is not using Qt.
I use BSD make (pmake?) which does lot of work for me (my lang is C, but I think no difference here). This is my common 'local.prog.mk', I never change it:
.PHONY: tags .depend
# .depend depends only on $(SRCS) in bsd.dep.mk, so we can't track changes of
# header's own dependencies properly. so .depend is .PHONY target here.
CSTD ?=c99
WARNS ?=9
.if !empty(PC_LIST)
PC_CF !=pkg-config --cflags $(PC_LIST)
PC_LD !=pkg-config --libs $(PC_LIST)
.endif
CFLAGS +=$(PC_CF) -fgnu89-inline
.if !defined(NO_DEBUG)
CFLAGS +=-O0 -ggdb3
.endif
LDFLAGS +=$(PC_LD)
CTAGS =exctags
NO_MAN=
NO_OBJ=
CLEANFILES+=$(PROG).core
.include <bsd.prog.mk>
$(PROG): $(SUBDIR)
build: clean cleandepend depend all
run: $(PROG)
./$(PROG)
Note 'bsd.prog.mk' inclusion -- this handles all, build, depend, clean targets. Project-specific BSDmakefiles are simple:
.SILENT:
PROG =hello
SRCS =hello.c world.c
PC_LIST =gtk+-2.0 gnet-2.0
.include "../local.prog.mk"
proto:
cproto -siv `pkg-config --cflags $(PC_LIST)` $(SRCS) > prototypes
CLEANFILES+=prototypes
I just make depend every time I insert/remove any #include directives.
Instead of the sed scripts, use gcc's -MT option to modify the target of the generated dependency rules. This blog post has more info.
With a more modern version of GCC, you can add the -MP flag to have GCC generate empty rules for the headers itself.
I top tip that I have found useful when building dependency files is to include the dependency file as a target in the generated rule:
file.d file.o : file.c header.h header2.h ...
Thus make will regenerate the dependencies if the source or any of the headers change. Including phony targets for the headers (GCC -MP) should then allow stable builds when headers are removed - the absense of required header remains a compilation error, not a make dependency error.
Assuming that dependency files are generated into the same directory as the object files, the following should work for GCC on Unix:
-include $(OBJ:.o=.d)
$(OBJDIR)/%d : $(SRCDIR)/%.cpp
mkdir -p $(#D)
echo -n "$# " > $#.tmp
$(CXX) $(CPPFLAGS) -MM -MP -MT $(#:.d=.o) $< >> $#.tmp
mv $#.tmp $#
(from memory)
I'm using GCC to generate a dependency file, but my build rules put the output into a subdirectory. Is there a way to tell GCC to put my subdirectory prefix in the dependency file it generates for me?
gcc $(INCLUDES) -E -MM $(CFLAGS) $(SRC) >>$(DEP)
I'm assuming you're using GNU Make and GCC. First add a variable to hold your list of dependency files. Assuming you already have one that lists all our sources:
SRCS = \
main.c \
foo.c \
stuff/bar.c
DEPS = $(SRCS:.c=.d)
Then include the generated dependencies in the makefile:
include $(DEPS)
Then add this pattern rule:
# automatically generate dependency rules
%.d : %.c
$(CC) $(CCFLAGS) -MF"$#" -MG -MM -MP -MT"$#" -MT"$(<:.c=.o)" "$<"
# -MF write the generated dependency rule to a file
# -MG assume missing headers will be generated and don't stop with an error
# -MM generate dependency rule for prerequisite, skipping system headers
# -MP add phony target for each header to prevent errors when header is missing
# -MT add a target to the generated dependency
"$#" is the target (the thing on the left side of the : ), "$<" is the prerequisite (the thing on the right side of the : ). The expression "$(<:.c=.o)" replaces the .c extension with .o.
The trick here is to generate the rule with two targets by adding -MT twice; this makes both the .o file and the .d file depend on the source file and its headers; that way the dependency file gets automatically regenerated whenever any of the corresponding .c or .h files are changed.
The -MG and -MP options keep make from freaking out if a header file is missing.
The answer is in the GCC manual: use the -MT flag.
-MT target
Change the target of the rule emitted by dependency generation. By default CPP takes the name of the main input file, deletes any directory components and any file suffix such as .c, and appends the platform's usual object suffix. The result is the target.
An -MT option will set the target to be exactly the string you specify. If you want multiple targets, you can specify them as a single argument to -MT, or use multiple -MT options.
For example, -MT '$(objpfx)foo.o' might give
$(objpfx)foo.o: foo.c
You may like this briefer version of Don McCaughey's answer:
SRCS = \
main.c \
foo.c \
stuff/bar.c
DEPS = $(SRCS:.c=.d)
Add -include $(DEPS) note the - prefix, which silences errors if the .d files don't yet exist.
There's no need for a separate pattern rule to generate the dependency files. Simply add -MD or -MMD to your normal compilation line, and the .d files get generated at the same time your source files are compiled. For example:
%.o: %.c
gcc $(INCLUDE) -MMD -c $< -o $#
# -MD can be used to generate a dependency output file as a side-effect of the compilation process.
Detailing on DGentry's answer, this has worked well for me:
.depend: $(SOURCES)
$(CC) $(CFLAGS) -MM $(SOURCES) | sed 's|[a-zA-Z0-9_-]*\.o|$(OBJDIR)/&|' > ./.depend
This also works in the case where there is only one dependency file that contains the dependency rules for all source files.
Ok, just to make sure I've got the question right: I'm assuming you have test.c which includes test.h, and you want to generate subdir/test.d (while not generating subdir/test.o) where subdir/test.d contains
subdir/test.o: test.c test.h
rather than
test.o: test.c test.h
which is what you get right now. Is that right?
I was not able to come up with an easy way to do exactly what you're asking for. However, looking at Dependency Generation Improvements, if you want to create the .d file while you generate the .o file, you can use:
gcc $(INCLUDES) -MMD $(CFLAGS) $(SRC) -o $(SUBDIR)/$(OBJ)
(Given SRC=test.c, SUBDIR=subdir, and OBJ=test.o.) This will create both subdir/test.o and subdir/test.d, where subdir/test.d contains the desired output as above.
If there is an argument to GCC to do this, I don't know what it is. We end up piping the dependency output through sed to rewrite all occurrences of <blah>.o as ${OBJDIR}/<blah>.o.
[GNU] make gets angry if you don't place the output in the current directory. You should really run make from the build directory, and use the VPATH make variable to locate the source code. If you lie to a compiler, sooner or later it will take its revenge.
If you insist on generating your objects and dependencies in some other directory, you need to use the -o argument, as answered by Emile.