Makefile: automatic generation of dependencies for binaries - c++

I would like to "completely" automate the process of dependency generation in a project's Makefile. So far, I have followed this tutorial on automatic dependencies, which works very nicely. There is only one catch: this only works for head-file dependencies, so it will automatically detect if say bin_1.o depends on header_1.h as well as header_2.h. However, I can directly translate header-file dependencies to object-file dependencies. Say, one of the automatically generated dependency files says
bin_1.o: bin_1.cpp header_1.h header_2.h
So, in the linking step, I can immediately conclude that I will need to link together the files bin_1.o, header_1.o, and header_2.o in order to generate the binary bin_1. In other words, a corresponding dependency file for bin_1 should have
bin_1: bin_1.o header_1.o header_2.o
That is exactly what I have tried to achieve. The translation described above is done by a small python script, extract_dependencies.py, and a stripped-down version of my Makefile then looks like this:
binaries = bin_1 bin_2 bin_3
SRCS := $(wildcard *.cpp)
all: $(binaries)
# dependencies
DEPDIR := .d
$(shell mkdir -p $(DEPDIR) >/dev/null)
DEPFLAGS = -MT $# -MMD -MP -MF $(DEPDIR)/$*.Td
COMPILE = $(CC) $(DEPFLAGS) $(CFLAGS) -c
POSTCOMPILE = mv -f $(DEPDIR)/$*.Td $(DEPDIR)/$*.d
GENERATE_BINARY_DEPS = python extract_dependencies.py $(DEPDIR)/$*.d > $(DEPDIR)/$*.bin.d
$(DEPDIR)/%.d: ;
$(DEPDIR)/%.bin.d: ;
.PRECIOUS: $(DEPDIR)/%.d $(DEPDIR)/%.bin.d
$(binaries): % : %.o $(DEPDIR)/%.bin.d
${CC} ${CFLAGS} $(shell cat .d/$#.bin.d | cut -d ' ' -f 2-) ${LDFLAGS} -o $#
%.o: %.cpp
%.o: %.cpp $(DEPDIR)/%.d
$(COMPILE) $<
$(POSTCOMPILE)
$(GENERATE_BINARY_DEPS)
clean:
rm -vf *.o
rm -vf bin_1 bin_2 bin_3
realclean:
$(MAKE) clean
rm -fr .d
-include $(patsubst %,$(DEPDIR)/%.d,$(basename $(SRCS)))
-include $(patsubst %,$(DEPDIR)/%.bin.d,$(basename $(SRCS)))
Now comes the problem: in order for that to work, I need to run "make" twice. The *.bin.d dependency files are generated, but only on the second run, Make actually "realizes" that say header_1.o is also necessary for bin_1, not only bin_1.o. On the first run, it tries to link all the object files together without actually building all the object files first.
Is there any way to come around that issue, i. e. get everything right on the first pass?
Thanks!

Since Make will read-in whole makefile at 1st, then perform targets. It's hard to get %.bin.d updated in one pass. Alternatively, we could try to trigger 2nd "make" internally like,
ifneq ($(STAGE),2)
$(binaries): % : %.o $(DEPDIR)/%.bin.d
#{MAKE} $(binaries) STAGE=2
else
#2nd pass
$(binaries):
${CC} ${CFLAGS} -o $# $(shell cat .d/$#.bin.d | cut -d ' ' -f 2-) ${LDFLAGS}
endif

Related

Getting wrong file name while pattern matching in makefile

The directory structure of my project is something like this:
|-build
|-src
|-lib
|-libfile1.hpp
|-libfile1.cpp
|-libfile2.hpp
|-libfile2.cpp
|-libfile3.hpp
|-Makefile(lib)
|-examples
|-pingpong
|-main.cpp
|-ping.cpp
|-ping.hpp
|-Makefile(ping)
My goal is to have many examples in the examples folder which all use the libfiles but are independent of each other. Hence I wanted to use separate makefiles in each of the example folders to link to objects of libfiles.
To that goal, here's makefile(lib):
SRC = .
BLD = ../../build
SOURCE = $(wildcard $(SRC)/*.cpp)
INCLUDE = $(wildcard $(SRC)/*.hpp)
OBJECT = $(patsubst %,$(BLD)/%, $(notdir $(SOURCE:.cpp=.o)))
CC = g++
#flags went here, removed for brevity
all: $(OBJECT)
$(BLD)/%.o: $(SRC)/%.cpp $(SRC)/%.hpp
mkdir -p $(dir $#)
$(CC) -c $< -o $#
.PHONY: clean
clean:
$(RM) -r $(BLD)
And here's makefile(ping), where I try to build local objects and pass the library object build pattern to makefile(lib):
TARGET = mainpingpong.out
SRC = ../../actorlib
BLD = ../../../build
LIBSOURCE = $(wildcard $(SRC)/*.cpp)
LIBINCLUDE = $(wildcard $(SRC)/*.hpp)
LIBOBJECT = $(patsubst %,$(BLD)/%, $(notdir $(LIBSOURCE:.cpp=.o)))
LOCSOURCE = $(wildcard *.cpp)
LOCINCLUDE = $(wildcard *.hpp)
LOCOBJECT = $(patsubst %,$(BLD)/%, $(notdir $(LOCSOURCE:.cpp=.o)))
CC = g++
$(TARGET) : $(LIBOBJECT) $(LOCOBJECT)
$(CC)-o $# $^
$(LOCOBJECT): $(LOCSOURCE) $(LOCINCLUDE)
mkdir -p $(dir $#)
$(CC) -c $< -o $#
$(LIBOBJECT): $(LIBSOURCE) $(LIBINCLUDE)
cd $(SRC) && $(MAKE)
.PHONY: clean
clean :
cd $(SRC) && $(MAKE) clean
The error occurs in my local file matching, where it associates both main.o and ping.o with main.cpp (everything else works fine):
mkdir -p ../../../build/
g++ -c main.cpp -o ../../../build/main.o
mkdir -p ../../../build/
g++ -c main.cpp -o ../../../build/ping.o
How do I correctly do what I'm trying to do? Should I have a different rule or target? Or is there a correct way to get the filename from the variable?
I want the .o files to end up in the same place but the executable to stay in the examples directory.
The problem is here:
$(LOCOBJECT): $(LOCSOURCE) $(LOCINCLUDE)
mkdir -p $(dir $#)
$(CC) -c $< -o $#
LOCSOURCE contains main.cpp ping.cpp, so this rule makes both source files prerequisites of each target. Whether Make is trying to build main.o or ping.o, the prerequisite list is the same: main.cpp ping.cpp ping.hpp. So the automatic variable $< always expands to main.cpp (since that is the first filename in the list).
One way to solve this is by means of a static pattern rule:
$(LOCOBJECT): $(BLD)/%.o: %.cpp $(LOCINCLUDE)
mkdir -p $(dir $#)
$(CC) -c $< -o $#
This is still imperfect, in that all local header files are prerequisites of each object from a local source. If you have a paddle.cpp and you modify ping.hpp, Make will rebuild paddle.o, even if paddle.cpp does not depend on ping.hpp in any way. This is not a serious problem, it simply means that Make will do some extra work, rebuilding things when there is no reason to. If you want to solve it, you can either write the dependencies into the makefile by hand, or use a more sophisticated method like automatic dependency generation.

Prevent GNU Make from generating protobuf code each build

I have below Makefile which runs Google's protoc on some proto files, generate the code and build the library
Everything works fine except the fact that it generates code every make run, despite there were no changes to the proto files or whatever. How do I prevent that?
GEN_FLAG:=$(PROJECT_ROOT)/build/.etcd_protocols_gen
ETCD=$(PROJECT_ROOT)/ext/etcd/
PROTOCOLS=$(ETCD)proto/
PROTO_FILES=$(shell find $(PROTOCOLS) -name \*.proto)
FILES=$(wildcard *.cc)
OBJS=$(patsubst %.cc,%.o,$(FILES))
LIB=$(BUILD_LIB_DIR)/libetcdclient.so
DIRS=proto
.PHONY: all
all: etcd-gen $(LIB)
-include $(FILES:.cc=.d)
.PHONY: etcd-gen
etcd-gen:
$(Q)echo "Generating ETCD gRPC C++ sources"
#ln -sf $(PROTOBUF_DIR)/libprotoc.so.16.0.0 $(PROTOBUF_DIR)/libprotoc.so.16
#ln -sf $(PROTOBUF_DIR)/libprotobuf.so.16.0.0 $(PROTOBUF_DIR)/libprotobuf.so.16
#LD_LIBRARY_PATH=$(PROTOBUF_DIR) $(PROTOBUF_DIR)/protoc -I $(PROTOCOLS) --cpp_out=. $(PROTO_FILES)
#LD_LIBRARY_PATH=$(PROTOBUF_DIR) $(PROTOBUF_DIR)/protoc -I $(PROTOCOLS) --grpc_out=. --plugin=protoc-gen-grpc=$(PROJECT_ROOT)/ext/grpc/bin/grpc_cpp_plugin $(PROTO_FILES)
#touch $(GEN_FLAG)
%.o: %.cc
#echo "CC $<"
$(Q)$(CC) $(CFLAGS) -c -o $# $< -ldl
$(Q)$(CC) -MM $(CFLAGS) -MF $*.d -MT $*.o $*.cc
$(LIB): $(OBJS)
#echo "LD $#"
$(Q)$(CXX) $(CXXFLAGS) -shared -Wl,-soname,$(notdir $(LIB)) -o $(LIB) $(OBJS) $(LFLAGS) -L$(PROTOBUF_DIR) -l:libprotobuf.so.16
clean:
$(Q)rm -f *.d *.o *.pb.cc *.pb.h
$(Q)rm -f $(BUILD_LIB_DIR)/libetcdclient.so
$(Q)rm -f GEN_FLAG
make considers the target to be the file to be generated. Any comparison of timestamps to determine whether or not to rebuild that file is therefore a comparison of those of the dependencies versus that of the target.
In your case make is checking the timestamp of a non-existing file etcd-gen, and since it doesn't exist or bears no relationship to the sources anyway (the sources aren't listed as dependencies) a rebuild is triggered each time.
What you could do instead is re-structure your makefile so your proto sources are dependencies of the $(GEN_FLAG) and the contents of $(GEN_FLAG) is a target. Then you can declare $(GEN_FLAG) a dependency of etcd-gen.

Makefile Confusion

I'm new to makefiles, but I finally managed to get one working for a project I'm working on. The original goal was to search the folder "Source" for any CPP files, then create an Object file in the folder Products/Objs. Instead, it finds the CPP files, but OBJECT_FILES is the same as SOURCE_FILES, and I'm not sure how it's working. It doesn't generate any .o files (that I could see), but still compiles successfully. Super confused but would love some help. Thanks!
##########################################
# Editable options #
##########################################
# Program Name
EXECUTABLE=net
# Compiler options
CC=g++
CFLAGS=
LDFLAGS=
# Folders
SRC=Source
OBJ=Products/Objs
#########################################################
# Do Not Touch This #
#########################################################
SOURCE_FILES := $(shell find $(SRC)/ -type f -name '*.cpp')
## This should not work
OBJECT_FILES := $(SOURCE_FILES: $(SRC)/%.cpp=%.o)
## Like at all --> It's the same as SOURCE_FILES
## But I guess???
build: $(EXECUTABLE)
## Deletes the executable (assumes small compile time)
clean:
#rm -r -f $(shell find $(SRC)/ -type f -name '*.o')
#rm $(EXECUTABLE)
.PHONY: build clean
$(EXECUTABLE): $(OBJECT_FILES)
#$(CC) $(LDFLAGS) -o $# $^
#echo "Build successful!"
$(OBJECTS_FILES): %.o : $(SRC)/%.cpp
#$(CC) $(CFLAGS) -o -c $# $^
Well first you have a typo; you create a variable OBJECT_FILES:
OBJECT_FILES := $(SOURCE_FILES: $(SRC)/%.cpp=%.o)
Also you shouldn't use spaces here (after the :). Then you use a variable named OBJECTS_FILES instead:
$(OBJECTS_FILES): %.o : $(SRC)/%.cpp
That variable is empty so this is essentially a no-op.
Your compiler invocation is wrong. The word after the -o option is the file the output should go to; yours is:
#$(CC) $(CFLAGS) -o -c $# $^
which means that the output of this command is going to the file -c. Then the input is an object file that doesn't exist... most likely you'll get an error. Also you only want to pass the source file to the compiler, not all the other prerequisites (headers etc.) This should be:
#$(CC) $(CFLAGS) -c -o $# $<

(GNU) Make: How does one set up a basic system?

I would like to try to establish a very small system of Makefiles. I have the following set up, but something is not quite right (this has been pieced together from reading a few SO posts about the topic, though somewhat project-specific, and some websites. I must not be catching something fundamental in having a "Makefile" call sub-makefiles.
This is even simpler than having the main Makefile call files in different subdirectories. Here are the following files I have prepared:
Makefile:
all:
$(MAKE) -f ./make_system.mk
$(MAKE) -f ./make_crc.mk
make_system.mk:
G = -g ## debug option
CC = gcc ## compiler
SRCS = source.c sink.c gateway.c
EXES = source sink gateway
OBJS = source.o sink.o gateway.o
CFLG =
LFLG =
LIB = #-lsocket -lnsl -lpthread
all: $(EXES)
%.o: %.c %.h
$(CC) -c $G $(CFLG) $<
source: source.o
$(CC) -o source source.o $(LIB) $(LFLG)
sink: sink.o
$(CC) -o sink sink.o $(LIB) $(LFLG)
gateway: gateway.o
$(CC) -o gateway gateway.o $(LIB) $(LFLG)
clean:
/bin/rm -f core *.o $(EXES) *~
make_crc.mk:
CC = gcc
CFLAGS = -g
LFLAGS =
HDR = crcmode.h
SRC = crcmodel.c crctest.c
OBJ = crcmodel.o crctest.o
EXE = crctest
all: $(EXE)
%.o: %.c %.h
$(CC) -c $(CLFAGS) $<
$(EXE): $(OBJ)
$(CC) $(LFLAGS) $(OBJ) -o $(EXE)
clean:
/bin/rm -f *.o *~ core $(EXE)
How would I set up Makefile to call the smaller sub-makefiles (of type *.mk)? This is a basic but important first step towards working with larger scale makefile systems (there is the manual to consult, though I do not think it has explicit basic examples). If someone who has done this could show me a small Makefile vignette, this would be greatly appreciated (again, this makefile system can exist within the same local directory).
Note: The individual makefiles "make_system" and "make_crc" work fine when they are themselves named "Makefile", but I want to try to call them as separate sub-makefiles with one single overall Makefile.
ADDENDUM (to solution below):
Once I implemented Carl's solution shown below, I realized that you need to always include some form of "make clean" or at least enter at in the command line before calling your Makefile. Otherwise, you see the appropriate output "nothing to be done". This was something I knew before, but neglected to do this time around (easy check: just look at your directory and you will see you have executables in the environment/directory).
I you want to "forward" the target, you'll need to do it explicitly. Something like this example should work (though I can't test easily right now, sorry):
default: all
%:
$(MAKE) -f ./make_system.mk $#
$(MAKE) -f ./make_crc.mk $#

How to build library and link it with sources with Make

I'm still quite new to make. I am trying to compile a project in which maketest.cpp and maketest.hpp rests on Users/wen/Projects/maketest with the Makefile itself. Problem is, I want to also compile and link the source files (.cc and .hh) in Users/wen/Projects/include/bigint.
# Macros
INCLUDE = -I/Users/wen/Projects/include/bigint
LIBS =
CC = g++
override FLAGS += -O2
HEADERS= $(wildcard *.hpp) $(wildcard Users/wen/Projects/include/bigint/*.hh)
# EXE Name
INSTALL = maketest
CC_FILES = %.cpp Users/wen/Projects/include/bigint/%.cc
OBJ_FILES = maketest.o $(wildcard Users/wen/Projects/include/bigint/*.o)
# Rules
$(INSTALL): $(OBJ_FILES)
$(CC) $(FLAGS) $^ -o $# $(LIBS)
%.o: $(CC_FILES) $(HEADERS)
$(CC) $(FLAGS) $(INCLUDE) -c $< -o $#
# Installation types
install: $(INSTALL)
release:
#echo "Re-run with parameter:"
#echo "FLAGS=-D_RELEASE"
debug:
#echo "Re-run with parameter:"
#echo "FLAGS=-D_DEBUG"
# Cleaning up
clean:
rm -f $(OBJ_FILES) $(INSTALL)
The code compiles maketest.cpp and links it, but not bigint.
What will be the right way to build and link the files from Users/wen/Projects/include/bigint? Many thanks!
I figured it out at the end, thank you Jay. The problem was I forgot a slash at the front of Users/wen/Projects/include/bigint, so it was not searching from root but instead the project folder. Now it works!
Try explicitly specifying one of the files from bigint in OBJ_FILES (perhaps "bigint.o" ?). I don't think your wildcard is getting what you want.