I have a binary which is linked using a large number of object files with many interdependencies. Whenever I recompile even one of these, I need to link the entire binary.
Do linkers (specifically, those of GCC or Clang) support some method of "differential linking", in which enough information is kept about the inter-relations between all the other linked parts, so that the only work that needs to be done when a single part is recompiled is its relations to the other parts + putting them together in the binary?
Note: I'm mostly interested in C++ but I guess this question generalizes at least to C and probably to other compiled languages.
In MSVC this is called "incremental linking". Interestingly, what I've found that GCC might support that to some extent, try using the "-Wl,-i" or "-Wl,-r" parameters to GCC (should be actually also supported by CLang, as these "-Wl" parameters are just passed to ld).
I never used it before, but I made this work with the following makefile:
OBJS := a.o b.o c.o main.o
all: test_app
test_app: test_app.reloc
g++ -o $# $^
# build a "relocatable" object for incremental linking (either -i or -r)
test_app.reloc: $(OBJS)
g++ -Wl,-i -nostdlib -nostartfiles -o $# $^
$(OBJS): makefile
%.o: %.cpp
g++ -c -o $# $<
This builds the app, but I'm not entirely sure what it does internally, if it really does something like "incremental linking" done in MSVC.
In particular, the parameter "-nostdlib" is necessary when using the "-Wl,-i" so that the default libs will not be passed to the ld (which then can't find them - without it I had the error "/usr/bin/ld: cannot find -lgcc_s").
Another version which might actually work better (not sure, this would need to be tested on a bigger application to see if there is some gain in the link time for single object updates):
OBJS := a.ro b.ro c.ro main.ro
all: test_app
test_app: $(OBJS)
g++ -o $# $^
%.o: %.cpp
g++ -c -o $# $<
%.ro: %.o
g++ -Wl,-i -nostdlib -nostartfiles -o $# $<
Basically creating relocatable file for each object (which might be perhaps a significant portion of the linkage of obj files into the executable) and then just updating the relocatables necessary. For the final link step using the relocatables to link everything together (but part of the linkage has been already done before).
It is also possible to create "groups" of object files to be grouped in a single relocatable, so that there will be less of them at the end (not sure if that would bring anything at the end though).
Related
I am trying to compile a TBB and OpenMp comparison program that I made. It is compiling fine with the default visual studio compiler. So, I know that TBB is installed correctly. However, I would like to use g++ instead. I have created a Makefile, and from what I read the -ltbb flag is needed.
My error is, "c:/mingw/bin/../lib/gcc/mingw32/9.2.0/../../../../mingw32/bin/ld.exe: cannot find -ltbb".
I am not an expert when it comes to making sure I have everything linked correctly and am still trying to learn.
This is my current Makefile
CPLUSPLUS = g++
OPTFLAGS = -O3
TBB_INCLUDE_PATH = C:\tbb-2020.1-win\tbb\include
TBB_LIBRARY_PATH = C:\tbb\tbb\lib\intel64\vc14
all: pps
pps: avl.o main.o parPlaneSweep.o
$(CPLUSPLUS) -I$(TBB_INCLUDE_PATH) -L$(TBB_LIBRARY_PATH) $(OPTFLAGS) -o $# $^ -ltbb
avl.o: avl.h avl.c
$(CC) -c $(OPTFLAGS) -fPIC avl.c
main.o: main.cpp parPlaneSweep.h
$(CPLUSPLUS) -c $(OPTFLAGS) -fopenmp main.cpp
parPlaneSweep.o: parPlaneSweep.h parPlaneSweep.cpp
$(CPLUSPLUS) -c $(OPTFLAGS) -fPIC -fopenmp parPlaneSweep.cpp
clean:
rm *.o
rm pps
Please update your question rather than pointing people at other websites.
First, you should never use backslashes in makefiles, even on Windows (there are exceptions to this on Windows but they're very rare). Always use forward slashes as directory separators.
Second, you define the variables TBB_INCLUDE_PATH and TBB_LIBRARY_PATH but then you never use them. Just mentioning the name of the variable doesn't use the variable. You have to include it in $(...) to use it, like $(TBB_INCLUDE_PATH).
Finally, all common linkers are single-pass linkers, which means the order in which you put the libraries and object files on the link line is critically important. You should always put the object files first, and the libraries last. If you have multiple libraries the order in which they appear may be important as well. Your link line should be something like this:
pps: avl.o main.o parPlaneSweep.o
$(CPLUSPLUS) -I$(TBB_INCLUDE_PATH) -L$(TBB_LIBRARY_PATH) $(OPTFLAGS) -o $# $^ -ltbb
If you want to know what $# and $^ mean, you can read about automatic variables.
Is there a way to have g++ check for C++98 syntax when compiling but at the same time compile as if no -std= has been given ? My goal is to make sure that my source code stays C++98 but I don't want to prevent g++ from using any newer optimisation or trick. For the moment, I compile my projet twice, once with CXXFLAGS=-std=c++98 and one with a final empty CXXFLAGS for release.
It looks like gcc 5 will have -Wc90-c99-compat and -Wc99-c11-compat, that something in that direction.
You will have to run the compiler twice, but you can save compiletime
on the -std=c++98 pass and avoid generating unwanted object files by specifying syntax-checking only. You do that by
passing the option -fsyntax-only.
You'd also need to modify your make to skip linkage for C++98, as there'll be nothing to link.
Probably the most efficient way you could do this is with a make on the
following lines:
.phony: all clean
SRCS = foo.cpp
FLAT_SRCS = $(patsubst %.cpp,%.ii,$(SRCS))
OBJS = $(patsubst %.ii,%.o,$(FLAT_SRCS))
%.ii: %.cpp
g++ -std=c++98 $(CPPFLAGS) -E $< > $# && g++ -std=c++98 -fsyntax-only $#
%.o: %.cpp
%.o: %.ii
g++ -c -o $# $(CPPFLAGS) $(CXXFLAGS) $<
all: foo
foo: $(OBJS)
g++ -o $# $(CXXFLAGS) $(LDFLAGS) $(OBJS) $(LDLIBS)
clean:
rm -f $(OBJS) $(FLAT_SRCS) foo
Here, the %.ii: %.cpp rule will first just preprocess the .cpp to the .ii,
then pass the already preprocessed source to a C++98 syntax-checking pass,
which produces no new file.
The empty %.o: %.cpp rule overrides the implicit rule that would otherwise
cause the .cpp to be compiled to .o, and it is replaced with the %.o: %.ii
rule to compile the .ii. g++ recognises .ii as denoting already-preprocessed
C++ source code, so it will not preprocess the source a second time.
Code is only preprocessed once, and object code is only generated once.
The linkage is as usual. Provided the C++98 syntax check passes, a make will look like:
$ make
g++ -std=c++98 -E foo.cpp > foo.ii && g++ -std=c++98 -fsyntax-only foo.ii
g++ -c -o foo.o foo.ii
g++ -o foo foo.o
rm foo.ii
You'll note that make automatically deletes the preprocessed .ii, which is fine:
it's just a conduit between the .cpp and the .o.
All that being said, I side with #Matt McNabb's observation that you have nothing to
gain by this! Given your code is C++98, the compiler won't optimize it any better
when instructed that it must be C++98 than when not. By time GCC gets to the
optimization stage of its business, it no longer cares what kind of source code
it started off with.
You may be supposing that -std=c++98, when given say to g++ 4.x, causes the
whole compiler to behave as if was g++ 2.x. Not so. It's still g++ 4.x, with the
g++ 4.x optimization tech, etc., just operating by the language definition of C++98.
There would certainly be a point in this if your code for some reason had to pass,
as C++98, on some older compiler than your release compiler, and in that case you
would need to distinguish the compilers in the makefile. But apparently this is not the
case. You might as well just compile conventionally with -std=C++98
I've been building a C++11 library, and the number of header/source files has grown to the point where compiling programs invoking it, entails passing 20+ .cpp files to g++. I've been reading up on shared libraries and it seems to be the best solution.
However, as headers/source change frequently, I'm hoping to create a makefile that would automatically generate all the .so files from the headers and source.
To better demonstrate what I'm trying to do, I'll take one of my sub-libraries, Chrono and show how I would do this manually.
I first create the object files like so,
$ g++ -std=c++11 -fPIC -g -c -Wall ../src/Chrono/cpp/DateTime.cpp
$ g++ -std=c++11 -fPIC -g -c -Wall ../src/Chrono/cpp/Schedule.cpp
$ g++ -std=c++11 -fPIC -g -c -Wall ../src/Chrono/cpp/Duration.cpp
$ g++ -std=c++11 -fPIC -g -c -Wall ../src/Chrono/cpp/DayCount.cpp
So that I now have DateTime.o, Schedule.o, Duration.o, and DayCount.o in the current directory. I then create the .so file,
$ g++ -shared -Wl,-soname,libChrono.so.1 -o libChrono.so.1.0.1 DateTime.o Schedule.o Duration.o DayCount.o -lc
I then go,
$ rm ./*.o && ldconfig -n ./
So that my working directory now contains, libChrono.so.1.0.1 and the symlink libChrono.so.1.
There are quite a few subdirectories I need to do this for, so you can see that this quickly grows inefficient whenever changes to headers/source are made. I would be grateful if anyone can help me design a makefile that accomplishes all this simply by invoking make.
Thanks!
UPDATE:
Based on goldilock's advice and some digging, I managed to bang together:
CXX=g++
CFLAGS=-std=c++11
TARGET=./lib/libChrono.so.1.0.1
CHRONODIR=./src/Chrono
CHRONOSRC=$(wildcard $(CHRONODIR)/cpp/*.cpp)
CHRONOOBJ=$(join $(addsuffix ../obj/, $(dir $(CHRONOSRC))), $(notdir (CHRONOSRC:.cpp=.o)))
all: $(TARGET)
#true
clean:
#-rm -f $(TARGET) $(CHRONOOBJ)
./lib/libChrono.so.1.0.1: $(CHRONOOBJ)
#echo "======================="
#echo "Creating library file $#"
#echo "======================="
#$(CXX) -shared -Wl,-soname,$(join $(basename $#), .1) -o $# $^ -l
#echo "-- $# file created --"
$(CHRONODIR)/cpp/../obj/%.o : $(CHRONOSRC)
#mkdir -p $(dir $#)
#echo "============="
#echo "Compiling $<"
#$(CXX) $(CFLAGS) -fPIC -g -Wall -c $< -o $#
4 .o files are produced in lib/ but I get multiple definition complaints from ld. Before I was compiling the object files separately, but this unwinds CHRONOOBJ on one line. Any ideas?
Fortunately you included the origin of your problem:
I've been building a C++11 library, and the number of header/source files has grown to the point where compiling programs invoking it, entails passing 20+ .cpp files to g++.
Because this reveals a potential XY problem. The straightforward solution to this is to put object files into an archive (aka. a static library) and use that.
GNU make has an implicit rule for creating C++ .o files. It amounts to this:
%.o: %.cpp
$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $<
Meaning, if you make DateTime.o in a directory with a makefile that doesn't redefine this, it will make DateTime.o. You may want to add things to $(CXXFLAGS) however, e.g.:
CXXFLAGS += -Wall -Wextra --std=c++11
If you intend to stick with the shared lib route, -fPIC can go there too. That one line could be your entire makefile.
However, you also want to put these together, so you must first declare all the objects and a rule for combining them:
OBJS = DateTime.o Schedule.o Duration.o
libChrono.a: $(OBJS)
ar crvs $# $^
This last line (see man ar) creates the archive (libChrono.a) containing all the objects in $(OBJS). You can then use this with whatever program by placing it in the same directory (or a directory in the library path) and linking -lChrono. Only the necessary parts will be extracted and compiled in. This saves you having to maintain a shared lib in a system directory.
If you still think you need a shared lib, $# and $^ are automatic variables; you can use similar methodology to create a .so, something along the lines of:
SO_FLAGS = -shared
libChrono.so.1.0.1: $(OBJS)
$(CXX) $(SO_FLAGS) -Wl,-soname,libChrono.so.1 -o $# $^ -lc
If that is your first rule, make will take care of everything: building first the objects and then the library. Notice this one has excluded your normal $(CXXFLAGS) to duplicate exactly the compiler line from the question.
I have a makefile that's been working pretty great for the past couple weeks. However, now that I've added another target to my project with "main", it starts to flip out a bit.
CC=g++
CCOPTS=-g -w
OBJS = $(BINDIR)/manager.o $(BINDIR)/rtngnode.o
TARGETS = $(BINDIR)/manager $(BINDIR)/rtngnode
BINDIR = build
all: $(TARGETS) $(OBJS)
clean:
rm -f $(TARGETS) $(OBJS)
.PHONY: all clean
$(TARGETS): $(OBJS)
$(CC) -o $# $^ $(CFLAGS) $(LIBS)
$(BINDIR)/%.o: %.cpp
$(CC) -c $(CCOPTS) -o $# $<
I don't quite understand makefiles still... so I'm not sure what's going on. I get these two errors:
build/rtngnode.o: In function `__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<int const, Node> > >::new_allocator()':
/home/caleb/Documents/dev/tote2/mp2/rtngnode.cpp:5: multiple definition of `main'
build/manager.o:/home/caleb/Documents/dev/tote2/mp2/manager.cpp:161: first defined here
build/manager.o: In function `main':
manager.cpp:(.text+0xefb): undefined reference to `NetworkConnection::NetworkConnection(char const*, char const*)'
In my rtngnode target, I have dependancies on the tcpcon class... and my manager target also has a main() reference. I'm pretty confused... so not even sure if I'm asking the right question.
I guess my question is:
For OBJS and TARGETS, what goes there? All of my .cpp files? All of my executables?
How do I like the two together?
The other answers are all accurate, but I think the key thing to understand is this rule:
$(TARGETS): $(OBJS)
$(CC) -o $# $^ $(CFLAGS) $(LIBS)
doesn't do what you seem to think that it does. This expands to:
$(BINDIR)/manager $(BINDIR)/rtngnode : $(BINDIR)/manager.o $(BINDIR)/rtngnode.o
$(CC) -o $# $^ $(CFLAGS) $(LIBS)
Make doesn't try to infer some magical matching up of targets to prerequisites here; instead make repeats the entire rule again for each target, but with all the prerequisites. The rule above is identical to writing this:
$(BINDIR)/manager : $(OBJS)
$(CC) -o $# $^ $(CFLAGS) $(LIBS)
$(BINDIR)/rtngnode : $(OBJS)
$(CC) -o $# $^ $(CFLAGS) $(LIBS)
Given this you can see what the problem is: you're trying to link ALL the objects into EACH executable.
Since your programs are built from a single object file with the same prefix as the program name, you actually don't need to write a rule for them at all; you can rely on make's built-in rule for building the programs. Just take the above rule out completely and it should work.
You should bind your executable with exactly one source file containing the "main" function.
As linker tells you, both manager.cpp and rtngnode.cpp have the definition of main function there. You probably do not want to combine them together. I advice you to start from manually building your code using the gcc commands (using gcc -c to produce object files and gcc -o to produce the executable). Once you understand the logic there - proceed writing makefiles.
By the way, if you can choose your build environment - start with cmake which is less cryptic than makefiles.
The most important thing here has nothing to do with your Makefile (which looks reasonably good except for the somewhat lacking use of compiler warnings):
You may have only one main() in your project, across all source files.
So basically, TARGETS should contain your source files (.cpp) which are then compiled to create object files, which should be in the OBJS variable so they are then passed to the linker to create the executable file.
As of right now your Makefile seems correct, the error tells you that you have 2 main() functions declared, one in the rtngnode.cpp file and again in the manager.cpp, this cannot happen or else the machine would not know which one to call when the program is executed.
Also undefined reference errors are most of the time due to a linker error, happening most of the time when you include a library to the compiler with #include but not to the linker usually using the LIBS variable in your makefile.
HERE is a small tutorial on Makefiles, which could also be a good read.
Currently, this is how I build my object files for my x86 compiler.
CC=g++
%.o: %.cpp
$(CC) $(CFLAGS) $(INCPATH) $(LIBPATH) -DPC -c $^ -o $#
How can I easily switch between g++ and my cross compiler? How can I compile both x86 and binaries for my target at the same time? I seem to be limited to compiling with only one compiler right now. Is there a way to come about this without having to list all my object files one by one?
You could pass the tool chain while giving make
Just an extra variable CROSS_COMPILE would do that
The makefile would rather look like this
CROSS_COMPILE :=
CC=$(CROSS_COMPILE)g++
%.o: %.cpp
$(CC) $(CFLAGS)$(INCPATH) $(LIBPATH) -DPC -c $^ -o $#
So if u just give make then CC is used as g++ and get compiled for x86, if you give make CROSS_COMPILE=arm-none-linux-gnueabi- then it would be compiled for ARM
You can use whatever the tool chain you want the concept remains the same
Sagar's answer does address how to switch between compilers, but does not address how to both do that and the main part of the question, which is how to do both targets at the same time.
To do both targets at the same time, you probably want to separate the objects into directories related to the compiler or target used, and then based on the directory of object, figure out what compiler to use.
So, something like below, in the simplest case, where the sources are in current directory, and the directory is the same as the compiler name. Fill in the user-defined COMPILER function with how you like to calculate your compiler name, based on the directory of the object file, which is the argument $1 to the function.
define COMPILER
$1
endef
.PRECIOUS: %/.
%/.:
mkdir -p $#
.SECONDEXPANSION:
%.o: $$(notdir $$*.cpp) | $$(#D)/.
$(call COMPILER, $(#D)) -c $< -o $#