How to properly use make with parallel builds? - build

I am having problems with parallel builds in make, namely having this definition
rmlink:
$(RM) $(ALIB)
$(RM) $(BLIB)
$(RM) $(CLIB)
$(RM) $(DLIB)
$(RM) $(ELIB)
mklink:
ln -sf $(ALIB_WITHPATH) $(ALIB)
ln -sf $(BLIB_WITH_PATH) $(BLIB)
ln -sf $(CLIB_WITH_PATH) $(CLIB)
ln -sf $(DLIB_WITH_PATH) $(DLIB)
ln -sf $(E_LIB) $(ELIB)
and launching build with make I will have those targets build
$ make
/bin/rm -f libA.so
/bin/rm -f libB.so
/bin/rm -f libC.so
/bin/rm -f libD.so
/bin/rm -f libE.so
ln -sf ../libA.so libA.so
ln -sf ../libB.so libB.so
ln -sf ../libC.so libC.so
ln -sf ../libD.so libD.so
ln -sf ../libE.so libE.so
But launching it with parallel build with make -j 2 (or more) I would get :
$ make -j 2
/bin/rm -f libA.so
ln -sf ../libB.so libB.so
/bin/rm -f libB.so
/bin/rm -f libC.so
/bin/rm -f libD.so
ln -sf ../libD.so libD.so
/bin/rm -f libE.so
ln -sf ../libA.so libA.so
ln -sf ../libC.so libC.so
ln -sf ../libE.so libE.so
g++: libB.so: No such file or directory
Which in this particular case would remove my link to libB.so and won't create it again - breaking the build in effect.
How can I fix this to be able to launch parallel builds ?
EDIT
Adding something like this to mklink target seems to fix the issue:
mklink: rmlink
Please comment if this is the correct way.

When considering parallel builds, make decides the order in which to build things solely by the prerequisite lists. If target FOO is not defined as a prerequisite of target BAR, then make will feel free to build both FOO and BAR at the same time.
Remember that having two targets listed as prerequisites to the same target, as in TARGET : FOO BAR, just means that both FOO and BAR must be built before TARGET. It does NOT mean that FOO and BAR have any relationship to each other, at all. So if you have:
foo: rmlink mklink
then make is free to run both rmlink and mklink at the same time. If you want to have rmlink always completed before mklink is started, you have to declare a prequisite relationship between them, like:
mklink: rmlink
And of course, remember you must ensure that the mklink target is a prerequisite of any linker target that needs those links to exist.
There may well be more wrong here but given the parts of the makefile you've shown and the question you asked, that should fix it.

Related

Build error: make: Nothing to be done for 'compile'

I need to add my own package to the openwrt image. On the wiki of the project I found this article.
I tried to follow the instructions for it, but in the end I did not manage to add my own package to the source code tree (the build ignored its presence).
Because of this, I tried to find some other way. And it turned out to be a this instruction. I followed the directions from there and compiled my own package.
But as you can see, the source code of that package does not depend on others and does not require any other build header files. Also, his Makefile completely includes instructions for compiling.
define Build/Compile
$(TARGET_CC) $(TARGET_CFLAGS) -o $(PKG_BUILD_DIR)/helloworld.o -c $(PKG_BUILD_DIR)/helloworld.c
$(TARGET_CC) $(TARGET_LDFLAGS) -o $(PKG_BUILD_DIR)/$1 $(PKG_BUILD_DIR)/helloworld.o
endef
But now this does not suit me, since another package that I want to add already has such dependencies.
I tried to bypass them like this (copy the source code to the build folder and call the makefile located there) but nothing came of it:
define Build/Prepare
echo $PKG_NAME
mkdir -p $(PKG_BUILD_DIR)
cp $(SOURCE_DIR)/* $(PKG_BUILD_DIR)
$(Build/Patch)
endef
define Build/Compile
$(PKG_BUILD_DIR) $(MAKE)
endef
I am getting next output:
$ make -C package/feeds/mypackages/helloworld compile TOPDIR=$PWD
make: Entering directory '/home/username/mypackages/examples/helloworld'
bash: mkhash: command not found
bash: mkhash: command not found
bash: mkhash: command not found
bash: mkhash: command not found
bash: mkhash: command not found
bash: mkhash: command not found
bash: mkhash: command not found
bash: mkhash: command not found
make: Nothing to be done for 'compile'.
make: Leaving directory '/home/username/mypackages/examples/helloworld'
My full Makefile for both package and binary:
include $(TOPDIR)/rules.mk
PKG_NAME:=helloworld
PKG_VERSION:=1.0
PKG_RELEASE:=1
SOURCE_DIR:=/home/username/helloworld
include $(INCLUDE_DIR)/package.mk
define Package/$(PKG_NAME)
SECTION:=utils
DEPENDS:= +libstdcpp
TITLE:=helloworld
endef
define Package/helloworld/description
A simple "Hello, world!" -application.
endef
define Build/Prepare
echo $PKG_NAME
mkdir -p $(PKG_BUILD_DIR)
cp $(SOURCE_DIR)/* $(PKG_BUILD_DIR)
$(Build/Patch)
endef
define Build/Compile
$(PKG_BUILD_DIR) $(MAKE)
endef
define Package/helloworld/install
# Install binary
#$(INSTALL_DIR) $(1)/usr/bin
#$(INSTALL_BIN) $(PKG_BUILD_DIR)/helloworld $(1)/usr/bin/
endef
$(eval $(call BuildPackage,$(PKG_NAME)))
,
TARGET = heloworld
OBJS = heloworld.o
CFLAGS += -Wall -Wextra
LDFLAGS += -lxsacpp -lxsac -lubus -lubox
CXXFLAGS += $(CFLAGS) -std=c++14
%.o : %.cpp
$(CXX) -c $(CXXFLAGS) $< -o $#
all: $(TARGET)
$(TARGET): $(OBJS)
$(CXX) $(LDFLAGS) -o $# $^
clean:
rm *.o $(TARGET)
And actually my question is, what needs to be set in the Makefile to copy files correctly and call the local Makefile for package binary?
In order to copy files and directories you can use below step:
# copy all files and directories using **cp -r -f **
define Build/Prepare
echo $PKG_NAME
mkdir -p $(PKG_BUILD_DIR)
cp -r -f $(SOURCE_DIR)/* $(PKG_BUILD_DIR)
$(Build/Patch)
endef
In order to execute the local makefile use below step:
# Execute local makefile by giving path using '-C' option
define Build/Compile
`$(MAKE) -C $(PKG_BUILD_DIR)`
endef

link static library with shared library problem

I'm new to Linux, so sorry if this my question is really stupid.
I have shared cross-platform library project, which uses third party static library (Libtorrent).
Windows/Android/macOS - builds and works fine.
But I can't figure out how to build it in Linux (Ubuntu) with GCC.
I'm getting the following linker errors:
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(create_torrent.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(disk_io_thread.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(peer_connection.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(session.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
I've tried rebuilding Boost and Libtorrent with -fPIC flag: nothing is changed.
Boost build script:
#!/bin/bash
set -e
BOOST_VERSION=1.63.0
BOOST_VERSION_UNDERSCORES=${BOOST_VERSION//./_}
echo "Boost version $BOOST_VERSION"
echo "Downloading..."
BOOST_ARCH_NAME=boost_${BOOST_VERSION_UNDERSCORES}.tar.gz
curl -O -L https://dl.bintray.com/boostorg/release/$BOOST_VERSION/source/$BOOST_ARCH_NAME
echo "Extracting..."
tar -xvzf $BOOST_ARCH_NAME
BOOST_FOLDER_NAME=boost_${BOOST_VERSION_UNDERSCORES}
cd $BOOST_FOLDER_NAME
chmod +x bootstrap.sh
./bootstrap.sh
chmod +x b2
./b2 headers
./b2 \
--layout=versioned \
--with-thread \
--with-date_time \
--with-filesystem \
--with-chrono \
--with-random \
toolset=gcc \
threading=multi \
link=static \
runtime-link=shared \
variant=release \
threadapi=pthread \
debug-symbols=off \
warnings=off \
warnings-as-errors=off \
architecture=x86 \
address-model=64 \
--stagedir=stage/linux \
cxxflags="-std=gnu++0x -fPIC" \
stage
echo "Copying prebuilt..."
mkdir -p ../prebuilt/linux/lib
cp -rf ./stage/linux/lib ../prebuilt/linux/
cd ../
echo "Cleaning up..."
rm $BOOST_ARCH_NAME
rm -rf $BOOST_FOLDER_NAME
echo "Done."
Libtorrent build script:
#!/bin/sh
set -e
MY_PWD=$PWD
LIBTORRENT_VERSION=1_1_8
LIBTORRENT_NAME=libtorrent-$LIBTORRENT_VERSION
export LIBTORRENT_FOLDER_NAME=libtorrent-$LIBTORRENT_NAME
curl -O -L "https://github.com/arvidn/libtorrent/archive/${LIBTORRENT_NAME}.tar.gz"
echo "Extracting..."
rm -rf ./$LIBTORRENT_FOLDER_NAME
tar xfz "${LIBTORRENT_NAME}.tar.gz"
echo "Building for linux..."
cd $LIBTORRENT_FOLDER_NAME
. ./autotool.sh
mkdir -p lib
if [ ! -d "./include/boost" ]; then
ln -s $PWD/../../boost-build/prebuilt/include/boost $PWD/include/boost
ln -s $PWD/../../boost-build/prebuilt/linux/lib/* $PWD/lib/
fi
if [ ! -d "./include/openssl" ]; then
ln -s $PWD/../../openssl-build/prebuilt/include/openssl $PWD/include/openssl
ln -s $PWD/../../openssl-build/prebuilt/linux/lib/* $PWD/lib/
fi
export CXX=g++
export CC=gcc
export CXXFLAGS="-std=gnu++0x -fPIC -fPIE"
export CPPFLAGS="-I"$PWD"/include"
./configure --enable-debug=no --with-boost-python=no --enable-logging=no \
--enable-tests=no --with-openssl=$PWD \
--with-boost=$PWD \
--enable-shared=no \
--enable-encryption
make -j4
mkdir -p ../prebuilt/linux/lib
cp -rf ./src/.libs/libtorrent-rasterbar.a ../prebuilt/linux/lib/
cd $MY_PWD
echo "cleaning up"
rm -rf $LIBTORRENT_FOLDER_NAME
rm -rf ${LIBTORRENT_NAME}.tar.gz
echo "done"
My shared lib itself is a Qt-based library. I build it using Qt Creator, providing all the required headers and libs in .pro file.
The problem was in Libtorrent build script. I removed -fPIE option and added
export CFLAGS="-fPIC"
Rebuilt and now it links without errors.

Makefile : How to install a project with a file pkg-config

I am trying to make a clean install of my c++ project :
exports the .h inside an include/ dir
compile a .dylib (macos) or .so (ubuntu) inside a lib/ dir
Now the question is : How do I write/export a pkg-config .pc file, with the makefile, that will correspond to my project. In Mac OS and Linux, so I can just call
LDFLAGS=$(shell pkg-config --libs my_project)
CPPFLAGS=$(shell pkg-config --cflags my_project)
In other makefiles, to get the flags to include.
This is what I done so far (macos):
install : $(OBJS)
mkdir -p $(INSTALL_LIB_DIR)
$(CXX) -dynamiclib $^ $(CPPFLAGS) $(LDFLAGS) -o $(INSTALL_LIB_DIR)/lib$(ENGINE_LIB_NAME).dylib
mkdir -p include
cp src/*.h include/
cp src/*.tpp include/
mv include $(INSTALL_DIR)
rm -rf include
rm -rf lib
uninstall:
#echo "Removing dir " $(INSTALL_DIR)
rm -r $(INSTALL_DIR)
Thank you very much.
If it can't be done, I will look into doing that with cmake instead...

Issue with Makefile - section executed twice

I may not be a Makefile-guru, but I have some experience with them in the past. However, I've got a (rather unimportant) issue, which still baffles me.
Here's my Makefile :
#-------------------------
# Definitions
#-------------------------
APP = lgm
# Tools & commands
CC = gcc
LEX = lex
YACC = /usr/local/bin/bison
RM = rm
CP = cp
MV = mv
DMD = dmd
# Paths
SRC = src
BIN = bin
TEST = test
LIB = lib
# C stuff
CC_HEADERS = logramm.tab.h
CC_SOURCES = lex.yy.c logramm.tab.c
CC_OBJECTS = lex.yy.o logramm.tab.o
CC_LEXER = logramm.l
CC_PARSER = logramm.y
# D stuff
D_SOURCES = main.d
D_OBJECTS = main.o
D_LFLAGS = -m64
#-------------------------
# Main Functions
#-------------------------
all: ${APP} clean
${APP}: ${CC_OBJECTS} ${D_OBJECTS}
${DMD} ${CC_OBJECTS} ${D_OBJECTS} -of${APP} ${D_FLAGS}
${MV} ${APP} ${BIN}
${D_OBJECTS}:
${DMD} -c ${D_SOURCES}
${CC_OBJECTS}: ${CC_SOURCES}
${CC} -g -c ${CC_SOURCES}
${CC_SOURCES}: setup
${LEX} ${CC_LEXER}
${YACC} -d ${CC_PARSER}
setup:
${CP} ${SRC}/*.d .
${CP} ${SRC}/*.l .
${CP} ${SRC}/*.y .
clean:
${RM} *.d *.y *.l *.o *.hh *.c *.h
The trouble is :
When I'm looking at the output of make all, the ${CC_SOURCES} part is executed twice (2 lex, 2 bison commands). E.g. It outputs :
cp src/*.d .
cp src/*.l .
cp src/*.y .
lex logramm.l
/usr/local/bin/bison -d logramm.y
lex logramm.l
/usr/local/bin/bison -d logramm.y
gcc -g -c lex.yy.c logramm.tab.c
dmd -c main.d
dmd lex.yy.o logramm.tab.o main.o -oflgm
mv lgm bin
rm *.d *.y *.l *.o *.hh *.c *.h
rm: *.hh: No such file or directory
make: *** [clean] Error 1
Why is that? What am I doing wrong?
UPDATE :
I've just managed to fix it, by taking setup from ${CC_SOURCES} and putting it at the all section, like :
all: setup ${APP} clean
So, it's ok. However, I still don't understand why. Have I misinterpreted the way Makefiles are structured? Any input would be more than welcome! :-)
You have a basic misunderstanding of how make interprets rules with multiple targets: this makefile is really not going to work. Let's expand the rule that runs the lex/yacc:
lex.yy.c logramm.tab.c: setup
${LEX} ${CC_LEXER}
${YACC} -d ${CC_PARSER}
Make interprets this as if it were this (in an explicit rule, multiple targets create a separate rule for each target):
lex.yy.c: setup
${LEX} ${CC_LEXER}
${YACC} -d ${CC_PARSER}
logramm.tab.c: setup
${LEX} ${CC_LEXER}
${YACC} -d ${CC_PARSER}
Now you can probably see your issue. Make wants to build lex.yy.o and that depends on lex.yy.c, so make tries to build that. It sees that it depends on setup, which doesn't exist (because nothing creates it, but if someone WERE to create that file in your directory it would break badly). So, it run the lex and yacc steps (both, since that's the recipe) to build lex.yy.c.
Then it goes through exactly the same thing to build logramm.tab.c... thus the rules are run twice.
You have an identical problem with your object files; expanding that rule we see:
lex.yy.o logramm.tab.o: lex.yy.c logramm.tab.c
${CC} -g -c lex.yy.c logramm.tab.c
which is identical to:
lex.yy.o: lex.yy.c logramm.tab.c
${CC} -g -c lex.yy.c logramm.tab.c
logramm.tab.o: lex.yy.c logramm.tab.c
${CC} -g -c lex.yy.c logramm.tab.c
which is clearly not right.
Also, your setup rule (since it never exists) will cause everything to rebuild every time.
Your solution is better WRT setup, since the source files don't depend on it so they won't always be rebuilt. You still have the broken rules for object files. Also your solution will not work if you ever wanted to enable parallel builds, because the setup may be run in parallel with the other targets: you really WANT the dependency to ensure ordering.
In general when writing makefiles you should (a) have one rule to build one target, and (b) ensure the rule builds EXACTLY the target you told make it would build.
Try using this instead:
all: ${BIN}/${APP} clean
${BIN}/${APP}: ${APP}
${CP} $< $#
${APP}: ${CC_OBJECTS} ${D_OBJECTS}
${DMD} $^ -of$# ${D_FLAGS}
%.o : %.d
${DMD} -c $<
%.o: %.c
${CC} -g -c $<
%: ${SRC}/%
${CP} $< $#
lex.yy.c: ${CC_LEXER}
${LEX} $<
%.tab.c %.tab.h: %.y
${YACC} -d $<
clean:
${RM} *.d *.y *.l *.o *.hh *.c *.h
The trouble is that CC_SOURCES = lex.yy.c logramm.tab.c so the target line (plus actions):
${CC_SOURCES}: setup
${LEX} ${CC_LEXER}
${YACC} -d ${CC_PARSER}
is equivalent to:
lex.yy.c: setup
${LEX} ${CC_LEXER}
${YACC} -d ${CC_PARSER}
logramm.tab.c: setup
${LEX} ${CC_LEXER}
${YACC} -d ${CC_PARSER}
The setup rule has no dependencies, but isn't a (GNU make extension) .PHONY target either (and I'm not sure that being a phony target would help, either), so it doesn't exist. Therefore, when make is trying to ensure that lex.yy.c is up to date, it has to execute the rules for setup; ditto for logramm.tab.c; hence the double execution.
Your solution looks reasonable. You could explore .PHONY and see whether that helps if you're using GNU make and don't mind being tied to GNU make.
This comes down to the way make handles dependencies. You can always examine exactly what make is doing by passing the -d flag (at least for GNU Make, ymmv, also consider adding -r if you aren't using any built-in rules as they clutter the output substantially).
Here's what happens in the case where setup is a dependency of ${CC_SOURCES}
...
Does lex.yy.c exist? no, okay check its prerequisites
Does setup exist? no, okay check its prerequisites
No prerequisites, run the recipe
Prerequisites for lex.yy.c done, run the recipe
Does logramm.tab.c exist? yes, but check its prerequisites
We already built setup, so prune this prerequisite, but treat it as newer than logramm.tab.c
Prerequisites for logramm.tab.c done, run the recipe
...
Here's what happens in the case where setup is a dependency of all
...
Does setup exist? no, okay check its prerequisites
No prerequisites, run the recipe
...
Does lex.yy.c exist? no, okay check its prerequisites
No prerequisites, run the recipe
Does logramm.tab.c exist? yes, but check its prerequisites
No prerequisites, so done
...

Makefile and linking .a libraries

I've got a src folder which, after running make, creates 4 libraries in .a format.
There's also a folder tests which contains tests.cpp. This file depends on the libraries previously mentioned.
I'm trying to get a Makefile inside tests that generates the libraries inside src (this it does) and then uses these to compile tests.cpp (this it doesn't - I get many many undefined references, as if it weren't linking properly).
The folder tests contains this Makefile:
include ../src/makefile.inc
DIR = ../src
OBJLIBS = ../src/CapaFisica.a ../src/CapaLogica.a ../src/CapaInterfaz.a ../src/Utilitarios.a
TARGETS = tests.cpp
EXE = tests
all : $(EXE)
%.a :
-for d in $(DIR); do (cd $$d; $(MAKE)); done
tests: $(OBJLIBS)
$(CC) $(CFLAGS) $(OBJLIBS) -o $# $(TARGETS)
clean :
$(ECHO) Borrando archivos.
-$(RM) -f ./tests
cd $(DIR); make clean
find . -name "*.o" -type f -print | xargs /bin/rm -f
What am I doing wrong?
gcc linker is sensitive about the order of .o files and static libraries specified on the command line.
Replacing
$(CC) $(CFLAGS) $(OBJLIBS) -o $# $(TARGETS)
with
$(CC) $(CFLAGS) -o $# $(TARGETS) $(OBJLIBS)
might help. Also make sure that .a files in $(OBJLIBS) are in the correct order if they depend on each other. Depending library must be on the command line before the library which defines the symbols.
For more details see this question: Why does the order in which libraries are linked sometimes cause errors in GCC?