Makefile: Error while compiling src and test directories - c++

LIB_DIRS = abcd xyz pqr mnq ghj
SER_DIRS = klm
.PHONY: default build lib service clean distclean
TEST_DIRS = abcd xyz pqr
test_lib :
for dir in $(TEST_DIRS); do \
$(MAKE) -C $$dir; \
done
TESTS := $(addprefix TEST_, $(TEST_DIRS))
run : $(TESTS)
TEST_%:
./$*/test/$*_test --log_level=message
test : test_lib run
default: all
all: build lib service
lib:
for dir in $(LIB_DIRS); do \
$(MAKE) -C $$dir; \
done
Each LIB_DIR has a src and test directory. With the make all command only src directories files should get compiled. But in my case the test directories files also get compiled along with src directories. While with make test command only test directories files get compiled. Since test files are dependent on src files, errors occur with make all for test files. Can some one help me resolve this?

Well, here is a starter:
You should provide Makefiles in each directory that know how to build lib and test. Obviously test depends on lib.
You could then change your lines to
lib:
for dir in $(LIB_DIRS); do \
$(MAKE) -C $$dir; \
done
to
lib:
for dir in $(LIB_DIRS); do \
$(MAKE) -C $$dir lib; \ #<- notice the target
done
and similar for test_lib. Otherwise, it is not possible with the current set of variables since the sub-make does not know what to do.

Related

Build C++ project with makefile (without Cmake) and run tests using Jenkins

I am trying to build my makefile C++ project using Jenkins.
See project structure below. Project is on a bitbucket repository and job profile is set Freestyle Project.
Project is successfully built on Jenkins server however it looks like it just uploads the project from repository to its workspace and says "Finshed with success" but does not run a makefile.
Console output:
Checking out Revision 6720229e2d82a9e958f69afabe361c65d1647398 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 6720229e2d82a9e958f69afabe361c65d1647398
Commit message: "My commit"
> git rev-list --no-walk 084977a421fc8fb064297f64407e2d137a1b32a1 # timeout=10
Finished: SUCCESS
On my local however both test and main projects are built successfully with make.
Basically there are two questions:
How to build my project (including the test one) with Makefile on Jenkins? (i.e. how to run a make command on Jenkins). I do not want to use a Cmake. Is it possible?
If both projects built successfully, how to run the test project and see test results in console/write to file in Jenkins?
My project structure:
MyProject
|+src/ <-- source files main project
|+include/ <-- header files main project
|+bin/ <-- binaries main project
|+test/ <-- test project
|~test/
| |+bin/ <-- test binaries
| |+gtest/ <-- gtest headers
| |+lib/ <-- gtest binaries
| |-test.cpp <-- test source
|-Makefile
My Makefile:
# Compiler options and variables def
CC=g++
CPPFLAGS= -c -Wall
GFLAGS = -g
INC_DIR = include
INC_DIR_TEST = test
TST += \
*.cpp
VPATH += test/
TESTLIB += \
*.a
SRC += \
*.cpp
BIN = bin
TSTBIN = test/bin
# build
all: program test
OBJ = $(patsubst %.cpp, $(BIN)/%.o, $(SRC))
TSTOBJ = $(patsubst %.cpp, $(TSTBIN)/%.o, $(TST))
program: $(OBJ)
$(CC) $(GFLAGS) $? -o $#
test: $(TSTOBJ) $(TESTLIB) $(BIN)/file1.o $(BIN)/file2.o
$(CC) $(GFLAGS) $(TESTLIB) $(BIN)/file1.o $(BIN)/file2.o $< -o $#
$(BIN)/%.o: src/%.cpp
$(CC) $(CPPFLAGS) -I$(INC_DIR) $< -o $#
$(TSTBIN)/%.o: test/%.cpp
$(CC) $(CPPFLAGS) -I$(INC_DIR_TEST) -I$(INC_DIR) $< -o $#
clean:
rm *.o *.exe bin/*.o test/bin/*.o
Update:
As I understand, this type of project has to be marked as pipeline rather than freestyle project that will allow to choose a build tool and run shell command from Jenkinsfile file that facilitates considerably the CI process.
However I cannot find any examples of building C++ project with make and GNU build tools.
This is a Jenkinsfile example from official documentation
pipeline {
agent { docker 'maven:3.3.3' }
stages {
stage('build') {
steps {
sh 'mvn --version'
}
}
}
}
I am wondering now how this should be modified in order to build a simplest C++ project with makefile. Am I on a right way?
Second question is still actual: how to run my gtests and record results in Jenkins after the make command works?
Update:
I have used a batchfile as it can be ran from java code in Jenkinsfile. Now my testing Jenkinsfile looks like:
pipeline {
agent any
stages {
stage('build') {
steps {
echo 'building..'
bat 'batchfile.cmd'
}
}
}
}
Batchfile code:
PATH = "C:\Program Files (x86)\GnuWin32\bin"
make
There are still some bugs but at least make command is ran and the commands from makefile are called.
So general conclusion:
As it looks like there is no any make plugin for jenkins, the pipeline stages can run a batchfile (or shell script for UNIX), and this batchfile can call make. Of course make has to be installed on a server and the path to environment variable has to be specified.
Maybe there are some better approaches or I am wrong, please correct me.

qmake generates unnecessary dependency chains in generated makefile

I have the following command involving qmake:
cd HmiLogging/ && ( test -e Makefile ||\
c:/Users/mureadr/Desktop/A/HMI_FORGF/qt5host/win32/x86/bin/qmake.exe\
C:/Users/mureadr/Desktop/A/HMI_FORGF/src/HmiLogging/HmiLogging.pro\
top_builddir=C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/\
top_srcdir=C:/Users/mureadr/Desktop/A/HMI_FORGF/ -Wall CONFIG+=release\
CONFIG+=qnx_build_release_with_symbols CONFIG+=rtc_build -o Makefile ) &&\
C:/Users/mureadr/Downloads/make-4.2.1/Release/make_msvc.net2003 -f Makefile
qmakegenerates the file Makefile and then regular make is called on that makefile.
The qmake-generated makefile has some entries like so:
deploy_al2hmi-mappings: deploy_fonts
#echo Copying application data... && $(MKDIR)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/al2hmi-mappings"\
&& $(COPY_DIR) $(wildcard C:/Users/mureadr/Desktop/A/HMI_FORGF/src/fordhmi/al2hmi-mappings/*)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/al2hmi-mappings"
deploy_data: deploy_al2hmi-mappings
#echo Copying application data... && $(MKDIR)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/data"\
&& $(COPY_DIR) $(wildcard C:/Users/mureadr/Desktop/A/HMI_FORGF/src/fordhmi/data/*)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/data"
deploy_qml: deploy_data
#echo Copying application data... && $(MKDIR)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/qml" &&\
$(COPY_DIR) $(wildcard C:/Users/mureadr/Desktop/A/HMI_FORGF/src/fordhmi/qml/*)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/qml"
You can see that each target depends on the one above it however, examining the recipes you'll also see that the directories created and used are unique to each target!
QUESTION
The generated makefile is chaining those targets, as dependencies, when none actually exists and making it unable to do them in parallel! Why?
I've grep-ed all *.pro and *.pri files - the files that qmake uses to generate makefiles - but deploy_qml doesn't appear in any of them so I'm guessing that these are tasks "internal" to qmake.
Is there any way to tell qmake to NOT dependency-chain them when there is no good reason?

Building multiple targets off same source tree with different preprocessor macros

My knowledge of make and autotools (which I'm not yet using for this project) is rudimentary at best despite plenty of googling and experimenting over a long period of time. I have a source hierarchy like below that I'm trying to find way to build has seamlessly as possible.
The application is made up of a main application with source in various subfolders under app/src. These are built with the respective Makefile in the root of that folder.
Then I have multiple other utilities that reside different folders under app/tools that each have their own Makefile.
app/src/module1/file1.cpp
app/src/module1/file1.hpp
app/src/module2/file2.cpp
app/src/module2/file2.hpp
app/src/module3/file3.cpp
app/src/module3/file3.hpp
app/src/main.cpp
app/src/main.hpp
app/src/Makefile
app/tools/util1/file1.cpp
app/tools/util1/file1.hpp
app/tools/util1/Makefile
app/tools/util2/file2.cpp
app/tools/util2/file2.hpp
app/tools/util2/Makefile
The problem for me is that some of these tools depend on source files inside the app/src source folder, but with a preprocess macro EXTERNAL_TOOL enabled. So the object files generated from compiling the main app and the varous utilities are not compatible.
Currently to build each portion of the project I'm having to clean the source tree in between. This is painful and certainly not what I want in the end. What would be the best way to go about solving this? Ideas I've had that I've no been able to put into practice are:
Separate build directory for each portion of the project
When building the external tools, tagging their object files in the main app source tree somehow (util.file1.o?)
I'm not too sure I have the time and patience needed to master make / autotools. Might one of the other build tools (scons? cmake?) make this kind of task easier to accomplish? If so which one?
UPDATE: This is what I've got now
SOURCES := util1.cpp util2.cpp util3.cpp \
../../src/module1/file1.cpp \
../../src/module1/file2.cpp \
../../src/module1/file3.cpp \
../../src/module2/file4.cpp \
../../src/module3/file5.cpp \
../../src/module3/file6.cpp \
../../src/module4/file7.cpp \
../../src/module4/file8.cpp \
../../src/module3/file9.cpp \
../../src/module4/file10.cpp \
../../src/module5/file11.cpp \
../../src/module3/file12.cpp \
../../src/module1/file13.cpp \
../../src/module3/file14.cpp \
../../src/module3/file15.cpp
OBJECTS = $(join $(addsuffix .util/, $(dir $(SOURCES))), $(notdir $(SOURCES:.cpp=.o)))
.PHONY: all mkdir
all: util
util: $(OBJECTS)
$(CXX) $(CXXFLAGS) $(OBJECTS) $(LIBS) -o util
$(OBJECTS): | mkdir
$(CXX) -c $(CXXFLAGS) -o $# $(patsubst %.o,%.cpp,$(subst .util/,,$#))
mkdir:
#mkdir -p $(sort $(dir $(OBJECTS)))
clean:
-#rm -f $(OBJECTS) util
-#rmdir $(sort $(dir $(OBJECTS))) 2>/dev/null
I came about this after extensive googling SO browsing. This seems to work, but this part doesn't really seem particular nice (feels like a bit of a hack):
$(OBJECTS): | mkdir
$(CXX) -c $(CXXFLAGS) -o $# $(patsubst %.o,%.cpp,$(subst .util/,,$#))
In particular I'm not too keen on the fact I'm creating the list of objects from sources earlier on and adding the suffix, only to do the reverse down here. I couldn't seem to get it working any other way.
CMake has add_definitions and remove_definitions commands. You can use them to define macros for different parts of your project:
# building tools #
add_definitions(-DEXTERNAL_TOOL)
add_subdirectory($TOOL1$ $BUILD_DIR$)
add_subdirectory($TOOL2$ $BUILD_DIR$)
...
# building main app #
remove_definitions(-DEXTERNAL_TOOL)
add_executable(...)
This can be done with SCons rather painlessly. You will definitely need a build directory hierarchy for the objects built with different preprocessor macros. In SCons terms, creating build directories like this is called variant_dir. I would recomend the following SCons Hierarchical build structure:
app/SConstruct
app/src/module1/file1.cpp
app/src/module1/file1.hpp
app/src/module2/file2.cpp
app/src/module2/file2.hpp
app/src/module3/file3.cpp
app/src/module3/file3.hpp
app/src/main.cpp
app/src/main.hpp
app/src/SConscript_modules
app/src/SConscript_main
app/tools/util1/file1.cpp
app/tools/util1/file1.hpp
app/tools/util2/file2.cpp
app/tools/util2/file2.hpp
app/tools/SConscript
app/build/main/
app/build/target1/modules/
app/build/target2/modules/
app/build/tools/utils/
To be able to build the same source files with different preprocessor macros, you will need to build the same file with several different Environments. These env's could be setup in the src/module SConscript scripts, or from the root SConstruct and passed down. I prefer the second option, since it will make the src/module SCons scripts modular, and unaware (agnostic) of the preprocessor macros.
Here is the root build script, which creates the different env's and orchestrates the sub-directory build scripts:
app/SConstruct
defines1 = ['MACRO1']
defines2 = ['MACRO2']
env1 = Environment(CPPDEFINES = defines1)
env2 = Environment(CPPDEFINES = defines2)
includePaths = [
'src/module1',
'src/module2',
'src/module3',
]
env1.Append(CPPPATH = includePaths)
env2.Append(CPPPATH = includePaths)
# Build different versions of the module libs
SConscript('src/SConscript_modules',
variant_dir = '#build/target1/modules',
exports = {'env':env1},
duplicate=0)
SConscript('src/SConscript_modules',
variant_dir = '#build/target2/modules',
exports = {'env':env2},
duplicate=0)
# Build main with env1
SConscript('src/SConscript_main',
variant_dir = '#build/main',
exports = {'env':env2},
duplicate=0)
# Build tools with env2
SConscript('tools/SConscript',
variant_dir = '#build/utils',
exports = {'env':env2},
duplicate=0)
This is the build script for main
app/src/SConscript_main
Import('env')
sourceFiles = ['main.cpp']
# If you want to modify the env here, Clone() it first, otherwise
# the changes will be visible to all other SConscripts
env.Program(target = 'main', source = sourceFiles)
This is the build script for the module libs, it will be called twice, each time with a different env
app/src/SConscript_modules
Import('env')
module1SourceFiles = ['file1.cpp']
module2SourceFiles = ['file2.cpp']
module3SourceFiles = ['file3.cpp']
# If you want to modify the env here, Clone() it first, otherwise
# the changes will be visible to all other SConscripts
env.Library(target = 'module1', source = module1SourceFiles)
env.Library(target = 'module2', source = module2SourceFiles)
env.Library(target = 'module3', source = module3SourceFiles)

automake and project dependencies

I have a project that I want to build using automake. The project consists of different components or modules, and there are inter module dependencies which require the project to be built in a specific order.
For example:
project dir/
module1 (core C shared lib)
module2 (C++ shared lib wrapper around module 1)
module3 (C++ application with dependency on module2)
module4 (C library with dependency on module1)
module5 (C application with dependency on module4)
I am relatively new to automake, but I (just about) know how to use it to successfully build a single project.
I would like to have a 'master' project file (if that's possible), which specifies the build order of the projects modules, runs unit tests and fails the entire build process if either:
One of the modules fails to build
One of the modules fails a unit test
How would I go about writing such a 'master project' file (or invoking any other mechanism) to build projects that have a lot of inter-modular dependencies?
If you're using autotools, then you might as well use automake. The top level Makefile.am can provide a list of subdirectories that are descended in order, e.g:
SUBDIRS = module1 module2 module3 module4 module5
The subdirectory Makefile.am can add targets for tests, invoked with 'make check', which will force the build if necessary:
check_PROGRAMS = t_module1
t_module1_SOURCES = t_module1.c
t_module1_LDADD = ./libmodule1.la
There's a lot to learn, and best current practice can be difficult to determine, but if you're using autotools, you'll be used to this situation.
EDIT:
info automake provides the reference documentation - but it makes a poor tutorial. One of the best guides I've come across can be found here.
I've encountered the same issue and found that a pure autotools solution is very hard to get running, because the configure script e.g. for module4 depends on the installation of module1.
A hand-rolled Makefile and configure script for this situation is fairly easy to generate. I've pasted below the rapidSTORM project Makefile. It is used for out-of-tree build (source directory and a build directory).
TARGETS=any_iterator libb64 readsif cs_units dStorm-doc simparm andorcamd rapidSTORM plugin-andorsif fitter master
all:
# Project dependencies: Any project whose configure run depends upon other projects has a line here
andorcamd.prerequisites-installed : $(addsuffix .installed-stamp,libb64 simparm cs_units)
rapidSTORM.prerequisites-installed : $(addsuffix .installed-stamp,simparm cs_units libb64 any_iterator)
plugin-andorsif.prerequisites-installed : $(addsuffix .installed-stamp,rapidSTORM readsif)
master.prerequisites-installed fitter.prerequisites-installed : $(addsuffix .installed-stamp,rapidSTORM)
# [Autoconf substitutions snipped here]
# The .options files control configuration of subdirectories. They are used in %.configured-stamp
vpath %.options $(srcdir)/options:$(builddir)
RULES = all check install installcheck dist distcheck
# All standard rules have a simple template: Execute them for each
# subdirectory after configuring it and installing all prerequisite
# packages, and re-execute them whenever
# the source files changed. install and distcheck are special and
# treated further below.
define recursive_rule_template
$(1) : $(foreach target,$(TARGETS),$(target).$(1)ed-stamp)
endef
define standard_rule_template
%.$(1)ed-stamp : %.source-change-stamp %.configured-stamp %.prerequisites-installed
make -j 4 -C $$* $(1) && touch $$#
endef
$(foreach rule,$(RULES),$(eval $(call recursive_rule_template,$(rule))))
$(foreach rule,$(filter-out install distcheck,$(RULES)),$(eval $(call standard_rule_template,$(rule))))
%.installed-stamp : %.alled-stamp
make -C $* install && touch $#
# This rule is probably the most complex. It collects option files named after a
# number of options and generates configure flags from them; this rule could be
# shortened considerably when you don't need project-specific configure/CFLAGS
# configuration.
%.configured-stamp : $(foreach i, all $(host_config) $(tag) $(host_config)-$(tag), global-$i.options) \
$(foreach i, all $(host_config) $(tag) $(host_config)-$(tag),%-$i.options) | %.prerequisites-installed
prefix="$(prefix)"; abs_builddir=$(abs_builddir); \
for i in $(filter %.options,$^); do . ./$$i; done; \
mkdir -p $* && cd $* \
&& echo "Configuring with $$OPTIONS CPPFLAGS=$$CPPFLAGS CFLAGS=$$CFLAGS CXXFLAGS=$$CXXFLAGS LDFLAGS=$$LDFLAGS PKG_CONFIG_PATH=$$PKG_CONFIG_PATH" INSTALL="$(INSTALL)" \
&& /bin/sh ../$(srcdir)/$*/configure --build=$(build_alias) --host=$(host_alias) --target=$(target_alias) --config-cache $$OPTIONS \
CPPFLAGS="$$CPPFLAGS" CFLAGS="$$CFLAGS" CXXFLAGS="$$CXXFLAGS" PKG_CONFIG_PATH="$$PKG_CONFIG_PATH" \
LDFLAGS="$$LDFLAGS" $(if $(CC),CC=$(CC),) $(if $(CXX),CXX=$(CXX),) \
INSTALL="$(INSTALL)"
touch $#
# The source change stamp is updated whenever a file in the source directory changes.
# It is used to prevent non-necessary sub-make invocations.
%.source-change-stamp : always-renew
{ test -e $# && find $(srcdir)/$* -newer $# -and -not -ipath '*/.svn*' -and -not -path '*/.libs*' | wc -l | grep -q '^0$$'; } \
|| touch $#
%.prerequisites-installed :
#true
%.distchecked-stamp : %.source-change-stamp %.configured-stamp %.prerequisites-installed
DISTCHECK_CONFIGURE_FLAGS=`./$*/config.status --config | sed -e "s/'--prefix=[^']*' //"` \
$(MAKE) -j 4 -C $* distcheck && touch $#
Makefile : $(srcdir)/Makefile.in config.status
./config.status $#
installcheck : dejagnu-tests-ran-stamp
dejagnu-tests-ran-stamp : $(foreach target,$(TARGETS),$(target).installed-stamp) testsuite.configured-stamp
make -C testsuite check
touch $#
always-renew :
#true
clean :
rm -rf *-stamp $(foreach target,$(TARGETS),$(target)/*.la $(target)/config.cache) deploy
realclean : clean
rm -rf $(TARGETS)
%.options :
touch $#
world : $(foreach target,$(TARGETS),$(foreach rule,$(RULES),$(target).$(rule)ed-stamp))
.PHONY : always-renew
.SECONDARY :
.DELETE_ON_ERROR :

Makefile for Unit Tests in C++

I'm struggling to write Makefiles that properly build my unit tests. As an example, suppose the file structure looks like this
src/foo.cpp
src/foo.hpp
src/main.cpp
tests/test_foo.cpp
tests/test_all.cpp
So, to build the executable test_all, I'd need to build test_foo.o which in turn depends on test_foo.cpp but also on src/foo.o.
What is the best practice in this case? One Makefile in the parent folder? One Makefile per folder? If so, how do I manage the dependencies across folders?
Common practice is a Makefile per directory. That's what I would have suggested before I read "Recursive Make Considered Harmfull" (http://miller.emu.id.au/pmiller/books/rmch/). Now I'd recommend one Makefile. Also check out the automatic dependency generation - now you don't even need to work out what your tests depends on. All you need is some targets.
The common practice is one Makefile for each folder. Here is a simple Makefile.am script for the root folder:
#SUBDIRS = src tests
all:
make -C ./src
make -C ./tests
install:
make -C ./src install
uninstall:
make -C ./src uninstall
clean:
make -C ./src clean
test:
make -C ./tests test
The corresponding Makefile.am for the src folder will look like this:
AM_CPPFLAGS = -I./
bin_PROGRAMS = progName
progName_SOURCES = foo.cpp main.cpp
LDADD = lib-to-link
progName_LDADD = ../libs/
Makefile.am for tests will look similar:
AM_CPPFLAGS = -I../src
bin_PROGRAMS = tests
tests_SOURCES = test_foo.cpp test_all.cpp
Use automake to generate Makefile.in files from the .am files. The configure script will use the .in files to produce the Makefiles. (For small projects you would like to directly hand-code the Makefiles).