I have the following command involving qmake:
cd HmiLogging/ && ( test -e Makefile ||\
c:/Users/mureadr/Desktop/A/HMI_FORGF/qt5host/win32/x86/bin/qmake.exe\
C:/Users/mureadr/Desktop/A/HMI_FORGF/src/HmiLogging/HmiLogging.pro\
top_builddir=C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/\
top_srcdir=C:/Users/mureadr/Desktop/A/HMI_FORGF/ -Wall CONFIG+=release\
CONFIG+=qnx_build_release_with_symbols CONFIG+=rtc_build -o Makefile ) &&\
C:/Users/mureadr/Downloads/make-4.2.1/Release/make_msvc.net2003 -f Makefile
qmakegenerates the file Makefile and then regular make is called on that makefile.
The qmake-generated makefile has some entries like so:
deploy_al2hmi-mappings: deploy_fonts
#echo Copying application data... && $(MKDIR)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/al2hmi-mappings"\
&& $(COPY_DIR) $(wildcard C:/Users/mureadr/Desktop/A/HMI_FORGF/src/fordhmi/al2hmi-mappings/*)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/al2hmi-mappings"
deploy_data: deploy_al2hmi-mappings
#echo Copying application data... && $(MKDIR)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/data"\
&& $(COPY_DIR) $(wildcard C:/Users/mureadr/Desktop/A/HMI_FORGF/src/fordhmi/data/*)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/data"
deploy_qml: deploy_data
#echo Copying application data... && $(MKDIR)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/qml" &&\
$(COPY_DIR) $(wildcard C:/Users/mureadr/Desktop/A/HMI_FORGF/src/fordhmi/qml/*)\
"C:/Users/mureadr/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/qml"
You can see that each target depends on the one above it however, examining the recipes you'll also see that the directories created and used are unique to each target!
QUESTION
The generated makefile is chaining those targets, as dependencies, when none actually exists and making it unable to do them in parallel! Why?
I've grep-ed all *.pro and *.pri files - the files that qmake uses to generate makefiles - but deploy_qml doesn't appear in any of them so I'm guessing that these are tasks "internal" to qmake.
Is there any way to tell qmake to NOT dependency-chain them when there is no good reason?
Related
I have a c++ makefile project. It works great for non-parallel building. It works 99% for parallel building... the only problem I have is that I can't get my final executable link-line to run last (it must be the last thing that happens).
I have some constraints: I don't want to have any PHONY dependencies on my link line because this causes it to re-link every time. I.e. once my target is built, when I re-build it should not be re-linked.
Here is (slightly contrived) minimal example. Please don't try to pick holes in it, its really here just to show the problem, its not real, but the problem I am showing is. You should be able to just run this and see the same issue that I am.
# Set the default goal to build.
.DEFAULT_GOAL = build
#pretend subdirs (these don't really exist but it does not matter so long as they always try to be built)
MAKE_SUB_DIRS = 1 2 3
#pretend shared objects that are created by the pretend makefile sub directories (above)
OUTPUTS = out1.so out2.so out3.so
# Top level build goal - depends on all of the subdir makes and the target.out
.PHONY: build
build: $(MAKE_SUB_DIRS) target.out
#echo build finished
# Takes 1 second to build each of these pretend sub make directories. PHONY so always runs
.PHONY: $(MAKE_SUB_DIRS)
$(MAKE_SUB_DIRS):
#if [ ! -f out$#.so ] ; then echo making $#... ; sleep 1 ; echo a > out$#.so ; fi
# The main target, pretending that it needs out1,2 and 3 to link
# Should only run when target.out does not exist
# No PHONY deps allowed here
target.out:
#echo linking $#...
#ls $(OUTPUTS) > /dev/null
#cat $(OUTPUTS) > target.out
# Clean for convinience
clean:
#rm -rf *.so target.out
Now, I don't really care about make working, what I want is make -j to work. Here is me trying to run it:
admin#osboxes:~/sandbox$ make clean
admin#osboxes:~/sandbox$
admin#osboxes:~/sandbox$ make -j - 1st attempt
making 1...
making 2...
linking target.out...
making 3...
ls: cannot access 'out1.so': No such file or directory
ls: cannot access 'out2.so': No such file or directory
ls: cannot access 'out3.so': No such file or directory
makefile:24: recipe for target 'target.out' failed
make: *** [target.out] Error 2
make: *** Waiting for unfinished jobs....
admin#osboxes:~/sandbox$
admin#osboxes:~/sandbox$ make -j - 2nd attempt
linking target.out...
build finished
admin#osboxes:~/sandbox$
admin#osboxes:~/sandbox$ make -j - 3rd attempt
build finished
admin#osboxes:~/sandbox$
So I highlighted my three attempts to run it.
Attempt 1: you can see all 4 dependencies of build are started at the same time (approx). Since each of the makeing x... take 1 second and the linking is nearly instant we see my error. However all the three "libraries" are build correctly.
Attempt 2: The libraries only get created if they don't already exists (that's bash code - pretending to do what a makefile might have done). In this case they are already created. So the Linking passes now since it just requires the libraries to exist.
Attempt 3: nothing happens because nothing needs to :)
So you can see all the steps are there, its simply a matter of ordering them. I would like the the make sub dirs 1, 2, 3 to build in any order in parallel and then only once they are all completed I want target.out to run (i.e. the linker).
I don't want to call it like this though: $(MAKE) target.out because in my real makefile I have lots of variables all setup...
I have tried looking at (from othe answers) .NOT_PARALLEL and using the dep order operator | (pipe), and I have tried order a load of rules to get target.out to be last.... but the -j option just ploughs through all of these and ruins my ordering :( ... there must be some simple way to do this?
EDIT: add an example of ways to pass variables to sub-makes. Optimized a bit by adding $(SUBDIRS) to the prerequisites of build instead of making them in its recipe.
I am not sure I fully understand your organization but one solution to deal with sub-directories is as follows. I assume, a bit like in your example, that building sub-directory foo produces foo.o in the top directory. I assume also that your top Makefile defines variables (VAR1, VAR2...) that you want to pass to the sub-makes when building your sub-directories.
VAR1 := some-value
VAR2 := some-other-value
...
SUBDIRS := foo bar baz
SUBOBJS := $(patsubst %,%.o,$(SUBDIRS))
.PHONY: build clean $(SUBDIRS)
build: $(SUBDIRS)
$(MAKE) top
$(SUBDIRS):
$(MAKE) -C $# VAR1=$(VAR1) VAR2=$(VAR2) ...
top: top.o $(SUBOBJS)
$(CXX) $(LDFLAGS) -o $# $^ $(LDLIBS)
top.o: top.cc
$(CXX) $(CXXFLAGS) -c $< -o $#
clean:
rm -f top top.o $(SUBOBJS)
for d in $(SUBDIRS); do $(MAKE) -C $$d clean; done
This is parallel safe and guarantees that the link will take place only after all sub-builds complete. Note that you can also export the variables you want to pass to sub-makes, instead of passing them on the command line:
VAR1 := some-value
VAR2 := some-other-value
...
export VAR1 VAR2 ...
Normally you would just add the lib files as prerequisites of target.out:
target.out: $(OUTPUTS)
#echo linking $#...
The thing is, this will relink target.out if any of the output lib files are newer. Normally this is what you want (if the lib has changed, you need to relink target), but you specifically say you do not.
GNU make provides an extension called "order only prerequisites", which you put after a |:
target.out: | $(OUTPUTS)
#echo linking $#...
now, target.out will only be relinked if it does not exist, but in that case, it will still wait until after $(OUTPUTS) have finished being built
If your $(OUTPUT) files are build by subsirectory makes, you may find you need a rule like:
.PHONY: $(OUTPUT)
$(OUTPUT):
$(MAKE) -C $$(dirname $#) $#
to invoke the recursive make, unless you have other rules that will invoke make in the subdirectories
Ok, so I have found "a" solution... but it goes a little bit against what I wanted and is therefore ugly (but not that that ugly):
The only way I can fathom to ensure order in parallel build (again from other answers I read) is like this:
rule: un ordered deps
rule:
#echo this will happen last
Here the three deps will be made (or maked?) in any order and then finally the echo line will be run.
However the thing that I want to do is a rule and specifically so, such that it checks if anything has changed or if the file does not exist - and then, and only then, runs the rule.
The only way I know of to run a rule from within the bode of another rule is to recursively call make on it. However I get the following issues just calling make recursively on the same makefile:
Variables are not passed in by default
Many of the same rules will be re-defined (not allowed or wanted)
So I came up with this:
makefile:
# Set the default goal to build.
.DEFAULT_GOAL = build
#pretend subdirs (these don't really exist but it does not matter so long as they always try to be built)
MAKE_SUB_DIRS = 1 2 3
#pretend shared objects that are created by the pretend makefile sub directories (above)
OUTPUTS = out1.so out2.so out3.so
# Top level build goal - depends on all of the subdir makes and the target.out
export OUTPUTS
.PHONY: build
build: $(MAKE_SUB_DIRS)
#$(MAKE) -f link.mk target.out --no-print-directory
#echo build finished
# Takes 1 second to build each of these pretend sub make directories. PHONY so always runs
.PHONY: $(MAKE_SUB_DIRS)
$(MAKE_SUB_DIRS):
#if [ ! -f out$#.so ] ; then echo making $#... ; sleep 1 ; echo a > out$#.so ; fi
# Clean for convinience
clean:
#rm -rf *.so target.out
link.mk:
# The main target, pretending that it needs out1,2 and 3 to link
# Should only run when target.out does not exist
# No PHONY deps allowed here
target.out:
#echo linking $#...
#ls $(OUTPUTS) > /dev/null
#cat $(OUTPUTS) > target.out
So here I put the linker rule into a separate makefile called link.mk, this avoids recursive make calling on the same file (and therefore with re-defined rules). But I have to export all the variables I need to pass through... which is ugly and adds a bit of a maintenance overhead if those variables change.
... but... it works :)
I will not mark this any time soon, because I am hopeful some genius will point out a neater/better way to do this...
I am trying to build my makefile C++ project using Jenkins.
See project structure below. Project is on a bitbucket repository and job profile is set Freestyle Project.
Project is successfully built on Jenkins server however it looks like it just uploads the project from repository to its workspace and says "Finshed with success" but does not run a makefile.
Console output:
Checking out Revision 6720229e2d82a9e958f69afabe361c65d1647398 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 6720229e2d82a9e958f69afabe361c65d1647398
Commit message: "My commit"
> git rev-list --no-walk 084977a421fc8fb064297f64407e2d137a1b32a1 # timeout=10
Finished: SUCCESS
On my local however both test and main projects are built successfully with make.
Basically there are two questions:
How to build my project (including the test one) with Makefile on Jenkins? (i.e. how to run a make command on Jenkins). I do not want to use a Cmake. Is it possible?
If both projects built successfully, how to run the test project and see test results in console/write to file in Jenkins?
My project structure:
MyProject
|+src/ <-- source files main project
|+include/ <-- header files main project
|+bin/ <-- binaries main project
|+test/ <-- test project
|~test/
| |+bin/ <-- test binaries
| |+gtest/ <-- gtest headers
| |+lib/ <-- gtest binaries
| |-test.cpp <-- test source
|-Makefile
My Makefile:
# Compiler options and variables def
CC=g++
CPPFLAGS= -c -Wall
GFLAGS = -g
INC_DIR = include
INC_DIR_TEST = test
TST += \
*.cpp
VPATH += test/
TESTLIB += \
*.a
SRC += \
*.cpp
BIN = bin
TSTBIN = test/bin
# build
all: program test
OBJ = $(patsubst %.cpp, $(BIN)/%.o, $(SRC))
TSTOBJ = $(patsubst %.cpp, $(TSTBIN)/%.o, $(TST))
program: $(OBJ)
$(CC) $(GFLAGS) $? -o $#
test: $(TSTOBJ) $(TESTLIB) $(BIN)/file1.o $(BIN)/file2.o
$(CC) $(GFLAGS) $(TESTLIB) $(BIN)/file1.o $(BIN)/file2.o $< -o $#
$(BIN)/%.o: src/%.cpp
$(CC) $(CPPFLAGS) -I$(INC_DIR) $< -o $#
$(TSTBIN)/%.o: test/%.cpp
$(CC) $(CPPFLAGS) -I$(INC_DIR_TEST) -I$(INC_DIR) $< -o $#
clean:
rm *.o *.exe bin/*.o test/bin/*.o
Update:
As I understand, this type of project has to be marked as pipeline rather than freestyle project that will allow to choose a build tool and run shell command from Jenkinsfile file that facilitates considerably the CI process.
However I cannot find any examples of building C++ project with make and GNU build tools.
This is a Jenkinsfile example from official documentation
pipeline {
agent { docker 'maven:3.3.3' }
stages {
stage('build') {
steps {
sh 'mvn --version'
}
}
}
}
I am wondering now how this should be modified in order to build a simplest C++ project with makefile. Am I on a right way?
Second question is still actual: how to run my gtests and record results in Jenkins after the make command works?
Update:
I have used a batchfile as it can be ran from java code in Jenkinsfile. Now my testing Jenkinsfile looks like:
pipeline {
agent any
stages {
stage('build') {
steps {
echo 'building..'
bat 'batchfile.cmd'
}
}
}
}
Batchfile code:
PATH = "C:\Program Files (x86)\GnuWin32\bin"
make
There are still some bugs but at least make command is ran and the commands from makefile are called.
So general conclusion:
As it looks like there is no any make plugin for jenkins, the pipeline stages can run a batchfile (or shell script for UNIX), and this batchfile can call make. Of course make has to be installed on a server and the path to environment variable has to be specified.
Maybe there are some better approaches or I am wrong, please correct me.
I am trying to compile my project which has the following structure
Project:
MakeFile
Executable
Source1
.cxx
.h
Source2
.cxx
.h
Build
*.o
And I'm having difficulty writting a Makefile to compile. I currently have commands like:
Src1 = $(wildcard $(SRCDIR1)/*.cxx)
Obj1 = $(patsubst $(SRCDIR1)/%.cxx, $(OBJDIR)/%.o, $(Src1))
But then I have difficulty making the compile rules for the object files a) Because I can no longer do:
$(Obj1): %.cxx
$(CXX) $(CFLAGS) -c $(#:.o=.cxx) -o $#
Because the '$#' command now includes the path of the build directory and b) because the prerequisites now include the build path and I should have a source path. I have read large bits of the make manual to try and find a solution but no luck.
Any help towards a solution appreciated!
Jack
From personal experience, after playing around a bit with "raw" Makefiles, I'd really recommend using some tool building the Makefiles for you, like automake or cmake.
You'll still have to specify all the source files manually - but at least I prefer that to manually fiddling around with the Makefiles.
One option I prefer is building an isomorphic directory structure in the build directory. That is, a source file ${src_dir}/project_x/main.cxx builds into ${build_dir}/project_x/main.o. This way you are protected from name clashes when there are source files with the same name in different source directories. The compiler rule would look something like:
${obj_dir}/%.o : ${src_dir}/%.cxx # % includes directory name, e.g. project_x/main
#-mkdir -p ${#D}
${CXX} -c -o $# ${CPPFLAGS} ${CXXFLAGS} $<
Notice how in the above it creates the target directory on the fly. This is a bit simplistic, in a real-world build system object files depend (using order-only dependency) on its directory, so that make automatically creates the destination directory if it does not exist instead of speculatively trying to create them for each target object file even if it already exists.
I have a project that I want to build using automake. The project consists of different components or modules, and there are inter module dependencies which require the project to be built in a specific order.
For example:
project dir/
module1 (core C shared lib)
module2 (C++ shared lib wrapper around module 1)
module3 (C++ application with dependency on module2)
module4 (C library with dependency on module1)
module5 (C application with dependency on module4)
I am relatively new to automake, but I (just about) know how to use it to successfully build a single project.
I would like to have a 'master' project file (if that's possible), which specifies the build order of the projects modules, runs unit tests and fails the entire build process if either:
One of the modules fails to build
One of the modules fails a unit test
How would I go about writing such a 'master project' file (or invoking any other mechanism) to build projects that have a lot of inter-modular dependencies?
If you're using autotools, then you might as well use automake. The top level Makefile.am can provide a list of subdirectories that are descended in order, e.g:
SUBDIRS = module1 module2 module3 module4 module5
The subdirectory Makefile.am can add targets for tests, invoked with 'make check', which will force the build if necessary:
check_PROGRAMS = t_module1
t_module1_SOURCES = t_module1.c
t_module1_LDADD = ./libmodule1.la
There's a lot to learn, and best current practice can be difficult to determine, but if you're using autotools, you'll be used to this situation.
EDIT:
info automake provides the reference documentation - but it makes a poor tutorial. One of the best guides I've come across can be found here.
I've encountered the same issue and found that a pure autotools solution is very hard to get running, because the configure script e.g. for module4 depends on the installation of module1.
A hand-rolled Makefile and configure script for this situation is fairly easy to generate. I've pasted below the rapidSTORM project Makefile. It is used for out-of-tree build (source directory and a build directory).
TARGETS=any_iterator libb64 readsif cs_units dStorm-doc simparm andorcamd rapidSTORM plugin-andorsif fitter master
all:
# Project dependencies: Any project whose configure run depends upon other projects has a line here
andorcamd.prerequisites-installed : $(addsuffix .installed-stamp,libb64 simparm cs_units)
rapidSTORM.prerequisites-installed : $(addsuffix .installed-stamp,simparm cs_units libb64 any_iterator)
plugin-andorsif.prerequisites-installed : $(addsuffix .installed-stamp,rapidSTORM readsif)
master.prerequisites-installed fitter.prerequisites-installed : $(addsuffix .installed-stamp,rapidSTORM)
# [Autoconf substitutions snipped here]
# The .options files control configuration of subdirectories. They are used in %.configured-stamp
vpath %.options $(srcdir)/options:$(builddir)
RULES = all check install installcheck dist distcheck
# All standard rules have a simple template: Execute them for each
# subdirectory after configuring it and installing all prerequisite
# packages, and re-execute them whenever
# the source files changed. install and distcheck are special and
# treated further below.
define recursive_rule_template
$(1) : $(foreach target,$(TARGETS),$(target).$(1)ed-stamp)
endef
define standard_rule_template
%.$(1)ed-stamp : %.source-change-stamp %.configured-stamp %.prerequisites-installed
make -j 4 -C $$* $(1) && touch $$#
endef
$(foreach rule,$(RULES),$(eval $(call recursive_rule_template,$(rule))))
$(foreach rule,$(filter-out install distcheck,$(RULES)),$(eval $(call standard_rule_template,$(rule))))
%.installed-stamp : %.alled-stamp
make -C $* install && touch $#
# This rule is probably the most complex. It collects option files named after a
# number of options and generates configure flags from them; this rule could be
# shortened considerably when you don't need project-specific configure/CFLAGS
# configuration.
%.configured-stamp : $(foreach i, all $(host_config) $(tag) $(host_config)-$(tag), global-$i.options) \
$(foreach i, all $(host_config) $(tag) $(host_config)-$(tag),%-$i.options) | %.prerequisites-installed
prefix="$(prefix)"; abs_builddir=$(abs_builddir); \
for i in $(filter %.options,$^); do . ./$$i; done; \
mkdir -p $* && cd $* \
&& echo "Configuring with $$OPTIONS CPPFLAGS=$$CPPFLAGS CFLAGS=$$CFLAGS CXXFLAGS=$$CXXFLAGS LDFLAGS=$$LDFLAGS PKG_CONFIG_PATH=$$PKG_CONFIG_PATH" INSTALL="$(INSTALL)" \
&& /bin/sh ../$(srcdir)/$*/configure --build=$(build_alias) --host=$(host_alias) --target=$(target_alias) --config-cache $$OPTIONS \
CPPFLAGS="$$CPPFLAGS" CFLAGS="$$CFLAGS" CXXFLAGS="$$CXXFLAGS" PKG_CONFIG_PATH="$$PKG_CONFIG_PATH" \
LDFLAGS="$$LDFLAGS" $(if $(CC),CC=$(CC),) $(if $(CXX),CXX=$(CXX),) \
INSTALL="$(INSTALL)"
touch $#
# The source change stamp is updated whenever a file in the source directory changes.
# It is used to prevent non-necessary sub-make invocations.
%.source-change-stamp : always-renew
{ test -e $# && find $(srcdir)/$* -newer $# -and -not -ipath '*/.svn*' -and -not -path '*/.libs*' | wc -l | grep -q '^0$$'; } \
|| touch $#
%.prerequisites-installed :
#true
%.distchecked-stamp : %.source-change-stamp %.configured-stamp %.prerequisites-installed
DISTCHECK_CONFIGURE_FLAGS=`./$*/config.status --config | sed -e "s/'--prefix=[^']*' //"` \
$(MAKE) -j 4 -C $* distcheck && touch $#
Makefile : $(srcdir)/Makefile.in config.status
./config.status $#
installcheck : dejagnu-tests-ran-stamp
dejagnu-tests-ran-stamp : $(foreach target,$(TARGETS),$(target).installed-stamp) testsuite.configured-stamp
make -C testsuite check
touch $#
always-renew :
#true
clean :
rm -rf *-stamp $(foreach target,$(TARGETS),$(target)/*.la $(target)/config.cache) deploy
realclean : clean
rm -rf $(TARGETS)
%.options :
touch $#
world : $(foreach target,$(TARGETS),$(foreach rule,$(RULES),$(target).$(rule)ed-stamp))
.PHONY : always-renew
.SECONDARY :
.DELETE_ON_ERROR :
I'm struggling to write Makefiles that properly build my unit tests. As an example, suppose the file structure looks like this
src/foo.cpp
src/foo.hpp
src/main.cpp
tests/test_foo.cpp
tests/test_all.cpp
So, to build the executable test_all, I'd need to build test_foo.o which in turn depends on test_foo.cpp but also on src/foo.o.
What is the best practice in this case? One Makefile in the parent folder? One Makefile per folder? If so, how do I manage the dependencies across folders?
Common practice is a Makefile per directory. That's what I would have suggested before I read "Recursive Make Considered Harmfull" (http://miller.emu.id.au/pmiller/books/rmch/). Now I'd recommend one Makefile. Also check out the automatic dependency generation - now you don't even need to work out what your tests depends on. All you need is some targets.
The common practice is one Makefile for each folder. Here is a simple Makefile.am script for the root folder:
#SUBDIRS = src tests
all:
make -C ./src
make -C ./tests
install:
make -C ./src install
uninstall:
make -C ./src uninstall
clean:
make -C ./src clean
test:
make -C ./tests test
The corresponding Makefile.am for the src folder will look like this:
AM_CPPFLAGS = -I./
bin_PROGRAMS = progName
progName_SOURCES = foo.cpp main.cpp
LDADD = lib-to-link
progName_LDADD = ../libs/
Makefile.am for tests will look similar:
AM_CPPFLAGS = -I../src
bin_PROGRAMS = tests
tests_SOURCES = test_foo.cpp test_all.cpp
Use automake to generate Makefile.in files from the .am files. The configure script will use the .in files to produce the Makefiles. (For small projects you would like to directly hand-code the Makefiles).