Just a disclaimer I'm not too experienced with Make files, so forgive me if the answer is simple! I have a Make file that compiles several .cpp & .h files into .o files, and then produces an executable. I am using g++ as the compiler currently. I'd like to adapt this file to allow for openmpi multi-core computing. To note: I'm on a Windows 11 x 64 architecture, using Cygwin to build run this cpp code. I also would like this to run on MacOS systems as well.
In particular, the parallelization macros are only in 1 .cpp file, what I'd consider the 'main' file of this project (mag_spec_tracker.cpp). I don't know if that makes a difference here but thought it could be useful to let be known.
My current Makefile is below - note I added a line to try and grab the mpicc compile flags, but I am unsure on how to use them here.
IDIR = include
CXX = g++
CXXFLAGS = -I$(IDIR) -std=c++17
ODIR = obj
_DEPS = my_functions.h particle.h beam.h threevector.h threematrix.h screen.h magnet.h
DEPS = $(patsubst %,$(IDIR)/%,$(_DEPS))
_OBJ = my_functions.o particle.o beam.o mag_spec_tracker.o screen.o magnet.o
OBJ = $(patsubst %,$(ODIR)/%,$(_OBJ))
$(ODIR)/%.o: %.cpp $(DEPS)
$(CXX) -c -o $# $< $(CXXFLAGS)
run: $(OBJ)
$(CXX) -o $# $^ $(CXXFLAGS)
.PHONY: clean
clean:
rm -f $(ODIR)/*.o *~ core $(IDIR)/*~
I have tried adding lines into the Makefile to get the mpi compile flags based on other reference I've found online, but was unsure how to adapt my current code to add these flags to allow for correct usage of openmpi. The lines I added to grab compile flags were:
MPI_COMPILE_FLAGS = $(shell mpicc --showme:compile)
MPI_LINK_FLAGS = $(shell mpicc --showme:link)
Thank you in advance!
Update (12/6/2022)
Responding to the comment from Matzeri ('for opempi the compiler is mpicc not gcc. for c++ is mpicxx not g++'), I replaced 'g++' in my makefile with 'mpicxx' and recieved the following error:
$ make
mpicxx -c -o obj/mag_spec_tracker.o mag_spec_tracker.cpp -Iinclude -std=c++17
mpicxx -o run obj/my_functions.o obj/particle.o obj/beam.o obj/mag_spec_tracker.o obj/screen.o obj/magnet.o -Iinclude -std=c++17
C:/Users/Jason/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lmpi_cxx: No such file or directory
C:/Users/Jason/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lmpi: No such file or directory
C:/Users/Jason/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lopen-rte: No such file or directory
C:/Users/Jason/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lopen-pal: No such file or directory
collect2.exe: error: ld returned 1 exit status
make: *** [Makefile:19: run] Error 1
I notice that its looking for an ld executable in 'mingw64'. I tried removing that directory (previously I had installed mingw64, but switched to cygwin), and got the following error:
$ make
mpicxx -o run obj/my_functions.o obj/particle.o obj/beam.o obj/mag_spec_tracker.o obj/screen.o obj/magnet.o -Iinclude -std=c++17
--------------------------------------------------------------------------
The Open MPI wrapper compiler was unable to find the specified compiler
g++ in your PATH.
Note that this compiler was either specified at configure time or in
one of several possible environment variables.
--------------------------------------------------------------------------
make: *** [Makefile:19: run] Error 1
I also added c:\cygwin\bin to my environment paths, and this issue still occurs. I confirmed that mpicxx is in that bin folder.
Any ideas?
Im trying to compile a C++ project using MinGW and can compile a simple main.cpp file with hello world without problems using g++ main.cpp -o main and also with external libraries using main.cpp extlib.cpp -o main.
But say im working on a rather large project with 10s of .cpp files organised inside of different files, how can I get the compiler to find all the cpp files that are needed? I know i can use main.cpp libs/*.cpp -o main but this will only compile all the source files inside of libs but not inside folders in libs.
Ive looked into make and cmake but dont understand how those automate the process if you still have to manually enter the directories. Is there no way to simply hit compile or at least a command line command to compile all the needed files inside a directory? This seems to work with #include without issues?
If you want to stick with MinGW and GNU Make I would probably use a Makefile that looks something like this to start with. You basically only need to maintain the srcs-variable by adding your source-files there. Usually you can use the wildcard-function for this if you have sub dirs. The rest of the Makefile (which can be left alone) sets up a build of an executable main.exe that depends on all the object-files. I also included dependency-handling via the deps-variable and the compiler flag -MMD which comes in handy when the project grows.
srcs := $(wildcard *.cpp) $(wildcard dir1/*.cpp) $(wildcard dir2/*.cpp)
objs := $(srcs:.cpp=.o)
deps := $(objs:.o=.d)
app := main.exe
CXXFLAGS := -MMD -Og -g -Wall -Werror -Wpedantic -std=c++2a
$(app): $(objs)
$(CXX) $(LDFLAGS) -o $# $^ $(LDLIBS)
-include $(deps)
.PHONY: clean
clean:
rm -f $(objs) $(deps)
I use CMake for simple projects.
Here's the simplest example I came with (CMakeLists.txt to put along your main.cpp in the root of your project):
cmake_minimum_required(VERSION 3.1)
SET(CMAKE_APP_NAME "Project")
project (${CMAKE_APP_NAME})
# list here your directories
INCLUDE_DIRECTORIES(dir1)
INCLUDE_DIRECTORIES(dir2)
# add an executable and list all files to compile
add_executable(${CMAKE_APP_NAME} main.cpp
dir1/file1.cpp
dir1/file1.h
dir2/file2.h
dir2/file2.cpp
)
Once your project becomes more complex, you could use file(GLOB*) to avoid writing all the files.
Overall, the most "automated" way to build a larger project is to use CMake. Keep learning it. You can use file(GLOB) to avoid listing every file in CMakeLists.txt. This is not recommended (see discussion here), but I do it anyway and never had any issues.
I have receive a makefile from another source. I am supposed to cross-compile on my machine (x86_64, GCC 5.3) it to use on an arm-embedded device (host). I tried to compile with a tool chain (arm_64, GCC7.5) which i installed in my /opt directory by making the following changes (those that are remarked out #) in the makefile of the project
#SYSROOT?=$(shell $(CXX) --print-sysroot)
SYSROOT= /opt/yocto/sysroots/
CXXFLAGS= -std=c++11 -I src/
CCFLAGS= -std=c++11 -I src/
#CXX_LDFLAGS= -L /usr/local/aarch64-linux/lib/ -lpthread -pthread
CXX_LDFLAGS= -L $(SYSROOT)/aarch64-yocto-linux/lib -lpthread -pthread
all: foo
foo: src/foo.cpp
$(CXX) -std=c++11 -g -o $# $^ $(CXX_LDFLAGS)
clean:
rm -rf .objs
However when the code was compiled, it could not be run on host device(arm) and there was warning during make that say along lines of skipping incompatible binaries.
I perform file on the compiled application. It read
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=99376bcc919a48224e063ec6c6892e1979b5d94e, not stripped
It is clear to me that I have compiled it with my native compiler,libraries etc.
What should i do in this case:
must i do the following before compilation or making the folder
./configure -build xxxxx -host yyyyyy
there is some files, called environment files (full of export settings) in the tool chain directory that seems to be like some configuration settings being exported. Should i use this file? or how should i use them
Regards
I don't often use makefiles therefore I only how to make basic ones. I've tried reading on how to solve my issue but I am lost. What I need is a makefile that has two targets all and test. Both of these targets need to compile the program with g++ -std=c+11 -Wall -Werror -ansi -pedantic and the only file that needs to be compiled is main.cpp when the executable is built by the compiler it needs to put the executable in a new directory called bin. And I'm assuming that if I were to use the makefile again it would need to check if bin was already created.
I know how to create a simple makefile such as
all:
g++ -std=c++11 -Wall -Werror -ansi -pedantic main.cpp
That creates an executable named a.out in the current directory, but that's about as far as my knowledge of makefileS go
All that a Makefile does is specify the build dependencies, and the commands to execute to build those dependencies.
Things like creating the output directories, et. al. is not make's job per se, but rather something that's up to the commands that makes executes to do. Neither does the Makefile specify where the output of the build commands go. That, of course, is specified by the actual commands that get executed. With gcc, for example, the -o option specifies where the compilation output goes. So:
all: bin/main
test: bin/main
bin/main: main.cpp
mkdir -p bin
g++ -std=c++11 -Wall -Werror -ansi -pedantic -o bin/main main.cpp
It's very convenient to use mkdir -p in these situations. If the given directory does not exist, it gets created. If it already exists, mkdir just quietly terminates without an error.
I am writing something like an interactive tutorial for C++. The tutorial will consist of two parts: one is compiled into a library (I'm using Scons to build that), and the other (the lessons) is shipped with the tutorial to be compiled by the end user. I'm currently looking for a good, easy way for people to build these lessons.
Basically, the second part is a directory with all the lessons in it, each in its own directory. Each lesson will have at least a lesson.cpp and a main.cpp file, there may be also other files, the existence of which I will not know until after it is shipped -- the end user will create these. It will look something like this:
all_lessons/
helloworld/
lesson.cpp
main.cpp
even_or_odd/
lesson.cpp
main.cpp
calculator/
lesson.cpp
main.cpp
user_created_add.cpp
Each of these will need to be compiled according to almost the same rules, and the command for compiling should be possible to run from one of the lesson directories (helloworld/, etc.).
Seeing as the rest of the project is built with Scons, it would make sense to use it for this part, too. However, Scons searches for the SConstruct file in the directory it is run from: would it be acceptable to put a SConstruct file in each lesson directory, plus a SConscript in the all_lessons/ directory that gives the general rules? This seems to go against the typical way Scons expects projects to be organised: what are the potential pitfalls of this approach? Could I put a SConstruct file instead of the SConscript one, and thereby make it possible to build from either directory (using exports to avoid endless recursion, I'm guessing)?
Also, I may at some point want to replace the lesson.cpp with a lesson.py that generates the necessary files; will Scons allow me to do this easily with builders, or is there a framework that would be more convenient?
In the end, I want to end up with the following (or equivalent with different build systems):
all_lessons/
SConstruct
helloworld/
SConstruct
lesson.cpp
main.cpp
even_or_odd/
SConstruct
lesson.py
main.cpp
calculator/
SConstruct
lesson.cpp
main.cpp
user_created_add.cpp
Running scons all in the all_lessons directory would need to:
Run even_or_odd/lesson.py to generate even_or_odd/lesson.cpp.
Realise that user_created_add.cpp also needs to be compiled.
Produce an executable for each lesson.
Running scons in even_or_odd/, or scons even_or_odd in all_lessons/ should produce an executable identical to the one above (same compile flags).
Summary:
Is Scons suitable for/capable of this?
Does Scons work well when SConscript files are above SConstruct files?
Does Scons work well with multiple SConstrcut files for one project, SConscripting each other?
Is the Scons builder system suitable for using Python scripts to generate C++ files?
Is there any advantage of using a different build system/writing my own build framework that I'm missing?
Any further comments are, of course, welcome.
Thanks.
You can actually do this with a few lines of GNU Make.
Below are two makefiles that allow building and cleaning from all_lessons directory and individual project directories. It assumes that all C++ sources in that directory comprise an executable file which gets named after its directory. When building and cleaning from the top level source directory (all_lessons) it builds and cleans all the projects. When building and cleaning from a project's directory it only builds and cleans the project's binaries.
These makefiles also automatically generate dependencies and are fully parallelizable (make -j friendly).
For the following example I used the same source file structure as you have:
$ find all_lessons
all_lessons
all_lessons/even_or_odd
all_lessons/even_or_odd/main.cpp
all_lessons/Makefile
all_lessons/helloworld
all_lessons/helloworld/lesson.cpp
all_lessons/helloworld/main.cpp
all_lessons/project.mk
all_lessons/calculator
all_lessons/calculator/lesson.cpp
all_lessons/calculator/user_created_add.cpp
all_lessons/calculator/main.cpp
To be able to build from individial project directories project.mk must be symlinked as project/Makefile first
[all_lessons]$ cd all_lessons/calculator/
[calculator]$ ln -s ../project.mk Makefile
[helloworld]$ cd ../helloworld/
[helloworld]$ ln -s ../project.mk Makefile
[even_or_odd]$ cd ../even_or_odd/
[even_or_odd]$ ln -s ../project.mk Makefile
Let's build one project:
[even_or_odd]$ make
make -C .. project_dirs=even_or_odd all
make[1]: Entering directory `/home/max/src/all_lessons'
g++ -c -o even_or_odd/main.o -Wall -Wextra -MD -MP -MF even_or_odd/main.d even_or_odd/main.cpp
g++ -o even_or_odd/even_or_odd even_or_odd/main.o
make[1]: Leaving directory `/home/max/src/all_lessons'
[even_or_odd]$ ./even_or_odd
hello, even_or_odd
Now build all projects:
[even_or_odd]$ cd ..
[all_lessons]$ make
g++ -c -o calculator/lesson.o -Wall -Wextra -MD -MP -MF calculator/lesson.d calculator/lesson.cpp
g++ -c -o calculator/user_created_add.o -Wall -Wextra -MD -MP -MF calculator/user_created_add.d calculator/user_created_add.cpp
g++ -c -o calculator/main.o -Wall -Wextra -MD -MP -MF calculator/main.d calculator/main.cpp
g++ -o calculator/calculator calculator/lesson.o calculator/user_created_add.o calculator/main.o
g++ -c -o helloworld/lesson.o -Wall -Wextra -MD -MP -MF helloworld/lesson.d helloworld/lesson.cpp
g++ -c -o helloworld/main.o -Wall -Wextra -MD -MP -MF helloworld/main.d helloworld/main.cpp
g++ -o helloworld/helloworld helloworld/lesson.o helloworld/main.o
[all_lessons]$ calculator/calculator
hello, calculator
[all_lessons]$ helloworld/helloworld
hello, world
Clean one project:
[all_lessons]$ cd helloworld/
[helloworld]$ make clean
make -C .. project_dirs=helloworld clean
make[1]: Entering directory `/home/max/src/all_lessons'
rm -f helloworld/lesson.o helloworld/main.o helloworld/main.d helloworld/lesson.d helloworld/helloworld
make[1]: Leaving directory `/home/max/src/all_lessons'
Clean all projects:
[helloworld]$ cd ..
[all_lessons]$ make clean
rm -f calculator/lesson.o calculator/user_created_add.o calculator/main.o even_or_odd/main.o helloworld/lesson.o helloworld/main.o calculator/user_created_add.d calculator/main.d calculator/lesson.d even_or_odd/main.d calculator/calculator even_or_odd/even_or_odd helloworld/helloworld
The makefiles:
[all_lessons]$ cat project.mk
all :
% : forward_ # build any target by forwarding to the main makefile
$(MAKE) -C .. project_dirs=$(notdir ${CURDIR}) $#
.PHONY : forward_
[all_lessons]$ cat Makefile
# one directory per project, one executable per directory
project_dirs := $(shell find * -maxdepth 0 -type d )
# executables are named after its directory and go into the same directory
exes := $(foreach dir,${project_dirs},${dir}/${dir})
all : ${exes}
# the rules
.SECONDEXPANSION:
objects = $(patsubst %.cpp,%.o,$(wildcard $(dir ${1})*.cpp))
# link
${exes} : % : $$(call objects,$$*) Makefile
g++ -o $# $(filter-out Makefile,$^) ${LDFLAGS} ${LDLIBS}
# compile .o and generate dependencies
%.o : %.cpp Makefile
g++ -c -o $# -Wall -Wextra ${CPPFLAGS} ${CXXFLAGS} -MD -MP -MF ${#:.o=.d} $<
.PHONY: clean
clean :
rm -f $(foreach exe,${exes},$(call objects,${exe})) $(foreach dir,${project_dirs},$(wildcard ${dir}/*.d)) ${exes}
# include auto-generated dependency files
-include $(foreach dir,${project_dirs},$(wildcard ${dir}/*.d))
As an exercise in learning scons, I've tried to answer your question. Unfortunately, I'm no expert, so I can't tell you what's the best/ideal way, but here's a way that works.
Scons is suitable for/capable of this. (This is exactly what build tools are for.)
Not applicable. (I don't know.)
Scons seems to work well with multiple SConstrcut files for one project, SConscripting each other.
The Scons builder system can use Python scripts to generate C++ files.
A different build system? To each his own.
Using the hierarchy you defined, there's a SConstruct file in each folder. You can run scons in a subfolder to build that project or at the top level to build all projects (not sure how you'd alias "all" to the default build). You can run scons -c to clean the project and scons automatically figures out which files it created and cleans them (including the generated lesson.cpp).
However, if you want compiler flags to propagate from the top-level file down, I think it's better to use SConscript files -- except I'm not sure about making these compile on their own.
./SConstruct
env = Environment()
env.SConscript(dirs=['calculator', 'even_or_odd', 'helloworld'], name='SConstruct')
./calculator/SConstruct and ./calculator/helloworld
env = Environment()
env.Program('program', Glob('*.cpp'))
./even_or_odd/SConstruct
env = Environment()
def add_compiler_builder(env):
# filename transformation
suffix = '.cpp'
src_suffix = '.py'
# define the build method
rule = 'python $SOURCE $TARGET'
bld = Builder(action = rule,
suffix = suffix,
src_suffix = src_suffix)
env.Append(BUILDERS = {'Lesson' : bld})
return env
add_compiler_builder(env)
env.Lesson('lesson.py')
env.Program('program', Glob('*.cpp'))
Using SConscripts
I convert the subfolder's SConstructs to SConscripts and can lift the code build specifics out of the subfolders, but then you need to run scons -u to build in a subfolder (to search upwards for the root SConstruct).
./SConstruct
def default_build(env):
env.Program('program', Glob('*.cpp'))
env = Environment()
env.default_build = default_build
Export('env')
env.SConscript(dirs=['calculator', 'even_or_odd', 'helloworld'])
./helloworld/SConscript, etc...
Import('env')
env.default_build(env)
Is it essential that the command for compiling be run from the lesson directory? If not then I would personally create all_lessons/makefile with the following contents:
lessons = helloworld even_or_odd calculator
all: $(lessons)
# for each $lesson, the target is $lesson/main built from $lesson/main.cpp and $lesson/lesson.cpp
# NB: the leading space on the second line *must* be a tab character
$(lessons:%=%/main): %/main: %/main.cpp %/lesson.cpp
g++ -W -Wall $+ -o $#
All lessons could then be built with "make" or "make all" in the all_lessons directory, or a specific lesson with e.g. "make helloworld/main".
As far as I have found, this is the best solution available:
The directory is structured in the same way, but instead of having multiple SConstruct files, the lessons have a SConscript file each, where defaults are overridden as necessary. The SConstruct files are generated by an external script as necessary, and SCons is invoked.
An overview:
all_lessons/
helloworld/
SConscript
lesson.cpp
main.cpp
even_or_odd/
SConscript
lesson.py
main.cpp
calculator/
SConscript
lesson.cpp
main.cpp
user_created_add.cpp
Using Glob, the SConscript file can make all files with the cpp extension be compiled. It can also use a builder (either one invoking a simple command, or a fully-fledged one) that will generate the lesson, meaning it's possible to even just store the lesson as metadata and have it generated on the spot.
So, to answer the questions:
Yes.
I don't know, but it is not required for the purpose of this.
As far as I have seen, no (issues with paths relative to SConstruct, amongst other things).
Yes, with several options being available.
I don't know.
Downsides to the suggested approach: this does require making a meta-build system separately. The number of files where options can be specified is higher, and the SConscript files give a lot of room for error.
Here is my way.
# SConstruct or SConscript
def getSubdirs(dir) :
lst = [ name for name in os.listdir(dir) if os.path.isdir(os.path.join(dir, name)) and name[0] != '.' ]
return lst
env = Environment()
path_to_lessons = '' # path to lessons
# configure your environment, set common rules and parameters for all lessons
for lesson in getSubdirs(path_to_lessons) :
lessonEnv = env.Clone()
# configure specific lesson, for example i'h ve checked SConscript file in lesson dir
# and if it exist, execute it with lessonEnv and append env specific settings
if File(os.path.join(path_to_lessons, lesson, 'lesson.scons')).exists() :
SConscript(os.path.join(lesson, 'lesson.scons', export = ['lessonEnv'])
# add lesson directory to include path
lessonEnv.Append(CPPPATH = os.path.join(path_to_lessons, lesson));
lessonEnv.Program(lesson, Glob(os.path.join(path_to_lessons, lesson, '*.cpp'))
Now you have :
env - core Environment that contain common rules and parameters for
all lessons
lessonEnv - clone of core env, but if you have
lesson.scons in specific lesson dir, you can additional configure
that environment or rewrite some parameters.