How to collect source files with CMake without globbing? - build

Question

The CMake documentation explicitly states that file(GLOB ...) is not
recommended to collect source files for a build, but it doesn't
mention what the recommended method actually is.
Specifying every source file manually sounds a little bit too manually
to me. So, what is the right method to collect source files, if not
file(GLOB ...)?

Answer 1

Manual is indeed the recommended method. By recommending against using GLOB, the documentation is simply warning against a build system that depends on files present. For example, you want to add a test executable, so you create mytest.cpp. Oops. Now your library compilation breaks. The documentation for AUX_SOURCE_DIRECTORY (similar purpose as globbing for for source files) gives the following warning:
It is tempting to use this command to avoid writing the list of source
files for a library or executable target. While this seems to work,
there is no way for CMake to generate a build system that knows when a
new source file has been added. Normally the generated build system
knows when it needs to rerun CMake because the CMakeLists.txt file is
modified to add a new source. When the source is just added to the
directory without modifying this file, one would have to manually
rerun CMake to generate a build system incorporating the new file.
If you're certain that you want all the contents of a directory, and don't plan on adding new ones, then by all means use a GLOB.
Also, don't forget listing files manually doesn't have to involve typing all the filenames. You could do, for example, ls *.cpp >> CMakeLists.txt, then use your editor to move the list of files to the correct place in the file.

Answer 2

I use GLOB for exactly that and every time I add a file I run
touch ../src/CMakeLists.txt
The next make command will re-scan the directories.
"There is no way for CMake to generate a build system that knows when a new source file has been added" Really? Okay, so tell it!
It's not 100% automatic but a damn sight easier than adding files manually.

Answer 3

I use cog, a python module. Here is a sample to collect .cpp file:
The CMakeLists.txt:
set(SRC_FILES "")
# [[[cog
# import cog, glob
# for src in glob.glob('*.cpp'):
# if "skeleton" in src: continue
# cog.outl("SET(SRC_FILES ${SRC_FILES} %s)" % src)
# ]]]
# [[[end]]]
add_library(mylib STATIC ${SRC_FILES})
And then, run:
python -m cogapp -r CMakeLists.txt
The CMakeLists.txt file will be updated in place.
For how to install cog and other usage, please read the article from the author.

Answer 4

I use a conventional CMakeLists.txt and a python script to update it. I run the python script manually after adding files.
import os
import re
def relFiles(base, sub):
fullSub = os.path.join(base,sub)
abs = [os.path.join(dp, f) for dp, dn, fn in os.walk(fullSub) for f in fn]
return [os.path.relpath(f, base) for f in abs]
def updateAddLibrary(cmakelistsDir, subs):
cmakelists = os.path.join(cmakelistsDir, "CMakeLists.txt")
listings = [relFiles(cmakelistsDir, sub) for sub in subs]
files = [f for listing in listings for f in listing] #flatten
with open(cmakelists, 'r') as file:
text = file.read()
sources = "".join([" %s\n" % f.replace('\\', '/') for f in files])
text = re.sub(r"add_library\s*\(\s*([^\s\)]+).*?\)",
r"add_library(\1\n%s)" % sources,
text, 1, re.DOTALL)
with open(cmakelists, "w") as file:
file.write(text)
dir = os.path.dirname(os.path.abspath(__file__))
updateAddLibrary(dir, ['inc','src'])
Example before:
...
add_library(MyLib
inc/a.h
)
...
after:
...
add_library(MyLib
inc/a.h
inc/sub/b.h
src/a.cpp
)
...

Answer 5

Following douyw's answer, thank you for your answer.
Not a Cmake expert, don't want to be one, I spent a fxxking 3 hours trying to deal with GLOB(disabled) and aux_source_directory(Not even close to GLOB), and douyw save my life.
I add the recursive file walking, and it's working in my project:
Firstly, install the cogapp python module (python -m pip install cogapp)
set(SRC_FILES "")
# [[[cog
# import cog, os
# for root, _, files in os.walk(".", topdown=False):
# for f in files:
# if not "_unittest" in f: continue
# if not f.endswith(".cpp"): continue
# cog.outl('SET(SRC_FILES ${SRC_FILES} "%s")' % os.path.join(root, f).replace('\\', '/'))
# ]]]
# [[[end]]]
run: python -m cogapp -r CMakeLists.txt
The upper lines add all "*_unittest.cpp" to the list
You can change the middle lines to make your own rule, just plain python.
Using regex is better, but simple string searching can do the job on the above situation.
Notice the last line, it needs to replace // to change to the usable universal separator. You may generate your CMakeList.txt in windows.
You may replace SRC_FILES with whatever you want.
The python command may be run under Jenkins/TeamCity, triggered by svn/git commit. Then we can automatically add new files.