Can I manually use CMake's cpp file dependency-scanner in my cmake code? - c++

I am trying to add a custom target with CMake that executes one command for each given .cpp file. The command should only be re-executed when the source file itself or one of the included source files changes. AFAIK to achieve this I need a list of all the included files and add them to the DEPENDS option of the add_custom_command() calls that belong to my custom target.
So is there a built-in way to get that list of included files?
I know about the IMPLICIT_DEPENDS option of the add_custom_command() function but it only works for Makefile generators. I would like to make this work for all generators.
Thank you for your time
Edit:
As requested I will post some cmake code to show what I want to achieve.
I want to add a custom target, that runs clang-tidy on all the given .cpp files. When incrementally building the custom target the clang-tidy commands should be re-run whenever a .cpp file or one of its directly or indirectly included header files is changed. Just like re-runs of the compiler are handled.
# ----------------------------------------------------------------------------------------
# mainTargetName The name of the target that shall be analyzed
# files A list of all the main targets .cpp files
#
function( addStaticAnalysisTarget mainTargetName files )
set(targetName runStaticAnalysis_${mainTargetName})
set(command "clang-tidy-4.0 -checks=* -p ${CMAKE_BINARY_DIR}")
foreach( file ${files} )
get_filename_component( baseName ${file} NAME_WE)
set(stampFile ${CMAKE_CURRENT_BINARY_DIR}/analyze_${baseName}.stamp )
set(fullFile ${CMAKE_CURRENT_SOURCE_DIR}/${file})
set(commandWithFile "${command} ${fullFile}")
separate_arguments_for_platform( commandList ${commandWithFile})
add_custom_command(
OUTPUT ${stampFile}
DEPENDS "${fullFile}"
IMPLICIT_DEPENDS CXX "${fullFile}"
COMMAND ${commandList}
COMMAND cmake -E touch "${stampFile}" # without creating a file as a touch-stone the command will always be re-run.
WORKING_DIRECTORY ${CPPCODEBASE_ROOT_DIR}
COMMENT "${commandWithFile}"
VERBATIM
)
list(APPEND stampFiles ${stampFile})
endforeach()
set_source_files_properties(${stampFiles} PROPERTIES GENERATED TRUE) # make the stamp files known to cmake as generated files.
add_custom_target(
${targetName}
DEPENDS ${stampFiles}
)
endfunction()
The problem with that is, that it does not seem to work. When I change included files clang-tidy is not re-run for the affected files.
I used the "Unix Makefile" generator for this example so it should work at least with make. Any hints why it doesn't?
My hopes where that I could achieve the desired behavior for all generators by somehow getting the file-dependencies at cmake time and then adding them to the ''''DEPENDS'''' list. But the dependency scanning must be done each time the command is run, so it can not be done at cmake time. This means that the scanning must be implemented by cmake which it currently is not.
A guy with similar problems:
https://gitlab.kitware.com/cmake/cmake/issues/16830
Edit 2:
I think the problem that the IMPLICIT_DEPENDS option was not working was because I did not use correct filenames. I changed that in the code snipped, but I have not yet tested if it works in the project.

I think the answer to my question is ...
No, you can not use cmakes dependency scanner in the cmake code.
That makes sense, because this problem can not be solved at cmake time, because the dependencies of a .cpp file may change without cmake being re-run.
The problem must be solved within cmake itself at make time. This is done when using the IMPLICIT_DEPENDS option.
Also, I tried to solve a Problem that I did not really have, because at this point I can only run clang-tidy on linux anyways. However, clang-tidy may become available on windows as well and then I may have the problem again.
To sum the comments up:
Tambre stated that CMake is not a compiler and therefore can not do that.
I think this is wrong. According to this article, CMake can parse cpp include dependencies because make has no such dependency searcher itself. That was news to me, but I mostly live on Windows so I am not that familiar with make. It could also be possible that in the meantime make was extended to do its own dependency searching. Also this explains why the IMPLICIT_DEPENDS option is only available for make.
Florian pointed out that it is not necessary to create an own custom target for running clang-tidy. Instead, one can use the CXX_CLANG_TIDY target property to run clang-tidy for each file after compiling it. This means however, that static-analysis can not be separated from the build which could lead to inacceptable buildtimes.
There is the cmake -E cmake_depends command line, that could be used to retrieve dependencies at cmake time. But as stated above, I erroneously thought that I needed the dependencies at cmake time, while I needed them at runtime.
The IMPLICIT_DEPENDS options did not work because I had an error in my cmake code.

Related

What is the proper way of using a source generator in CMake

In my C++ project I'm using a source generator to embed some resources into the binary.
I use CMake to build my project and my code works but had some issues.
I am pretty sure that what I want to accomplish is possible but I didn't find any answer online.
The current problems I have are:
The generator runs every time, even if the input files did not change.
This is not too big of a deal because it is really fast, but I hopped there was a better way to do it
While using Ninja the generator runs at every build (as described above) without rebuilding every time.
I think that Ninja sees that the file has not changed and does not build it again,
but when I make changes in the resources change it still uses the old version.
It takes another build to "realize" that the generated file has changed and rebuild it
While using Make the code rebuilds every time, even when the generated file does not change, resulting in wasted build time
In both cases (looking at the output) the generator runs before the compiler.
This situation is not unsustainable but I was wondering if a better solution was possible.
Here's a code snippet from my CMakeLists.txt
add_subdirectory(Generator)
file(GLOB RESOURCES Resources/*)
add_custom_command(
OUTPUT src/Resources/Generated.hpp src/Resources/Generated.cpp
COMMAND Generator ${RESOURCES}
DEPENDS ${RESOURCES}
DEPENDS Generator
)
add_custom_target(Generated DEPENDS src/Resources/Generated.hpp src/Resources/Generated.cpp)
add_dependencies(${PROJECT_NAME} Generated)
Here's a minimal reproducible example, sorry for not having it before.
EDIT: I implemented the functional solution from the correct answer in the fix branch, on the same repo. It might be useful for future reference :)
This one is interesting, because there are multiple errors and stylistic issues, which partially overlap each other.
First off:
file(GLOB_RECURSE SRC src/*.cpp src/*.hpp)
add_executable(${PROJECT_NAME} ${SRC})
While convenient in the beginning, globbing your sources is not a good idea. At some point you will have a testme.cpp in there that should not be built with the rest, or a conditionally_compiled.cpp that should only be compiled if a certain option is set. You end up compiling sources that you really did not intended to.
In this case, the file src/Generated.hpp from your git repository. That file is supposed to be generated, not checked out from repo. How did it even get in there?
add_custom_command(
OUTPUT src/Generated.hpp
COMMAND ${PROJECT_SOURCE_DIR}/generator.sh
${PROJECT_SOURCE_DIR}/Resources/data.txt
> ${PROJECT_SOURCE_DIR}/src/Generated.hpp
DEPENDS Resources/data.txt
DEPENDS ${PROJECT_SOURCE_DIR}/generator.sh
)
Do you see the output redirection there? You wrote to ${PROJECT_SOURCE_DIR}. That is not a good idea. Your source tree should never have anything compiled or generated in it. Because these things end up being committed with the rest... like it happened to you.
Next issue:
add_custom_target(Generated DEPENDS src/Generated.hpp)
This creates a make target Generated. Try it: make Generated. You keep getting the following output:
[100%] Generating src/Generated.hpp
[100%] Built target Generated
Obviously it does not realize that Generated.hpp is already up-to-date. Why not?
Let's look at your custom command again:
add_custom_command(
OUTPUT src/Generated.hpp
COMMAND ${PROJECT_SOURCE_DIR}/generator.sh
${PROJECT_SOURCE_DIR}/Resources/data.txt
> ${PROJECT_SOURCE_DIR}/src/Generated.hpp
DEPENDS Resources/data.txt
DEPENDS ${PROJECT_SOURCE_DIR}/generator.sh
)
What if I told you that your OUTPUT is never actually generated?
Quoting from CMake docs on add_custom_command, emphasis mine:
OUTPUT
Specify the output files the command is expected to produce. If an output name is a relative path it will be interpreted relative to the build tree directory corresponding to the current source directory.
So your output claims to be to the binary tree, but your command's redirection is to the source tree... no wonder the generator keeps getting re-run.
Having your header generated in the wrong location does not give you a compiler error, because that header from your source tree gets picked up by your GLOB_RECURSE. As that one keeps getting re-generated, your executable keeps getting recompiled as well.
Try this from your build directory:
mkdir src && touch src/Generated.hpp && make Generated
Output:
[100%] Built target Generated
You see that make has nothing to do for the Generated target, because it now sees an OUTPUT of your custom command that is newer than its dependencies. Of course, that touched file isn't the generated one; we need to bring it all together.
Solution
Don't write to your source tree.
Since ${PROJECT_BINARY_DIR}/src does not exist, you need to either create it, or live with the created files on the top dir. I did the latter for simplicity here. I also removed unnecessary ${PROJECT_SOURCE_DIR} uses.
add_custom_command(
OUTPUT Generated.hpp
COMMAND ${PROJECT_SOURCE_DIR}/generator.sh \
${PROJECT_SOURCE_DIR}/Resources/data.txt \
> Generated.hpp
DEPENDS Resources/data.txt
DEPENDS generator.sh
)
Don't glob, keep control over what actually gets compiled:
add_executable( ${PROJECT_NAME} src/main.cpp )
Add the binary tree to your target's include path. After add_executable:
target_include_directories( ${PROJECT_NAME} PRIVATE ${PROJECT_BINARY_DIR} )
That's it, things work as expected now.

Generate a source file that may or may not be updated

I have a CMakeLists.txt in which I want to generate several source files (namely, versiondata.cpp and version.rc.inc, included by res.rc) that depends on the general environment (current git HEAD, gcc -v output, CMakeCache.txt itself, and so on).
If it depended just on some files, I would generate it using an add_custom_command directive with the relevant DEPENDS and OUTPUT clauses; however, it's a bit tricky to pinpoint exactly its file dependencies; ideally, I'd want to run my script every time I call make, updating the files only if needed; if the generated files have actually been touched, then the targets depending from them should be rebuilt (the script is careful not to overwrite the files if they would have the same content as before).
My first attempt was using an add_custom_command with a fake main output, like this:
add_custom_command(OUTPUT versiondata.cpp.fake versiondata.cpp version.rc.inc
COMMAND my_command my_options
COMMENT "Generating versiondata.cpp"
)
# ...
# explicitly set the dependencies of res.rc, as they are not auto-deduced
set_source_files_properties(res.rc PROPERTIES OBJECT_DEPENDS "${PROJECT_BINARY_DIR}/version.rc.inc;${PROJECT_SOURCE_DIR}/other_stuff.ico")
# ...
add_executable(my_executable WIN32 ALL main.cpp versiondata.cpp res.rc)
versiondata.cpp.fake is never really generated, so the custom command is always run. This worked correctly, but always rebuilt my_executable, as CMake for some reasons automatically touches the output files (if generated) even though my script left them alone.
Then I thought I might make it work using an add_custom_target, that is automatically "never already satisfied":
add_custom_target(versiondata BYPRODUCTS versiondata.cpp version.rc.inc
COMMAND my_command my_options
COMMENT "Generating versiondata.cpp"
)
# ...
# explicitly set the dependencies of res.rc, as they are not auto-deduced
set_source_files_properties(res.rc PROPERTIES OBJECT_DEPENDS "${PROJECT_BINARY_DIR}/version.rc.inc;${PROJECT_SOURCE_DIR}/other_stuff.ico")
# ...
add_executable(my_executable WIN32 ALL main.cpp versiondata.cpp res.rc)
The idea here is that the versiondata target should be "pulled in" from the targets that depend on its BYPRODUCTS, and should be always executed. This seems to work on CMake 3.20, and the BYPRODUCTS seem to have some effect because if I remove the dependencies from my_executable my script doesn't get called.
However, on CMake 3.5 I get
make[2]: *** No rule to make target 'version.rc.inc', needed by 'CMakeFiles/my_executable.dir/res.rc.res'. Stop.
and if I remove the explicit dependency from version.rc.inc it doesn't get generated at all
[ 45%] Building RC object CMakeFiles/my_executable.dir/res.rc.res
/co/my_executable/res.rc:386:26: fatal error: version.rc.inc: No such file or directory
#include "version.rc.inc"
^
compilation terminated.
/opt/mingw32-dw2/bin/i686-w64-mingw32-windres: preprocessing failed.
CMakeFiles/my_executable.dir/build.make:5080: recipe for target 'CMakeFiles/my_executable.dir/res.rc.res' failed
make[2]: *** [CMakeFiles/my_executable.dir/res.rc.res] Error 1
so I suspect that the fact that this works in 3.20 is just by chance.
Long story short: is there some way to make this work as I wish?
In CMake there are two types of dependencies:
Target-level dependency, between targets.
A target can be build only after unconditional building of all targets it depends on.
File-level dependency, between files.
If some file is older than one of its dependencies, the file will be regenerated using corresponded COMMAND.
The key factor is that checking for timestamp of dependent files is performed strictly after building of dependent targets.
For correct regeneration of versiondata.cpp file and executable based on it, one need both dependencies:
Target-level, which would ensure that versiondata custom target
will be built before the executable.
add_dependencies(my_executable versiondata)
File-level, which will ensure that the executable will be rebuilt whenever
file versiondata.cpp will be updated.
This dependency is created automatically by listing versiondata.cpp
among the sources for the executable.
Now about BYPRODUCTS.
Even without explicit add_dependencies, your code works on CMake 3.20 because BYPRODUCTS generates needed target-level dependency automatically.
This could be deduced from the description of DEPENDS option in add_custom_target/add_custom_command:
Changed in version 3.16: A target-level dependency is added if any dependency is a byproduct of a target or any of its build events in the same directory to ensure the byproducts will be available before this target is built.
and noting, that add_executable effectively depends on every of its source files.
Because given comment for DEPENDS is applicable only for CMake 3.16 and later,
in older CMake versions BYPRODUCTS does not create target-level dependency automatically, and one need to resort to explicit add_dependencies.

CMake: compilation speed when including external makefile

I have a c++ cmake project. In this project I build (among other) one example, where I need to use another project, call it Foo. This Foo project does not offer a cmake build system. Instead, it has a pre-made Makefile.custom.in. In order to build an executable that uses Foo's features, one needs to copy this makefile in his project, and modify it (typically setting the SOURCES variable and a few compiler flags). Basically, this Makefile ends up having the sources for your executable and also all the source files for the Foo project. You will not end up using Foo as a library.
Now, this is a design I don't like, but for the sake of the question, let's say we stick with it.
To create my example inside my cmake build I added a custom target:
CONFIGURE_FILE( ${CMAKE_CURRENT_SOURCE_DIR}/Makefile.custom.in Makefile.custom)
ADD_CUSTOM_TARGET(my_target COMMAND $(MAKE) -f Makefile.custom
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
This works. I can specify some variables to cmake, which get resolved in the call to CONFIGURE_FILE, and I end up with a working Makefile.custom. Then, invoking make my_target from the build directory, I can build the executable. I can even add it to the all target (to save me the effort of typing make my_target) with
SET_TARGET_PROPERTIES(my_target PROPERTIES EXCLUDE_FROM_ALL FALSE)
Sweet. However, cmake appears to assign a single job to the custom target, slowing down my compilation time (the Foo source folder contains a couple dozens cpp files). On top of that, the make clean target does not forward to the custom makefile. I end up having to add another target:
ADD_CUSTOM_TARGET(really-clean COMMAND "$(MAKE)" clean
COMMAND "$(MAKE)" -f Makefile.custom clean
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
which, unlike my_target with all, I can't include in the clean target (can I?).
Now, I know that a cleaner solution would be to have the Foo project be built as an external project, and then link to it. However, I've been 'recommended' to use their Makefile.custom.in makefile, modifying the few lines I need (adding my sources, specifying compiler flags, and few other minor modifications). So, regardless of how neat and clean this design pattern is, my questions are:
is there a way to tell cmake that make should use more than 1 job when making the target my_target?
is there a cleaner way to include a pre-existing makefile in a cmake project? Note that I don't want (can't?) use Foo as a library (and link against it). I want (need?) to compile it together with my executable using a makefile not generated by cmake (well, cmake can help a bit, through CONFIGURE_FILE, by resolving some variables, but that's it).
Note: I am aware of ExternalProject (as suggested also in this answer), but I think it's not exactly what I need here (since it would build Foo and then use it as a library). Also, both my project and Foo are written exclusively in C++ (not sure this matter at all).
I hope the question makes sense (regardless of how ugly/annoying/unsatisfactory the resulting design would be).
Edit: I am using cmake version 3.5.2
First, since you define your own target, you can assign more cores to the build process for the target my_target, directly inside your CMakeLists.txt.
You can include the Cmake module ProcessCount to determine the number of cores in your machine and then use this for a parallel build.
include(ProcessorCount)
ProcessorCount(N)
if(NOT N EQUAL 0)
# given that cores != 0 you could modify
# math(EXPR N "${N}+1") # modify (increment/decrement) N at your will, in this case, just incrementing N by one
set(JOBS_IN_PARALLEL -j${N})
endif(NOT N EQUAL 0)
and when you define your custom target have something like the following:
ADD_CUSTOM_TARGET(my_target
COMMAND ${CMAKE_MAKE_PROGRAM} ${JOBS_IN_PARALLEL} -f Makefile.custom
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
by the way, I don't think there's the need for you to include also CMAKE_BUILD_TOOL among the COMMANDs in your target.
I believe that instead of modifying the lines as above, you could call
make -j8 my_target
and it might start 8 jobs (just an example) without modifying the CMakeLists.txt, but I cannot guarantee this works having defined the COMMAND the way you have, just try if that's enough.
For the second point, I cannot think right now of a "cleaner" way.

Compile a single file under CMake project?

I'm developing a C++ project which is going to be enclosed on a bigger one.
I've seen that on the bigger project (is a Qt application and it's being generated from qmake) I am able to compile a single file from the linux command line, just entering the relative path to the specific file as an argument to make.
On the other hand, I'm using CMake for my own project. When I modify some code for a compilation unit and I have to modify its header file, I have to wait a long time to compile its dependencies and then its own source file. But there are some situations in which I would prefer to check whether the source code in the *.cc file is compilable without errors.
Is there a way to generate a Makefile from CMake the way qmake does this? Switching to qmake is not an option anymore.
You do not have to add extra custom targets to your CMake scripts, as the Makefiles generated by CMake already contain .o targets for each .cc file. E.g. if you have a source file called mySourceFile.cc, there will be a Makefile in your build directory that defines a target called <Some Path>/mySourceFile.cc.o. If you cd into your build directory, you can use grep or ack-grep to locate the Makefile that defines this target, then cd into that Makefile's directory and build it.
E.g. suppose the command ack-grep mySourceFile.cc.o prints something like:
foo/bar/Makefile
119:x/y/z/mySourceFile.o: x/y/z/mySourceFile.cc.o
123:x/y/z/mySourceFile.cc.o:
124: # recipe for building target
Then you can build mySourceFile.cc.o by doing:
cd foo/bar && make x/y/z/mySourceFile.cc.o
CMake doesn't have a generic built-in way of doing this (it's an open issue), but if you're using the Ninja generator, you can can use a special Ninja syntax for building just the direct outputs of a given source file. For example, to compile just foo.o you would use:
ninja /path/to/foo.cpp^
Not out-of-the box. CMake does not expose those "internal" makefile rules in the main makefile.
You can do this only if you consider what kind of file structure CMake uses internally. You can e.g. for compiling a single .obj files using CMake generated makefiles call
make -f CMakeFiles/myProg.dir/build.make CMakeFiles/myProg.dir/main.cc.obj
when you have something like
cmake_minimum_required(VERSION 3.1)
project(myProg CXX)
file(WRITE "main.cc" "int main()\n{\nreturn 0;\n}")
add_executable(myProg main.cc)
To build src/foo.cpp alone:
cmake --build . --target src/foo.cpp.o
No, CMake does not offer built-in support to compile single files.
You have to add a target for each object file, maybe by a function iterating over all files of a directory.
Others have suggested ways to find the target name (ending in .cpp.o) from the .cpp filename, but if you already know the name of a target that will trigger compilation of the .cpp file and you're using ninja this suggestion should be easier.
First build the target:
ninja TriggersCppCompilationLib
Assuming your file was changed or was not yet built, ninja will print the full target name. When you see the name come up, hit enter so it is not overwritten. Then simply copy the name from the terminal (e.g. using tmux copy mode).

clion run cmake on each build

I am not sure if its possible to do this with clion, I was testing out the program and enjoy using it to write c code because the ctags and etags support is really nice.
I am copying some files over from the cmake source tree to the bin location on each build. While using clion if I update some of the files that I am copying the results aren't updated within clion.
If I instead go back to the terminal and just run the typical cmake steps
cmake ../ && make && bin/./program
that copies the updated files and I am able to see my results.
This is the CMake command that I am using in my build.
FILE(COPY ${CMAKE_CURRENT_SOURCE_DIR}/resources/ DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/bin/resources/)
I might be able to restructure the cmake command to make it copy every time or it might just be a clion issue. I am unsure and would like to be able to take care of this all from within clion, instead of going back to the terminal to run the cmake command for updates to take effect.
If you want CMake to make some action whenever some file is changed, you should create a rule using add_custom_command and pass file via DEPENDS argument. Note, that only file-level dependencies are supported by CMake, for make dependency on directory you need to list all files in it.
# Collect list of files within directory.
FILES(GLOB files_list RELATIVE ${CMAKE_CURRENT_SOURCE_DIR}/resources/
"${CMAKE_CURRENT_SOURCE_DIR}/resources/*")
# This will contain full paths to files in binary directory.
set(binary_files_list)
foreach(file ${files_list})
set(source_file ${CMAKE_CURRENT_SOURCE_DIR}/resources/${file})
set(binary_file ${CMAKE_CURRENT_BINARY_DIR}/bin/resources/${file})
add_custom_command(OUTPUT ${binary_file}
COMMAND cmake -E ${source_file} ${binary_file}
DEPENDS ${source_file})
list(APPEND binary_files_list ${binary_file})
endforeach()
add_custom_target(resources DEPENDS ${binary_files_list})
Note, that in case of adding/removing files you should to run cmake explicitely. That's why hardcoded files list is preferred to GLOBing.