Copy out plain .o files with cmake - build

I'm trying to get cmake (on linux) to create some static object (.o) files and install them to an external directory. To that end I have a list, object_sources, containing the project path of the sources, and put this in the top level CMakeLists.txt:
set(local_objects "")
foreach(file ${object_sources})
get_filename_component(cur ${file} NAME_WE)
add_library("${cur}.o" OBJECT ${file})
list(APPEND local_objects "${cur}.o")
# To confirm these variables are what I think they are:
message("---> ${cur} | ${file}")
endforeach()
install(
# Also tried FILES, see below.
TARGETS ${local_objects}
CONFIGURATIONS Debug
# Also tried nothing instead of 'OBJECTS' (same outcome)
# and LIBRARY (cmake fails on error).
OBJECTS
DESTINATION "/some/external/dir"
)
With regard to using FILES, it cannot work because in fact a foo.o is never created, so there is no point in me trying to get the correct project path.
When I get to the actual make command in the cmake build directory, it spits out Built target foo.o, etc., during the process. However, as mentioned, no foo.o is ever created anywhere, and the install creates an objects-Debug directory in /some/external/dir containing directories for each target using its name (e.g., foo.o/) and inside that some more nested directories at the bottom of which is the object I want, with the wrong name: foo.cpp.o.
Cmake can structure its build tree however it wants and name its temporaries whatever it likes, but I would really like to be able to ask for a final product object foo.o (not foo.cpp.o) and have it installed that way.
Is that possible? I'm beginning to think this is all very contrary to the (implicit) purposes of cmake, and that I should keep these kinds of tasks separate in a shell script or makefile (it takes about three lines in either), and just use cmake for the normative libs and executables.
Or: I believe I could get this to work if I created each object as a static archive (lib.a) instead, but part of the issue here is that would require modifying the build scripts of something else, or else extracting them via a script (in which case I might as well do the whole thing in the script).

I'm not aware of a good way to do this in CMake, but maybe the least fussy would be to use an OBJECT library. This would allow you to get the object files for a target using generator expressions, more specifically the TARGET_OBJECTS expression. This is a minimal example with a single foo.c file containing an empty main:
cmake_minimum_required(VERSION 3.11)
project(foo C)
set(SOURCES foo.c)
add_library(foo_obj OBJECT ${SOURCES})
add_executable(foo $<TARGET_OBJECTS:foo_obj>)
set(OBJS $<TARGET_OBJECTS:foo_obj>)
message(STATUS "these are my objects: ${OBJS}") # generator expression printed here, not evaluated yet
add_custom_target(print_foo_objs
COMMAND ${CMAKE_COMMAND} -E echo $<TARGET_OBJECTS:foo_obj>)
set(OBJ_ROOT_DIR $<TARGET_PROPERTY:foo,BINARY_DIR>)
add_custom_target(copy_foo_objs
COMMAND ${CMAKE_COMMAND} -E make_directory ${OBJ_ROOT_DIR}/myobjects/
COMMAND ${CMAKE_COMMAND} -E copy_if_different $<TARGET_OBJECTS:foo_obj> ${OBJ_ROOT_DIR}/myobjects/
COMMAND_EXPAND_LISTS) # UPDATE
Please note that as a generator expression, you will not have access to it during configuration (hence, that message will contain the generator expression as it was textually declared), but only during the generation phase.
If you still need to rename the object files (i.e. from foo.c.o to foo.o) you can maybe call a custom external script to do that for you.
Moreover, there might be issues with cleanup, etc. which will force you to add more custom targets and/or commands (and associated dependencies) to deal with it. Another downside of this approach.
UPDATE: COMMAND_EXPAND_LISTS allows the expansion of lists in the generator expression and it was added in CMake 3.8 and later (see here).

Related

cmake post-process static library target

I am using cmake to create my static libraries with something along the lines of
add_library(library library.cpp)
install(TARGETS library DESTINATION lib)
which creates liblibrary.a which is what I want. However I would like to bundle that with a library, let's say vendor/proprietary.a by doing something custom like
tmp=$(mktemp -d)
cd $tmp
ar -x $<TARGET_FILE:library>
ar -x vendor/proprietary.a
ar -qc $<TARGET_FILE:library> *
rm -rf $tmp
Can I do that with cmake without it forgetting that the target library is actually a library (eg by using add_custom_command/add_custom_target).
This is, disappointingly, quite hard. We managed to do it on the Halide team, but only because it was a hard requirement from a corporate client. To other readers without such constraints, I say this: here be dragons. Use CMake's usual targets and dependencies and let it put all the static libraries on the end-product's link line.
To OP, I say, try this:
First, create a CMakeLists.txt in your vendor directory with the following content:
# 0. Convenience variable
set(proprietary_lib "${CMAKE_CURRENT_SOURCE_DIR}/proprietary.a")
# 1. Get list of objects inside static lib
execute_process(COMMAND "${CMAKE_AR}" -t "${proprietary_lib}"
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}"
OUTPUT_VARIABLE proprietary_objects)
string(STRIP "${proprietary_objects}" proprietary_objects)
string(REPLACE "\n" ";" proprietary_objects "${proprietary_objects}")
# 2. Attach configure dependency to the static lib
set_property(DIRECTORY . APPEND PROPERTY
CMAKE_CONFIGURE_DEPENDS proprietary.a)
# 3. Extract the lib at build time
add_custom_command(
OUTPUT ${proprietary_objects}
COMMAND "${CMAKE_AR}" -x "${proprietary_lib}"
DEPENDS "${proprietary_lib}")
# 4. Get absolute paths to the extracted objects
list(TRANSFORM proprietary_objects
PREPEND "${CMAKE_CURRENT_BINARY_DIR}/")
# 5. Attach the objects to a driver target so the
# custom command doesn't race
add_custom_target(proprietary.extract DEPENDS ${proprietary_objects})
# 6. Add a target to encapsulate this
add_library(proprietary OBJECT IMPORTED GLOBAL)
set_target_properties(proprietary PROPERTIES
IMPORTED_OBJECTS "${proprietary_objects}")
# TODO: add usage requirements
# target_include_directories(proprietary INTERFACE ...)
# 7. Force proprietary to run completely after extraction
add_dependencies(proprietary proprietary.extract)
There's a lot going on here, but ultimately the steps are straightforward and the complications are with explaining the dependencies to CMake. Also, this comes with the caveat that it is Linux-only (or at least GNU-ar-compatible archiver only). It is possible to do something similar for MSVC, but it would be too much for this answer.
So first we ask the archiver which objects are in the library and we lightly process its one-object-per-line output into a CMake list. That's step 1 above.
Step 2 tells CMake that if the timestamp on proprietary.a is ever updated, then it will need to re-run CMake (and thereby get a new list of objects).
Step 3 creates a custom command which will, at build time, run the archiver tool to extract the objects into the vendor build directory.
Step 4 turns the (relative) list of objects into a list of absolute paths to those objects after the custom command runs. This is for the benefit of add_custom_target which expects absolute paths (or rather, does weird things with relative paths if certain policies are enabled).
Step 5 creates a custom target to drive the archive extraction.
Step 6 creates an imported object library to encapsulate the extracted library. It has to be global because imported targets are directory-scoped by default and this is an abuse of the imported-library feature. You can add additional usage requirements here.
Finally, step 7 puts a dependency to the driver target on the object library.
This can then be used transparently. Here's an example:
cmake_minimum_required(VERSION 3.16)
project(example)
add_subdirectory(vendor)
add_library(library library.cpp)
target_link_libraries(library PRIVATE proprietary)

Create CMake target that copies files/dirs but excludes some items

I am trying to generate a CMake target for a C++ project using add_custom_target that copies the contents of the directory in which the CMakeLists.txt resides into ${CMAKE_BINARY_DIR}, but excludes a given list of files.
While it looks like a quite easy task, I'm facing problems here. My attempts so far:
1) Generate a list of all files/dirs, remove the items to be excluded and use the CMake copy command:
file(GLOB files_to_copy LIST_DIRECTORIES TRUE *)
list(REMOVE_ITEM files_to_copy
${CMAKE_CURRENT_SOURCE_DIR}/file_to_exclude.txt
${CMAKE_CURRENT_SOURCE_DIR}/dir_to_exclude
# ...
)
add_custom_target(my-target
COMMAND ${CMAKE_COMMAND} -E copy ${files_to_copy} ${CMAKE_BINARY_DIR}
)
Problems:
Removing items this way works on a string comparison level, which could lead to problems when using symbolic links, for example
The copy command line tool apparently supports copying directories, however it doesn't seem to work on my machine, therefore directories do not get copied.
2) Use the file command to copy the files, excluding some files
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/" DESTINATION ${CMAKE_BINARY_DIR}
PATTERN "file_to_exclude.txt" EXCLUDE
PATTERN "dir_to_exclude" EXCLUDE
)
Problems:
This does exactly what I want, except that the command is not bound to a target and therefore is always executed. But I only need the copy operation to be a target.
Is there any possibility to bind the file(COPY ...) command to a target? Or any other straightforward solution to achieve what I want?
Please note that I only want to use CMake built-in tools and not execute any OS-specific shell commands (working on a cross-platform project).
CMake scripting commands work only in CMake context and are executed immediately.
But a COMMAND in add_custom_command (or add_custom_target) is executed in the context of a build tool (e.g. Makefile), not in CMake context.
However, you may put CMake scripting commands into separate CMake script, and call this script from the COMMAND. This solution has the same platform-independent properties as CMakeLists.txt itself.
You may parameterize separate script either:
With configure_file CMake command.
By passing -D parameters to CMake when call the script.
The first approach is quite simple: you write the script as would you write CMakeLists.txt itself. But it generates additional files for every parametrization set.
The second approach is useful for multi-purpose (or multi-usable) scripts, as it doesn't create additional copy of the script for every usage. But it requires some design of the parameters.
Using 'configure_file'
copy_sources.cmake.in (as if commands are written in CMakeLists.txt):
file(COPY "${CMAKE_CURRENT_SOURCE_DIR}/" DESTINATION ${CMAKE_BINARY_DIR}
PATTERN "file_to_exclude.txt" EXCLUDE
PATTERN "dir_to_exclude" EXCLUDE
)
CMakeLists.txt:
# Instantiate the parameterized script in the binary directory.
configure_file(copy_sources.cmake.in copy_sources.cmake)
add_custom_target(my-target
COMMAND ${CMAKE_COMMAND} -P ${CMAKE_BINARY_DIR}/copy_sources.cmake
)
Using '-D' parameters
copy_sources.cmake:
# Expect the script to be called from the source directory.
# This saves one parameter.
#
# DEST_DIR denotes the directory for copy to.
file(COPY "${CMAKE_SOURCE_DIR}/" DESTINATION ${DEST_DIR}
PATTERN "file_to_exclude.txt" EXCLUDE
PATTERN "dir_to_exclude" EXCLUDE
)
CMakeLists.txt:
add_custom_target(my-target
COMMAND ${CMAKE_COMMAND}
-DDEST_DIR=${CMAKE_BINARY_DIR} # all '-D' options should preceed '-P' one
-P ${CMAKE_CURRENT_SOURCE_DIR}/copy_sources.cmake
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} # The script assumes proper current directory
)

Copying assets directory from source to build directory in CMake

I am trying to write a game for practice in C++. I am using CMake for this, and my project gets built in a separate build directory automatically by my IDE. However, my assets folder is not being copied over to the new directory.
Currently, I am attempting to resolve this with the following:
add_custom_command(TARGET ${PROJECT_NAME}
POST_BUILD
COMMAND cp -r ${PROJECT_SOURCE_DIR}/assets ${CMAKE_BINARY_DIR}
)
However, this seems to have absolutely no effect on the outcome and the assets directory is still not present in my build directory.
How can I actually make it copy this asset directory to my build location? Is there a smarter way to point my program to my assets location (perhaps one that will also work with assets in /usr/share?)
After quite a while of more searching, I managed to find the file(COPY {files} DESTINATION {directory}) command:
file(COPY assets DESTINATION ${CMAKE_BINARY_DIR})
Two ways I've found, both ensuring that it copies on target build and stays up to date even if the target doesn't need to re-build.
The first uses the CMake command line tool copy_directory. It's unclear from the documentation whether this copy always happens, regardless if files in the assets directory were updated or not, or if it only copies when the asset files don't exist or an asset file was updated:
add_custom_target(copy_assets
COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_CURRENT_LIST_DIR}/assets ${CMAKE_CURRENT_BINARY_DIR}/assets
)
add_dependencies(mytarget copy_assets)
The second requires a small additional CMake script file, but it allows us to use the file(COPY ...) CMake comamnd, which states in the documentation that "copying preserves input file timestamps, and optimizes out a file if it exists at the destination with the same timestamp."
First, create the CMake script file copy-assets.cmake in the same directory as your CMakeLists.txt, with the contents being (similar to Ethan McTague's answer)
file(COPY ${CMAKE_CURRENT_LIST_DIR}/assets DESTINATION ${CMAKE_CURRENT_BINARY_DIR})
And then in your CMakeLists.txt, add the following:
add_custom_target(copy_assets
COMMAND ${CMAKE_COMMAND} -P ${CMAKE_CURRENT_LIST_DIR}/copy-assets.cmake
)
add_dependencies(mytarget copy_assets)
The solution above using add_custom_target is suitable for most cases, but has a few problems.
Does not support dependent generator expressions (e.g. $<TARGET_FILE_DIR:tgt>)
Re-copies every build (slow for numerous/large files)
Is not particularly readable, especially for copying from multiple locations
Instead, I've been using file GENERATE
set(ASSETS
assetfile
...)
foreach(ASSET ${ASSETS})
file(GENERATE OUTPUT ${ASSET} INPUT ${ASSET})
endforeach()
Note I haven't needed generator expressions yet, but it looks like the latest version of CMake (3.19) adds generator expression support to this function.
Here is what I do with vulkan shaders. Simply replace the compilation step with a copy command:
macro(add_shaders)
cmake_parse_arguments("MY" "TARGET" "SOURCES" ${ARGN})
foreach(src ${MY_SOURCES})
set(OUTF "${CMAKE_CURRENT_BINARY_DIR}/shaders/${src}.spv")
get_filename_component(PARENT_DIR "${OUTF}" DIRECTORY)
add_custom_command(
OUTPUT "${OUTF}"
COMMAND ${CMAKE_COMMAND} -E make_directory "${PARENT_DIR}"
COMMAND "${Vulkan_GLSLC_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/${src}" -o "${OUTF}"
DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/${src}"
VERBATIM
)
# src in order to get listed in the IDE, OUTF to declare the build relationship
target_sources("${MY_TARGET}" PRIVATE "${src}" "${OUTF}")
endforeach()
endmacro()
add_shaders(TARGET my_target SOURCES asset1 asset2 asset3 ...)
You'll have to specify each asset on its own (or loop over a list). Including wildcards into a build is a bad idea [tm]. This will also only copy/process stuff if it has changed.

CMake: compilation speed when including external makefile

I have a c++ cmake project. In this project I build (among other) one example, where I need to use another project, call it Foo. This Foo project does not offer a cmake build system. Instead, it has a pre-made Makefile.custom.in. In order to build an executable that uses Foo's features, one needs to copy this makefile in his project, and modify it (typically setting the SOURCES variable and a few compiler flags). Basically, this Makefile ends up having the sources for your executable and also all the source files for the Foo project. You will not end up using Foo as a library.
Now, this is a design I don't like, but for the sake of the question, let's say we stick with it.
To create my example inside my cmake build I added a custom target:
CONFIGURE_FILE( ${CMAKE_CURRENT_SOURCE_DIR}/Makefile.custom.in Makefile.custom)
ADD_CUSTOM_TARGET(my_target COMMAND $(MAKE) -f Makefile.custom
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
This works. I can specify some variables to cmake, which get resolved in the call to CONFIGURE_FILE, and I end up with a working Makefile.custom. Then, invoking make my_target from the build directory, I can build the executable. I can even add it to the all target (to save me the effort of typing make my_target) with
SET_TARGET_PROPERTIES(my_target PROPERTIES EXCLUDE_FROM_ALL FALSE)
Sweet. However, cmake appears to assign a single job to the custom target, slowing down my compilation time (the Foo source folder contains a couple dozens cpp files). On top of that, the make clean target does not forward to the custom makefile. I end up having to add another target:
ADD_CUSTOM_TARGET(really-clean COMMAND "$(MAKE)" clean
COMMAND "$(MAKE)" -f Makefile.custom clean
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
which, unlike my_target with all, I can't include in the clean target (can I?).
Now, I know that a cleaner solution would be to have the Foo project be built as an external project, and then link to it. However, I've been 'recommended' to use their Makefile.custom.in makefile, modifying the few lines I need (adding my sources, specifying compiler flags, and few other minor modifications). So, regardless of how neat and clean this design pattern is, my questions are:
is there a way to tell cmake that make should use more than 1 job when making the target my_target?
is there a cleaner way to include a pre-existing makefile in a cmake project? Note that I don't want (can't?) use Foo as a library (and link against it). I want (need?) to compile it together with my executable using a makefile not generated by cmake (well, cmake can help a bit, through CONFIGURE_FILE, by resolving some variables, but that's it).
Note: I am aware of ExternalProject (as suggested also in this answer), but I think it's not exactly what I need here (since it would build Foo and then use it as a library). Also, both my project and Foo are written exclusively in C++ (not sure this matter at all).
I hope the question makes sense (regardless of how ugly/annoying/unsatisfactory the resulting design would be).
Edit: I am using cmake version 3.5.2
First, since you define your own target, you can assign more cores to the build process for the target my_target, directly inside your CMakeLists.txt.
You can include the Cmake module ProcessCount to determine the number of cores in your machine and then use this for a parallel build.
include(ProcessorCount)
ProcessorCount(N)
if(NOT N EQUAL 0)
# given that cores != 0 you could modify
# math(EXPR N "${N}+1") # modify (increment/decrement) N at your will, in this case, just incrementing N by one
set(JOBS_IN_PARALLEL -j${N})
endif(NOT N EQUAL 0)
and when you define your custom target have something like the following:
ADD_CUSTOM_TARGET(my_target
COMMAND ${CMAKE_MAKE_PROGRAM} ${JOBS_IN_PARALLEL} -f Makefile.custom
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
by the way, I don't think there's the need for you to include also CMAKE_BUILD_TOOL among the COMMANDs in your target.
I believe that instead of modifying the lines as above, you could call
make -j8 my_target
and it might start 8 jobs (just an example) without modifying the CMakeLists.txt, but I cannot guarantee this works having defined the COMMAND the way you have, just try if that's enough.
For the second point, I cannot think right now of a "cleaner" way.

How do I add objects with a custom extension to a cmake library?

I'd like to add some object files to a CMake static library, but they have a custom extension.
Here's what I've tried:
set(SRCS testfile.cxx jsobj.js)
add_library(testlib STATIC ${SRCS})
When made, CMake invokes ar testfile.cxx.o (ie the other file is completely ignored). How do I get it included in the archive? Here are some other tricks I've tried:
list(APPEND CMAKE_CXX_SOURCE_FILE_EXTENSIONS js)
list(APPEND CMAKE_C_SOURCE_FILE_EXTENSIONS js) # no luck
add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/jsobj.js.o
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/jsobj.js
${CMAKE_CURRENT_BINARY_DIR}/jsobj.js.o
DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/jsobj.js.o) # still no luck
(In case you're interested, I'm using the emscripten compiler, which can accept C/C++ files as source input, and JavaScript files are essentially "precompiled objects". I want to find a way to get CMake to add them to the ar commandline, that's all!)
For the record, this is how I solved my problem in a hacky way: "proper" solutions would be gladly accepted.
I made up a new file extension for my special pre-compiled objects, "jso", then added it to the list of input files CMake understands:
list(APPEND CMAKE_CXX_SOURCE_FILE_EXTENSIONS jso)
Then, I add my object files with the extension ".jso" to the CMake sources for inclusion in a static library target.
Finally, I hacked the compiler by setting CC=mycc, where mycc is a Python script which checks if the input has the extension ".jso": if not, it simply re-invokes the standard compiler; otherwise it copies the input to the output with no changes at all, so that mycc -c input.jso -o output.jso.o is just a file copy.
This isn't pretty, but it picks up all the dependencies perfectly for incremental builds. I can't pretend it's pretty, but doing things the way CMake likes seems to work. Here, we're just pretending all inputs are source files, even if they're actually already compiled.