Ninja build system: refer to multiple targets with a variable - build

I have the following build.ninja file
$ cat build.ninja
rule mycmd
command = echo $in
build target1: mycmd
build target2: mycmd
build target3: mycmd target1 target2
targets = target1 target2
build target4: mycmd $targets
ninja target3 works correctly. However, ninja target4 raises error
$ ninja target4
ninja: error: 'target1 target2', needed by 'target4', missing and no known rule to make it
Looks like it thinks target1 target2 is a single target name, rather than two separate targets.
How should I solve this problem?

Related

In Meson, can I avoid to continuously jump from the source to the build directory and back?

To do an out-of-source build in Meson:
cd /path/to/source/
mkdir ../builddir
Then:
cd /path/to/source/
meson ../builddir
cd ../builddir
ninja
Is it possible to do anything like this (from builddir):
meson --pathToSource ../source // pseudocode
ninja
I.e. avoid jumping from the source to the build directory and back.
For CMake, this is the default.
Once you've run meson to create build directory (which meson can create automatically), there is no need to run it everytime you change meson.build. When you run ninja, meson can regenerate build configurations itself depending on the changes in the sources.
To run ninja in other places than build directory, you can you -C option (from ninja -h):
-C DIR change to DIR before doing anything else
Given your example, it will be:
$ cd /path/to/source/
$ meson ../builddir
$ ninja -C ../builddir

How can I statically link Arrow when building parquet-cpp?

From the parquet-cpp home page:
By default, Parquet links to Arrow's shared libraries. If you wish to statically-link the Arrow symbols instead, pass -DPARQUET_ARROW_LINKAGE=static.
I do want to statically link Arrow, because I want to use my program on other servers that won't have Arrow installed. I tried -DPARQUET_ARROW_LINKAGE=static, but I get an error about "missing transitive dependencies":
# cmake -DPARQUET_BUILD_TESTS=Off -DCMAKE_BUILD_TYPE=Release -DPARQUET_MINIMAL_DEPENDENCY=ON -DPARQUET_ARROW_LINKAGE=static .
-- The C compiler identification is GNU 4.8.5
...
-- [ /usr/local/share/cmake-3.9/Modules/FindBoost.cmake:1717 ] Boost_FOUND = 1
-- Boost version: 1.55.0
...
-- THRIFT_HOME:
-- Thrift compiler/libraries NOT found: (THRIFT_INCLUDE_DIR-NOTFOUND, THRIFT_STATIC_LIB-NOTFOUND). Looked in system search paths.
-- Thrift include dir: /root/tmp/parquet-cpp-master/thrift_ep/src/thrift_ep-install/include
-- Thrift static library: /root/tmp/parquet-cpp-master/thrift_ep/src/thrift_ep-install/lib/libthrift.a
-- Thrift compiler: /root/tmp/parquet-cpp-master/thrift_ep/src/thrift_ep-install/bin/thrift
-- Checking for module 'arrow'
-- No package 'arrow' found
-- Could not find the Arrow library. Looked for headers in , and for libs in
-- Building Apache Arrow from commit: 501d60e918bd4d10c429ab34e0b8e8a87dffb732
-- CMAKE_CXX_FLAGS: -O3 -DNDEBUG -Wall -std=c++11
-- Found cpplint executable at /root/tmp/parquet-cpp-master/build-support/cpplint.py
CMake Error at CMakeLists.txt:515 (message):
Missing transitive dependencies for Arrow static linking
So I found the code that generates the error:
if (NOT DEFINED ENV{BROTLI_STATIC_LIB_ENC} OR
NOT DEFINED ENV{BROTLI_STATIC_LIB_DEC} OR
NOT DEFINED ENV{BROTLI_STATIC_LIB_COMMON} OR
NOT DEFINED ENV{SNAPPY_STATIC_LIB} OR
NOT DEFINED ENV{ZLIB_STATIC_LIB} OR
NOT DEFINED ENV{LZ4_STATIC_LIB} OR
NOT DEFINED ENV{ZSTD_STATIC_LIB})
message(FATAL_ERROR "Missing transitive dependencies for Arrow static linking")
But that doesn't really help me, since I don't know what to do to get those environment variable defined.
Do I need to compile Arrow and install myself first? (I would rather have parquet-cpp do it for me.)
I arranged a script to download dependencies sources, set the environment variables and run your cmake line at the end. Just change the DEPDIR variable value, setting it to a directory of choice.
#!/bin/bash
CMKDIR=$PWD
DEPDIR=/tmp
cd $DEPDIR
#snappy
git clone https://github.com/google/snappy.git
cd snappy
mkdir build
cd build
cmake ..
make
export SNAPPY_STATIC_LIB=$DEPDIR/snappy/build/libsnappy.a
cd $DEPDIR
#brotli
git clone https://github.com/google/brotli.git
cd brotli
mkdir out
cd out
../configure-cmake
make
export BROTLI_STATIC_LIB_ENC=$DEPDIR/brotli/out/libbrotlienc-static.a
export BROTLI_STATIC_LIB_DEC=$DEPDIR/brotli/out/libbrotlidec-static.a
export BROTLI_STATIC_LIB_COMMON=$DEPDIR/brotli/out/libbrotlicommon-static.a
cd $DEPDIR
#zlib
git clone https://github.com/madler/zlib.git
cd zlib
./configure
make
export ZLIB_STATIC_LIB=$DEPDIR/zlib/libz.a
cd $DEPDIR
#lz4
git clone https://github.com/lz4/lz4.git
cd lz4
make
export LZ4_STATIC_LIB=$DEPDIR/lz4/lib/liblz4.a
cd $DEPDIR
#zstd
git clone https://github.com/facebook/zstd.git
cd zstd
make
export ZSTD_STATIC_LIB=$DEPDIR/zstd/lib/libzstd.a
cd $CMKDIR
cmake -DPARQUET_BUILD_TESTS=Off -DCMAKE_BUILD_TYPE=Release -DPARQUET_MINIMAL_DEPENDENCY=ON -DPARQUET_ARROW_LINKAGE=static
This script is very simple but should be effective. Just copy it in a new file (in the same CMakeLists.txt directory), give the file the execute permissions (i.e. sudo chmod +x filename) and execute it like this:
./filename.sh
About the fPIC option issue, you have to edit some files:
snappy: add this line in CMakeLists.txt, at the beginning, after the first two lines:
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
lz4 and zstd: edit the Makefile in lib sub-directory, after this line
CFLAGS += $(DEBUGFLAGS) $(MOREFLAGS)
add this line:
CFLAGS += -fPIC
zlib: edit the Makefile, after this line
CFLAGS=-O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN
add this line:
CFLAGS += -fPIC
brotli: as far as I can see from make output, the option is already set.
Before running make again, execute this script:
#!/bin/bash
DEPDIR=/tmp
cd $DEPDIR/snappy/build
cmake ..
make clean
make
cd $DEPDIR/lz4
make clean
make
cd $DEPDIR/zstd
make clean
make

building from source Qpid Messaging API (C++, bindings) [Qpid C++ 1.37.0] - missing files

I am trying to build from sources Qpid Messaging API (C++, bindings).
Build is triggered on RHEL and all prerequisites mentioned on link below are installed.
https://git-wip-us.apache.org/repos/asf?p=qpid-cpp.git;a=blob_plain;f=INSTALL.txt;hb=HEAD
yet when reaching step: make all
what i get is this
error:
Linking CXX shared library libqpidclient.so
[ 86%] Built target qpidclient
Scanning dependencies of target qpidmessaging
[ 86%] Building CXX object src/CMakeFiles/qpidmessaging.dir/qpid/messaging/amqp/AddressHelper.cpp.o
In file included from /home/m020390/qpid_cpp/qpid-cpp-1.37.0/src/qpid/messaging/amqp/AddressHelper.cpp:22:0:
/home/m020390/qpid_cpp/qpid-cpp-1.37.0/src/qpid/messaging/amqp/PnData.h:27:27: fatal error: proton/engine.h: No such file or directory
#include <proton/engine.h>
^
compilation terminated.
make[3]: *** [src/CMakeFiles/qpidmessaging.dir/qpid/messaging/amqp/AddressHelper.cpp.o] Error 1
make[2]: *** [src/CMakeFiles/qpidmessaging.dir/all] Error 2
make[1]: *** [examples/messaging/CMakeFiles/client.dir/rule] Error 2
make: *** [client] Error 2
that would indicate that "Qpid proton-c " might be missing but its installed....
any ideas ?
It worked for me using these steps.
$ wget http://apache.claz.org/qpid/proton/0.18.1/qpid-proton-0.18.1.tar.gz
$ tar -xf qpid-proton-0.18.1.tar.gz
$ cd qpid-proton-0.18.1/
$ mkdir bld
$ cd bld
$ cmake .. -DCMAKE_INSTALL_PREFIX=/tmp/testprefix
$ make -j8
$ make install
$ cd ../..
$ wget http://apache.claz.org/qpid/cpp/1.37.0/qpid-cpp-1.37.0.tar.gz
$ tar -xf qpid-cpp-1.37.0.tar.gz
$ cd qpid-cpp-1.37.0/
$ mkdir bld
$ cd bld
$ cmake .. -DCMAKE_INSTALL_PREFIX=/tmp/testprefix
>>>> Here check for "Found Proton" in the cmake output <<<<
$ make -j8
$ make install
Check that proton/engine.h is where you think it should be. In your case, it looks like that would be /usr/local/include/proton/engine.h.
You can use the following command to see what your include search path is.
cpp -v /dev/null -o /dev/null
As stated in the very INSTALL file you provided the link to:
Note: If Proton is installed in a non-standard location, there are two
ways to locate it:
Recommended: use proton 0.7 or later and use the same install prefix for both Proton and Qpid.
Using pkg-config: set the PKG_CONFIG_PATH environment variable to /lib[64]/pkgconfig before running cmake.
I'd go for method 1 and rebuild Proton specifiyng an install prefix to cmake, this way:
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local
Then use the same prefix for Qpid (i.e. run cmake again, just like the above).

How to build the latest clang-tidy?

I've tried to build clang-tidy from sources but it complains about an undefined CMake command:
CMake Error at clang-apply-replacements/CMakeLists.txt:5 (add_clang_library):
Unknown CMake command "add_clang_library".
CMake Warning (dev) in CMakeLists.txt:
No cmake_minimum_required command is present. A line of code such as
cmake_minimum_required(VERSION 3.9)
should be added at the top of the file. The version specified may be lower
if you wish to support older CMake versions for this project. For more
information run "cmake --help-policy CMP0000".
This warning is for project developers. Use -Wno-dev to suppress it.
-- Configuring incomplete, errors occurred!
How can I build clang-tidy or, alternatively, how can I install the latest version on macOS?
Up-to-date steps:
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" ../llvm
make install clang-tidy
Reference, ninja, and other details: my own blog post.
EDIT: this answer is out of date — the LLVM project has moved to a single git repository at https://github.com/llvm/llvm-project. See answers below for updated instructions.
clang-tidy is intended to be built inside a checkout of llvm/clang, and depends on CMake macros from the llvm project. You should check out the llvm repo, then the clang repo inside llvm/tools/clang, then the clang-tools-extra repo inside llvm/tools/clang/tools/extra. Then you can run CMake on the top-level directory, and make clang-tidy should work.
If you're not interested in building it yourself, it looks like the Homebrew formula for LLVM also includes the extra tools: https://github.com/Homebrew/homebrew-core/blob/382d3defb5bc48ce2dccd17261be70c4ada9a124/Formula/llvm.rb#L181
I had same problem as Per Mildner. Got is solved with slightly modified code YvesgereY posted (I don't have enough reputation to post a comment to that answer, hence a new answer instead).
In short, I added -G "Unix Makefiles" to cmake. Without this option, no makefile will be generated. Also, I used -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;". It didn't work when just clang-tools-extra was specified.
Here is the whole snippet:
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build
cd build
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;" ../llvm
make -j8 install-clang-tidy
#jtbandes: Thank you for the information.
I'd like to share these explicit steps for us noobs:
1. Download the released sources from LLVM Download Page
LLVM source code -> Links to the file llvm-6.0.0.src.tar.xz
Clang source code -> Links to the file cfe-6.0.0.src.tar.xz
clang-tools-extra -> Links to the file clang-tools-extra-6.0.0.src.tar.xz
2. Detar each of these into the proper directory:
$ tar -zxvf <download_dir_path>/llvm-6.0.1.src.tar.xz
$ cd llvm-6.0.1.src/tools
$ tar -zxcf <download_dir_path>/cfe-6.0.1.src.tar.xz
$ cd llvm-6.0.1.src/tools/cfe-6.0.1.src/tools
$ tar -zxvf <download_dir_path>/clang-tools-extra-6.0.1.src.tar.xz
Results in a directory llvm-6.0.1.src/tools/cfe-6.0.1.src/tools/clang-tools-extra-6.0.1.src/clang-tidy; Which is incorrect. The lang-tools-extra-6.0.1.src needs to be renamed to extra (as mentioned by #jtbandes).
3. So rename it or provide a symbolic link:
$ cd llvm-6.0.1.src/tools/cfe-6.0.1.src/tools
$ mv clang-tools-extra-6.0.1.src extra
or
$ ln -s clang-tools-extra-6.0.1.src extra
The path llvm-6.0.1.src/tools/cfe-6.0.1.src/tools/extra/clang-tidy should now be valid
4. Build it:
$ cd llvm-6.0.1.src
$ mkdir build
$ cd build
$ cmake ..
$ make
Everything should make without errors or warnings.
5. Build Output:
The build output can be found in llvm-6.0.1.src/build/bin.
For everyone who are looking for latest (LLVM 11) Windows build instructions (ensure CMake, Visual Studio 2019 and git are installed and set in PATH):
git clone --config core.autocrlf=false https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build
cd build
cmake -G "Visual Studio 16 2019" -Thost=x64 -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" ../llvm
cmake --build . --target clang-tidy --config RelWithDebInfo --parallel
cmake --build . --target clang-query --config RelWithDebInfo --parallel
This worked for me:
mkdir build
files="
llvm-12.0.1.src.tar.xz
clang-12.0.1.src.tar.xz
clang-tools-extra-12.0.1.src.tar.xz
"
for f in $files; do
echo "Untar $f"
tar xf $f
done
mv llvm-12.0.1.src llvm
mv clang-12.0.1.src llvm/tools/clang
mv clang-tools-extra-12.0.1.src llvm/tools/clang/tools/extra
cd build
cmake -G "Unix Makefiles" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local/llvm \
-DCLANG_BUILD_TOOLS=ON \
../llvm
make -j16 install
As of LLVM 14.0.0, sparse checkouts do no longer work (at least temporarily) and the top-level directory contains no CMakeLists.txt. I believe the tree layout has changed after LLVM 13.0.1. In consequence, none of the approaches here worked without quite some modification.
Here is how you can build version 15.0.0git (the most recent at the time of this writing). See related issue: https://github.com/llvm/llvm-project/issues/53281.
First, get the compressed code or clone with git (slower)
$ wget "https://github.com/llvm/llvm-project/archive/refs/heads/main.zip" -O llvm.zip
$ unzip llvm.zip
As usual, create a build directory and run cmake in the llvm directory.
$ mkdir /build
$ cd /build
$ cmake -G "Unix Makefiles" \
-DCMAKE_INSTALL_PREFIX=/usr/local/llvm \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" \
/llvm-project-main/llvm
Navigate downwards in the generated files and only build clang-tidy.
$ cd /build/tools/clang/tools/extra/clang-tidy
$ make install
Cmake will install to /usr/local/llvm. Also, if you want to check out a specific version, use tags in the first step like this:
$ wget "https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-14.0.0.zip"
Note that you need to supply the matching builtin headers for running clang-tidy, which are located in GitHub under llvm-project/clang/lib/Headers and can by pointed to with -extra-arg=-I/path/to/builtin/headers.

qmake-generated Makefile for win32-g++ platform contains unix commands

I'm trying to build a cross-compiler with Qt 5.4.1 to generate Raspberry Pi executables from a Windows 8.1 64-bit machine.
I'm using mingw64 and a gcc 4.9 cross-compiler for the Pi on the Windows machine and I am using the Qt 5.4.1 source. PATH points to the mingw64 binaries, the gcc 4.9 cross-compiler binaries, perl binaries, and %windir%\system32.
I use the following command to build the Qt cross-compiler:
configure.bat -platform win32-g++ -opengl es2 -device linux-rasp-pi-g++ \
-device-option CROSS_COMPILE=arm-linux-gnueabihf- -sysroot \dev\qtxc\sysroot \
-opensource -confirm-license -release -make libs -prefix d:\dev\qtxc \
-platform win32-g++ -xplatform linux-arm-gnueabi-g++
Everything works fine until the very end when I get a few error messages containing Unix commands (again, I'm on a Windows machine in a Windows command prompt):
Running configuration tests...
process_begin: CreateProcess(NULL, rm -f arch.obj, ...) failed.
make (e=2): The file cannot be found
(...)
Then when I look at the generated Makefile, I can see that it erroneously defines Unix commands instead of Windows commands:
# Makefile for building: qt
# Generated by qmake (3.0) (Qt 5.4.1)
# Project: qt.pro
# Template: subdirs
# Command: D:\dev\qtxc\qtbase\bin\qmake -o Makefile qt.pro
MAKEFILE = Makefile
first: make_first
QMAKE = D:\dev\qtxc\qtbase\bin\qmake
DEL_FILE = rm -f
CHK_DIR_EXISTS= test -d
MKDIR = mkdir -p
COPY = cp -f
COPY_FILE = cp -f
COPY_DIR = cp -f -R
(...)
Then, of course, when I try to run mingw32-make as instructed in the output from the configure command, I get error messages related to the Unix commands that don't exist in Windows:
module-qtbase-qmake_all: FORCE
#test -d qtbase\ || mkdir -p qtbase\
cd qtbase\ && $(QMAKE) D:\dev\qtxc\qtbase\qtbase.pro -o Makefile
cd qtbase\ && $(MAKE) -f Makefile qmake_all
(#test doesn't work in Windows, of course).
I have tried modifying the qtbase\mkspecs\win32-g++\qmake.conf file to remove the Unix conditional definitions (seems like nonsense to have those in the win32 file anyway), forcing the definition of QMAKE_SH or QMAKE_OS_WIN32, to no avail.
Any suggestions?
I had the very same problem myself while setting up a Gitlab-CI runner to compile a Qt project on windows.
Using the -d (debug) option on qmake (see Running qmake) I discovered that qmake looks for sh in PATH and then determines which command set to use by setting QMAKE_SH.
If you seach for QMAKE_SH in the debug output of qmake bottom up, you should find the line where qmake makes the above described decision. In case sh was found, searching further up will take you to the line where the actual path to sh is printed out.
Mine was found in the path to Git for Windows (<path to git>\bin), since there is a 'sh.exe' in the bin folder next to git.exe.
I found two solutions for my particular case:
Temporarily add the unix tools to the PATH. In Git for Windows 2.5.3 they are located in <path to git>\usr\bin.
Temporarily remove the path to sh from PATH. You can do this by issuing SET PATH=%PATH:<path to remove>;=% to the command line.