Question
I'm trying to tell my package to use gcc to compile the C++ code in stead of clang. Why aren't my CXX flags in my Makevars file being used?
(I am expecting/hoping the solution to be something really simple that I've overlooked.)
It's my understanding (see references) that I can specify CXX* flags in either
src/Makevars in the package, or
~/.R/Makevars
However, I can't get option 1 to work, only option 2.
Example Builds
Here are screenshots showing the build options I'm using. In each case I'm showing both the Makevars and the /.R/Makevars files to show which one I'm using in each case.
Option 1: src/Makevars
Here I specify CXX11 = /usr/local/bin/g++-7 inside src/Makevars. The build messages say it's building with clang
Option 2: ~/.R/Makevars
Here I specify CXX11 = /usr/local/bin/g++-7 inside ~/.R/Makevars. The build message shows it builds with g++
Example package
I've noticed this on a few packages I've built recently using Rcpp, but if you want an example to test my googlePolylines package is here on github.
References
Building R packages using Alternate GCC
Understanding the contents of Makevars
Session Info
devtools::session_info()
Session info ----------------------------------------------------------------------------------------------
setting value
version R version 3.4.3 (2017-11-30)
system x86_64, darwin15.6.0 ## Mac OS
ui RStudio (1.1.414)
language (EN)
collate en_AU.UTF-8
tz Australia/Melbourne
date 2018-03-11
Quoting Duncan Murdoch in https://stat.ethz.ch/pipermail/r-package-devel/2017q4/002087.html:
According to section 1.2.1 "Using Makevars" in Writing R Extensions,
R_HOME/etcR_ARCH/Makeconf is included after Makevars, so what you're
seeing is by design. I believe this is so that packages are built with
tools compatible with those that built R. (Remember, packages are
designed for distribution to diverse systems.)
So you might change various flags within src/Makevars via for example PKG_CXXFLAGS, but you cannot overwrite CXXFLAGS or CXX itself. And there is no PKG_CXX.
Related
I'm doing a R package which is linked to the RcppCGAL package, which contains the C++ library CGAL, which depends on the C++ library Boost. If I link my package to the BH package (in order to include Boost), then R CMD CHECK throws a warning on Windows ("found significant warnings...") and then my package is not acceptable to CRAN. But with R-4.2.0, I don't need to link to BH, because Rtools42 contains the Boost library. Then I have my solution for Windows (I will just be obliged to force the R >= 4.2.0 dependency). But if I don't link to BH, the compilation fails on Linux (I tested on Linux with the help of a Github action).
So, is there a way to link or not to BH according to the OS? Or another solution?
Also, if there is such a conditional solution, it would be necessary in addition to include or not // [[Rcpp::depends(BH)]] in the C++ code according to the OS.
EDIT
I found a possible way: I put BH in the Suggests field of the DESCRIPTION file, not in the LinkingTo field, and I use the following Makevars file (BH_PATH is the relevant variable, the rest was already there):
CXX_STD = CXX14
PKG_LIBS = -lmpfr -lgmp -lgmpxx
BH_PATH=`echo 'cat(system.file("include", package="BH", mustWork=TRUE))' | "${R_HOME}/bin/R" --vanilla --no-echo`
PKG_CXXFLAGS = -DCGAL_HEADER_ONLY=1 -I$(BH_PATH)
This has worked on Ubuntu (tested via Github action). Now it remains to check whether this is portable. I gonna check on Rhub and ask to CRAN.
As part of a research project I'm trying to use clang 6.0.1 with Xcode 9.4.1. I've built and installed clang in a custom location (/opt/llvm-6_0_1/clang). I wrote a simple xcplugin compiler specification to integrate my clang version with Xcode.
Now I can open projects in Xcode, select my proxy compiler and use it to build instead of Apple's default clang.
There were some minor additions that I had to make to the xcplugin's xcspec file to get this to work that probably won't be interesting to most people, so I won't provide the details here unless asked.
This all works with most of the projects I've played with, but I'm running into an odd problem where an implicitly linked static library cannot be found by my copy of clang. Specifically I get this error:
ld: file not found: /opt/llvm-6_0_1/clang/Toolchains/LLVM6.0.1.xctoolchain/usr/lib/arc/libarclite_macosx.a
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
Note that the libarclite_macosx.a file is not explicitly included by the Xcode project. I figured it must be implicitly included, perhaps because this project enables ARC?
After pouring over the Xcode generated link command line (it's complex) I decided to look at the MyProject__dependency_info.dat file, which is passed in via the -dependency_info option. Apparently this data file (the path is defined as env var LD_DEPENDENCY_INFO_FILE) is created during the linking process, not as an input to the linker. Perhaps it exists because of a hack workaround using symlinks that I used to get a link to work (described at the end).
In any case the format appears to be binary, but I was able to see a text reference to libarclite_macosx.a in the file:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/arc/libarclite_macosx.a
After enabling the -Xlinker -v option I could see that my built clang was not searching the default toolchain lib or arc paths so I added them:
-L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
-L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/arc
Now I can see the search paths in the verbose output, but clang still cannot find the library:
Library search paths:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/arc
I've tried adding the paths to the frameworks search paths. I also tried defining the various link path env vars. Nothing has worked.
To try to get a sense of what clang is actually doing, I used fs_usage while getting the link error:
sudo fs_usage -e -w -f filesys | grep "lib/arc"
14:11:00.461965 stat64 [ 2] /opt/llvm-6_0_1/clang/Toolchains/LLVM6.0.1.xctoolchain/usr/lib/arc>>>>>>>>>>>>>>>>>>>>>> 0.000006 ld.1421614
14:11:00.461968 stat64 [ 2] /opt/llvm-6_0_1/clang/Toolchains/LLVM6.0.1.xctoolchain/usr/lib/arc>>>>>>>>>>>>>>>>>>>>>> 0.000002 ld.1421614
Clearly clang really wants to look for this file in the installed location, not the location indicated in the -dependency_info, nor in the search paths that I'm providing.
At this stage the only way I can get a build to work is to add a symlink to Xcode's "arc" directory to my installed clang lib directory. That "works", but is fragile and nasty.
Any thoughts as to how how I can get clang find the static library where it actually lives?
I am trying to compile tensorflow with a custom clang/llvm toolchain and using clang's native libc++ (instead of borrowing Gcc's stdlibc++).
It looks like bazel plain assumes that every clang will use Gcc's libraries because I get these errors:
$ bazel build --cxxopt=-std=c++11 --cxxopt=-stdlib=libc++ tensorflow:libtensorflow.so
INFO: Found 1 target...
INFO: From Compiling
external/protobuf/src/google/protobuf/compiler/js/embed.cc [for host]:
external/protobuf/src/google/protobuf/compiler/js/embed.cc:37:12:
warning: unused variable 'output_file' [-Wunused-const-variable]
const char output_file[] = "well_known_types_embed.cc";
^
1 warning generated.
ERROR: /home/hbucher/.cache/bazel/_bazel_hbucher/ad427c7fddd5b68de5e1cfaa7cd8c8cc/external/com_googlesource_code_re2/BUILD:11:1: undeclared inclusion(s) in rule '#com_googlesource_code_re2//:re2':
this rule is missing dependency declarations for the following files included by 'external/com_googlesource_code_re2/re2/bitstate.cc':
'/home/hbucher/install/include/c++/v1/stddef.h'
'/home/hbucher/install/include/c++/v1/__config'
I tried to hack into tools/cpp/CROSSTOOL inside bazel as some posts suggested to add the line
cxx_builtin_include_directory: "/home/hbucher/install/include/c++/v1"
but to no avail, it does not seem to make any difference.
Then I tried to follow a bazel tutorial to create a custom toolchain. The text does not help much because they are actually writing a cross tool while what I am trying to do is to tweak the existing host rules and somehow bazel seems to undo every attempt I try to tweak its parameters.
I have got to the point that is currently in my github repository https://github.com/HFTrader/BazelCustomToolchain
However it does not compile and I cannot even figure out how to start debugging this message.
$ bazel build --crosstool_top=#hbclang//:toolchain tensorflow:libtensorflow.so
.....................
ERROR: The crosstool_top you specified was resolved to
'#hbclang//:toolchain', which does not contain a CROSSTOOL file. You can
use a crosstool from the depot by specifying its label.
INFO: Elapsed time: 2.216s
I have appended these lines to my tensorflow/WORKSPACE
new_local_repository(
name="hbclang",
path="/home/hbucher/BazelCustomToolchain",
build_file = "/home/hbucher/BazelCustomToolchain/BUILD",
)
I have asked this question on bazel's google groups but they redirected me to stackoverflow. At this point I am about to give up.
Have someone attempted to do this or I'm breaking ground here?
Thank you.
Solved. Not in the intended way but it works for me.
export INSTALL_DIR="$HOME/install"
export CC=$INSTALL_DIR/bin/clang
export CXX=$INSTALL_DIR/bin/clang++
export CXXFLAGS="-stdlib=libc++ -L$INSTALL_DIR/lib"
export LDFLAGS="-L$INSTALL_DIR/lib -lm -lrt"
export LD_LIBRARY_PATH="/usr/lib:/lib/x86_64-linux-gnu/:$INSTALL_DIR/lib"
git clone https://github.com/tensorflow/tensorflow.git tensorflow-github
cd tensorflow-github
mkdir build-tmp && cd build-tmp
cmake ../tensorflow/contrib/cmake/
make -j4
Easy as 1-2-3 with cmake
[2020-05-24: Edit to make the answer up to date.]
TLDR: To build a project with Bazel with a specific Clang binary, and with libc++, this works for me (where INSTALL_DIR is where I've installed llvm):
CC="$INSTALL_DIR/bin/clang" \
BAZEL_CXXOPTS="-stdlib=libc++:-isystem$INSTALL_DIR/include" \
BAZEL_LINKOPTS="-stdlib=libc++" \
BAZEL_LINKLIBS="-L$INSTALL_DIR/lib:-Wl,-rpath,$INSTALL_DIR/lib:-lc++:-lm" \
bazel test //...
Background:
You can use --repo_env option, e.g. --repo_env=CC=clang, to put these defaults into your project- or system-wide .bazelrc.
This approach uses Bazel's C++ toolchain autoconfiguration which doesn't attempt to declare all the toolchain inputs in BUILD files. This is to simplify the configuration for the user. Therefore whenever you modify the C++ toolchain in a way that Bazel cannot know about (rebuild llvm etc.), you have to run bazel clean --expunge to flush the cache and rerun the autoconfiguration the next time.
The robust solution to specifying C++ toolchain in Bazel is to use the CcToolchainConfigInfo. See the documentation at https://docs.bazel.build/versions/master/tutorial/cc-toolchain-config.html and https://docs.bazel.build/versions/master/cc-toolchain-config-reference.html.
In my project (which uses waf/wscript based build system), I am now adding mongodb c++ driver APIs. I figured out that 'libmongoclient.a' is not getting added as a linker option (at compile time) and I get all undefined reference to the mongodb c++ driver API calls.
I want to understand, how do I modify my wscript so that it picks up the mongoclient related library by itself and links it properly. It perhaps involves updating the configuration function of wscript. I am new to the waf build system, and not sure how to change it.
I have built and installed the mongodb c++ driver as follows:
- INCLUDE: /usr/local/include/mongo/
- LIB: /usr/local/lib/libmongoclient.a
I posted a similar question earlier in this regard, and the above one is more specific problem statement.
https://stackoverflow.com/questions/30020574/building-project-with-waf-script-and-eclipse
Since I am just invoking ./waf from within eclipse, I believe, the options that I specify into Eclipse's build environment are not being picked up by the waf (and hence the library option for mongoclient).
I figured this out and the steps are as follows:
Added following check in the configure command/function.
conf.check_cfg(package='libmongoclient', args=['--cflags', '--libs'],
uselib_store='MONGOCLIENT', mandatory=True)
After this step, we need to add a package configuration file (.pc) into /usr/local/lib/pkgconfig path. This is the file where we specify the paths to lib and headers. Pasting the content of this file below.
prefix=/usr/local
libdir=/usr/local/lib
includedir=/usr/local/include/mongo
Name: libmongoclient
Description: Mongodb C++ driver
Version: 0.2
Libs: -L${libdir} -lmongoclient
Cflags: -I${includedir}
Added the above library into the build function to the sepcific program which depends on the above dependency (i.e. MongoClient).
mobility = bld(
target='bin/mobility',
features='cxx cxxprogram',
source='src/main.cpp',
use='mob-objects MONGOCLIENT',
)
I would like to do the following: If CCache is present in PATH, use "ccache g++" for compilation, else use g++. I tried writing a small my-cmake script containing
CC="ccache gcc" CXX="ccache g++" cmake $*
but it does not seem to work (running make still does not use ccache; I checked this using CMAKE_VERBOSE_MAKEFILE on).
Update:
As per this link I tried changing my script to
cmake -D CMAKE_CXX_COMPILER="ccache" -D CMAKE_CXX_COMPILER_ARG1="g++" -D CMAKE_C_COMPILER="ccache" -D CMAKE_C_COMPILER_ARG1="gcc" $*
but cmake bails out complaining that a test failed on using the compiler ccache (which can be expected).
As of CMAKE 3.4 you can do:
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
It is now possible to specify ccache as a launcher for compile commands and link commands (since cmake 2.8.0). That works for Makefile and Ninja generator. To do this, just set the following properties :
find_program(CCACHE_FOUND ccache)
if(CCACHE_FOUND)
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)
set_property(GLOBAL PROPERTY RULE_LAUNCH_LINK ccache) # Less useful to do it for linking, see edit2
endif(CCACHE_FOUND)
It is also possible to set these properties only for specific directories or targets.
For Ninja, this is possible since version 3.4.
For XCode, Craig Scott gives a workaround in his answer.
Edit : Thanks to uprego and Lekensteyn's comment, I edited the answer to check if ccache is available before using it as launcher and for which generators is it possible to use a compile launcher.
Edit2: #Emilio Cobos recommended to avoid doing that for the linking part as ccache doesn't improve linking speed and can mess with other types of cache like sccache
I personally have /usr/lib/ccache in my $PATH. This directory contains loads of symlinks for every possible name the compiler could be called from (like gcc and gcc-4.3), all pointing to ccache.
And I didn't even create the symlinks. That directory comes pre-filled when I install ccache on Debian.
From CMake 3.1, it is possible to use ccache with the Xcode generator and Ninja is supported from CMake 3.4 onwards. Ninja will honour RULE_LAUNCH_COMPILE just like the Unix Makefiles generator (so #Babcool's answer gets you there for Ninja too), but getting ccache working for the Xcode generator takes a little more work. The following article explains the method in detail, focussing on a general implementation which works for all three CMake generators and making no assumptions about setting up ccache symlinks or the underlying compiler used (it still lets CMake decide the compiler):
https://crascit.com/2016/04/09/using-ccache-with-cmake/
The general gist of the article is as follows. The start of your CMakeLists.txt file should be set up something like this:
cmake_minimum_required(VERSION 2.8)
find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
# Support Unix Makefiles and Ninja
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${CCACHE_PROGRAM}")
endif()
project(SomeProject)
get_property(RULE_LAUNCH_COMPILE GLOBAL PROPERTY RULE_LAUNCH_COMPILE)
if(RULE_LAUNCH_COMPILE AND CMAKE_GENERATOR STREQUAL "Xcode")
# Set up wrapper scripts
configure_file(launch-c.in launch-c)
configure_file(launch-cxx.in launch-cxx)
execute_process(COMMAND chmod a+rx
"${CMAKE_BINARY_DIR}/launch-c"
"${CMAKE_BINARY_DIR}/launch-cxx")
# Set Xcode project attributes to route compilation through our scripts
set(CMAKE_XCODE_ATTRIBUTE_CC "${CMAKE_BINARY_DIR}/launch-c")
set(CMAKE_XCODE_ATTRIBUTE_CXX "${CMAKE_BINARY_DIR}/launch-cxx")
set(CMAKE_XCODE_ATTRIBUTE_LD "${CMAKE_BINARY_DIR}/launch-c")
set(CMAKE_XCODE_ATTRIBUTE_LDPLUSPLUS "${CMAKE_BINARY_DIR}/launch-cxx")
endif()
The two script template files launch-c.in and launch-cxx.in look like this (they should be in the same directory as the CMakeLists.txt file):
launch-c.in:
#!/bin/sh
export CCACHE_CPP2=true
exec "${RULE_LAUNCH_COMPILE}" "${CMAKE_C_COMPILER}" "$#"
launch-cxx.in:
#!/bin/sh
export CCACHE_CPP2=true
exec "${RULE_LAUNCH_COMPILE}" "${CMAKE_CXX_COMPILER}" "$#"
The above uses RULE_LAUNCH_COMPILE alone for Unix Makefiles and Ninja, but for the Xcode generator it relies on help from CMake's CMAKE_XCODE_ATTRIBUTE_... variables support. The setting of the CC and CXX user-defined Xcode attributes to control the compiler command and LD and LDPLUSPLUS for the linker command is not, as far as I can tell, a documented feature of Xcode projects, but it does seem to work. If anyone can confirm it is officially supported by Apple, I'll update the linked article and this answer accordingly.
I didn't like to set a symlink from g++ to ccache. And CXX="ccache g++" didn't work for me as some cmake test case wanted to have just the compiler program without attributes.
So I used a small bash script instead:
#!/bin/bash
ccache g++ "$#"
and saved it as an executable in /usr/bin/ccache-g++.
Then C configured cmake to use /usr/bin/ccache-g++ as C++ compiler.
This way it passes the cmake test cases and I feel more comfortable than having symlinks that I might forget about in 2 or 3 weeks and then maybe wonder if something doesn't work...
I verified the following works (source: this link):
CC="gcc" CXX="g++" cmake -D CMAKE_CXX_COMPILER="ccache" -D CMAKE_CXX_COMPILER_ARG1="g++" -D CMAKE_C_COMPILER="ccache" -D CMAKE_C_COMPILER_ARG1="gcc" $*
Update: I later realized that even this does not work. Strangely it works every alternate time (the other times cmake complains).
Let me add one important item that was not mentioned here before.
While bootstrapping a minimalistic build system from the ubuntu:18.04 docker image, I've found that order of installation makes a difference.
In my case ccache worked fine when calling gcc, but failed to catch invocations of the same compiler by the other names: cc and c++.
To fully install ccache, you need to make sure all compilers are installed first, or add a call to update-ccache symlinks to be safe.
sudo /usr/sbin/update-ccache-symlinks
export PATH="/usr/lib/ccache/:$PATH"```
... and then (due to updated symlinks) also calls to cc and c++ get caught!
In my opinion the best way is to symlink gcc,g++ to ccache, but if you would like to use within cmake, try this:
export CC="ccache gcc" CXX="ccache g++" cmake ...
Here are 2 methods I think are clean/robust, and also don't pollute your CMake code.
1.) Set environment variables
This method is nice since you don't have to individually set it up for each CMake project. The con is you may not want ccache for each CMake project.
# Requires CMake 3.17 (https://cmake.org/cmake/help/latest/envvar/CMAKE_LANG_COMPILER_LAUNCHER.html)
export CMAKE_CXX_COMPILER_LAUNCHER=/usr/bin/ccache
export CMAKE_C_COMPILER_LAUNCHER=/usr/bin/ccache
2.) Pass in cache variables during project configuration
Con a bit annoying to do for each project. This can be negated by your IDE though.
# Requires CMake 3.4
$ cmake ... -D CMAKE_CXX_COMPILER_LAUNCHER=/usr/bin/ccache \
-D CMAKE_C_COMPILER_LAUNCHER=/usr/bin/ccache
NOTE: It isn't really necessary to specify the full path.
If ccache is in your path you can just specify ccache instead.
export CMAKE_CXX_COMPILER_LAUNCHER=ccache
export CMAKE_C_COMPILER_LAUNCHER=ccache
It is extending #Nicolas answer.
Add following line to your cmake file:
list(PREPEND CMAKE_PROGRAM_PATH /usr/lib/ccache)
Or add it as argument to cmake configuration step:
cmake -DCMAKE_PROGRAM_PATH=/usr/lib/ccache