I have a package I'm writing in R that has a boatload of complicated C++ code which takes a while to compile.
When I change the code, I'd like to quickly rebuild the package so I can test it. However, R CMD build seems to start from scratch each time, rather than using my code's makefiles to do only what is needed.
Is there a way to quickly do repeated builds of a package in R for testing?
I am a little overdue a short blog post on this, but I mentioned it a couple of times before: use ccache. It helps dramatically when the files don't change (ie when you just alter help pages), or when few files changes. Caching is a very clever trick, and package is robust.
On Ubuntu/Debian: sudo apt-get install ccache followed by e.g. this in your ~/.R/Makevars:
VER=
CCACHE=ccache
#CCACHE=
CC=$(CCACHE) gcc$(VER)
CXX=$(CCACHE) g++$(VER)
CXX11=$(CCACHE) g++$(VER)
CXX14=$(CCACHE) g++$(VER)
That also allows to switch between g++ versions. Changing it to clang++ is left as an exercise to the reader ;-)
Besides this, see the options to R CMD build and R CMD INSTALL to skip vignette and/or manual building to further speed up re-builds.
Illustration: Here is re-install of Rcpp itself (fresh from a git pull) where the first installation takes 21.9 seconds on my (decent) machine at work, the second then only takes 1.4 seconds thanks to ccache:
~/git/rcpp(master)$ time R CMD INSTALL .
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘Rcpp’ ...
** libs
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c Date.cpp -o Date.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c Module.cpp -o Module.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c Rcpp_init.cpp -o Rcpp_init.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c api.cpp -o api.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c attributes.cpp -o attributes.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c barrier.cpp -o barrier.o
g++ -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o Rcpp.so Date.o Module.o Rcpp_init.o api.o attributes.o barrier.o -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/Rcpp/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (Rcpp)
real 0m21.917s
user 0m21.388s
sys 0m2.304s
~/git/rcpp(master)$ ./cleanup
~/git/rcpp(master)$ time R CMD INSTALL .
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘Rcpp’ ...
** libs
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c Date.cpp -o Date.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c Module.cpp -o Module.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c Rcpp_init.cpp -o Rcpp_init.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c api.cpp -o api.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c attributes.cpp -o attributes.o
ccache g++ -I/usr/share/R/include -DNDEBUG -I../inst/include/ -fpic -g -O3 -Wall -pipe -pedantic -Wextra -Wno-empty-body -Wno-unused -c barrier.cpp -o barrier.o
g++ -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o Rcpp.so Date.o Module.o Rcpp_init.o api.o attributes.o barrier.o -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/Rcpp/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (Rcpp)
real 0m1.444s
user 0m1.380s
sys 0m1.452s
~/git/rcpp(master)$
Dirk's answer above pointed me in the right direction, but was insufficient. Since he has requested that I not append the final steps I required, I do so here.
My question could not have been answered without the following.
Dirk's answer did not work off the bat for me. If compilation still seems slow to you, try running:
ccache -s
The result will look a little like this
cache directory /home/myuser/.ccache
primary config /home/myuser/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 0
cache hit (preprocessed) 0
cache miss 989
cache hit rate 0 %
called for link 12
preprocessor error 12
cleanups performed 16
files in cache 177
cache size 31.1 MB
max cache size 5.0 GB
Note that cache isn't getting hit, which means ccache isn't doing anything.
You can use:
export CCACHE_LOGFILE=ccache.log
R CMD build .
to do some debugging, though this wasn't helpful for me.
What fixed things was to run:
export CCACHE_NOHASHDIR=true
R CMD build .
As it turns out, ccache sometimes takes the location of a file into account. R CMD build . appears to build in a temporary directory, so the location of the files was changing each time.
Related
I am trying to install a python library but I am getting an error
building '_pyFlask' extension
/home/lisa/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/lisagoh/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/lisa/anaconda3/include -fPIC -DFLASKCOMMIT="33a2752" -DHEALPIX_DATA="/mnt/c/Users/lisa_/OneDrive/Documents/Healpix_3.70/data" -DUSEXCOMPLEX=0 -DUSEMAP2TGA=0 -I/home/lisa/anaconda3/include/python3.8 -I/mnt/c/Users/lisa_/OneDrive/Documents/Healpix_3.70/include/healpix_cxx -c src/pyFlask_wrap.cxx -o build/temp.linux-x86_64-3.8/src/pyFlask_wrap.o -fopenmp
x86_64-conda_cos6-linux-gnu-cc: error trying to exec 'cc1plus': execvp: No such file or directory
error: command '/home/lisa/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1
I've tried sudo apt-get g++, sudo apt-get upgrade,sudo apt-get update and they tell me I have the latest versions but it still doesn't work.
When I try g++ --print-prog-name=cc1plus I get /usr/lib/gcc/x86_64-linux-gnu/9/cc1plus but when I try to find the cc1plus file it does not exist. I'm using the Ubunto 20.04 on a Windows OS.
I'm new to Linux and any help will be greatly appreciated thank you!
I'm trying to compile a C++ library on Mac OS which compiles on Linux. (This is done via installing an R package via a Makevars file, see: https://cran.r-project.org/doc/manuals/r-release/R-exts.html) On Linux, I see the following when I do make:
g++ -std=c++11 -I/usr/share/R/include -DNDEBUG -I../inst/include/ -I"/usr/lib/R/site-library/Rcpp/include" -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
g++ -std=c++11 -I/usr/share/R/include -DNDEBUG -I../inst/include/ -I"/usr/lib/R/site-library/Rcpp/include" -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c packagename.cpp -o packagename.o
g++ -std=c++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o packagename.so RcppExports.o packagename.o -llib1 -llib2 -llib3 -L/usr/lib/R/lib -lR
installing to /home/username/R/x86_64-pc-linux-gnu-library/3.2/packagename/libs
** R
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (packagename)
When doing this with Mac OS, I notice it's using clang++ instead, which appears to cause errors:
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I/usr/local/include -fPIC -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I/usr/local/include -fPIC -Wall -g -O2 -c packagename.cpp -o packagename.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I/usr/local/include -fPIC -Wall -g -O2 -c rcpp_hello_world.cpp -o rcpp_hello_world.o
clang++ -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o packagename.so RcppExports.o packagename.o rcpp_hello_world.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/packagename/libs
** R
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error: package or namespace load failed for ‘packagename’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Library/Frameworks/R.framework/Versions/3.4/Resources/library/packagename/libs/packagename.so':
Error: loading failed
Execution halted
ERROR: loading failed
How do I specify in the Makefile to use g++ and not clang++?
I have tried CXX=g++ but this doesn't appear to work.
I have written a few c++ function for integration into an R package by Rcpp, and these functions reply on the boost filesystem library. I have specified these in src/Makevar:
PKG_LIBS = -lboost_system -lboost_filesystem
And the package compiles ok:
==> R CMD INSTALL --preclean --no-multiarch --with-keep.source txtutils2
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c RcppExports.cpp -o RcppExports.o
* installing to library ‘/Library/Frameworks/R.framework/Versions/3.1/Resources/library’
* installing *source* package ‘txtutils2’ ...
** libs
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c bedcollc.cpp -o bedcollc.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c bedcollr.cpp -o bedcollr.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c countlines.cpp -o countlines.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c fileSize.cpp -o fileSize.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c ncols.cpp -o ncols.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c printlines.cpp -o printlines.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c readbed.cpp -o readbed.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c readcol.cpp -o readcol.o
clang++ -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.1/Resources/library/RcppArmadillo/include" -fPIC -std=c++11 -c readcols.cpp -o readcols.o
clang++ -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/usr/local/lib -o txtutils2.so RcppExports.o bedcollc.o bedcollr.o countlines.o fileSize.o ncols.o printlines.o readbed.o readcol.o readcols.o -lboost_system -lboost_filesystem -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /Library/Frameworks/R.framework/Versions/3.1/Resources/library/txtutils2/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (txtutils2)
But when I try to load it, something goes wrong:
> library(txtutils2)
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/Library/Frameworks/R.framework/Versions/3.1/Resources/library/txtutils2/libs/txtutils2.so':
dlopen(/Library/Frameworks/R.framework/Versions/3.1/Resources/library/txtutils2/libs/txtutils2.so, 6): Library not loaded: libboost_system.dylib
Referenced from: /Library/Frameworks/R.framework/Versions/3.1/Resources/library/txtutils2/libs/txtutils2.so
Reason: image not found
Error: package or namespace load failed for ‘txtutils2’
I am on osx 10.10, if that matters.
General instructions on usage of external shared libs in R is also most welcome.
As far as I know, what you have met is not the problem of Rcpp even not the problem of R. It may be about linking on OSX.
I don't use Mac, so I just give you some advice. Try
otool -L yourpackagename.so
You should see a link to the boost library you use. You need to check whether the path is right.
I would like to set the C++ compiler flag to -O0 in the Makevars of an Rcpp project.
If I take a look at /etc/R/Makeconf, I see that the compilation command seems to be
$(CXX) $(ALL_CPPFLAGS) $(ALL_CXXFLAGS) -c $< -o $#
Since
ALL_CXXFLAGS = $(R_XTRA_CXXFLAGS) $(PKG_CXXFLAGS) $(CXXPICFLAGS) $(SHLIB_CXXFLAGS) $(CXXFLAGS)
I can edit in the Makevars the variable $(PKG_CXXFLAGS) to add headers for specific libraries, but I am not satisfied with CXXFLAGS = -O3 -pipe -g $(LTO). I would also like to be able to do that directly in the Makevars, so as to tune each project according to my needs.
When I edit CXXFLAGS in the Makevar, nothing happens. Is is possible to adjust that variable ? Is another approach possible ? I know that I can edit ~/.R/Makevars, and switch as requested. I wondered if there was a more robust approach.
You generally want the PKG_* variants in your local file, e.g. ~/.R/Makevars.
Here is a (shortened, edited) portion of mine:
## for C code
CFLAGS= -O3 -g0 -Wall -pipe -pedantic -std=gnu99
## for C++ code
#CXXFLAGS= -g -O3 -Wall -pipe -Wno-unused -pedantic -std=c++11
CXXFLAGS= -g -O3 -Wall -pipe -Wno-unused -pedantic
## for Fortran code
#FFLAGS=-g -O3 -Wall -pipe
FFLAGS=-O3 -g0 -Wall -pipe
## for Fortran 95 code
#FCFLAGS=-g -O3 -Wall -pipe
FCFLAGS=-O3 -g0 -Wall -pipe
VER=-4.8
CC=ccache gcc$(VER)
CXX=ccache g++$(VER)
SHLIB_CXXLD=g++$(VER)
FC=ccache gfortran
F77=ccache gfortran
MAKE=make -j8
The other (system-global) approach is to create and edit /etc/R/Makeconf.site (or, when /etc/R/ does not exist, $RHOME/etc/R/Makeconf.site.
When compiling my code I run into an issue as follows:
io.cpp:21: undefined reference to `PQconnectdb'
as well as all other instances of missing postgres function calls occurring in my code. Obviously this is a linking problem, I'm just not sure what the link issue is.
I'm compiling with the following:
mpiCC -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ decisioning_mpi.cpp
g++ -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ io.cpp
g++ -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ calculations.cpp
g++ -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ rules.cpp
g++ -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ Instrument.cpp
g++ -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ Backtest_Parameter_CPO.cpp
g++ -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ Backtest_Trade_CPO.cpp
g++ -c -O2 -g -Wall -Werror -I /usr/include/postgresql/ Data_Bar.cpp
mpiCC -o decisioning_mpi -O2 -g -Wall -Werror -L/usr/lib -lm -lpq decisioning_mpi.o
io.o calculations.o rules.o Instrument.o Backtest_Parameter_CPO.o Backtest_Trade_CPO.o Data_Bar.o
It should be noted that this is the correct directory for libpq-fe.h and that I'm linking pq, so I'm not exactly sure why the postgres functions aren't linking correctly. I'm running Ubuntu 12.04 and installed psql (PostgreSQL) 9.1.6 from synaptic. As well I'll short circuit this, I am using #include "libpq-fe.h".
Any ideas on how I can get this linking issue resolved?
put -L/usr/lib/ -lm -lpq in the end of link command, the linker can then find the symbols
mpiCC -o decisioning_mpi -O2 -g -Wall -Werror decisioning_mpi.o io.o \
calculations.o rules.o Instrument.o Backtest_Parameter_CPO.o \
Backtest_Trade_CPO.o Data_Bar.o -L/usr/lib -lm -lpq
GCC Link Reference:
http://www.network-theory.co.uk/docs/gccintro/gccintro_18.html