Using an alternate compiler for Travis-CI R project builds - c++

Travis-CI's official R project build support on Ubuntu uses (at the time of this question) gcc version 4.6.
CRAN uses gcc 4.9 and some packages that build fine on CRAN will not build on Travis with gcc 4.6.
How does one change the default gcc compiler for R project/package builds to more closely mirror CRAN builds?

I really wanted to be able to use Travis to test my ndjson package but the C++ library I'm using won't compile under gcc 4.6.
The ndjson package is on CRAN and the CRAN builds are fine (except for r-oldrel on Windows which doesn't bother me a bit), so I needed a way to change the compiler that R uses on Travis.
I'm using gcc 5 in the example below, but you can use any version available in the toolchain test builds. Ideally, one should mimic CRAN's gcc version and this may be something the Travis folks might consider making default for R builds.
The .travis.yml starts off the same:
language: r
warnings_are_errors: true
sudo: required
env:
global:
- CRAN: http://cran.rstudio.com
I added a matrix build configuration to add the new package source as well as specify the package(s) that needed to be installed. I left it in a matrix configuration since I'm going to try to (eventually) add clang.
matrix:
include:
- os: linux
compiler: gcc
addons:
apt:
sources: ['ubuntu-toolchain-r-test']
packages: ['g++-5']
env:
- COMPILER=g++-5
- CC=gcc=5
- CXX=g++-5
Next, I made sure the auto default compiler is set to this newer gcc and also made doubly sure that R would use it by creating a local Makevars:
before_install:
- sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 100
- mkdir -p ~/.R
- echo "VkVSPS01CkNDPWdjYyQoVkVSKSAtc3RkPWMxMSAKQ1hYPWcrKyQoVkVSKQpTSExJQl9DWFhMRD1nKyskKFZFUikKRkM9Z2ZvcnRyYW4KRjc3PWdmb3J0cmFuCg==" | base64 -d > ~/.R/Makevars
- cat ~/.R/Makevars
The base64 string equates to:
VER=-5
CC=gcc$(VER) -std=c11
CXX=g++$(VER)
SHLIB_CXXLD=g++$(VER)
FC=gfortran
F77=gfortran
and is just (IMO) cleaner this way.
In theory, all I would have had to do is create the Makevars (i.e. not have to change the default gcc with update-alternatives) but it turned out that Travis used the Makevars gcc setting when installing the dependencies but not for the actual package build itself. So, the update-alternatives is necessary. I also had to add the -std=c11 to ensure a few of the dependencies compiled (the build errored w/o it).
After these modifications to the .travis.yml configuration, ndjson built fine.

Just to give an alternate approach, I use a custom shell script in one of my packages sourcetools. My goal there was to ensure the package would build using the (now ancient) gcc-4.4 compiler. In .travis.yml, I have:
language: r
cache: packages
sudo: false
warnings_are_errors: true
before_install:
- source travis/before-install.sh
addons:
apt:
packages:
- gcc-4.4
- g++-4.4
- clang
r:
- oldrel
- release
- devel
env:
- COMPILER=gcc-4.4
- COMPILER=gcc
- COMPILER=clang
And in travis/before-install.sh, I have:
#!/usr/bin/env sh
mkdir -p ~/.R
if [ "${COMPILER}" = "gcc-4.4" ]; then
echo "CC=gcc-4.4 -std=gnu99" >> ~/.R/Makevars
echo "CXX=g++-4.4" >> ~/.R/Makevars
echo "CXX1X=g++-4.4 -std=c++0x" >> ~/.R/Makevars
fi
if [ "${COMPILER}" = "gcc" ]; then
echo "CC=gcc -std=gnu99" >> ~/.R/Makevars
echo "CXX=g++" >> ~/.R/Makevars
echo "CXX1X=g++ -std=c++0x" >> ~/.R/Makevars
fi
if [ "${COMPILER}" = "clang" ]; then
echo "CC=clang -std=gnu99" >> ~/.R/Makevars
echo "CXX=clang++" >> ~/.R/Makevars
echo "CXX1X=clang++ -std=c++0x" >> ~/.R/Makevars
fi
The end result is basically the same, but IMHO it's somewhat cleaner to separate the 'setup' logic into its own script that's driven entirely off of environment variables. It also makes it easier to construct the R matrix builds, since Travis automatically combines the permutations of r and env here (no need to do it 'by hand').
I imagine the before-install.sh script I use could be cleaned up / made more general, but I haven't had the need to do that yet.

Related

How to choose linux compiler and simplify this azure-pipeline script?

Setup different compilers for cmake;
Colorize build results.
my current pipeline script for linux ('Hosted Ubuntu 1604'):
steps:
- script: 'cp CMakeLists.txt build/'
displayName: 'cp cmakelist'
- script: 'export CC=/usr/local/bin/gcc-7'
displayName: 'set gcc version'
- script: 'export CXX=/usr/local/bin/g++-7'
displayName: 'set g++ version'
- script: 'cmake CMakeLists.txt'
displayName: 'cmake'
- script: 'make AllTest'
displayName: 'Build AllTest'
- script: 'cd lib/all_test'
displayName: 'set all test directory'
- script: './AllTest'
workingDirectory: lib/all_test/
displayName: 'run AllTest'
My problems:
CMake#1 works well in windows, but this command expect the cMakeLists.txt in the build directory
I've listed the available compilers and gcc 7 is available, but even I tried to setup to use it, the cmake uses gcc 5.4, how can I select different compilers e.g gcc 7, clang (in windows I use cmake -G Visual Studio...
I use a standalone test library (rili tests/ c++) , and on the azure servers the test results has no color, how can I colorize my test results?

Travis CI build matrix contains spurious entry?

I've a Travis CI build matrix intended to test recent g++ and clang. In addition to those two intended entries, I see an entry, AFAIK, not specified in the include matrix. How can I either suppress or remove this third build configuration?
The .travis.yml:
language: generic
os: linux
script: ./bootstrap && ./configure && make all && make check && make distcheck
matrix:
include:
- env: COMPILER_NAME=gcc CXX=g++-5 CC=gcc-5
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- autotools-dev
- g++-5
- env: COMPILER_NAME=clang CXX=clang++-3.8 CC=clang-3.8
addons:
apt:
sources:
- ubuntu-toolchain-r-test
- llvm-toolchain-precise-3.8
packages:
- autotools-dev
- clang-3.8
Snippet of Travis' output taken from the full details:
From the docs, "the os key also multiplies the matrix":
If your code is used on multiple operating systems it probably should
be tested on multiple operating systems. Travis CI can test on Linux
and OS X.
To enable testing on multiple operating systems add the os key to your
.travis.yml:
os:
linux
osx
The value of the $TRAVIS_OS_NAME variable is set to linux or osx
according to the operating system a particular build is running on, so
you can use it to conditionalize your build scripts.
If you are already using a build matrix to test multiple versions, the
os key also multiplies the matrix.
Source: https://docs.travis-ci.com/user/multi-os/
Builds with multiple os keys:
For example, for each separate "os" key, it creates another build. See here: https://github.com/jaredsburrows/cplusplus-cmake-template/blob/master/.travis.yml#L3 or https://github.com/jaredsburrows/open-virus/blob/master/.travis.yml#L3.
Since the default "os" is linux, it is implied in your own question that you have 3 separate builds, that are all use "os: linux".
Matrix builds with different os keys:
For example, each key is now specified in the matrix on separate lines. See here: https://github.com/jaredsburrows/cs-interview-questions/blob/master/.travis.yml#L3 and https://github.com/jaredsburrows/android-gif-example/blob/master/.travis.yml#L8.
In your question, you are not only specifying a "matrix" build but also a plain build with a different os. When specifying "matrix", you are overriding the default "os" build. But you explicitly wrote both "os" and "matrix", implying that you wanted 3 separate builds.
Moving the language and os portions into the matrix include works. I've still no insight as to why, however.
https://github.com/RhysU/descendu/commit/ab275ea79b4ff315f99604d6d7f26dd2488aa266

Travis for C++ always ignore script for customize testing

I've been experimenting this and spend more than 25 commit with only small change in .travis.yml but travis always do either:
./configure: if I put the language: cpp in the first line. travis will ignore everything written in .travis.yml and will go straight ./configure && make && make test
language: cpp
addons:
apt:
sources:
- ubuntu-sdk-team
addons:
apt:
packages:
- ubuntu-sdk
before_script: ls
script: qmake demo/QtGoodiesDemo/QtGoodiesDemo.pro && make
sudo: false
rake: if I reorder the placement of language: cpp or remove it. this point, travis succeeded install the dependency (apt addons executed)
addons:
apt:
sources:
- ubuntu-sdk-team
addons:
apt:
packages:
- ubuntu-sdk
before_script: ls
script: qmake demo/QtGoodiesDemo/QtGoodiesDemo.pro && make
language: cpp
sudo: false
I have read the documentation and using what it told just lead me to the point (1) problem.
I've searched but nobody got this problem, I wonder how people do this.
(This is the link)
Attempt 1
using nested apt source still gets me to problem(1) and ignore the custom script (https://travis-ci.org/imakin/QtGoodies/builds/85143452). Note that in problem 2 when rake is called, it succeeded in installing ubuntu-sdk for the Qt dependencies
language: cpp
sudo: false
addons:
apt:
sources:
- ubuntu-sdk-team
packages:
- ubuntu-sdk
before_script: ls
script: qmake demo/QtGoodiesDemo/QtGoodiesDemo.pro && make
Attempt 2
So then I try to play it and added configure file. Having it all rwxrwxrwx permission too. and pushed it. But travis still execute to problem (1) again. No Qt Installing, even failed to run the configure telling it has no permission to execute it. I can't do anything to chmod it in testing process though as before_script is ignored too. This one occurred in https://travis-ci.org/imakin/QtGoodies/builds/85146543
I think your .travis.yml file is invalid and prevents it to be parsed properly, hence you default to the Ruby language.
Can you try merging both addons.apt sections like so:
addons:
apt:
sources:
- ubuntu-sdk-team
packages:
- ubuntu-sdk
Hope this helps!
Update: it turns out the .travis.yml file started with a hidden character that prevented Travis CI from parsing it properly.

Protobuf 3.0.0-alpha-1 not compiling on windows

I'm trying to compile protobuf 3.0.0 alpha 1 on Windows using MinGW 4.9.2 & MSYS.
According to the instructions I'm supposed to:
./configure
make
make check
make install
I added --prefix=/c/path/to/mingw to configure (How to build Google's protobuf in Windows using MinGW?) but it didnt help.
It fails at make with the message:
CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /home/Markus/protobuf-3.0.0-alpha-
1/missing aclocal-1.14 -I m4
/home/Markus/protobuf-3.0.0-alpha-1/missing: line 81: aclocal-1.14: command not
found
WARNING: 'aclocal-1.14' is missing on your system.
You should only need it if you modified 'acinclude.m4' or
'configure.ac' or m4 files included by 'configure.ac'.
The 'aclocal' program is part of the GNU Automake package:
<http://www.gnu.org/software/automake>
It also requires GNU Autoconf, GNU m4 and Perl in order to run:
<http://www.gnu.org/software/autoconf>
<http://www.gnu.org/software/m4/>
<http://www.perl.org/>
make: *** [aclocal.m4] Error 127
I've tried installing Automake but it doesnt come with aclocal.
Also today at work i managed to get it working on the first try with a bare MinGW & MSYS.
It looks like you need to install autoconf (which is separate from automake, though they are often used together). You may also need to install libtool.

How do I install and build against OpenSSL 1.0.0 on Ubuntu?

You can consider this a follow-up question to How do I install the OpenSSL C++ library on Ubuntu?
I'm trying to build some code on Ubuntu 10.04 LTS that requires OpenSSL 1.0.0.
Ubuntu 10.04 LTS comes with OpenSSL 0.9.8k:
$ openssl version
OpenSSL 0.9.8k 25 Mar 2009
So after running sudo apt-get install libssl-dev and building, running ldd confirms I've linked in 0.9.8:
$ ldd foo
...
libssl.so.0.9.8 => /lib/i686/cmov/libssl.so.0.9.8 (0x00110000)
...
libcrypto.so.0.9.8 => /lib/i686/cmov/libcrypto.so.0.9.8 (0x002b0000)
...
How do I install OpenSSL 1.0.0 and the 1.0.0 development package?
Update: I'm writing this update after reading SB's answer (but before trying it), because it's clear I need to explain that the obvious solution of downloading and installing OpenSSL 1.0.0 doesn't work:
After successfully doing the following (recommended in the INSTALL file):
$ ./config
$ make
$ make test
$ make install
...I still get:
OpenSSL 0.9.8k 25 Mar 2009
...and:
$ sudo apt-get install libssl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libssl-dev is already the newest version.
The following packages were automatically installed and are no longer required:
linux-headers-2.6.32-21 linux-headers-2.6.32-21-generic
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
...and (just to make sure) after rebuilding my code, ldd still returns the same thing.
Update #2: I added the "-I/usr/local/ssl/include" and "-L/usr/local/ssl/lib" options (suggested by SB) to my makefile, but I'm now getting a bunch of undefine reference compile errors, for example:
/home/dspitzer/foo/foo.cpp:86: undefined reference to `BIO_f_base64'
/home/dspitzer/foo/foo.cpp:86: undefined reference to `BIO_new'
/usr/local/ssl/include/ contains only an openssl directory (which contains numerous .h files), so I also tried "-I/usr/local/ssl/include/openssl" but got the same errors.
Update #3: I tried changing the OpenSSL includes from (for example):
#include <openssl/bio.h>
...to:
#include "openssl/bio.h"
...in the .cpp source file but still get the same undefined reference errors.
Update #4: I now realize those undefined reference errors are linker errors. If I remove the "-L/usr/local/ssl/lib" from my Makefile, I don't get the errors (but it links to OpenSSL 0.9.8). The contents of /usr/local/ssl/lib/ are:
$ ls /usr/local/ssl/lib/
engines libcrypto.a libssl.a pkgconfig
I added -lcrypto, and the errors went away.
Get the 1.0.0a source from here.
# tar -xf openssl-1.0.0a.tar.gz
# cd openssl-1.0.0a
# ./config
# sudo make install
Note: if you have man pages build errors on modern systems, use make install_sw instead of make install.
This puts it in /usr/local/ssl by default
When you build, you need to tell gcc to look for the headers in /usr/local/ssl/include and link with libs in /usr/local/ssl/lib. You can specify this by doing something like:
gcc test.c -o test -I/usr/local/ssl/include -L/usr/local/ssl/lib -lssl -lcrypto
EDIT DO NOT overwrite any system libraries. It's best to keep new libs in /usr/local. Overwriting Ubuntu defaults can be hazardous to your health and break your system.
Additionally, I was wrong about the paths as I just tried this in Ubuntu 10.04 VM. Fixed.
Note, there is no need to change LD_LIBRARY_PATH since the openssl libs you link against by default are static libs (at least by default - there might be a way to configure them as dynamic libs in the ./config step)
You may need to link against libcrypto because you are using some calls that are built and defined in the libcrypto package. Openssl 1.0.0 actually builds two libraries, libcrypto and libssl.
EDIT 2 Added -lcrypto to gcc line.
Instead of:
$ ./config
$ make
$ make test
$ make install
Do:
$ sudo ./config --prefix=/usr
$ sudo make
$ sudo make test
$ sudo make install
This will help you update to openssl 1.0.1g to patch for CVE-2014-0160 (Heartbleed).
OpenSSL Security Advisory [07 Apr 2014]
TLS heartbeat read overrun (CVE-2014-0160)
A missing bounds check in the handling of the TLS heartbeat extension can be
used to reveal up to 64k of memory to a connected client or server.
Only 1.0.1 and 1.0.2-beta releases of OpenSSL are affected including
1.0.1f and 1.0.2-beta1.
Thanks for Neel Mehta of Google Security for discovering this bug and to
Adam Langley and Bodo Moeller for
preparing the fix.
Affected users should upgrade to OpenSSL 1.0.1g. Users unable to immediately
upgrade can alternatively recompile OpenSSL with -DOPENSSL_NO_HEARTBEATS.
1.0.2 will be fixed in 1.0.2-beta2.
Source: https://www.openssl.org/news/secadv_20140407.txt
Here's what solved it for me:
Upgrade latest version OpenSSL on Ubuntu
Transcribing the main information:
Download the OpenSSL v1.0.0g source:
$ wget http://www.openssl.org/source/openssl-1.0.0g.tar.gz
Unpack the archive and install:
$ tar xzvf openssl-1.0.0g.tar.gz
$ cd openssl-1.0.0g
$ ./config
$ make
$ make test
$ sudo make install
All files, including binaries and man pages are install under the directory /usr/local/ssl. To ensure users use this version of OpenSSL instead of the previous version you must update the paths for man pages and binaries.
Edit the file /etc/manpath.config adding the following line before the first MANPATH_MAP:
MANPATH_MAP /usr/local/ssl/bin /usr/local/ssl/man
Update the man database (I honestly can't remember and don't know for sure if this command was necessary - maybe try without it and at the end when testing if the man pages are still the old versions come back and run mandb):
sudo mandb
Edit the file /etc/environment and insert the path for OpenSSL binaries (/usr/local/ssl/bin) before the path for Ubuntu's version of OpenSSL (/usr/bin). My environment file looks like this:
PATH="/usr/local/sbin:/usr/local/bin:/usr/local/ssl/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
Logout and login and test:
$ openssl version
OpenSSL 1.0.0g 18 Jan 2012
Also test the man pages by running man openssl and at the very bottom in the left hand corner it should report 1.0.0g.
Note that although the users will now automatically use the new version of OpenSSL, existing programs (e.g. Apache) may not as they are linked against the libraries from the Ubuntu version.