Related
It seems like there are a few answers that kind-of, sort-of make sense, but that I don't know how to carry out. And I haven't found a comprehensive answer.
The First Problem
Google Test should not be an installed library, it should be built with the project. (See the FAQ.) As far as I can tell, this means the Google Test libraries are a dependency of my unit tests, and should be built when I run "make check" within my project for the first time. This should build Google Test libraries in some directory. I don't know how to do this. It mentions some autotools script that's deprecated, and I'm not sure what they're talking about or how to point my build at it properly.
The Second Problem
Assuming the build is successful, how do I write a test that uses my locally-compiled version of Google Test to run tests? I assume that there are a bunch of Makefile.am commands I put in my tests directory. But what are they? And what's an example of a unit test that uses Google Test?
I have solved the problem to my satisfaction! I will move on entirely now. This is basically asking for a tutorial. There are a lot of decisions that must be made, hopefully logically, so that Google Test dovetails nicely into autotools. So I apologize in advance for the long answer, but all the details should be there.
The First Problem
In order to understand the answer, the question needs to be rephrased a little. We are compiling Google Test as a library which our test code will link to. The library will not be installed. The question we want to ask is
"How do we configure autotools to compile Google Test as a library
which our test code can link against?"
In order to do that, we need to download Google Test and place it into our project. I use Github, so I do that by adding a submodule in the root path of my project:
$ git submodule add git#github.com:google/googletest.git
$ git submodule init
$ git submodule update
This downloads googletest into my root of my project:
/:
Makefile.am
configure.ac
src/:
(files for my project)
tests/:
(test files)
googletest/:
googletest/:
include/:
(headers, etc., to be included)
gtest/:
gtest.h
m4/:
(directory for m4 scripts and things)
src/:
(source files for Google Test)
I need to compile per the instructions. I only want the Google Test library to be built upon running make check, so I will use check_LTLIBRARIES. I add the following to my tests Makefile.am in /tests:
check_LTLIBRARIES = libgtest.la
libgtest_la_SOURCES = ../googletest/googletest/src/gtest-all.cc
libgtest_la_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/googletest/googletest
libgtest_la_LDFLAGS = -pthread
This requires subdir-objects to be enabled in configure.ac. That is accomplished by adding it to the AM_INIT_AUTOMAKE line. I also need to include the makefile in AC_CONFIG_FILES. We also want to use libtool, because we are compiling library files (I'll explain why and how that works in a moment). To use libtool, we add AM_PROG_AR, LT_INIT. We want autoreconf to install m4 macros to /m4, and then we want automake to find them, so we need AC_CONFIG_MACRO_DIRS. My configure.ac has lines updated:
AM_INIT_AUTOMAKE([-Wall -Werror subdir-objects])
...
AM_PROG_AR
LT_INIT
AC_CONFIG_MACRO_DIRS([m4])
...
AC_CONFIG_FILES([Makefile
src/Makefile
tests/Makefile
])
I also need to include the subdirectory and a line pointing to the macros in the /m4 macros directory in my /Makefile.am:
ACLOCAL_AMFLAGS = -I m4
SUBDIRS = src tests
What has this done? Libtool has been enabled with AM_PROG_AR and LT_INIT. The check_LTLIBRARIES means we will use libtool to create what's called a convenience library called libgtest.la. With subdir-objects enabled, it will be built into the /tests directory, but not installed. This means that, whenever we want to update our tests, we don't have to recompile the Google Test library libgtest.la. This will save time when testing and help us iterate faster. Then, we will want to compile our unit tests against it later as we update them. The library will only be compiled upon running make check, saving time by not compiling it if all we want to do is make or make install.
The Second Problem
Now, the second problem needs to be refined: How do you (a) create a test (b) that is linked to the Google Test libraries and thus uses them? The questions are kind of intertwined, so we answer them at once.
Creating a test is just a matter of putting the following code into a gtest.cpp file located at /tests/gtest.cpp:
#include "gtest/gtest.h" // we will add the path to C preprocessor later
TEST(CategoryTest, SpecificTest)
{
ASSERT_EQ(0, 0);
}
int main(int argc, char **argv)
{
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
This runs only the simple test 0=0. To create a test for your library, you need to read the primer. You'll notice we don't need a header for this (yet). We are linking to the file "gtest/gtest.h", so we'll need to make sure that we tell automake to include a directory that has gtest/gtest.h.
Next, we need to tell automake that we want to build a test and run it. The test is going to build into an executable that we don't want to install. Then automake is going to run that executable. It will report whether that executable says the tests passed or failed.
Automake does that by looking in the makefile for the variable check_PROGRAMS. These are the programs it will compile, but it won't necessarily run them. So we add to /tests/Makefile.am:
check_PROGRAMS = gtest
gtest_SOURCES = gtest.cpp
gtest_LDADD = libgtest.la
gtest_LDFLAGS = -pthread
gtest_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/googletest/googletest -pthread
The gtest_SOURCES finds the /tests/gtest.cpp file and compiles it. gtest_LDADD links against libgtest.la which will be compiled into the /tests directory. Google wants us to use the gtest_LDFLAGS line to enable pthreads. Finally, we need to include the location where the header "gtest/gtest.h" will be found, and that is the gtest_CPPFLAGS line. Google also wants us to include the /googletest/googletest location, and include the
The state of things: The Google Test library libgtest.la will compile with make into the directory /tests, but not be installed. The binary gtest will only be compiled with make check, but will not be installed.
Next we want to tell automake to actually run the compiled binary gtest and report errors. This is accomplished by adding a line to /tests/Makefile.am:
TESTS = gtest
The final /tests/Makefile.am looks like this:
check_LTLIBRARIES = libgtest.la
libgtest_la_SOURCES = ../googletest/googletest/src/gtest-all.cc
libgtest_la_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/googletest/googletest -pthread
check_PROGRAMS = gtest demo
gtest_SOURCES = gtest.cpp ../src/fields.cpp
gtest_LDADD = libgtest.la
gtest_LDFLAGS = -pthread
gtest_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/src
demo_SOURCES = demo.cpp ../src/fields.cpp
demo_CPPFLAGS = -I$(top_srcdir)/src
TESTS = gtest
Now, autoreconf -fiv (note any errors and hopefully fix them) from /, and make check and you should get a test that runs:
build(dev)$ make check
Making check in tests
/Applications/Xcode.app/Contents/Developer/usr/bin/make gtest
make[2]: `gtest' is up to date.
/Applications/Xcode.app/Contents/Developer/usr/bin/make check-TESTS
PASS: gtest
============================================================================
Testsuite summary for IonMotion 0.0.1
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
Here is a sample Makefile.am for the unit test project (project name: TestProject). It depends on GTEST and GMOCK:
Makefile.am
#######################################
# The list of executables we are building seperated by spaces
# the 'bin_' indicates that these build products will be installed
# in the $(bindir) directory. For example /usr/bin
#bin_PROGRAMS=exampleProgram
# Because a.out is only a sample program we don't want it to be installed.
# The 'noinst_' prefix indicates that the following targets are not to be
# installed.
noinst_PROGRAMS=utTestProject
#######################################
# Build information for each executable. The variable name is derived
# by use the name of the executable with each non alpha-numeric character is
# replaced by '_'. So a.out becomes a_out and the appropriate suffex added.
# '_SOURCES' for example.
# Sources for the a.out
utTestProject_SOURCES= \
utTestProject.cpp
# Library dependencies
utTestProject_LDADD = \
$(top_srcdir)/../TestProject/build/${host}/libTestProject/.libs/libTestProject.a \
../$(PATH_TO_GTEST)/lib/libgtest.a \
../$(PATH_TO_GMOCK)/lib/libgmock.a
# Compiler options for a.out
utTestProject_CPPFLAGS = \
-std=c++11 \
-I../$(PATH_TO_GTEST)/include \
-I../$(PATH_TO_GMOCK)/include \
-I$(top_srcdir)/include \
-I$(top_srcdir)/..
TESTS = utTestProject
TESTS_ENVIRONMENT = export UT_FOLDER_PATH=$(top_srcdir)/utTestProject; \
export GTEST_OUTPUT="xml";
Compiling gtest:
# Useful vars
SourceVersionedArchiveFolderName="gtest-1.7.0"
#
# Make it
#
pushd .
cd ./${SourceVersionedArchiveFolderName}/make
make gtest.a
if [ $? != 0 ]; then
echo "$0: Make failed"
exit 1
fi
popd
It's worth noting that Googletest doesn't officially maintain its Autotools integration anymore:
Before settling on CMake, we have been providing hand-maintained build
projects/scripts for Visual Studio, Xcode, and Autotools. While we
continue to provide them for convenience, they are not actively
maintained any more. We highly recommend that you follow the
instructions in the above sections to integrate Google Test with your
existing build system.
https://github.com/google/googletest/tree/master/googletest#legacy-build-scripts
It's now recommended to build Googletest with CMake.
Making GoogleTest's source code available to the main build can be
done a few different ways:
Download the GoogleTest source code manually and place it at a known
location. This is the least flexible approach and can make it more
difficult to use with continuous integration systems, etc.
Embed the
GoogleTest source code as a direct copy in the main project's source
tree. This is often the simplest approach, but is also the hardest to
keep up to date. Some organizations may not permit this method.
Add
GoogleTest as a git submodule or equivalent. This may not always be
possible or appropriate. Git submodules, for example, have their own
set of advantages and drawbacks.
Use CMake to download GoogleTest as
part of the build's configure step. This is just a little more
complex, but doesn't have the limitations of the other methods.
https://github.com/google/googletest/tree/master/googletest#incorporating-into-an-existing-cmake-project
I'm really eager to start using Google's new Tensorflow library in C++. The website and docs are just really unclear in terms of how to build the project's C++ API and I don't know where to start.
Can someone with more experience help by discovering and sharing a guide to using tensorflow's C++ API?
To get started, you should download the source code from Github, by following the instructions here (you'll need Bazel and a recent version of GCC).
The C++ API (and the backend of the system) is in tensorflow/core. Right now, only the C++ Session interface, and the C API are being supported. You can use either of these to execute TensorFlow graphs that have been built using the Python API and serialized to a GraphDef protocol buffer. There is also an experimental feature for building graphs in C++, but this is currently not quite as full-featured as the Python API (e.g. no support for auto-differentiation at present). You can see an example program that builds a small graph in C++ here.
The second part of the C++ API is the API for adding a new OpKernel, which is the class containing implementations of numerical kernels for CPU and GPU. There are numerous examples of how to build these in tensorflow/core/kernels, as well as a tutorial for adding a new op in C++.
To add to #mrry's post, I put together a tutorial that explains how to load a TensorFlow graph with the C++ API. It's very minimal and should help you understand how all of the pieces fit together. Here's the meat of it:
Requirements:
Bazel installed
Clone TensorFlow repo
Folder structure:
tensorflow/tensorflow/|project name|/
tensorflow/tensorflow/|project name|/|project name|.cc (e.g. https://gist.github.com/jimfleming/4202e529042c401b17b7)
tensorflow/tensorflow/|project name|/BUILD
BUILD:
cc_binary(
name = "<project name>",
srcs = ["<project name>.cc"],
deps = [
"//tensorflow/core:tensorflow",
]
)
Two caveats for which there are probably workarounds:
Right now, building things needs to happen within the TensorFlow repo.
The compiled binary is huge (103MB).
https://medium.com/#jimfleming/loading-a-tensorflow-graph-with-the-c-api-4caaff88463f
If you are thinking into using Tensorflow c++ api on a standalone package you probably will need tensorflow_cc.so ( There is also a c api version tensorflow.so ) to build the c++ version you can use:
bazel build -c opt //tensorflow:libtensorflow_cc.so
Note1: If you want to add intrinsics support you can add this flags as: --copt=-msse4.2 --copt=-mavx
Note2: If you are thinking into using OpenCV on your project as well, there is an issue when using both libs together (tensorflow issue) and you should use --config=monolithic.
After building the library you need to add it to your project.
To do that you can include this paths:
tensorflow
tensorflow/bazel-tensorflow/external/eigen_archive
tensorflow/bazel-tensorflow/external/protobuf_archive/src
tensorflow/bazel-genfiles
And link the library to your project:
tensorflow/bazel-bin/tensorflow/libtensorflow_framework.so (unused if you build with --config=monolithic)
tensorflow/bazel-bin/tensorflow/libtensorflow_cc.so
And when you are building your project you should also specify to your compiler that you are going to use c++11 standards.
Side Note: Paths relative to tensorflow version 1.5 (You may need to check if in your version anything changed).
Also this link helped me a lot into finding all this infos: link
First, after installing protobuf and eigen, you'd like to build Tensorflow:
./configure
bazel build //tensorflow:libtensorflow_cc.so
Then Copy the following include headers and dynamic shared library to /usr/local/lib and /usr/local/include:
mkdir /usr/local/include/tf
cp -r bazel-genfiles/ /usr/local/include/tf/
cp -r tensorflow /usr/local/include/tf/
cp -r third_party /usr/local/include/tf/
cp -r bazel-bin/libtensorflow_cc.so /usr/local/lib/
Lastly, compile using an example:
g++ -std=c++11 -o tf_example \
-I/usr/local/include/tf \
-I/usr/local/include/eigen3 \
-g -Wall -D_DEBUG -Wshadow -Wno-sign-compare -w \
-L/usr/local/lib/libtensorflow_cc \
`pkg-config --cflags --libs protobuf` -ltensorflow_cc tf_example.cpp
One alternative to using Tensorflow C++ API I found is to use cppflow.
It's a lightweight C++ wrapper around Tensorflow C API. You get very small executables and it links against the libtensorflow.so already compiled file. There are also examples of use and you use CMAKE instead of Bazel.
If you wish to avoid both building your projects with Bazel and generating a large binary, I have assembled a repository instructing the usage of the TensorFlow C++ library with CMake. You can find it here. The general ideas are as follows:
Clone the TensorFlow repository.
Add a build rule to tensorflow/BUILD (the provided ones do not include all of the C++ functionality).
Build the TensorFlow shared library.
Install specific versions of Eigen and Protobuf, or add them as external dependencies.
Configure your CMake project to use the TensorFlow library.
If you don't mind using CMake, there is also tensorflow_cc project that builds and installs TF C++ API for you, along with convenient CMake targets you can link against. The project README contains an example and Dockerfiles you can easily follow.
You can use this ShellScript to install (most) of it's dependencies, clone, build, compile and get all the necessary files into ../src/includes folder:
https://github.com/node-tensorflow/node-tensorflow/blob/master/tools/install.sh
If you don't want to build Tensorflow yourself and your operating system is Debian or Ubuntu, you can download prebuilt packages with the Tensorflow C/C++ libraries. This distribution can be used for C/C++ inference with CPU, GPU support is not included:
https://github.com/kecsap/tensorflow_cpp_packaging/releases
There are instructions written how to freeze a checkpoint in Tensorflow (TFLearn) and load this model for inference with the C/C++ API:
https://github.com/kecsap/tensorflow_cpp_packaging/blob/master/README.md
Beware: I am the developer of this Github project.
I use a hack/workaround to avoid having to build the whole TF library myself (which saves both time (it's set up in 3 minutes), disk space, installing dev dependencies, and size of the resulting binary). It's officially unsupported, but works well if you just want to quickly jump in.
Install TF through pip (pip install tensorflow or pip install tensorflow-gpu). Then find its library _pywrap_tensorflow.so (TF 0.* - 1.0) or _pywrap_tensorflow_internal.so (TF 1.1+). In my case (Ubuntu) it's located at /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so. Then create a symlink to this library called lib_pywrap_tensorflow.so somewhere where your build system finds it (e.g. /usr/lib/local). The prefix lib is important! You can also give it another lib*.so name - if you call it libtensorflow.so, you may get better compatibility with other programs written to work with TF.
Then create a C++ project as you are used to (CMake, Make, Bazel, whatever you like).
And then you're ready to just link against this library to have TF available for your projects (and you also have to link against python2.7 libraries)! In CMake, you e.g. just add target_link_libraries(target _pywrap_tensorflow python2.7).
The C++ header files are located around this library, e.g. in /usr/local/lib/python2.7/dist-packages/tensorflow/include/.
Once again: this way is officially unsupported and you may run in various issues. The library seems to be statically linked against e.g. protobuf, so you may run in odd link-time or run-time issues. But I am able to load a stored graph, restore the weights and run inference, which is IMO the most wanted functionality in C++.
answers above are good enough to show how to build the library, but how to collect the headers are still tricky. here I share the little script I use to copy the necessary headers.
SOURCE is the first param, which is the tensorflow source(build) direcoty;
DST is the second param, which is the include directory holds the collected headers. (eg. in cmake, include_directories(./collected_headers_here)).
#!/bin/bash
SOURCE=$1
DST=$2
echo "-- target dir is $DST"
echo "-- source dir is $SOURCE"
if [[ -e $DST ]];then
echo "clean $DST"
rm -rf $DST
mkdir $DST
fi
# 1. copy the source code c++ api needs
mkdir -p $DST/tensorflow
cp -r $SOURCE/tensorflow/core $DST/tensorflow
cp -r $SOURCE/tensorflow/cc $DST/tensorflow
cp -r $SOURCE/tensorflow/c $DST/tensorflow
# 2. copy the generated code, put them back to
# the right directories along side the source code
if [[ -e $SOURCE/bazel-genfiles/tensorflow ]];then
prefix="$SOURCE/bazel-genfiles/tensorflow"
from=$(expr $(echo -n $prefix | wc -m) + 1)
# eg. compiled protobuf files
find $SOURCE/bazel-genfiles/tensorflow -type f | while read line;do
#echo "procese file --> $line"
line_len=$(echo -n $line | wc -m)
filename=$(echo $line | rev | cut -d'/' -f1 | rev )
filename_len=$(echo -n $filename | wc -m)
to=$(expr $line_len - $filename_len)
target_dir=$(echo $line | cut -c$from-$to)
#echo "[$filename] copy $line $DST/tensorflow/$target_dir"
cp $line $DST/tensorflow/$target_dir
done
fi
# 3. copy third party files. Why?
# In the tf source code, you can see #include "third_party/...", so you need it
cp -r $SOURCE/third_party $DST
# 4. these headers are enough for me now.
# if your compiler complains missing headers, maybe you can find it in bazel-tensorflow/external
cp -RLf $SOURCE/bazel-tensorflow/external/eigen_archive/Eigen $DST
cp -RLf $SOURCE/bazel-tensorflow/external/eigen_archive/unsupported $DST
cp -RLf $SOURCE/bazel-tensorflow/external/protobuf_archive/src/google $DST
cp -RLf $SOURCE/bazel-tensorflow/external/com_google_absl/absl $DST
Tensorflow itself only provides very basic examples about C++ APIs.
Here is a good resource which includes examples of datasets, rnn, lstm, cnn and more
tensorflow c++ examples
We now provide a pre-built library and a Docker image for easy installation and usage of the TensorFlow C++ API at https://github.com/ika-rwth-aachen/libtensorflow_cc
We provide the pre-built libtensorflow_cc.so including accompanying headers as a one-command-install deb-package.
We provide a pre-built Docker image based on the official TensorFlow Docker image. Our Docker image has both TensorFlow Python and TensorFlow C++ installed.
Try it out yourself by running the example application:
git clone https://github.com/ika-rwth-aachen/libtensorflow_cc.git && \
cd libtensorflow_cc && \
docker run --rm \
--volume $(pwd)/example:/example \
--workdir /example \
rwthika/tensorflow-cc:latest \
./build-and-run.sh
While we currently only support x86_64 machines running Ubuntu, this could easily be extended to other OS and platforms in the future. Except for a some exceptions, all TensorFlow versions from 2.0.0 through 2.9.2 are available, 2.10.0 coming soon.
If you want to use the TensorFlow C++ API to load, inspect, and run saved models and frozen graphs in C++, we suggest that you also check out our helper library tensorflow_cpp.
Im trying to run a c++ program on github. (available at the following link https://github.com/mortehu/text-classifier)
I have a mac, and am trying to run it in the terminal. I think I have downloaded autoconf and automake but am not sure. To run the program I am going to the correct folder in terminal then running
./configure && make
But I get the error:
WARNING: 'aclocal-1.15' is missing on your system.
You should only need it if you modified 'acinclude.m4' or
'configure.ac' or m4 files included by 'configure.ac'.
The 'aclocal' program is part of the GNU Automake package:
http://www.gnu.org/software/automake
It also requires GNU Autoconf, GNU m4 and Perl in order to run:
http://www.gnu.org/software/autoconf
http://www.gnu.org/software/m4/
http://www.perl.org/ make: *** [aclocal.m4] Error 127
I have xcode and g++ and all the things required to run c programs, but as is probably obvious, I have no idea what Im doing.
What is the easiest, simplest way to run the program in the above link? I realise it comes with a readme and example usage but I can not get that to work.
Before running ./configure try running autoreconf -f -i. The autoreconf program automatically runs autoheader, aclocal, automake, autopoint and libtoolize as required.
Edit to add: This is usually caused by checking out code from Git instead of extracting it from a .zip or .tar.gz archive. In order to trigger rebuilds when files change, Git does not preserve files' timestamps, so the configure script might appear to be out of date. As others have mentioned, there are ways to get around this if you don't have a sufficiently recent version of autoreconf.
Another edit: This error can also be caused by copying the source folder extracted from an archive with scp to another machine. The timestamps can be updated, suggesting that a rebuild is necessary. To avoid this, copy the archive and extract it in place.
Often, you don't need any auto* tools and the simplest solution is to simply run touch aclocal.m4 configure in the relevant folder (and also run touch on Makefile.am and Makefile.in if they exist). This will update the timestamp of aclocal.m4 and remind the system that aclocal.m4 is up-to-date and doesn't need to be rebuilt. After this, it's probably best to empty your build directory and rerun configure from scratch after doing this. I run into this problem regularly. For me, the root cause is that I copy a library (e.g. mpfr code for gcc) from another folder and the timestamps change.
Of course, this trick isn't valid if you really do need to regenerate those files, perhaps because you have manually changed them. But hopefully the developers of the package distribute up-to-date files.
And of course, if you do want to install automake and friends, then use the appropriate package-manager for your distribution.
Install aclocal which comes with automake:
brew install automake # for Mac
apt-get install automake # for Ubuntu
Try again:
./configure && make
You can install the version you need easily:
First get source:
$ wget https://ftp.gnu.org/gnu/automake/automake-1.15.tar.gz
Unpack it:
$ tar -xzvf automake-1.15.tar.gz
Build and install:
$ cd automake-1.15
$ ./configure --prefix=/opt/aclocal-1.15
$ make
$ sudo mkdir -p /opt
$ sudo make install
Use it:
$ export PATH=/opt/aclocal-1.15/bin:$PATH
$ aclocal --version
aclocal (GNU automake) 1.15
Now when aclocal is called, you get the right version.
A generic answer that may or not apply to this specific case:
As the error message hint at, aclocal-1.15 should only be required if you modified files that were used to generate aclocal.m4
If you don't modify any of those files (including configure.ac) then you should not need to have aclocal-1.15.
In my case, the problem was not that any of those files was modified but somehow the timestamp on configure.ac was 6 minutes later compared to aclocal.m4.
I haven't figured out why, but a clean clone of my git repo solved the issue for me. Maybe something linked to git and how it created files in the first place.
Rather than rerunning autoconf and friends, I would just try to get a clean clone and try again.
It's also possible that somebody committed a change to configure.ac but didn't regenerate the aclocal.m4, in which case you indeed have to rerun automake and friends.
The whole point of Autotools is to provide an arcane M4-macro-based language which ultimately compiles to a shell script called ./configure. You can ship this compiled shell script with the source code and that script should do everything to detect the environment and prepare the program for building. Autotools should only be required by someone who wants to tweak the tests and refresh that shell script.
It defeats the point of Autotools if GNU This and GNU That has to be installed on the system for it to work. Originally, it was invented to simplify the porting of programs to various Unix systems, which could not be counted on to have anything on them. Even the constructs used by the generated shell code in ./configure had to be very carefully selected to make sure they would work on every broken old shell just about everywhere.
The problem you're running into is due to some broken Makefile steps invented by people who simply don't understand what Autotools is for and the role of the final ./configure script.
As a workaround, you can go into the Makefile and make some changes to get this out of the way. As an example, I'm building the Git head of GNU Awk and running into this same problem. I applied this patch to Makefile.in, however, and I can sucessfully make gawk:
diff --git a/Makefile.in b/Makefile.in
index 5585046..b8b8588 100644
--- a/Makefile.in
+++ b/Makefile.in
## -312,12 +312,12 ## distcleancheck_listfiles = find . -type f -print
# Directory for gawk's data files. Automake supplies datadir.
pkgdatadir = $(datadir)/awk
-ACLOCAL = #ACLOCAL#
+ACLOCAL = true
AMTAR = #AMTAR#
AM_DEFAULT_VERBOSITY = #AM_DEFAULT_VERBOSITY#
-AUTOCONF = #AUTOCONF#
-AUTOHEADER = #AUTOHEADER#
-AUTOMAKE = #AUTOMAKE#
+AUTOCONF = true
+AUTOHEADER = true
+AUTOMAKE = true
AWK = #AWK#
CC = #CC#
CCDEPMODE = #CCDEPMODE#
Basically, I changed things so that the harmless true shell command is substituted for all the Auto-stuff programs.
The actual build steps for Gawk don't need the Auto-stuff! It's only involved in some rules that get invoked if parts of the Auto-stuff have changed and need to be re-processed. However, the Makefile is structured in such a way that it fails if the tools aren't present.
Before the above patch:
$ ./configure
[...]
$ make gawk
CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/bash /home/kaz/gawk/missing aclocal-1.15 -I m4
/home/kaz/gawk/missing: line 81: aclocal-1.15: command not found
WARNING: 'aclocal-1.15' is missing on your system.
You should only need it if you modified 'acinclude.m4' or
'configure.ac' or m4 files included by 'configure.ac'.
The 'aclocal' program is part of the GNU Automake package:
<http://www.gnu.org/software/automake>
It also requires GNU Autoconf, GNU m4 and Perl in order to run:
<http://www.gnu.org/software/autoconf>
<http://www.gnu.org/software/m4/>
<http://www.perl.org/>
make: *** [aclocal.m4] Error 127
After the patch:
$ ./configure
[...]
$ make gawk
CDPATH="${ZSH_VERSION+.}:" && cd . && true -I m4
CDPATH="${ZSH_VERSION+.}:" && cd . && true
gcc -std=gnu99 -DDEFPATH='".:/usr/local/share/awk"' -DDEFLIBPATH="\"/usr/local/lib/gawk\"" -DSHLIBEXT="\"so"\" -DHAVE_CONFIG_H -DGAWK -DLOCALEDIR='"/usr/local/share/locale"' -I. -g -O2 -DNDEBUG -MT array.o -MD -MP -MF .deps/array.Tpo -c -o array.o array.c
[...]
gcc -std=gnu99 -g -O2 -DNDEBUG -Wl,-export-dynamic -o gawk array.o awkgram.o builtin.o cint_array.o command.o debug.o dfa.o eval.o ext.o field.o floatcomp.o gawkapi.o gawkmisc.o getopt.o getopt1.o int_array.o io.o main.o mpfr.o msg.o node.o profile.o random.o re.o regex.o replace.o str_array.o symbol.o version.o -ldl -lm
$ ./gawk --version
GNU Awk 4.1.60, API: 1.2
Copyright (C) 1989, 1991-2015 Free Software Foundation.
[...]
There we go. As you can see, the CDPATH= command lines there are where the Auto-stuff was being invoked, where you see the true commands. These report successful termination, and so it just falls through that junk to do the darned build, which is perfectly configured.
I did make gawk because there are some subdirectories that get built which fail; the trick has to be repeated for their respective Makefiles.
If you're running into this kind of thing with a pristine, official tarball of the program from its developers, then complain. It should just unpack, ./configure and make without you having to patch anything or install any Automake or Autoconf materials.
Ideally, a pull of their Git head should also behave that way.
I think the touch command is the right answer e.g. do something like
touch --date="`date`" aclocal.m4 Makefile.am configure Makefile.in
before [./configure && make].
Sidebar I: Otherwise, I agree with #kaz: adding dependencies for aclocal.m4 and/or configure and/or Makefile.am and/or Makefile.in makes assumptions about the target system that may be invalid. Specifically, those assumptions are
1) that all target systems have autotools,
2) that all target systems have the same version of autotools (e.g. automake.1.15 in this case).
3) that if either (1) or (2) are not true for any user, that the user is extracting the package from a maintainer-produced TAR or ZIP format that maintains timestamps of the relevant files, in which case all autotool/configure/Makefile.am/Makefile.in dependencies in the configure-generated Makefile will be satisfied before the make command is issued.
The second assumption fails on many Mac systems because automake.1.14 is the "latest" for OSX (at least that is what I see in MacPorts, and apparently the same is true for brew).
The third assumption fails spectacularly in a world with Github. This failure is an example of an "everyone thinks they are normative" mindset; specifically, the maintainers, who are the only class of users that should need to edit Makefile.am, have now put everyone into that class.
Perhaps there is an option in autowhatever that keeps these dependencies from being added to Makefile.in and/or Makefile.
Sidebar II [Why #kaz is right]: of course it is obvious, to me and other cognoscenti, to simply try a sequence of [touch] commands to fool the configure-created Makefile from re-running configure and the autotools. But that is not the point of configure; the point of configure is to ensure as many users on as many different systems as as possible can simply do [./configure && make] and move on; most users are not interested in "shaving the yak" e.g. debugging faulty assumptions of the autotools developers.
Sidebar III: it could be argued that ./configure, now that autotools adds these dependencies, is the wrong build tool to use with Github-distributed packages.
Sidebar IV: perhaps configure-based Github repos should put the necessary touch command into their readme, e.g. https://github.com/drbitboy/Tycho2_SQLite_RTree.
2018, yet another solution ...
https://github.com/apereo/mod_auth_cas/issues/97
in some cases simply running
$ autoreconf -f -i
and nothing else .... solves the problem.
You do that in the directory /pcre2-10.30 .
What a nightmare.
(This usually did not solve the problem in 2017, but now usually does seem to solve the problem - they fixed something. Also, it seems your Dockerfile should now usually start with "FROM ibmcom/swift-ubuntu" ; previously you had to give a certain version/dev-build to make it work.)
The problem is not automake package, is the repository
sudo apt-get install automake
Installs version aclocal-1.4, that's why you can't find 1.5 (In Ubuntu 14,15)
Use this script to install latest
https://github.com/gp187/nginx-builder/blob/master/fix/aclocal.sh
2017 - High Sierra
It is really hard to get autoconf 1.15 working on Mac. We hired an expert to get it working. Everything worked beautifully.
Later I happened to upgrade a Mac to High Sierra.
The Docker pipeline stopped working!
Even though autoconf 1.15 is working fine on the Mac.
How to fix,
Short answer, I simply trashed the local repo, and checked out the repo again.
This suggestion is noted in the mix on this QA page and elsewhere.
It then worked fine!
It likely has something to do with the aclocal.m4 and similar files. (But who knows really). I endlessly massaged those files ... but nothing.
For some unknown reason if you just scratch your repo and get the repo again: everything works!
I tried for hours every combo of touching/deleting etc etc the files in question, but no. Just check out the repo from scratch!
I am attempting to build Google Breakpad for Mac OS X as a part of porting an application, based on the trunk revision 782.
The Breakpad wiki specifies that one should build client/mac/Breakpad.xcodeproj, which produces a Breakpad.framework including a dynamically linked lib if I understand correctly. There is also sample code on how to use this from an Objective-C application, but all this seem very different from what seems to be the normal way of doing things on other platforms, including the use of plists and other things that are not part of my application. I would much rather do things as similar as possible across platforms.
For instance, this appears to be the way that Firefox uses Breakpad:
// include exception_handler.h from client/<platform>/handler,
// using ... here for brevity
#include "... exception_handler.h"
...
gExceptionHandler = new google_breakpad::
ExceptionHandler(tempPath.get(),
nsnull,
MinidumpCallback,
nsnull,
#if defined(XP_WIN32)
google_breakpad::ExceptionHandler::HANDLER_ALL);
#else
true);
#endif
In my project, we are doing the same thing and just link against exception_handler.lib on Windows. It seems that on Linux, Breakpad generates a corresponding libbreakpad_client.a that can be linked against in the same way, but not on Mac OS X. If I do
./configure
make
from the breakpad root directory a libbreakpad.a is generated that does not contain the exception handler, and the libbreakpad_client.a that should is not being built. I may very well have misunderstood just about anything on both the normal way of using Breakpad as well as the normal procedure for building external libraries on the Mac, so any help is appreciated.
How do I build libbreakpad_client.a on Mac OS X?
This might not be helpful in your case, but I've found this to work for a project I'm working on that needs to work on both Linux and OSX. On Linux we use the "normal" autotools way of doing things; on OSX we invoke xcodebuild to create Breakpad.framework and then link against that.
Here's the relevant portion of the CMakeLists.txt file (for CMake see here):
IF(LINUX)
add_custom_target(
GOOGLE_BREAKPAD_MAKEFILE
COMMAND cd ../google-breakpad/ && ([ -e Makefile ] || (autoreconf && bash configure))
)
add_custom_target(
GOOGLE_BREAKPAD
COMMAND cd ../google-breakpad/ && $(MAKE)
)
add_dependencies(GOOGLE_BREAKPAD GOOGLE_BREAKPAD_MAKEFILE)
ENDIF()
IF(APPLE)
add_custom_target(
GOOGLE_BREAKPAD
COMMAND cd ../google-breakpad/src/client/mac/ && xcodebuild -sdk macosx
)
ENDIF()
If your application is built with clang -stdlib=libc++ (which is pretty normal if you make heavy use of C++11), you should append the phrase GCC_VERSION=com.apple.compilers.llvm.clang.1_0 OTHER_CFLAGS=-stdlib=libc++ OTHER_LDFLAGS=-stdlib=libc++ to the end of the xcodebuild line. This will force xcodebuild to do the right thing.
If your application is built with GCC and GNU libstdc++, you don't need to add anything to the xcodebuild line.
There is no solution in the Breakpad source for this, unfortunately. The XCode projects simply build the Breakpad framework, as that's the more-supported client API. You can build the code with your own set of Makefiles or whatever build setup you desire the same way Firefox does by looking at the set of Mozilla makefiles:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/common/Makefile.in
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/common/mac/Makefile.in
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/client/Makefile.in
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/client/mac/handler/Makefile.in
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/client/mac/crash_generation/Makefile.in
and gathering the set of files referenced in CSRCS/CPPSRCS/CMSRCS/CMMSRCS, and building all of those.
You might also file a bug in the Breakpad issue tracker to ask that the XCode project build this static library as well. It would not be a difficult patch.
How do I build libbreakpad_client.a on Mac OS X?
You could only build dynamic framework for macOS Intel64 & Apple Silicon
Checkout https://github.com/Sunbreak/cli-breakpad.trial
Fetch code
mkdir $BREAKPAD && cd $BREAKPAD
fetch breakpad
Setting up and build
cd $BREAKPAD/src
./configure && make
cd $BREAKPAD/src/client/mac && xcodebuild -target Breakpad
cd $BREAKPAD/src/tool/mac/dump_syms && xcodebuild -target dump_syms
Install library && tools
mkdir -p ./breakpad/mac/$(arch)
cp -r $BREAKPAD/src/src/client/mac/build/Release/Breakpad.framework ./breakpad/mac/
cp $BREAKPAD/src/src/tools/mac/dump_syms/build/Release/dump_syms ./breakpad/mac/
cp $BREAKPAD/src/src/processor/minidump_stackwalk ./breakpad/mac/$(arch)
mkdir Frameworks && cd Frameworks && ln -s ../breakpad/mac/Breakpad.framework .
I would like to do the following: If CCache is present in PATH, use "ccache g++" for compilation, else use g++. I tried writing a small my-cmake script containing
CC="ccache gcc" CXX="ccache g++" cmake $*
but it does not seem to work (running make still does not use ccache; I checked this using CMAKE_VERBOSE_MAKEFILE on).
Update:
As per this link I tried changing my script to
cmake -D CMAKE_CXX_COMPILER="ccache" -D CMAKE_CXX_COMPILER_ARG1="g++" -D CMAKE_C_COMPILER="ccache" -D CMAKE_C_COMPILER_ARG1="gcc" $*
but cmake bails out complaining that a test failed on using the compiler ccache (which can be expected).
As of CMAKE 3.4 you can do:
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
It is now possible to specify ccache as a launcher for compile commands and link commands (since cmake 2.8.0). That works for Makefile and Ninja generator. To do this, just set the following properties :
find_program(CCACHE_FOUND ccache)
if(CCACHE_FOUND)
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)
set_property(GLOBAL PROPERTY RULE_LAUNCH_LINK ccache) # Less useful to do it for linking, see edit2
endif(CCACHE_FOUND)
It is also possible to set these properties only for specific directories or targets.
For Ninja, this is possible since version 3.4.
For XCode, Craig Scott gives a workaround in his answer.
Edit : Thanks to uprego and Lekensteyn's comment, I edited the answer to check if ccache is available before using it as launcher and for which generators is it possible to use a compile launcher.
Edit2: #Emilio Cobos recommended to avoid doing that for the linking part as ccache doesn't improve linking speed and can mess with other types of cache like sccache
I personally have /usr/lib/ccache in my $PATH. This directory contains loads of symlinks for every possible name the compiler could be called from (like gcc and gcc-4.3), all pointing to ccache.
And I didn't even create the symlinks. That directory comes pre-filled when I install ccache on Debian.
From CMake 3.1, it is possible to use ccache with the Xcode generator and Ninja is supported from CMake 3.4 onwards. Ninja will honour RULE_LAUNCH_COMPILE just like the Unix Makefiles generator (so #Babcool's answer gets you there for Ninja too), but getting ccache working for the Xcode generator takes a little more work. The following article explains the method in detail, focussing on a general implementation which works for all three CMake generators and making no assumptions about setting up ccache symlinks or the underlying compiler used (it still lets CMake decide the compiler):
https://crascit.com/2016/04/09/using-ccache-with-cmake/
The general gist of the article is as follows. The start of your CMakeLists.txt file should be set up something like this:
cmake_minimum_required(VERSION 2.8)
find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
# Support Unix Makefiles and Ninja
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${CCACHE_PROGRAM}")
endif()
project(SomeProject)
get_property(RULE_LAUNCH_COMPILE GLOBAL PROPERTY RULE_LAUNCH_COMPILE)
if(RULE_LAUNCH_COMPILE AND CMAKE_GENERATOR STREQUAL "Xcode")
# Set up wrapper scripts
configure_file(launch-c.in launch-c)
configure_file(launch-cxx.in launch-cxx)
execute_process(COMMAND chmod a+rx
"${CMAKE_BINARY_DIR}/launch-c"
"${CMAKE_BINARY_DIR}/launch-cxx")
# Set Xcode project attributes to route compilation through our scripts
set(CMAKE_XCODE_ATTRIBUTE_CC "${CMAKE_BINARY_DIR}/launch-c")
set(CMAKE_XCODE_ATTRIBUTE_CXX "${CMAKE_BINARY_DIR}/launch-cxx")
set(CMAKE_XCODE_ATTRIBUTE_LD "${CMAKE_BINARY_DIR}/launch-c")
set(CMAKE_XCODE_ATTRIBUTE_LDPLUSPLUS "${CMAKE_BINARY_DIR}/launch-cxx")
endif()
The two script template files launch-c.in and launch-cxx.in look like this (they should be in the same directory as the CMakeLists.txt file):
launch-c.in:
#!/bin/sh
export CCACHE_CPP2=true
exec "${RULE_LAUNCH_COMPILE}" "${CMAKE_C_COMPILER}" "$#"
launch-cxx.in:
#!/bin/sh
export CCACHE_CPP2=true
exec "${RULE_LAUNCH_COMPILE}" "${CMAKE_CXX_COMPILER}" "$#"
The above uses RULE_LAUNCH_COMPILE alone for Unix Makefiles and Ninja, but for the Xcode generator it relies on help from CMake's CMAKE_XCODE_ATTRIBUTE_... variables support. The setting of the CC and CXX user-defined Xcode attributes to control the compiler command and LD and LDPLUSPLUS for the linker command is not, as far as I can tell, a documented feature of Xcode projects, but it does seem to work. If anyone can confirm it is officially supported by Apple, I'll update the linked article and this answer accordingly.
I didn't like to set a symlink from g++ to ccache. And CXX="ccache g++" didn't work for me as some cmake test case wanted to have just the compiler program without attributes.
So I used a small bash script instead:
#!/bin/bash
ccache g++ "$#"
and saved it as an executable in /usr/bin/ccache-g++.
Then C configured cmake to use /usr/bin/ccache-g++ as C++ compiler.
This way it passes the cmake test cases and I feel more comfortable than having symlinks that I might forget about in 2 or 3 weeks and then maybe wonder if something doesn't work...
I verified the following works (source: this link):
CC="gcc" CXX="g++" cmake -D CMAKE_CXX_COMPILER="ccache" -D CMAKE_CXX_COMPILER_ARG1="g++" -D CMAKE_C_COMPILER="ccache" -D CMAKE_C_COMPILER_ARG1="gcc" $*
Update: I later realized that even this does not work. Strangely it works every alternate time (the other times cmake complains).
Let me add one important item that was not mentioned here before.
While bootstrapping a minimalistic build system from the ubuntu:18.04 docker image, I've found that order of installation makes a difference.
In my case ccache worked fine when calling gcc, but failed to catch invocations of the same compiler by the other names: cc and c++.
To fully install ccache, you need to make sure all compilers are installed first, or add a call to update-ccache symlinks to be safe.
sudo /usr/sbin/update-ccache-symlinks
export PATH="/usr/lib/ccache/:$PATH"```
... and then (due to updated symlinks) also calls to cc and c++ get caught!
In my opinion the best way is to symlink gcc,g++ to ccache, but if you would like to use within cmake, try this:
export CC="ccache gcc" CXX="ccache g++" cmake ...
Here are 2 methods I think are clean/robust, and also don't pollute your CMake code.
1.) Set environment variables
This method is nice since you don't have to individually set it up for each CMake project. The con is you may not want ccache for each CMake project.
# Requires CMake 3.17 (https://cmake.org/cmake/help/latest/envvar/CMAKE_LANG_COMPILER_LAUNCHER.html)
export CMAKE_CXX_COMPILER_LAUNCHER=/usr/bin/ccache
export CMAKE_C_COMPILER_LAUNCHER=/usr/bin/ccache
2.) Pass in cache variables during project configuration
Con a bit annoying to do for each project. This can be negated by your IDE though.
# Requires CMake 3.4
$ cmake ... -D CMAKE_CXX_COMPILER_LAUNCHER=/usr/bin/ccache \
-D CMAKE_C_COMPILER_LAUNCHER=/usr/bin/ccache
NOTE: It isn't really necessary to specify the full path.
If ccache is in your path you can just specify ccache instead.
export CMAKE_CXX_COMPILER_LAUNCHER=ccache
export CMAKE_C_COMPILER_LAUNCHER=ccache
It is extending #Nicolas answer.
Add following line to your cmake file:
list(PREPEND CMAKE_PROGRAM_PATH /usr/lib/ccache)
Or add it as argument to cmake configuration step:
cmake -DCMAKE_PROGRAM_PATH=/usr/lib/ccache