Can I build/get static library (.a files instead of .so files) of tensorflow C API? I need to include/link tensorflow c api in my C++ software xyz and then I build shared library xyz.so that will be sent to other party. The other party should be able to run xyz.so without installing/building tensorflow.
Download tensorflow c api and link .so files using CMakeLists.txt file.
I have gotten some things to work by compiling Tensorflow with the following bazel command:
bazel build --config=monolithic -j 1 //tensorflow:libtensorflow.so
And then linking together the object files with libtool for MacOS:
libtool -static -o libtensorflow_arm64_libtool_SO3.a $(cat bazel-bin/tensorflow/libtensorflow_cc.so.*.params | sed -e 's!-Wl,-force_load,!!' | grep -e '\.a$' -e '\.lo$' | sort -t: -u -k1,1)
Or ar for linux:
ar -cr libtensorflow.a $(cat bazel-bin/tensorflow/libtensorflow_cc.so.*.params | grep '\.o$')
I have not figured out all the details, but this does make at least version control and tensor allocation work.
EDIT:
In the end I opted for going with TFLite as it supports almost all TF operations and have Cmake support for building static libraries. For the full solution see this thread:
Has anyone experience in building a static library for the Tensorflow C++ API?
Related
I tried the approach in this question, but it seems the linux version of ar is not the same as the mac version since I failed to combine the object files again.
What I basically want to do is is merge another static library into my Xcode static library build product via a run-script build phase.
Unfortunately I can't compile the other library directly into my project because it has it's own build system (therefore I use the compiled libs).
I think it should be possible to merge the other library via ar into the Xcode generated library without decompiling the build product. How do I accomplish this?
you can use libtool to do it
libtool -static -o new.a old1.a old2.a
If you're dealing with multi-architecture static libraries, a bit of extra manipulation is required to thin each library, combine the thinned versions, and then fatten the result. Here's a handy script which you can edit to your satisfaction which does all that in one. The example takes three iOS libraries path/to/source/libs/libone.a, path/to/source/libs/libtwo.a, and path/to/source/libs/libthree.a and merges them into a single library libcombined.a.
#! /bin/bash
INPATH="path/to/source/libs"
LIBPREFIX="lib"
LIBS="one two three"
LIBEXT=".a"
OUT="combined"
ARCHS="armv7 armv7s arm64"
for arch in $ARCHS
do
for lib in $LIBS
do
lipo -extract $arch $INPATH/$LIBPREFIX$lib$LIBEXT -o $LIBPREFIX$lib-$arch$LIBEXT
done
INLIBS=`eval echo $LIBPREFIX\{${LIBS// /,}\}-$arch$LIBEXT`
libtool -static -o $LIBPREFIX$OUT-$arch$LIBEXT $INLIBS
rm $INLIBS
done
OUTLIBS=`eval echo $LIBPREFIX$OUT-\{${ARCHS// /,}\}$LIBEXT`
lipo -create $OUTLIBS -o $LIBPREFIX$OUT$LIBEXT
rm $OUTLIBS
You should just be able to link one to the other. So... just use ld to merge the images.
You should use ar -r to create an archive on MacOS:
ar -x libabc.a
ar -x libxyz.a
ar -r libaz.a *.o
I downloaded and built gcc 4.8.1 on my desktop, running 64-bit Ubuntu 12.04. I built it out of source, like the docs recommend, and with the commands
../../gcc-4.8.1/configure --prefix=$HOME --program-suffix=-4.8
make
make -k check
make install
It seemed to pass all the tests, and I installed everything into my home directory w/ the suffix -4.8 to distinguish from the system gcc, which is version 4.6.3.
Unfortunately when I compile c++ programs using g++-4.8 it links to the system libc and libstdc++ rather than the newer ones compiled from gcc-4.8.1. I downloaded and built gcc 4.8 because I wanted to play around with the new C++11 features in the standard library, so this behaviour is definitely not what I wanted. What can I do to get gcc-4.8 to automatically link to the standard libraries that came with it rather than the system standard libraries?
When you link with your own gcc you need to add an extra run-time linker search path(s) with -Wl,-rpath,$(PREFIX)/lib64 so that at run-time it finds the shared libraries corresponding to your gcc.
I normally create a wrapper named gcc and g++ in the same directory as gcc-4.8 and g++-4.8 which I invoke instead of gcc-4.8 and g++-4.8, as prescribed in Dynamic linker is unable to find GCC libraries:
#!/bin/bash
exec ${0}SUFFIX -Wl,-rpath,PREFIX/lib64 "$#"
When installing SUFFIX and PREFIX should be replaced with what was passed to configure:
cd ${PREFIX}/bin && rm -f gcc g++ c++ gfortran
sed -e 's#PREFIX#${PREFIX}#g' -e 's#SUFFIX#${SUFFIX}#g' gcc-wrapper.sh > ${PREFIX}/bin/gcc
chmod +x ${PREFIX}/bin/gcc
cd ${PREFIX}/bin && ln gcc g++ && ln gcc c++ && ln gcc gfortran
(gcc-wrapper.sh is that bash snippet).
The above solution does not work with some versions of libtool because g++ -Wl,... -v assumes linking mode and fails with an error.
A better solution is to use specs file. Once gcc/g++ is built, invoke the following command to make gcc/g++ add -rpath to the linker command line (replace ${PREFIX}/lib64 as necessary):
g++ -dumpspecs | awk '/^\*link:/ { print; getline; print "-rpath=${PREFIX}/lib64", $0; next } { print }' > $(dirname $(g++ -print-libgcc-file-name))/specs
I just had the same problem when building gcc-4.8.2. I don't have root access on that machine and therefore need to install to my home directory. It took several attempts before I figured out the magic required to get this to work so I will reproduce it here so other people will have an easier time. These are the commands that I used to configure gcc:
prefix=/user/grc/packages
export LDFLAGS=-Wl,-rpath,$prefix/lib
export LD_RUN_PATH=$prefix/lib
export LD_LIBRARY_PATH=$prefix/lib
../../src/gmp-4.3.2/configure --prefix=$prefix
../../src/mpfr-2.4.2/configure --prefix=$prefix
../../src/mpc-0.8.1/configure --prefix=$prefix --with-mpfr=$prefix --with-gmp=$prefix
../../src/gcc-4.8.2/configure --prefix=$prefix --with-mpfr=$prefix --with-gmp=$prefix --with-mpc=$prefix --enable-languages=c,c++
That got me a working binary but any program I built with that version of g++ wouldn't run correctly unless I built it with the -Wl,-rpath,$prefix/lib64 option. It is possible to get g++ to automatically add that option by providing a specs file. If you run
strace g++ 2>&1 | grep specs
you can see which directories it checks for a specs file. In my case it was $prefix/lib/gcc/x86_64-unknown-linux-gnu/4.8.2/specs so I ran g++ -dumpspecs to create a new specs file:
cd $prefix/lib/gcc/x86_64-unknown-linux-gnu/4.8.2
$prefix/bin/g++ -dumpspecs > xx
mv xx specs
and then edited that file to provide the -rpath option. Search for the lines like this:
*link_libgcc:
%D
and edit to add the rpath option:
*link_libgcc:
%D -rpath /user/grc/packages/lib/%M
The %M expands to either ../lib or ../lib64 depending on whether you are building a 32-bit or a 64-bit executable.
Note that when I tried this same trick on an older gcc-4.7 build it didn't work because it didn't expand the %M. For older versions you can remove the %M and just hardcode lib or lib64 but that is only a viable solution if you only ever build 32-bit executables (with lib) or only ever build 64-bit executables (with lib64).
gcc -print-search-dirs will tell you where your compiler is looking for runtime libraries, etc. You can override this with the -B<prefix> option.
I am trying to build gcc 4.7.2 using a custom prefix $PREFIX
I have built and installed all the prerequisites into my prefix location, and then successfully configured, built and installed gcc.
The problem that I now have is that $PREFIX is not in the library search path, and therefore the shared libraries cannot be found.
$PREFIX/bin $ ./g++ ~/main.cpp
$PREFIX/libexec/gcc/x86_64-suse-linux/4.7.2/cc1plus: \
error while loading shared libraries: \
libcloog-isl.so.1: \
cannot open shared object file: No such file or directory
What works, but isn't ideal
If I export LD_LIBRARY_PATH=$PREFIX/lib then it works, but I'm looking for something which works without having to set environment variables.
If I use patchelf to set the RPATH on all the gcc binaries then it also works; however this involves searching out all elf binaries and iterating over them calling patchelf, I would rather have something more permanent.
What I think would be ideal for my purposes
So I'm hoping there is a way to have -Wl,-rpath,$PREFIX/lib passed to make during the build process.
Since I know the paths won't need to be changed this seems like the most robust solution, and can be also be used for when we build the next gcc version.
Is configuring the build process to hard code the RPATH possible?
What I have tried, but doesn't work
Setting LDFLAGS_FOR_TARGET prior to calling configure:
All of these fail:
export LDFLAGS_FOR_TARGET="-L$PREFIX/lib -R$PREFIX/lib"
export LDFLAGS_FOR_TARGET="-L$PREFIX/lib"
export LDFLAGS_FOR_TARGET="-L$PREFIX/lib -Wl,-rpath,$PREFIX/lib"
Setting LDFLAGS prior to calling configure:
export LDFLAGS="-L$PREFIX/lib -Wl,-rpath,$PREFIX/lib"
In any event I worry that these will override any of the LDFLAGS gcc would have had, so I'm not sure these are a viable option even if they could be made to work?
My configure line
For completeness here is the line I pass to configure:
./configure \
--prefix=$PREFIX \
--build=x86_64-suse-linux \
--with-pkgversion='SIG build 12/10/2012' \
--disable-multilib \
--enable-cloog-backend=isl \
--with-mpc=$PREFIX \
--with-mpfr=$PREFIX \
--with-gmp=$PREFIX \
--with-cloog=$PREFIX \
--with-ppl=$PREFIX \
--with-gxx-include-dir=$PREFIX/include/c++/4.7.2
I've found that copying the source directories for gmp, mpfr, mpc, isl, cloog, etc. into the top level gcc source directory (or using symbolic links with the same name) works everywhere. This is in fact the preferred way.
You need to copy (or link) to those source directory names without the version numbers for this to work.
The compilers do not need LD_LIBRARY_PATH (although running applications built with the compilers will need an LD_LIBRARY_PATH to the $PREFIX/lib64 or something like that - but that's different)
Start in a source directory where you'll keep all your sources.
In this source directory you have your gcc directory either by unpacking a tarball or svn...
I use subversion.
Also in this top level directory you have, say, the following source tarballs:
gmp-5.1.0.tar.bz2
mpfr-3.1.1.tar.bz2
mpc-1.0.1.tar.gz
isl-0.11.1.tar.bz2
cloog-0.18.0.tar.gz
I just download these and update to the latest tarballs periodically.
In script form:
# Either:
svn checkout svn://gcc.gnu.org/svn/gcc/trunk gcc_work
# Or:
bunzip -c gcc-4.8.0.tar.bz2 | tar -xvf -
mv gcc-4.8.0 gcc_work
# Uncompress sources.. (This will produce version numbered directories).
bunzip -c gmp-5.1.0.tar.bz2 | tar -xvf -
bunzip -c mpfr-3.1.1.tar.bz2 | tar -xvf -
gunzip -c mpc-1.0.1.tar.gz | tar -xvf -
bunzip -c isl-0.11.1.tar.bz2 | tar -xvf -
gunzip -c cloog-0.18.0.tar.gz | tar -xvf -
# Link outside source directories into the top level gcc directory.
cd gcc_work
ln -s ../gmp-5.1.0 gmp
ln -s ../mpfr-3.1.1 mpfr
ln -s ../mpc-1.0.1 mpc
ln -s ../isl-0.11.1 isl
ln -s ../cloog-0.18.0 cloog
# Get out of the gcc working directory and create a build directory. I call mine obj_work.
# I configure the gcc binary and other outputs to be bin_work in the top level directory. Your choice. But I have this:
# home/ed/projects
# home/ed/projects/gcc_work
# home/ed/projects/obj_work
# home/ed/projects/bin_work
# home/ed/projects/gmp-5.1.0
# home/ed/projects/mpfr-3.1.1
# home/ed/projects/mpc-1.0.1
# home/ed/projects/isl-0.11.1
# home/ed/projects/cloog-0.18.0
mkdir obj_work
cd obj_work
../gcc_work/configure --prefix=../bin_work <other options>
# Your <other options> shouldn't need to involve anything about gmp, mpfr, mpc, isl, cloog.
# The gcc build system will find the directories you linked,
# then configure and compile the needed libraries with the necessary flags and such.
# Good luck.
I've been using this configure option with gcc-4.8.0, on FreeBSD, after building and installing gmp, isl and cloog:
LD_LIBRARY_PATH=/path/to/isl/lib ./configure (lots of other options) \
--with-stage1-ldflags="-rpath /path/to/isl/lib -rpath /path/to/cloog/lib -rpath /path/to/gmp/lib"
and the resulting gcc binary does not need any LD_LIBRARY_PATH. The LD_LIBRARY_PATH for configure is needed because it compiles a test program to check for the ISL version, which would fail if it didn't find the ISL shared lib.
I tried it on Linux (Ubuntu) where it failed during configuring because the -rpath args were passed to gcc instead of ld. I could fix this by using
--with-stage1-ldflags="-Wl,-rpath,/path/to/isl/lib,-rpath,/path/to/cloog/lib,-rpath,/path/to/gmp/lib"
instead.
Just using configure --with-stage1-ldflags="-Wl,-rpath,/path/to/lib" was not enough for me to build gcc 4.9.2, bootstrap failed in stage 2. What works is to pass he flags directly to make via
make BOOT_LDFLAGS="-Wl,-rpath,/path/to/lib"
I got this from https://gcc.gnu.org/ml/gcc/2008-09/msg00214.html
While it still involves setting environment variables, what I do is that I define LD_RUN_PATH, which sets the rpath. That way the rest of the system can keep using the system provided libraries instead of using the ones that your gcc build generates.
I am going to make a suggestion that I believe solves your problem, although it definitely does not answer your question. Let's see how many downvotes I get.
Writing a generic wrapper script to set LD_LIBRARY_PATH and then to run the executable is easy; see https://stackoverflow.com/a/7101577/768469.
The idea is to pass something like --prefix=$PREFIX/install to configure, building an install tree that looks like this:
$PREFIX/
install/
lib/
libcloogXX.so
libgmpYY.so
...
bin/
gcc
emacs
...
bin/
.wrapper
gcc -> .wrapper
emacs -> .wrapper
.wrapper is a simple shell script:
#!/bin/sh
here="${0%/*}" # or use $(dirname "$0")
base="${0##*/}" # or use $(basename "$0")
libdir="$here"/../install/lib
if [ "$LD_LIBRARY_PATH"x = x ] ; then
LD_LIBRARY_PATH="$libdir"
else
LD_LIBRARY_PATH="$libdir":"$LD_LIBRARY_PATH"
fi
export LD_LIBRARY_PATH
exec "$here"/../install/bin/"$base" "$#"
This will forward all arguments correctly, handle spaces in arguments or directory names, and so forth. For practical purposes, it is indistinguishable from setting the rpath like you want.
Also, you can use this approach not only for gcc, but for your entire my-personal-$PREFIX tree. I do this all the time in environments where I want an up-to-date suite of GNU tools, but I do not have (or want to admit to have) root access.
Try to add your $PREFIX to /etc/ld.so.conf and then run ldconfig:
# echo $PREFIX >> /etc/ld.so.conf
# ldconfig
This will recreate cache that is used by runtime linker and it will pick up your libraries.
WARNING: This operation will cause ALL applications to use your newly compiled libraries in $PREFIX instead of default location
I download the archive with libconfig, but I do not wont to install it into the system. I want to compile it as static library and link it from program girectory or directly link it into my program on the program compile time?
Any ideas how to do this with Makefile? (FreeBSD(UNIX), GCC)
or suggest other unix config libraries that support compilation into the program.
On linux system untar then configure and make:
> ls
libconfig-1.4.8.tar.gz
> tar -zxvf libconfig-1.4.8.tar.gz
> cd libconfig-1.4.8
> ./configure prefix=/somewhere/in/your/homedir/doesnotmatter
> make
> ls -al lib/.libs
libconfig.a
libconfig++.a
...
other versions of lib
Then link your application against the static .a library.
gcc -I/dir/with/libconfigheader -o yourapp main.cpp libconfig.a
If you don't do make install it will not get installed, the libraries will be in the output directory, you can copy them where you need them.
I hope this helps.
NOTE I haven't tried to use the library, I just downloaded and compiled it ...
Being perfectly satisfied with old-style Makefiles, I am looking for a simple alternative to libtool. I do not want to switch to automake, and I keep running into problems with libtool when I try to use it directly. The latest one is 'unsupported hardcode properties', and I am getting fed up with the lack of complete documentation that just tells me what is wrong this time...
I only want to compile a bunch of .o files with the right flags and then link them into a shared library, such that it works on as many platforms as possible. Is there anything out there that does just that and not force me to switch all of my other tools at the same time?
I not sure if it would fit info your workflow but I'd recommend looking at CMake. It works on Windows, Linux and Mac and should not force you to change any of your other tools. You'll have to judge its suitability yourself though.
There's jlibtool (which has nothing to do with java).
It's written in C, and can just be bundled with your source.
It was originally an apache project, but whoever was working it there seems to of abandoned it around 2004.
It was taken over by FreeRADIUS project maintainer Alan Dekok, who modernised the code and fixed a few niggling issues. We use it for the FreeRADIUS project (>= 3.0.0) to do all the build time linking.
Given your description in the comment to Milliams' answer,
I just want one tool that I tell: "give me the compiler flags so that I can compile these n files for use in a shared library, and then give me the commands to link them together",
then libtool may well be the simplest tool for the job. I know of no other alternative.
You are right that the documentation for using libtool with plain makefiles is practically nonexistent, but libtool certainly does not require you to switch to automake. Cross-platform libraries are difficult, and the price you have to pay for them is libtool. (Or maybe the discount price is libtool+automake+autoconf or CMake or Jam.)
slibtool (dl.midipix.org/slibtool, git://midipix.org/slibtool) is a libtool drop-in replacement, written in C. A single slibtool binary aims to seamlessly support both native and cross-builds, and the utility also provides some additional features (installation of .la files is optional, optional color-coded annotation, etc). The following minimal plain makefile demonstrates how to (cross-) build a library using slibtool.
CC = cc
LIBTOOL = slibtool
DESTDIR = destdir
all: libfoo.la
a.lo:
$(LIBTOOL) --mode=compile --tag=CC $(CC) -c a.c
libfoo.la: a.lo
$(LIBTOOL) --mode=link --tag=CC $(CC) -o libfoo.la -rpath /lib
install: all
mkdir -p destdir
$(LIBTOOL) --mode=install cp libfoo.la $(DESTDIR)
# the -rpath argument is required for semantic compatibility with libtool.
native build, default (both shared library and static library)
$ make
$ make install
native build, shared library only
$ make LIBTOOL=slibtool-shared
$ make install
native build, static library only
$ make LIBTOOL=slibtool-static
$ make install
cross-build, default
$ make CC=some-target-tuple-gcc
$ make install
cross-build, default, with lots of colors
$ make LIBTOOL=dlibtool CC=some-target-tuple-gcc
$ make install