I'll start off by saying that I'm new to Docker and a novice with cmake so forgive any ignorance. I'm working on porting a series of applications and a single shared library to a docker environment. From my understanding Docker does not play well with shared libraries so I'd need to static link. My goal is to use docker-compose to build multiple different application containers but each of these containers using the same custom library (that was setup as a .so in previous environment). I'm currently setup to do cross-compile w/ Docker and can build and run each app w/o the library. Now I need to add in the library.
Current folder/file setup is as follows:
app1:
inc:
app1.h <--includes mysharedlib.h
src:
app1.cpp
CMakeLists.txt
custom-toolchain.cmake
Dockerfile
app2:
inc:
app2.h <--includes mysharedlib.h
src:
app2.cpp
CMakeLists.txt
custom-toolchain.cmake
Dockerfile
MySharedLib:
inc:
mysharedlib.h
src:
mysharedlib.cpp
custom-toolchain.cmake
CMakeLists.txt
Can someone please help explain how I go about including the library into each container and how to include it correctly with CMake?
current working app implementation is as follows:
dockerfile for app1:
ARG IMAGE_TAG=2-bullseye
ARG TOOLCHAIN_ARCH=aarch64
ARG PKG_ARCH=arm64
ARG IMAGE_ARCH=linux/arm64
FROM torizon/debian-cross-toolchain-arm64:2-bullseye AS build
ARG PKG_ARCH
ARG TOOLCHAIN_ARCH
RUN apt-get -y update && apt-get install -y \
cmake \
python-dev \
gcc \
make \
gcc \
binutils \
build-essential \
wget\
libasound2-dev:arm64 \
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*
WORKDIR /mypjsip
RUN mkdir out_libs
RUN wget https://github.com/pjsip/pjproject/archive/2.10.tar.gz && tar -xvf 2.10.tar.gz
RUN cd pjproject-2.10/ && ./configure --host=aarch64-linux-gnu --disable-libwebrtc && make dep && make && make DESTDIR=/mypjsip/out_libs install
COPY . app1_src
COPY $TOOLCHAIN_ARCH-toolchain.cmake app1_src/$TOOLCHAIN_ARCH-toolchain.cmake
RUN mkdir app1_src/build
RUN cd app1_src/build && cmake .. -DCMAKE_TOOLCHAIN_FILE=./$TOOLCHAIN_ARCH-toolchain.cmake
RUN cd app1_src/build && make install
FROM --platform=linux/arm64 torizon/debian:$IMAGE_TAG
RUN apt-get -y update && apt-get install -y \
libasound2 \
alsa-utils \
&& apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*
COPY --from=build /mypjsip/pjproject-2.10/pjsip-apps/bin/pjsua* /usr/local/bin/pjsua
COPY --from=build /mypjsip/out_libs/* /
COPY --from=build /usr/bin/app1 /usr/bin/
ENTRYPOINT [ "/usr/bin/app1" ]
cmake for app1:
cmake_minimum_required(VERSION 3.13)
project(app1)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED True)
set(INC_DIR ./inc)
include_directories(${INC_DIR})
AUX_SOURCE_DIRECTORY(src SOURCE_FILES)
add_executable(app1 ${SOURCE_FILES})
set(WARNINGS "\
-pedantic \
-pedantic-errors \
-Wextra \
-Wcast-align \
-Wcast-qual \
-Wdisabled-optimization \
-Werror \
-Wfloat-equal \
-Wformat=2 \
-Wformat-nonliteral \
-Wformat-security \
-Wformat-y2k \
-Wimport \
-Winit-self \
-Winline \
-Winvalid-pch \
-Wmissing-field-initializers \
-Wmissing-format-attribute \
-Wmissing-include-dirs \
-Wmissing-noreturn \
-Wpacked \
-Wpointer-arith \
-Wredundant-decls \
-Wshadow \
-Wstack-protector \
-Wstrict-aliasing=2 \
-Wswitch-default \
-Wswitch-enum \
-Wunreachable-code \
-Wvariadic-macros \
-Wwrite-strings")
if ( CMAKE_COMPILER_IS_GNUCXX )
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${WARNINGS} ")
endif(CMAKE_COMPILER_IS_GNUCXX)
set(CMAKE_INSTALL_PREFIX /usr)
install(TARGETS app1 DESTINATION bin)
Related
Using a Dockerfile, I'm attempting to build Clang version v8 along with its dependencies on Ubuntu 18.04. The reason I'm going through all this trouble is because I can't find a 32-bit version of the LLVM libraries. Even the LLVM packages themselves have only 64-bit variants of everything. Because of this, I'm not able to build my 32-bit applications using the LLVM toolchain at version 8. Using version 8 of LLVM is mandatory for reasons I won't go into here.
So far, here is what I have:
FROM ubuntu:18.04 AS build
ARG NUM_PARALLEL=8
RUN true \
&& dpkg --add-architecture i386 \
&& apt-get -qq update \
&& apt-get -qq install \
software-properties-common \
build-essential \
gcc-multilib \
g++-multilib \
git \
wget \
autoconf \
pkg-config \
m4 \
python-dev:i386 \
libcurl4-gnutls-dev:i386 \
libncurses-dev:i386 \
uuid-dev:i386 \
libx11-dev:i386 \
libxext-dev:i386 \
libtinfo-dev:i386 \
libedit-dev:i386 \
swig \
libedit-dev
python-dev
ENV CMAKE_BUILD_PARALLEL_LEVEL=$NUM_PARALLEL
# Ninja
RUN true \
&& git clone --depth 1 --branch v1.8.2 https://github.com/ninja-build/ninja.git \
&& cd ninja \
&& ./configure.py --bootstrap \
&& cp ninja /usr/local/bin
# CMake
RUN true \
&& git clone --depth 1 --branch v3.13.4 https://gitlab.kitware.com/cmake/cmake.git \
&& cd cmake \
&& ./bootstrap --parallel=$NUM_PARALLEL \
&& make -j$NUM_PARALLEL install
# Clang (See: https://clang.llvm.org/get_started.html)
RUN true \
&& git clone --depth 1 --branch llvmorg-8.0.0 https://github.com/llvm/llvm-project.git \
&& cd llvm-project \
&& cmake -G Ninja -B build -S llvm \
-D CMAKE_BUILD_TYPE=Release \
-D LLVM_BUILD_32_BITS:BOOL=ON \
-D LLVM_ENABLE_PROJECTS=all \
-D LLVM_BUILD_TESTS:BOOL=OFF \
-D LLVM_BUILD_EXAMPLES:BOOL=OFF \
-D LLVM_INCLUDE_EXAMPLES:BOOL=OFF \
-D LLVM_INCLUDE_TESTS:BOOL=OFF \
-D LLVM_INCLUDE_BENCHMARKS:BOOL=OFF \
&& cmake --build build -j $NUM_PARALLEL --target install
I get through compiling a little over 5200 translation units, which takes over an hour, only for it to fail:
[5232/6435] Linking CXX shared library lib/readline.so
FAILED: lib/readline.so
: && /usr/bin/c++ -fPIC -fPIC -fvisibility-inlines-hidden -m32 -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -Wno-deprecated-declarations -Wno-unknown-pragmas -Wno-strict-aliasing -Wno-deprecated-register -Wno-vla-extension -Wno-macro-redefined -O3 -DNDEBUG -Wl,-z,defs -Wl,-z,nodelete -m32 -shared -Wl,-soname,readline.so -o lib/readline.so tools/lldb/scripts/Python/modules/readline/CMakeFiles/readline.dir/readline.cpp.o /usr/lib/x86_64-linux-gnu/libpython2.7.so /usr/lib/x86_64-linux-gnu/libedit.so && :
/usr/lib/x86_64-linux-gnu/libpython2.7.so: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
I followed the getting started page to understand how to build LLVM so far, but I must be missing something. They don't exactly go into 32-bit compilation in their examples. Can someone help me get this building? Or at least (and probably better), point me to a version of Clang v8 that has 32-bit LLVM libraries bundled with it?
Seems like you are linking against the wrong Python: usr/lib/x86_64-linux-gnu/libpython2.7.so.
You may want to for it and add it to the CMake commands:
-DPYTHON_EXECUTABLE=PATH_TO_PYTHON-DEV:i686
You may also need to set PYTHON_LIBRARY and PYTHON_INCLUDE_DIR to make sure that the i386 versions are used.
I'm new to Linux, so sorry if this my question is really stupid.
I have shared cross-platform library project, which uses third party static library (Libtorrent).
Windows/Android/macOS - builds and works fine.
But I can't figure out how to build it in Linux (Ubuntu) with GCC.
I'm getting the following linker errors:
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(create_torrent.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(disk_io_thread.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(peer_connection.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /home/user/Desktop/project/libtorrent-build/prebuilt/linux/lib/libtorrent-rasterbar.a(session.o): relocation R_X86_64_TPOFF32 against symbol `_ZN5boost4asio6detail15keyword_tss_ptrINS1_10call_stackINS1_15task_io_serviceENS1_27task_io_service_thread_infoEE7contextEE6value_E' can not be used when making a shared object; recompile with -fPIC
I've tried rebuilding Boost and Libtorrent with -fPIC flag: nothing is changed.
Boost build script:
#!/bin/bash
set -e
BOOST_VERSION=1.63.0
BOOST_VERSION_UNDERSCORES=${BOOST_VERSION//./_}
echo "Boost version $BOOST_VERSION"
echo "Downloading..."
BOOST_ARCH_NAME=boost_${BOOST_VERSION_UNDERSCORES}.tar.gz
curl -O -L https://dl.bintray.com/boostorg/release/$BOOST_VERSION/source/$BOOST_ARCH_NAME
echo "Extracting..."
tar -xvzf $BOOST_ARCH_NAME
BOOST_FOLDER_NAME=boost_${BOOST_VERSION_UNDERSCORES}
cd $BOOST_FOLDER_NAME
chmod +x bootstrap.sh
./bootstrap.sh
chmod +x b2
./b2 headers
./b2 \
--layout=versioned \
--with-thread \
--with-date_time \
--with-filesystem \
--with-chrono \
--with-random \
toolset=gcc \
threading=multi \
link=static \
runtime-link=shared \
variant=release \
threadapi=pthread \
debug-symbols=off \
warnings=off \
warnings-as-errors=off \
architecture=x86 \
address-model=64 \
--stagedir=stage/linux \
cxxflags="-std=gnu++0x -fPIC" \
stage
echo "Copying prebuilt..."
mkdir -p ../prebuilt/linux/lib
cp -rf ./stage/linux/lib ../prebuilt/linux/
cd ../
echo "Cleaning up..."
rm $BOOST_ARCH_NAME
rm -rf $BOOST_FOLDER_NAME
echo "Done."
Libtorrent build script:
#!/bin/sh
set -e
MY_PWD=$PWD
LIBTORRENT_VERSION=1_1_8
LIBTORRENT_NAME=libtorrent-$LIBTORRENT_VERSION
export LIBTORRENT_FOLDER_NAME=libtorrent-$LIBTORRENT_NAME
curl -O -L "https://github.com/arvidn/libtorrent/archive/${LIBTORRENT_NAME}.tar.gz"
echo "Extracting..."
rm -rf ./$LIBTORRENT_FOLDER_NAME
tar xfz "${LIBTORRENT_NAME}.tar.gz"
echo "Building for linux..."
cd $LIBTORRENT_FOLDER_NAME
. ./autotool.sh
mkdir -p lib
if [ ! -d "./include/boost" ]; then
ln -s $PWD/../../boost-build/prebuilt/include/boost $PWD/include/boost
ln -s $PWD/../../boost-build/prebuilt/linux/lib/* $PWD/lib/
fi
if [ ! -d "./include/openssl" ]; then
ln -s $PWD/../../openssl-build/prebuilt/include/openssl $PWD/include/openssl
ln -s $PWD/../../openssl-build/prebuilt/linux/lib/* $PWD/lib/
fi
export CXX=g++
export CC=gcc
export CXXFLAGS="-std=gnu++0x -fPIC -fPIE"
export CPPFLAGS="-I"$PWD"/include"
./configure --enable-debug=no --with-boost-python=no --enable-logging=no \
--enable-tests=no --with-openssl=$PWD \
--with-boost=$PWD \
--enable-shared=no \
--enable-encryption
make -j4
mkdir -p ../prebuilt/linux/lib
cp -rf ./src/.libs/libtorrent-rasterbar.a ../prebuilt/linux/lib/
cd $MY_PWD
echo "cleaning up"
rm -rf $LIBTORRENT_FOLDER_NAME
rm -rf ${LIBTORRENT_NAME}.tar.gz
echo "done"
My shared lib itself is a Qt-based library. I build it using Qt Creator, providing all the required headers and libs in .pro file.
The problem was in Libtorrent build script. I removed -fPIE option and added
export CFLAGS="-fPIC"
Rebuilt and now it links without errors.
I'm building my first OpenCV based application in C++. My goal is to build an intermediate docker image that can compile the application statically so that it can run standalone in the resulting smaller image. I'm open to using any docker image for this step, but just so that you can see exactly what I have, here's the dockerfile to reproduce the entire environment:
FROM ubuntu:18.04 as compiler
ENV OPENCV_VERSION='3.4.2' DEBIAN_FRONTEND=noninteractive
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get -y dist-upgrade && \
apt-get -y autoremove && \
apt-get install -y build-essential cmake
RUN apt-get install -y qt5-default libvtk6-dev
RUN apt-get install -y zlib1g-dev libjpeg-dev libwebp-dev libpng-dev libtiff5-dev libopenexr-dev libgdal-dev
RUN apt-get install -y libdc1394-22-dev libavcodec-dev libavformat-dev libswscale-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev yasm libopencore-amrnb-dev libopencore-amrwb-dev libv4l-dev libxine2-dev
RUN apt-get install -y unzip wget
RUN wget --progress=dot:giga https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip && \
unzip -q ${OPENCV_VERSION}.zip && \
rm ${OPENCV_VERSION}.zip && \
mv opencv-${OPENCV_VERSION} OpenCV && \
cd OpenCV && \
mkdir build && \
cd build && \
cmake \
-D BUILD_SHARED_LIBS=OFF \
-D WITH_QT=ON \
-D WITH_OPENGL=ON \
-D FORCE_VTK=ON \
-D WITH_TBB=ON \
-D WITH_GDAL=ON \
-D WITH_XINE=ON \
-D BUILD_EXAMPLES=OFF \
-D ENABLE_PRECOMPILED_HEADERS=OFF \
-D BUILD_DOCS=OFF \
-D BUILD_PERF_TESTS=OFF \
-D BUILD_TESTS=OFF \
-D BUILD_opencv_apps=OFF \
.. && \
make -j4 && \
make install && \
ldconfig
COPY compile-test.cpp compile-test.cpp
RUN g++ -std=c++11 -static compile-test.cpp -o /app $(pkg-config --cflags --libs opencv)
I can currently compile my c++ apps without issue using the dyanmic libs, but this creates a massive docker image, and I really want to be able to build standalone binaries for distribution, with minimal size.
As you can see I'm compiling OpenCV from source including the flag BUILD_SHARED_LIBS=OFF to make sure I get the .a static libs, rather than the .so dynamic libs. I took a hint from a highly recommended build script, and modified it for use with docker omitting a few python things as I'm using c++.
Because I was having so much trouble with my real application, I've gone ahead and created a much simpler app, which also blows up during compilation. I believe this has something to do with the included cflags and libs. The problem is currently beyond my comprehension. I get mountains of errors that seem to change when I adjust a single include on my compile command. Here's the simplest app I'm trying to compile. It really doesn't do anything, but it does include a lib.
#include "opencv2/imgcodecs.hpp"
using namespace cv;
Mat img;
int main( int argc, char** argv ) {
img = cv::imread( argv[1], IMREAD_COLOR );
}
Then I try to compile this like so:
g++ -std=c++11 -static compile-test.cpp -o /app $(pkg-config --cflags --libs opencv)
And it ends up in a pile of errors much too long to completely paste here.
//usr/local/lib/libopencv_imgcodecs.a(grfmt_jpeg.cpp.o): In function `cv::JpegEncoder::write(cv::Mat const&, std::vector<int, std::allocator<int> > const&)':
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0xf8): undefined reference to `jpeg_CreateCompress'
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0x105): undefined reference to `jpeg_std_error'
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0x2b5): undefined reference to `jpeg_set_defaults'
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0x2d0): undefined reference to `jpeg_set_quality'
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0x2fe): undefined reference to `jpeg_quality_scaling'
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0x30d): undefined reference to `jpeg_quality_scaling'
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0x367): undefined reference to `jpeg_default_qtables'
grfmt_jpeg.cpp:(.text._ZN2cv11JpegEncoder5writeERKNS_3MatERKSt6vectorIiSaIiEE+0x379): undefined reference to `jpeg_start_compress'
grfmt_jpeg.cpp:
...
collect2: error: ld returned 1 exit status
Some Things I've Already Tried
Beginning to google each of the seemingly unique compile errors and adding related flags to the end of my compile code.
Reordering some of the include flags, but there are just too many to do this effectively
Using the opencv-dev package instead of compiling it myself, but it seems you can't do this and expect to use static libs.
After a lot of experimentation I finally got something working! There were a few issues that are all fixed in this Dockerfile. In order to reproduce this, create a Dockerfile with the following contents, and create another file called app.cpp with the simple code from my question above, in the same folder.
I will explain what the issues were below:
FROM alpine:3.8 as compiler
RUN echo -e '#edgunity http://nl.alpinelinux.org/alpine/edge/community \
#edge http://nl.alpinelinux.org/alpine/edge/main \
#testing http://nl.alpinelinux.org/alpine/edge/testing \
#community http://dl-cdn.alpinelinux.org/alpine/edge/community' \
>> /etc/apk/repositories
RUN apk add --update --no-cache \
build-base \
openblas-dev \
unzip \
wget \
cmake \
g++ \
libjpeg \
libjpeg-turbo-dev \
libpng-dev \
jasper-dev \
tiff-dev \
libwebp-dev \
clang-dev \
linux-headers
ENV CC /usr/bin/clang
ENV CXX /usr/bin/g++
ENV OPENCV_VERSION='3.4.2' DEBIAN_FRONTEND=noninteractive
RUN mkdir /opt && cd /opt && \
wget https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip && \
unzip ${OPENCV_VERSION}.zip && \
rm -rf ${OPENCV_VERSION}.zip
RUN mkdir -p /opt/opencv-${OPENCV_VERSION}/build && \
cd /opt/opencv-${OPENCV_VERSION}/build && \
cmake \
-D BUILD_DOCS=OFF \
-D BUILD_EXAMPLES=OFF \
-D BUILD_opencv_apps=OFF \
-D BUILD_opencv_python2=OFF \
-D BUILD_opencv_python3=OFF \
-D BUILD_PERF_TESTS=OFF \
-D BUILD_SHARED_LIBS=OFF \
-D BUILD_TESTS=OFF \
-D CMAKE_BUILD_TYPE=RELEASE \
-D ENABLE_PRECOMPILED_HEADERS=OFF \
-D FORCE_VTK=OFF \
-D WITH_FFMPEG=OFF \
-D WITH_GDAL=OFF \
-D WITH_IPP=OFF \
-D WITH_OPENEXR=OFF \
-D WITH_OPENGL=OFF \
-D WITH_QT=OFF \
-D WITH_TBB=OFF \
-D WITH_XINE=OFF \
-D BUILD_JPEG=ON \
-D BUILD_TIFF=ON \
-D BUILD_PNG=ON \
.. && \
make -j$(nproc) && \
make install && \
rm -rf /opt/opencv-${OPENCV_VERSION}
RUN wget --progress=dot:giga https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-1.0.0-linux-x86-64.tar.gz && \
pwd && \
tar -xzf libwebp-1.0.0-linux-x86-64.tar.gz && \
mv /libwebp-1.0.0-linux-x86-64/lib/libwebp.a /usr/lib && \
rm -rf /libwebp*
RUN wget --progress=dot:giga http://www.ece.uvic.ca/~frodo/jasper/software/jasper-2.0.10.tar.gz && \
tar -xzf jasper-2.0.10.tar.gz && \
cd jasper-2.0.10 && \
mkdir BUILD && \
cd BUILD && \
cmake -DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_SKIP_INSTALL_RPATH=YES \
-DCMAKE_INSTALL_DOCDIR=/usr/share/doc/jasper-2.0.10 \
-DJAS_ENABLE_SHARED=FALSE \
.. && \
make install && \
rm -rf /jasper-2.0.10*
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/lib/pkgconfig
COPY app.cpp app.cpp
RUN g++ -Wl,-Bstatic -static-libgcc -std=c++11 \
app.cpp \
-o /app \
$(pkg-config --cflags --libs -static opencv) \
-lgfortran -lquadmath
FROM alpine
COPY --from=compiler /app /bin/app
Problems
Linker
There were indeed files that needed linking that weren't present, there were two reasons for this:
The pkg-config command is supposed to emit all of the necessary flags for compilation, but in my earlier attempt I hadn't included the -static flag to pkg-config. When you add the -static flag it makes sure to link the extra required packages. I saw a few people run into this problem with the solution of adding the extra flags like -pthread, but I found that the -static flag did this for me and so was preferable.
ld: cannot find -lgcc_s error. This appeared to be fixed by adding the -static-libgcc flag to g++. Some of this is still a mystery to me.
Missing Static Libraries
There were two libraries that I wanted to be included as static which needed to be acquired from sources other than apk. These were libjasper and libwebp. There are build steps above that acquire and build these as necessary and copy the resources into the required place.
More missing links
For reasons I can't yet explain pkg-config didn't provide the last two necessary flags. Those were -lgfortran and -lquadmath.
Notes about this Solution
I switched to alpine linux, just because I had read that some people had success with that, I'm sure the same could be done with Ubuntu. It did result in a much smaller image, so I do like that. This is about 900mb for the intermediate image, which, while huge, is much smaller than the 1.9GB Ubuntu image.
The actual resulting image is about 44mb including all of the statically linked OpenCV libs. This seems like a good solution for those that need a small docker image to run a single C++ bin.
I want to test a complex shared library (cxsc) with the help of cmake. After a lot of trial and error i managed to create a html coverage report but the lines I test, with the boost testing framework, are still red.
This is the script I use to create the coverage report:
rm -fr build_coverage
mkdir build_coverage
cd build_coverage
cmake \
-DCMAKE_BUILD_TYPE=Coverage \
..
make
cd ../tests
make clean
make
cd ../build_coverage
make cxsc_coverage
The cmake part where the coverage report gets created:
# Cleanup lcov
COMMAND ${LCOV_PATH} --zerocounters --directory ${CMAKE_CURRENT_SOURCE_DIR}/build_coverage/CMakeFiles/cxsc.dir/src
COMMAND ${LCOV_PATH} --capture --initial --no-external --directory ${CMAKE_CURRENT_SOURCE_DIR}/build_coverage/CMakeFiles/cxsc.dir/src --output-file ${_outputname}.before
# Run tests
COMMAND LD_LIBRARY_PATH=${CMAKE_CURRENT_SOURCE_DIR}/build_coverage ${_testrunner} ${ARGV3}
# Capturing lcov counters and generating report
COMMAND ${LCOV_PATH} --capture --no-checksum --no-external --directory ${CMAKE_CURRENT_SOURCE_DIR}/build_coverage/CMakeFiles/cxsc.dir/src --output-file ${_outputname}.after
COMMAND ${LCOV_PATH} --add-tracefile ${_outputname}.before --add-tracefile ${_outputname}.after --output-file ${_outputname}.info
COMMAND ${GENHTML_PATH} -o ${_outputname} ${_outputname}.info
Makefile of the test binary (I think this is where the error is):
g++ \
-o test_runner \
main.cpp \
test_interval.cpp \
-I./../src \
-I./../src/rts \
-I./../src/asm \
-I./../src/fi_lib \
-I./../build_coverage \
-L./../build_coverage \
-lcxsc \
-Wall -Winline \
-lboost_unit_test_framework
This is what happens:
cmake -DCMAKE_BUILD_TYPE=Coverage .. build_coverage directory is created with a target.dir directory and *.gcno files in it
cd tests && make Test executable is created and linked against the shared library in the build_coverage directory (Maybe here is my mistake)
make coverage Coverage data is collected and tests are executed
Edit:
To clarify my problem, there are some lines covered but only global consts or one helper function which is used pretty often. But not the functions/methods I call in my tests.
Files in build_coverage/CMakeFiles/cxsc.dir/src/:
After cmake -DC...: A few *.cmake and *.make files
After make in build_coverage dir: *.gcno files
After lcov -c -i ...: Still *.gcno files
After running tests: *.gcda and *.gcno files
After lcov -c ...: A lot of movement in the directory but still the same filenames
I don't see any gcov flags in your makefile. Try this one for test_runner :
g++ \
-Wall -Winline -fprofile-arcs -ftest-coverage \
main.cpp \
test_interval.cpp \
-I./../src \
-I./../src/rts \
-I./../src/asm \
-I./../src/fi_lib \
-I./../build_coverage \
-L./../build_coverage \
-lcxsc \
-lboost_unit_test_framework \
-lgcov \
-o test_runner
Your shared library need those flags too :
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fprofile-arcs -ftest-coverage")
SET(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -lgcov")
See this post with a similar problem.
NB : "--coverage" can substitute "-fprofile-arcs -ftest-coverage" and "-lgcov"
I am trying to build FFMPEG with libx264 for Android.
I can successfully build and use FFMPEG for Android but I realized that I need the ability to encode, therefore I am trying to build FFMPEG with x264.
I am using this tutorial to build FFmpeg for Android http://www.roman10.net/how-to-build-ffmpeg-for-android/
When trying to build FFMPEG I get an error:
"ERROR: libx264 not found"
And in my log it says:
"/usr/local/lib/libx264.a: could not read symbols: Archive has no
index; run ranlib to add one..."
I have the latest versions of both FFMPEG and x264.
I understand that FFMPEG looks for the header and libraries in usr/lib and usr/include, so in order to make it find x264 I use the cflags and ldflags:
--extra-cflags = " -I/usr/local/include "
--extra-ldflags = " -L/usr/local/lib "
I have tried building x264 with many different options that other people on the internet have said that i need. eg. --enable-shared, --enable-static, --disable-pthreads etc.
Some forums say enable this, others say no disable that.
Any help would be much appreciated,
Thanks
EDIT:
If I build FFmpeg with the simplest commands to include libx264 then it works.
ie.
./configure --enable-gpl --enable-libx264 --extra-cflags="-I/usr/local/include" --extra-ldflags="-L/usr/local/lib" --enable-static --enable-shared
However I need it to work for Android. The script I am using is:
NDK=~/Desktop/android-ndk-r7
PLATFORM=$NDK/platforms/android-8/arch-arm/
PREBUILT=$NDK/toolchains/arm-linux-androideabi-4.4.3/prebuilt/darwin-x86
function build_one
{
./configure --target-os=linux \
--prefix=$PREFIX \
--enable-cross-compile \
--enable-shared \
--enable-static \
--extra-libs="-lgcc" \
--arch=arm \
--cc=$PREBUILT/bin/arm-linux-androideabi-gcc \
--cross-prefix=$PREBUILT/bin/arm-linux-androideabi- \
--nm=$PREBUILT/bin/arm-linux-androideabi-nm \
--sysroot=$PLATFORM \
--extra-cflags=" -O3 -fpic -DANDROID -DHAVE_SYS_UIO_H=1 -Dipv6mr_interface=ipv6mr_ifindex -fasm -Wno-psabi -fno-short-enums -fno-strict-aliasing -finline-limit=300 $OPTIMIZE_CFLAGS -I/usr/local/include" \
--extra-ldflags="-Wl,-rpath-link=$PLATFORM/usr/lib -L $PLATFORM/usr/lib -nostdlib -lc -lm -ldl -llog -L/usr/local/lib " \
--enable-gpl \
--enable-libx264 \
--disable-everything \
--enable-demuxer=mov \
--enable-demuxer=h264 \
--disable-ffplay \
--enable-protocol=file \
--enable-avformat \
--enable-avcodec \
--enable-decoder=rawvideo \
--enable-decoder=mjpeg \
--enable-decoder=h263 \
--enable-decoder=mpeg4 \
--enable-decoder=h264 \
--enable-encoder=mjpeg \
--enable-encoder=h263 \
--enable-encoder=mpeg4 \
--enable-encoder=h264 \
--enable-parser=h264 \
--disable-network \
--enable-zlib \
--disable-avfilter \
--disable-avdevice \
$ADDITIONAL_CONFIGURE_FLAG
make clean
make -j4 install
$PREBUILT/bin/arm-linux-androideabi-ar d libavcodec/libavcodec.a inverse.o
$PREBUILT/bin/arm-linux-androideabi-ld -rpath-link=$PLATFORM/usr/lib -L$PLATFORM/usr/lib -soname libffmpeg.so -shared -nostdlib -z,noexecstack -Bsymbolic --whole-archive --no-undefined -o $PREFIX/libffmpeg.so libavcodec/libavcodec.a libavformat/libavformat.a libavutil/libavutil.a libswscale/libswscale.a -lc -lm -lz -ldl -llog --warn-once --dynamic-linker=/system/bin/linker $PREBUILT/lib/gcc/arm-linux-androideabi/4.4.3/libgcc.a
}
CPU=armv7-a
OPTIMIZE_CFLAGS="-mfloat-abi=softfp -mfpu=vfpv3-d16 -marm -march=$CPU "
PREFIX=./android/$CPU
ADDITIONAL_CONFIGURE_FLAG=
build_one
I am guessing that some option in my configure command is conflicting with enabling libx264
NOTE: If I remove --enable-libx264 then it works
I had the same problem. But after downgrading NDK to version 5c it works as described by halfninja. (ubuntu 64bit). there seem to be some changes in the toolchain from 5 to 7.
timo#serverplusplus:/tmp/android-ffmpeg-x264/Project/jni$ ndk-build
Compile thumb : ffmpeg <= ffmpeg.c
Compile thumb : ffmpeg <= cmdutils.c
Executable : ffmpeg
Install : ffmpeg => libs/armeabi/ffmpeg
Compile thumb : videokit <= uk_co_halfninja_videokit_Videokit.c
Compile thumb : videokit <= ffmpeg.c
Compile thumb : videokit <= cmdutils.c
SharedLibrary : libvideokit.so
Install : libvideokit.so => libs/armeabi/libvideokit.so
The ffmpeg source code seems to be updated, and I could compile ffmpeg with x264 for Android NDK as the following.
1 Download the halfninja's android-ffmpeg-x264 git file from https://github.com/halfninja/android-ffmpeg-x264
2 At "halfninja-android-ffmpeg-x264-fe12be0/Project/jni" directory, modify "configure_ffmpeg.sh" to link "libgcc.a" for solving a problem that can not resolve "__aeabi_f2uiz".
./configure $DEBUG_FLAG --enable-cross-compile \
--arch=arm5te \
--enable-armv5te \
--target-os=linux \
--disable-stripping \
--prefix=../output \
--disable-neon \
--enable-version3 \
--disable-shared \
--enable-static \
--enable-gpl \
--enable-memalign-hack \
--cc=arm-linux-androideabi-gcc \
--ld=arm-linux-androideabi-ld \
--extra-cflags="-fPIC -DANDROID -D__thumb__ -mthumb -Wfatal-errors -Wno-deprecated -I../x264 -Ivideokit" \
$featureflags \
--disable-ffmpeg \
--disable-ffplay \
--disable-ffprobe \
--disable-ffserver \
--disable-network \
--enable-filter=buffer \
--enable-filter=buffersink \
--disable-demuxer=v4l \
--disable-demuxer=v4l2 \
--disable-indev=v4l \
--disable-indev=v4l2 \
--extra-ldflags="-L../x264 -L../toolchain/lib/gcc/arm-linux-androideabi/4.4.3" \
--extra-libs="-lgcc"
3 Modify "Android.mk" to link new library "libswresample.a".
FFMPEG_LIBS := $(addprefix ffmpeg/, \
libavdevice/libavdevice.a \
libavformat/libavformat.a \
libavcodec/libavcodec.a \
libavfilter/libavfilter.a \
libswscale/libswscale.a \
libavutil/libavutil.a \
libswresample/libswresample.a \
libpostproc/libpostproc.a )
4 Replace ffmpeg.c and cmdutils.c in videokit directory with ones in ffmpeg directory.
5 Follow a procedure described in README.textile.
These are my working flags:
x264 (a recent stable):
./configure --cross-prefix=arm-linux-androideabi- \
--enable-pic \
--enable-static \
--disable-cli \
--disable-asm \
--host=arm-linux
ffmpeg (release/0.10):
./configure --enable-cross-compile \
--arch=arm5te \
--enable-armv5te \
--target-os=linux \
--disable-stripping \
--prefix=../output \
--disable-neon \
--enable-version3 \
--disable-shared \
--enable-static \
--enable-gpl \
--enable-memalign-hack \
--cc=arm-linux-androideabi-gcc \
--ld=arm-linux-androideabi-gcc \
--extra-cflags="-fPIC -DANDROID -D__thumb__ -mthumb -Wfatal-errors -Wno-deprecated" \
--disable-everything \
--enable-decoder=h264 \
--enable-demuxer=mov \
--enable-muxer=mp4 \
--enable-encoder=libx264 \
--enable-libx264 \
--enable-protocol=file \
--enable-decoder=aac \
--enable-encoder=aac \
--disable-ffmpeg \
--disable-ffplay \
--disable-ffprobe \
--disable-ffserver \
--disable-network \
--enable-filter=buffer \
--enable-filter=buffersink \
--enable-filter=scale \
--disable-demuxer=v4l \
--disable-demuxer=v4l2 \
--disable-indev=v4l \
--disable-indev=v4l2 \
--extra-cflags="-I../x264" \
--extra-ldflags="-L../x264" \
--extra-libs="-lgcc"
Obviously you will have to adjust the paths.
I've put together a Android build system for ffmpeg+x264 here:
https://github.com/guardianproject/android-ffmpeg
We're working on some wrapper Java too for running it, but that's not really usable yet.
I've discovered the --enable-static option does not have any effect on ffmpeg linking behaviour for libx264. I managed to build a copy of ffmpeg with libx264 included statically by editing the config.mak after I'd run ./configure
Build x264 from source in another directory
Add libx264.a to EXTRALIBS line in config.mak
Remove -lx264 from EXTRALIBS line in config.mak
Before
EXTRALIBS=-ldl -lX11 -lx264 etc.
After
EXTRALIBS=/home/adam/x264sourcebuild/libx264.a -ldl -lX11 etc.
I have met the same issue before, and I tried several times to find out the reason:
You must make sure the x264 and the ffmpeg are using same way to compile. like: using Android NDK. Using the same gcc compiler.
So you can not compile ffmpeg with this command:"--cross-prefix=$PREBUILT/darwin-x86_64/bin/arm-linux-androideabi-", but compile the x264 without that.
Here is my x264 compile script:
./configure --prefix=$PREFIX \
--enable-static \
--enable-pic \
--disable-asm \
--disable-cli \
--host=arm-linux \
--cross-prefix=$PREBUILT/darwin-x86_64/bin/arm-linux-androideabi- \
--sysroot=$PLATFORM
make
sudo make install
sudo ldconfig