shrinking a Docker Image - from debian to scratch - how to migrate? - dockerfile

i am trying to build a minimalistic docker image for one of my applicatoins
in my "usual" builds i do not rely on 3rd party applications. This time I need to include a precompiled executeable (xpdf) to the build; My go applications are prebuilt in a builder Docker and then copied over (no dependencies).
my current Dockerimage file looks like this: (working!) application launches
FROM debian:buster-slim
COPY ./bin64/pdftopng .
COPY ./config/xpdfrc .
# ... copy FONT files
# copy Test PDF
COPY ./test.pdf .
RUN chmod +x pdftopng
RUN apt-get update && apt-get install -y libfontconfig1
# execute pdftopng
ENTRYPOINT ["/pdftopng", "-cfg", "xpdfrc", "/test.pdf", "/test.pdf"]
as this results in a image file with over 100mb i tried to use scratch as a base (which usually results in images around 10-20mb)
FROM scratch
# ... copy my binaries to the docker
COPY --from=builder /dist/main /
# copy the XPDF -> pdf to png executeable (linux x64)
COPY ./bin64/pdftopng .
# copy font files (for this example i only copy one)
COPY ./fonts/n021003l.pfb /fonts/n021003l.pfb
# copy config file
COPY ./config/xpdfrc .
# copy test file
COPY ./test.pdf .
# execute the XPDF converter and convert the test.pdf to png
ENTRYPOINT ["/pdftopng", "-cfg", "xpdfrc", "/test.pdf", "test"]
outputs:
standard_init_linux.go:219: exec user process caused: no such file or directory
this error is the result of a missing library (something related to libfontconfig1)
how can i find out which files are missing (as i do have a working container this should be possible)

Solution
Step 1 - create libsource
create a docker image where you can grab all required libraries from
FROM debian:buster-slim AS libsource
COPY ./bin64/pdftopng .
COPY ./config/xpdfrc .
# ... copy FONT files
# copy Test PDF
COPY ./test.pdf .
RUN chmod +x pdftopng
RUN apt-get update && apt-get install -y libfontconfig1
Step 2 - find requirements
then obtain a list of libraries which the application [./pdftopng] needs to run
docker run --rm -it ldd ./pdftopng
this command lists all required libraries (direct use only):
linux-vdso.so.1 (0x00007ffce1304000)
libbz2.so.1 => /lib/x86_64-linux-gnu/libbz2.so.1 (0x00007fa1104ea000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa1102cc000)
libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fa110286000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa110265000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa1100e1000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa10ff5e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa10ff42000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa10fd81000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fa10fcc5000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fa10fc88000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007fa10fc7f000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa110501000)
libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007fa10fc44000)
Step 3 - copy requirements
in the last step copy the libraries from step 2 to the scratch image
COPY --from=libsource /lib/x86_64-linux-gnu/libbz2.so.1 /lib/x86_64-linux-gnu/libbz2.so.1
COPY --from=libsource /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1
COPY --from=libsource /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
COPY --from=libsource /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=libsource /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib/x86_64-linux-gnu/libstdc++.so.6
COPY --from=libsource /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6
COPY --from=libsource /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/libgcc_s.so.1
COPY --from=libsource /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
COPY --from=libsource /usr/lib/x86_64-linux-gnu/libfreetype.so.6 /usr/lib/x86_64-linux-gnu/libfreetype.so.6
COPY --from=libsource /lib/x86_64-linux-gnu/libexpat.so.1 /lib/x86_64-linux-gnu/libexpat.so.1
COPY --from=libsource /lib/x86_64-linux-gnu/libuuid.so.1 /lib/x86_64-linux-gnu/libuuid.so.1
COPY --from=libsource /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=libsource /usr/lib/x86_64-linux-gnu/libpng16.so.16 /usr/lib/x86_64-linux-gnu/libpng16.so.16
Result
in my case the result is a much smaller image 15.45 MB instead of 101.90 MB
which is a reduction by 84,84 %

Related

execute binary using google protobuf error

I'm developing an C++ application under Ubuntu 16 using google protobuf. When I compile and run at the same machine, it works fine, but when I try to run it at another Ubuntu 16 I get the following error
./APP: Symbol `_ZTVN6google8protobuf11MessageLiteE' has different size in shared object, consider re-linking
./APP: Symbol `_ZTVN6google8protobuf7MessageE' has different size in shared object, consider re-linking
If I compile the same source at this second machine it works fine, but if I run at the first machine, I get the error too.
I tried to copy all the libraries listed with ldd to an path at the other machine and I user the export LD_LIBRARY_PATH, and not working too.
ldd result at both machines are the same.
linux-vdso.so.1 => (0x00007ffd40fda000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f011b4b3000)
libzmq.so.5 => /usr/lib/x86_64-linux-gnu/libzmq.so.5 (0x00007f011b24d000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f011b030000)
libprotobuf.so.22 => /usr/local/lib/libprotobuf.so.22 (0x00007f011ab87000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f011a805000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f011a5ef000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f011a225000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f011a021000)
libsodium.so.18 => /usr/lib/x86_64-linux-gnu/libsodium.so.18 (0x00007f0119dc3000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0119bbb000)
/lib64/ld-linux-x86-64.so.2 (0x00007f011b788000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f01198b2000)

Error while loading shared libraries: libpython2.7.so.1.0

Since a few days I get the following error when I try to open python that was installed with conda. The only thing that changed is that I changed the group of the files located in /software and gave the group read and write writes.
/software/local/share/bcbio/anaconda/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
I tried many different things. I first look at the output of ldd
ldd /software/local/share/bcbio/anaconda/bin/python
linux-vdso.so.1 => (0x00007ffd11bfc000)
libpython2.7.so.1.0 => /software/local/share/bcbio/anaconda/bin/../lib/libpython2.7.so.1.0 (0x00007f276fa9b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000036be600000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000033bd800000)
libutil.so.1 => /lib64/libutil.so.1 (0x00000033bec00000)
libm.so.6 => /lib64/libm.so.6 (0x00000033be000000)
libc.so.6 => /lib64/libc.so.6 (0x00000033bd400000)
/lib64/ld-linux-x86-64.so.2 (0x00000033bd000000)
And looked at the permission of the librarie, which seemed to be correct:
ls -lah /software/local/share/bcbio/anaconda/bin/../lib/libpython2.7.so.1.0
-rwxrwsr-x+ 1 vsc lp_neuro 7.7M Jun 7 2018 /software/local/share/bcbio/anaconda/bin/../lib/libpython2.7.so.1.0
I also tried to export different environment variables:
export LD_LIBRARY_PATH=/software/local/share/bcbio/anaconda/bin/../lib/;
export PYTHONHOME=/software/local/share/bcbio/anaconda/bin/../lib/;
export PYTHONPATH=/software/local/share/bcbio/anaconda/bin/../lib/;
Since it is a conda installation of python, solutions needing root access (which I do not have) are of no uses. E.g. editing /etc/ld.so.conf is no option for me.
Is there a way to fix this issue?

Qt Mysql build Driver in ubuntu [duplicate]

In Ubuntu 15.04 64 bits I installed Qt5.6 (online installer) and while trying to move my development environment from Windows 7 to Linux I faced the following:
SqlDatabase: QMYSQL driver not loaded
Following this, I managed to find ~/Qt/5.6/gcc_64/plugins/sqldrivers/libqsqlmysql.so and then:
$ ldd libqsqlmysql.so
linux-vdso.so.1 => (0x00007ffffd571000)
libmysqlclient_r.so.16 => not found
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe94ef24000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fe94ecec000)
libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007fe94ead2000)
libssl.so.10 => not found
libcrypto.so.10 => not found
libQt5Sql.so.5 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libQt5Sql.so.5 (0x00007fe94e88d000)
libQt5Core.so.5 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libQt5Core.so.5 (0x00007fe94e17a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe94df5c000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe94dc4d000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe94d944000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe94d72e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe94d364000)
libicui18n.so.56 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libicui18n.so.56 (0x00007fe94cec9000)
libicuuc.so.56 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libicuuc.so.56 (0x00007fe94cb11000)
libicudata.so.56 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libicudata.so.56 (0x00007fe94b12e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe94af29000)
libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0 (0x00007fe94ad27000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe94ab1f000)
libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fe94a80f000)
/lib64/ld-linux-x86-64.so.2 (0x000056024837f000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fe94a5a2000)
Tells libmysqlclient_r.so.16 => not found. In fact, it seems I have a newer version:
find / -name libmysqlclient_r*
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.so.18
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.so.18.1.0
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.so
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.a
Perhaps that's the problem. Could some one confirm? How should I proceed?
First and foremost, double check that you have the packages containing libssl.so, libcrypto.so and libmysqlclient_r.so installed (looks like you have this last one, it's extremely likely you also have the first two, but just double check).
Then, your problem is that you have those shared objects with a different SONAME, sign that they're binary incompatible with the plugin shipped with Qt, which therefore needs to be recompiled.
Therefore:
install the development versions of the packages found above (libssl-dev, mysql-client-dev or similar).
run the MaintenanceTool from your Qt installation, and be sure to select to install Qt's source code too.
Go in QTDIR/5.6/Src/qtbase/src/plugins/sqldrivers/mysql/.
Run the right qmake, i.e. the one coming from that installation of Qt (not the system wide one or similar). Best way to be sure is providing the full path to it: QTDIR/5.6/gcc_64/bin/qmake.
Run make. Hopefully this will just work™; if it complains about some missing libraries, install them and rerun make.
This should now have produced a new libqsqlmysql.so plugin; overwrite the old one with this new one.
I am using Ubuntu 18.04.4 and compiling my project using system's Qt5. The driver is successfully loaded after I install libqt5sql5-mysql package.
sudo apt-get install libqt5sql5-mysql
There are two fixes for this issue. First try and locate the qtbase folder located within your qt directory and try run ./configure -plugin-sql-mysql which will activate the driver if they're missing.
If this doesn't solve your issue, I suggest that you double check your code and try running one of the Qt examples which makes a connection to the Sql databases. I.e. modify the example code which connects to a local SQLite database changing the parameter to MySQL. If this example doesn't throw 'Driver not loaded error' then follow the step below.
Make sure you're using the static function of the QSqlDatabase i.e. rather than using
QSqlDatabase *db = new QSqlDatabase();
db->addDatabase("QMYSQL");
you should be doing
QSqlDatabase db = QSqlDatabase::addDatabase("QMYSQL");
I also have the same problem with Qt5.11.1 and my OS is Ubuntu16.04 and I have resolve the problem by installing libmysqlclient18 from here
https://launchpad.net/ubuntu/xenial/amd64/libmysqlclient18/5.6.25-0ubuntu1
After download run the command from download directory,
sudo dpkg -i libmysqlclient18_5.6.25-0ubuntu1_amd64.deb
For use MySql with PyQt5 on Raspberry Pi, it's fine to install:
sudo apt-get install libqt5sql5-mysql
and
sudo apt-get install libqt5sql5-mysql

QSqlDatabase: QMYSQL driver not loaded on Ubuntu 15.04 64bits

In Ubuntu 15.04 64 bits I installed Qt5.6 (online installer) and while trying to move my development environment from Windows 7 to Linux I faced the following:
SqlDatabase: QMYSQL driver not loaded
Following this, I managed to find ~/Qt/5.6/gcc_64/plugins/sqldrivers/libqsqlmysql.so and then:
$ ldd libqsqlmysql.so
linux-vdso.so.1 => (0x00007ffffd571000)
libmysqlclient_r.so.16 => not found
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe94ef24000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fe94ecec000)
libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007fe94ead2000)
libssl.so.10 => not found
libcrypto.so.10 => not found
libQt5Sql.so.5 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libQt5Sql.so.5 (0x00007fe94e88d000)
libQt5Core.so.5 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libQt5Core.so.5 (0x00007fe94e17a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe94df5c000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe94dc4d000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe94d944000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe94d72e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe94d364000)
libicui18n.so.56 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libicui18n.so.56 (0x00007fe94cec9000)
libicuuc.so.56 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libicuuc.so.56 (0x00007fe94cb11000)
libicudata.so.56 => /home/user/Qt/5.6/gcc_64/plugins/sqldrivers/../../lib/libicudata.so.56 (0x00007fe94b12e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe94af29000)
libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0 (0x00007fe94ad27000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe94ab1f000)
libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fe94a80f000)
/lib64/ld-linux-x86-64.so.2 (0x000056024837f000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fe94a5a2000)
Tells libmysqlclient_r.so.16 => not found. In fact, it seems I have a newer version:
find / -name libmysqlclient_r*
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.so.18
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.so.18.1.0
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.so
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.a
Perhaps that's the problem. Could some one confirm? How should I proceed?
First and foremost, double check that you have the packages containing libssl.so, libcrypto.so and libmysqlclient_r.so installed (looks like you have this last one, it's extremely likely you also have the first two, but just double check).
Then, your problem is that you have those shared objects with a different SONAME, sign that they're binary incompatible with the plugin shipped with Qt, which therefore needs to be recompiled.
Therefore:
install the development versions of the packages found above (libssl-dev, mysql-client-dev or similar).
run the MaintenanceTool from your Qt installation, and be sure to select to install Qt's source code too.
Go in QTDIR/5.6/Src/qtbase/src/plugins/sqldrivers/mysql/.
Run the right qmake, i.e. the one coming from that installation of Qt (not the system wide one or similar). Best way to be sure is providing the full path to it: QTDIR/5.6/gcc_64/bin/qmake.
Run make. Hopefully this will just work™; if it complains about some missing libraries, install them and rerun make.
This should now have produced a new libqsqlmysql.so plugin; overwrite the old one with this new one.
I am using Ubuntu 18.04.4 and compiling my project using system's Qt5. The driver is successfully loaded after I install libqt5sql5-mysql package.
sudo apt-get install libqt5sql5-mysql
There are two fixes for this issue. First try and locate the qtbase folder located within your qt directory and try run ./configure -plugin-sql-mysql which will activate the driver if they're missing.
If this doesn't solve your issue, I suggest that you double check your code and try running one of the Qt examples which makes a connection to the Sql databases. I.e. modify the example code which connects to a local SQLite database changing the parameter to MySQL. If this example doesn't throw 'Driver not loaded error' then follow the step below.
Make sure you're using the static function of the QSqlDatabase i.e. rather than using
QSqlDatabase *db = new QSqlDatabase();
db->addDatabase("QMYSQL");
you should be doing
QSqlDatabase db = QSqlDatabase::addDatabase("QMYSQL");
I also have the same problem with Qt5.11.1 and my OS is Ubuntu16.04 and I have resolve the problem by installing libmysqlclient18 from here
https://launchpad.net/ubuntu/xenial/amd64/libmysqlclient18/5.6.25-0ubuntu1
After download run the command from download directory,
sudo dpkg -i libmysqlclient18_5.6.25-0ubuntu1_amd64.deb
For use MySql with PyQt5 on Raspberry Pi, it's fine to install:
sudo apt-get install libqt5sql5-mysql
and
sudo apt-get install libqt5sql5-mysql

Strange Python compilation results with "--enable-shared" flag

I am using Debian (which comes with Python-2.7.3), trying to compile Python-2.7.6 from source for use with mod_wsgi alongside Apache.
Apparently you must use --enable-shared when compiling for mod_wsgi usage, according to numerous answers.
Following these steps:
./configure --enable-shared --prefix=/usr/local/bin/python-2.7.6
make
make install
And then checking the interactive shell #
/usr/local/bin/python-2.7.6/bin/python
I am greeted with "Python 2.7.3 (default, Jan 2 2013, 14:09:21)" etc
Why is it resulting in Python 2.7.3?
I tried ldd against the executable and this is the result:
linux-vdso.so.1 => (0x00007fff271ff000)
libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0x00007f1545638000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f154541c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1545217000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f1545014000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1544d92000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1544a06000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f15447ef000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f15445d9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1545b40000)
How do I stop it from using the system library and instead use the locally compiled version?
I know that it is a lot easier for me to just revert to using the system installed Python version, and that the real-world difference is zero. But this behaviour seems strange.
When you do the make of Python, run it as:
LD_RUN_PATH=/usr/local/lib make
Setting the environment variable LD_RUN_PATH forces 'python' executable generated to look in /usr/local/lib before /usr/lib for Python shared library.
This is mentioned in the mod_wsgi documentation.
http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Python_Shared_Library
Before doing this again, make sure you do a 'make distclean' and rerun configure to make sure you haven't got old build products around.