I'm building a special-purpose embedded Python interpreter and want to avoid having dependencies on dynamic libraries so I want to compile the interpreter with static libraries instead (e.g. libc.a not libc.so).
I would also like to statically link all dynamic libraries that are part of the Python standard library. I know this can be done using Freeze.py, but is there an alternative so that it can be done in one step?
I found this (mainly concerning static compilation of Python modules):
http://bytes.com/groups/python/23235-build-static-python-executable-linux
Which describes a file used for configuration located here:
<Python_Source>/Modules/Setup
If this file isn't present, it can be created by copying:
<Python_Source>/Modules/Setup.dist
The Setup file has tons of documentation in it and the README included with the source offers lots of good compilation information as well.
I haven't tried compiling yet, but I think with these resources, I should be successful when I try. I will post my results as a comment here.
Update
To get a pure-static python executable, you must also configure as follows:
./configure LDFLAGS="-static -static-libgcc" CPPFLAGS="-static"
Once you build with these flags enabled, you will likely get lots of warnings about "renaming because library isn't present". This means that you have not configured Modules/Setup correctly and need to:
a) add a single line (near the top) like this:
*static*
(that's asterisk/star the word "static" and asterisk with no spaces)
b) uncomment all modules that you want to be available statically (such as math, array, etc...)
You may also need to add specific linker flags (as mentioned in the link I posted above). My experience so far has been that the libraries are working without modification.
It may also be helpful to run make with as follows:
make 2>&1 | grep 'renaming'
This will show all modules that are failing to compile due to being statically linked.
CPython CMake Buildsystem offers an alternative way to build Python, using CMake.
It can build python lib statically, and include in that lib all the modules you want. Just set CMake's options
BUILD_SHARED OFF
BUILD_STATIC ON
and set the BUILTIN_<extension> you want to ON.
Using freeze doesn't prevent doing it all in one run (no matter what approach you use, you will need multiple build steps - e.g. many compiler invocations). First, you edit Modules/Setup to include all extension modules that you want. Next, you build Python, getting libpythonxy.a. Then, you run freeze, getting a number of C files and a config.c. You compile these as well, and integrate them into libpythonxy.a (or create a separate library).
You do all this once, for each architecture and Python version you want to integrate. When building your application, you only link with libpythonxy.a, and the library that freeze has produced.
You can try with ELF STATIFIER. I've been used it before and it works fairly well. I just had problems with it in a couple of cases and then I had to use another similar program called Ermine. Unfortunately this one is a commercial program.
Related
I have written a haxe program that tries to communicate with a remote server. I was able to compile to the C++ target successfully. The executable runs just fine on my system. However, when I try to run the same on another windows box, it fails with the following error
Error: Could not load module std#socket_init__0
I then installed haxe and hxcpp which worked like a charm. I was able to run the exe. I understand now that there is dependency on hxcpp.
That still did not solve my problem as I want to create a stand-alone application. After some research I found a file (ExampleMain.CPP) with the following instructions that I think might solve my problem. However, I am a novice and do not quite follow. Can some one walk me through with this? Thanks
ExampleMain.CPP
This is an example mainline that can be used to link a static version.
First you need to build the static version of the standard libs, with:
cd $HXCPP/runtime
haxelib run hxcpp BuildLibs.xml -Dstatic_link
Then the static verion of your application with (note: extra space before 'static_link'):
haxe -main YourMain -cpp cpp -D static_link
You then need to link the above libraries with this (or a modified version) main.
You may choose to create a VisualStudio project, and add the libraries from
$HXCPP/bin/Windows/(std,regexp,zlib).lib and your application library.
Note also, that if you compile with the -debug flag, your library will have a different name.
Linking from the command line for windows (user32.lib only required for debug version):
cl ExampleMain.cpp cpp/YourMain.lib $HXCPP/bin/Windows/std.lib $HXCPP/bin/Windows/zlib.lib $HXCPP/bin/Windows/regexp.lib user32.lib
From other OSs, the compile+link command will be different. Here is one for mac:
g++ ExampleMain.cpp cpp/Test-debug.a $HXCPP/bin/Mac/regexp.a $HXCPP/bin/Mac/std.a $HXCPP/bin/Mac/zlib.a
If you wish to add other static libraries besides these 3 (eg, nme) you will
need to compile these with the "-Dstatic_link" flag too, and call their "register_prims"
init call. The inclusion of the extra static library will require the library
in the link line, and may requires additional dependencies to be linked.
Also note, that there may be licensing implications with static linking
thirdparty libraries.
I'm not sure, but it seems that you are taking the same extra steps hxcpp does for you already. When you compile your standalone application it is actually standalone and doesn't have a dependency on hxcpp per se - but it has a dependency on the standard libraries within hxcpp you may have used. For instance, if you use regular expressions, you will need the regexp.dll that hxcpp has for it, as you noted. The haxe standard library is in the std.dll and the zlib is if you used compression from the zip packages.
If I am not mistaken, the default is to reference these components dynamically. In order for your application to be standalone as you suggest, you simply have to copy these dll's alongside your binary.
If you want to link to these library components statically, automatically from your haxe code, just import the types from the cpp.link package. This instructs hxcpp to automatically bring its libraries as part of the compilation, linking it statically into your binary instead of dynamically. No extra steps are necessary!
Short answer: add import cpp.link.StaticStd; and any other library components in the link package somewhere to your code. It can be anywhere as long as it's imported, it will be linked in.
Let's say I have created and compiled a simple program using the MinGW 64 (g++ compiler). Running this program on my computer and looking in Process Explorer for what DLL files the program is using I find (among many others):
libgcc_s_seh-1.dll
libstdc++6.dll
libwinpthread-1.dll
These are the only ones that reside under my MinGW installation folder. The rest of the DLL files used reside under C:\Windows.
Question 1:
Are the MinGW DLL files the MinGW C++ runtime libraries (so to speak)? Do they serve the same purpose as for example msvcrXXX.dll (XXX = version of Microsoft runtime library).
Question 2:
If I want to run the application on a different computer which does not have MinGW installed, is it sufficient to include those DLL files listed above (i.e. placing them in the same folder as my executable) to have it run on the other computer (we assume the other computer is also a 64-bit Windows machine). If yes, does this mean we basically ship the MinGW C++ runtime with our executable. If no, why?
libstdc++6.dll is the C++ standard library, like you said.
libwinpthread-1.dll is for C++11 threading support. MinGW-W64 has two possible thread variants: Either use the native Windows functions like CreateThread, but C++11 stuff like std::thread won´t be available then; or include this library and use the C++11 classes (too).
Note that to switch the thread model, you´ll need to reinstall MinGW. Just removing the DLL and not using the C++11 stuff won´t work, the DLL will be required nonetheless with your current install.
libgcc_s_seh-1.dll is something about C++ exception handling.
Yes, it should be sufficient to deliver the DLLs too
(or use static linking and deliver only your program file).
For complicated projects where you're not exactly sure which DLL files need to be included to distribute your application, I made a handy dandy Bash script (for MSYS2 shells) that can tell you exactly what DLL files you need to include. It relies on the Dependency Walker binary.
#!/usr/bin/sh
depends_bin="depends.exe"
target="./build/main.exe" # Or wherever your binary is
temp_file=$(mktemp)
output="dll_list.txt"
MSYS2_ARG_CONV_EXCL="*" `cygpath -w $depends_bin` /c /oc:`cygpath -w $temp_file` `cygpath -w $target`
cat $temp_file | cut -d , -f 2 | grep mingw32 > $output
rm $temp_file
Note that this script would need to be modified slightly for use in regular MSYS (the MSYS2_ARG_CONV_EXCL and cygpath directives in particular). This script also assumes your MinGW DLL files are located in a path which contains MinGW.
You could potentially even use this script to automatically copy the DLL files in question into your build directory as part of an automatic deploy system.
You may like to add the options -static-libgcc and -static-libstdc++ to link the C and C++ standard libraries statically and thus remove the need to carry around any separate copies of those.
I used ntldd to get a list of dependencies.
https://github.com/LRN/ntldd
I'm using msys2 so i just installed it with pacman. Use that and then copy all the needed dependencies
There are several major challenges to distributing compiled software:
Compiling the code for all target processors (remember, when it comes to compiled code, you need to produce separate downloads/distributions for each type of instruction set architecture).
Ensuring that the builds are reproducible, consistent, and can be easily correlated with a specific version of the code (and versions of the dependencies).
Ensuring that the build output is self-contained and includes all of its dependencies within it (so that it is not dependent on any other installations that happen to exist on just your system).
Making sure that your code is built and distributed regularly, with updates distributed automatically so that -- in the event of security issues -- you can push out new patched versions.
For convenience and to increase reach, it is nice for non-savvy users to have a prebuilt version that they can install. However, I would recommend sharing the source code as a first step.
Most of these requirements are fairly non-trivial to hit and often require automating not only build process, but also automating the instantiation / configuration of VMs in which the build should take place. However, there are open source projects that can help... for example, check out Gitian.
In terms of bullet point #3, the key thing here is to use static linking... while this does make the binary you distribute much larger (because its dependencies are now baked into the output), it also makes your binary isolated from the version of the libraries on the system (avoiding "dependency hell").
Point #4 is very tricky, but thankfully there are also opensource tools to help here, as well such as cloudup, which provides a way to add auto-updating capability to your application distribution.
I feel somewhat ridiculous, but I'm trying to import the OpenBLAS libraries into a project. They were built with gfortran as the Fortran compiler. My early builds had no issue just pulling libopenblas.so in, but on another system, it's choking on libgfortran.so when I try to run our program, which doesn't exist there. My impression has been that this is a standard library on most, if not all, Linux systems. I could probably add a copy of libgfortran.so to Artifactory and let Apache Ivy pull it in, but it seems like it would make more sense to use the standard version if possible. Is there a good way to pull it in via Ivy when doing an ant resolve command if it doesn't exist on the system?
An alternate solution may be to statically link libgfortran.a in on the compiling system, but my attempts to do so by adding -static RELATIVE_PATH_TO_LIBS/libgfortran.a compile and link fine, but I still get errors when running said program on the system which lacks the library.
Thank you for whatever help you can provide.
If the executable file format is the "ELF" file format (default on Linux systems) you can use "readelf" to display the dynamic section of the executable:
readelf -d my_executable_file
It should contain a list of all shared libraries required. This is a possibility to check if the executable still requires this library.
If "libgfortran.so" is the problem and "libgfortran.a" is available I would rename "libgfortran.a" to "libxxxx.a" and use the linker switches:
-Lpath_containing_libxxxx.a -lxxxx
instead of "-lgfortran". I would not use the "-static" switch because in this case the linker also tries to link all the other libraries statically. The linker should automatically link "-lxxxx" statically because no dynamic library with this name is available.
I'm totally spinning my wheels with getting a couple of 3rd party libraries to work with my c++ programs. I'm looking for some general advice (40,000 foot level) about the general steps that one needs to take when implementing libraries.
First, some specifics: I am using code::blocks in Windows as my IDE. I like this IDE and really don't want to switch to anything else if I don't have to (I've tried visual c++ and also some things in linux). The libraries that I am trying to use are GMP and crypto++.
OK. What I think I know is this: After downloading the library, I unzip the file to a folder. I've been unzipping directly to C:\ with each zip file extracted to its own folder (e.g. c:\cryptopp and c:\gmp). I think that the next step is to build the library, but this is where I get totally stuck. How is this done? There are no executable files among those extracted. From what I can tell, I believe that I do this in code::blocks, but I have no idea how?
Finally, assuming that I can get this done, which I believe creates the .lib files, the last step before actually using the library in my code, is to link into the library. This part, I believe that I understand.
So, my question is broad: do I understand this process overall? And if so, how do I go about building these libraries, if in fact that it the thing that I am missing.
Thanks very much for indulging my ignorance. I'm totally rudderless right now and despite hours and hours on google, I'm making no progress. Also, feel free to correct anything that I have stated as fact that is not correct. Thanks a lot!
Usually libraries have a special file called makefile in them, and are built with a utility called Make (or one of it's variations, whatever works uder windows).
Usually all you have to do is to run Make in the directory where you have unpacked the source files, and it will do the rest itself.
If those libraries you mention (GMP and crypto++; disclaimer: I'm not familiar with either of them) don't have project files for code::blocks then you may still be able to compile them under Windows with MinGW.
If you have installed MinGW you use the MinGW shell to navigate to the appropriate directories which would be /c/cryptopp/ and /c/gmp in your examples - The MinGW shell works like a Unix shell, and has different naming conventions.
Then you need to compile the libraries. Check whether there's a Makefile in those directories, if there isn't you can check whether there's a configure script, which will generate the Makefile. If you have the Makefile you can type make which will compile the libraries with MinGW's port of the GCC compiler.
When the compilation is complete you should have a library in the form of a .a file (say libcryptopp.a) that you can link to your project. In code::blocks you would set the linker path (the -L command line option in GCC) to C:\cryptopp\bin or wherever the library has been compiled, and then add libcryptopp.a to the list of libraries you want to link (this is associated with the -l option in GCC). The convention is to leave out the lib prefix and the .a extension, so you would just add cryptopp to your library list. In the end your linker options should look like -LC:\cryptopp\bin -lcryptopp along with the
Also, to be able to use the libraries you need to add the path to the headers directory to the include path of your project. This is associated to the -I command line option in GCC, so your compiler's command line options would have something like -IC:\cryptopp\include somewhere.
Of course, all of the above assumes that you use code::blocks with GCC. If you're using it with VisualC++ then the principles are the same, but the specific steps differ.
I have started an C++ SFML project for linux. I was wondering where the .so's should go.
Should they go inside the project folder so a user can simply run the program after they get it? Or should the user have the SFML library installed on there linux machine before they run my program?
Even if you did include the .so files, you have no guarantee that the user will be able to run it (different architecture, libraries linked against different libc, ...). Either link statically, or better yet, just let them provide the supporting libraries themselves.
#Joel J. Adamson's answer to use autoconf is a good idea. SFML doesn't come with a pkg-config file, so you will check for SFML as follows:
dnl Checking for a C++ compiler
AC_PROG_CXX
dnl Checking C++ features. This tells configure to use the C++ compiler for checks.
AC_LANG_PUSH([C++])
dnl Check for a SFML header.
AC_CHECK_HEADER([SFML/Config.hpp], [], [AC_MSG_ERROR([SFML headers not found.])])
AC_LANG_POP([C++])
Checking for the libraries is a bit more difficult because of name mangling and so on. Tyler McHenry wrote a good article on this part, if you want to be thorough.
At the distribution level SFML will be a dependency, i.e. a user will have to install it (or their package manager will have to install it) before compiling your program. If a user wants to compile it, they will also need the header files (often there's a separate "devel" package to install). You shouldn't have to distribute the .so files, and it's probably better (for everybody) if you don't.
You will need to check if the user has it, e.g. using autoconf you need to check for the relevant headers in your configure.ac for the application project. For example, to check for the math library, because I'm using the exp() function, I'll use
AC_CHECK_LIB([m], [exp])
to create a check during the configure step. Luckily autoscan can check this for you and create a skeleton configure.ac file called configure.scan. See also Cmake.
HTH.
You should make your program depend on the specific library (and version) that it needs. If you're planning to package it into an rpm/deb file you should add the dependency there too so that it can be checked and applied by package managers (e.g. apt can install all the dependency packages of a given package)