Linux c++ portable binary issue - c++

I try to package my binary and all its shared libraries into an archive. I want user only extracts the archive to make the binary work. The executable is launched via a shell script where I set LD_LIBRARY_PATH to shared library path contained in the archive.
On a first time I wanted to make it work on a RHEL 6.7 distribution so I have compiled my binary with this distribution, and when I tested it it works well.
The problem is that I have now to make it work on a RHEL 7.2 (and keep RHEL 6.7 working) and when I launched the binary it don't work... it crashes inside a glibc function (isspace with no garbage in argument).
I see that behind these two RHEL releases glibc version has changed.
In the shared libraries that are contained in the archive there was not glibc shared library, so I tried to add it and now I got the following error :
./XXX: ���:ELF: zR: Error 892688562
It seems to be an ELF error (verbose is present each time I launch it), I check all my shared libraries and they are x64 libraries (like the binary)...
I don't want to compile on RHEL 7.2 because I have a lot of dependancies and I don't want to compile all of it, and also it is easier for me to maintain one version.
I tried many things like Statifier and Ermine, that both work but the first has a bug and have to had Address Space Layout Randomization disabled and the second one is a shareware and I prefer a free solution. I also tried CDE that also work, but it generates such a big package, it is a bit a mess...
Why with this kind of solution it works and with my homemade solution it don't work ? What am I not doing well ?
Thank you for reading,
I hope someone has a solution for me because I search for a long time...
EDIT :
I found solution, before my shell script was like :
# Binary location
LOCATION=$(dirname $0)
# Shared libraries directory
BINDIR=${LOCATION}/bin/
# Define LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${BINDIR}:${LD_LIBRARY_PATH}
# Launch binary
${LOCATION}/XXX $*
I changed by :
# Binary location
LOCATION=$(dirname $0)
# Shared libraries directory
BINDIR=${LOCATION}/bin/
# Changed shared library default location and launch binary
${BINDIR}/ld-linux-x86-64.so.2 --library-path ${BINDIR} ${LOCATION}/XXX $*
I don't really understand why but it works, someone can explain me please ? (ld-linux-x86-64.so.2 came from RHEL 6.7 distribution)

I don't really understand why but it works, someone can explain me please ?
It's explained here. Your solution is "explicit loader invocation" mentioned here.
BTW, this is wrong: ${LOCATION}/XXX $* You should do this instead:
${LOCATION}/XXX "$#"
(Your variant will not handle arguments with embedded spaces correctly.)

Related

Where can i find library "version.lib" in order to successfully call GetFileVersionInfoSize()?

I'm developing a project on DevC++ which uses MinGW64. On Windows 7 (i don't know if this can be related to my issue).
I had a problem compiling a C++ program where I call the function GetFileVersionInfoSize(), which is:
main.cpp:(.text+0x51): undefined reference to `GetFileVersionInfoSizeA'
After two days of researching, I understood that I have to include in the parameters of the linker the "version.lib" file, which is missing in my computer, I searched it everywhere.
I can't even find a download mirror on the web, so I'm asking, does anybody know where I can find version.lib? Maybe somewhere hidden in my PC or in the web? Maybe with a new installation of MinGW64? I don't know, since my installation of MinGW64 came with DevC++.
Thanks for reading.
Thank all you guys for the suggestions you gave me
In this answer there is how to find Version.lib in this case.
The following is about accomplishing using GetFileVersionInfoSize(), GetFileVersionInfo() and VerQueryValue() to get the Product Name of an executable file, which is why I needed Version.lib, actually being called libversion.a on my machine.
LAST-EDIT: I managed to achieve what I wanted, this is the code that works in my case:
// filename contains the path of the .exe we want to get Product Name
int version_info_size = GetFileVersionInfoSize(filename, NULL);
if(version_info_size > 0) {
BYTE *version_info_buffer = new BYTE[version_info_size];
if(GetFileVersionInfo(filename, 0, version_info_size, version_info_buffer)) {
char *product_name = NULL;
UINT pLenFileInfo = 0;
if(VerQueryValue(version_info_buffer, TEXT("\\StringFileInfo\\040904e4\\ProductName"),
(LPVOID*)&product_name, &pLenFileInfo)) cout << product_name;
}
}
Notice that if you want to compare product_name with another value, you have to do something like string product_name_str = product_name and compare that string variable, otherwise the comparison will always return false. Or maybe just do (string)product_name, I should try.
040904e4 is the lang-code hex translation that I needed in order to do this, I found it thanks to the code in this answer.
Since it's a environment built for GCC, library names will follow Nix conventions:
Start with lib: libversion
Extension can be either .so or .a (depending on the library being dynamic or static)
Typically (depending on default MinGW installation), it should reside in ${MINGW_INSTALL_DIR}/x86_64-w64-mingw32/lib/libversion.a. One installation example is: f:\Install\Qt\Qt\Tools\mingw730_64\x86_64-w64-mingw32\lib\libversion.a.
According to [Archive.Web - MinGW]: HOWTO Specify the Location of Libraries for use with MinGW (Determining MinGW's Default Library Search Path section), it can be retrieved by:
[cfati#cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow]> x86_64-w64-mingw32-ld.exe --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("=/usr/x86_64-w64-mingw32/lib")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")
But since it's a system library, you shouldn't care about its path, MinGW should find it automatically, all you have to do is pass it to the linker (using the standard ways).

Using connector c++ to access MYSQL database (How to add a path to my standard search directory)

I am relatively new to programming&C++ and just began learning about MYSQL. I have been searching for a couple of days but couldn't find a solution. I usually use Cygwin&VIM&g++ to write, compile and run codes that I write.
My goal is to retrieve data from MYSQL database that I've set up on my laptop and be able to run a simple algorithm on that data and possibly update the database.
I went into MYSQL's website and went through tutorials and found an example here. I have Boost, Connector, MYSQL server lib downloaded. When I run it, I get an error saying 'connection.h' was not found. I'm sure this is because when I compile it, g++ does not have the location of the library added to the search path that it goes through. The example has something like:
#include <cppconn/conneciton.h>
where these angled brackets means it is going to search (after the current directory) the standard search directory. I am guessing I have to add a new path (the location of where the libraries are) so that the compiler looks at that standard directory and compiles the header.
Through googling I am thinking in g++ I can use -Ldir -I to add a new search path but I don't get how to use it. Can anyone kindly explain/show me (by example) how to get over with this problem? :(
An #include issue would be solved with -I (-L is for a library if it is needed but first you have to get through the compile).
You need to figure out where the headers were installed (likely somewhere under either /usr/include or /usr/local/include/), and then add a -I with that directory, it may also be that you need to install a development package, many *nix distributions split based on runtime-build time distinctions.
And note also with the -I that the package may do something like install to /usr/include/libraryname/libname_version/libname in such a case you would only want to add -I/usr/include/libraryname/libname_version/ and then in your source file have #include

Change Linux shared library (.so file) version after it was compiled

I'm compiling Linux libraries (for Android, using NDK's g++, but I bet my question makes sense for any Linux system). When delivering those libraries to partners, I need to mark them with a version number. I must also be able to access the version number programatically (to show it in an "About" dialog or a GetVersion function for instance).
I first compile the libraries with an unversioned flag (version 0.0) and need to change this version to a real one when I'm done testing just before sending it to the partner. I know it would be easier to modify the source and recompile, but we don't want to do that (because we should then test everything again if we recompile the code, we feel like it would be less error prone, see comments to this post and finally because our development environment works this way: we do this process for Windows binaries: we set a 0.0 resources version string (.rc) and we later change it by using verpatch...we'd like to work with the same kind of process when shipping Linux binaries).
What would be the best strategy here?
To summarize, requirements are:
Compile binaries with "unset" version (0.0 or anything else)
Be able to modify this "unset" version to a specific one without having to recompile the binary (ideally, run a 3rd party tool command, as we do with verpatch under Windows)
Be able to have the library code retrieve it's version information at runtime
If your answer is "rename the .so", then please provide a solution for 3.: how to retrieve version name (i.e.: file name) at runtime.
I was thinking of some solutions but have no idea if they could work and how to achieve them.
Have a version variable (one string or 3 int) in the code and have a way to change it in the binary file later? Using a binary sed...?
Have a version variable within a resource and have a way to change it in the binary file later? (as we do for win32/win64)
Use a field of the .so (like SONAME) dedicated to this and have a tool allowing to change it...and make it accessible from C++ code.
Rename the lib + change SONAME (did not find how this can be achieved)...and find a way to retrieve it from C++ code.
...
Note that we use QtCreator to compile the Android .so files, but they may not rely on Qt. So using Qt resources is not an ideal solution.
I am afraid you started to solve your problem from the end. First of all SONAME is provided at link time as a parameter of linker, so in the beginning you need to find a way to get version from source and pass to the linker. One of the possible solutions - use ident utility and supply a version string in your binary, for example:
const char version[] = "$Revision:1.2$"
this string should appear in binary and ident utility will detect it. Or you can parse source file directly with grep or something alike instead. If there is possibility of conflicts put additional marker, that you can use later to detect this string, for example:
const char version[] = "VERSION_1.2_VERSION"
So you detect version number either from source file or from .o file and just pass it to linker. This should work.
As for debug version to have version 0.0 it is easy - just avoid detection when you build debug and just use 0.0 as version unconditionally.
For 3rd party build system I would recommend to use cmake, but this is just my personal preference. Solution can be easily implemented in standard Makefile as well. I am not sure about qmake though.
Discussion with Slava made me realize that any const char* was actually visible in the binary file and could then be easily patched to anything else.
So here is a nice way to fix my own problem:
Create a library with:
a definition of const char version[] = "VERSIONSTRING:00000.00000.00000.00000"; (we need it long enough as we can later safely modify the binary file content but not extend it...)
a GetVersion function that would clean the version variable above (remove VERSIONSTRING: and useless 0). It would return:
0.0 if version is VERSIONSTRING:00000.00000.00000.00000
2.3 if version is VERSIONSTRING:00002.00003.00000.00000
2.3.40 if version is VERSIONSTRING:00002.00003.00040.00000
...
Compile the library, let's name it mylib.so
Load it from a program, ask its version (call GetVersion), it returns 0.0, no surprise
Create a little program (did it in C++, but could be done in Python or any other languauge) that will:
load a whole binary file content in memory (using std::fstream with std::ios_base::binary)
find VERSIONSTRING:00000.00000.00000.00000 in it
confirms it appears once only (to be sure we don't modify something we did not mean to, that's why I prefix the string with VERSIONSTRING, to make it more unic...)
patch it to VERSIONSTRING:00002.00003.00040.00000 if expected binary number is 2.3.40
save the binary file back from patched content
Patch mylib.so using the above tool (requesting version 2.3 for instance)
Run the same program as step 3., it now reports 2.3!
No recompilation nor linking, you patched the binary version!

vtkMy python wrapping

VTK offers the possibility to extend the library with C++ classes so that they can be used in Python (or in any language supported). As far as I know, there are two way to do that, using the examples folders vtkLocal or vtkMy provided with the library (Examples/Build/).
I would like to use this functionnality, but I don't understand the installation process (vtk 5.8 on Debian). I follow the README instruction on vtkMy, it compiles and generetate 3 files : vtk*.so vtk*Python.so and vtk*PythonD.so.
What I am suppose to do next ? I've tried appending those file's path to $PATH, $PYTHONPATH or $LD_LIBRARY_PATH as suggested, but I've never been able to import anything into Python.
Any insight or even some instructions on how to compile/import/use the dummy classes provided in vtkMy would be a tremendous help.

Py_Initialize fails - unable to load the file system codec

I am attempting to put together a simple c++ test project that uses an embedded python 3.2 interpreter. The project builds fine but Py_Initialize raises a fatal error:
Fatal Python error: Py_Initialize: unable to load the file system codec
LookupError: no codec search functions registered: can't find encoding
Minimal code:
#include <Python.h>
int main (int, char**)
{
Py_Initialize ();
Py_Finalize ();
return 0;
}
The OS is 32bit Vista.
The python version used is a python 3.2 debug build, built from sources using VC++ 10.
The python_d.exe file from the same build runs without any problems.
Could someone explain the problem and how to fix it? My own google-fu fails me.
EDIT 1
After going through the python source code I've found that, as the error says, no codec search functions have been registered. Both codec_register and PyCodec_Register are as they should be. It's just that nowhere in the code are any of these functions called.
I don't really know what this means as I still have no idea when and from where these functions should have been called. The code that raises the error is entirely missing from the source of my other python build (3.1.3).
EDIT 2
Answered my own question below.
Check the PYTHONPATH and PYTHONHOME environment variables and make sure they don't point to Python 2.x.
http://bugs.python.org/issue11288
Parts of this have been mentioned before, but in a nutshell this is what worked for my environment where I have multiple Python installs and my global OS environment set-up to point to a different install than the one I attempt to work with when encountering the problem.
Make sure your (local or global) environment is fully set-up to point to the install you aim to work with, e.g. you have two (or more) installs of, let's say a python27 and python33 (sorry these are windows paths but the following should be valid for equivalent UNIX-style paths just as well, please let me know about anything I'm missing here (probably the DLLs path might differ)):
C:\python27_x86
C:\python33_x64
Now, if you intend to work with your python33 install but your global environment is pointing to python27, make sure you update your environment as such (while PATH and PYTHONHOME may be optional (e.g. if you temporarily work in a local shell)):
PATH="C:\python33_x64;%PATH%"
PYTHONPATH="C:\python33_x64\DLLs;C:\python33_x64\Lib;C:\python33_x64\Lib\site-packages"
PYTHONHOME=C:\python33_x64
Note, that you might need/want to append any other library paths to your PYTHONPATH if required by your development environment, but having your DLLs, Lib and site-packages properly set-up is of prime importance.
Hope this helps.
The core reason is quite simple: Python does not find its modules directory, so it can of course not load encodings, too
Python doc on embedding says "Py_Initialize() calculates the module search path based upon its best guess" ... "In particular, it looks for a directory named lib/pythonX.Y"
Yet, if the modules are installed in (just) lib - relative to the python binary - above guess is wrong.
Although docs says that PYTHONHOME and PYTHONPATH are regarded, we observed that this was not the case; their actual presence or content was completely irrelevant.
The only thing that had an effect was a call to Py_SetPath() with e.g. [path-to]\lib as argument before Py_Initialize().
Sure this is only an option for an embedding scenario where one has direct access and control over the code; with a ready-made solution, special steps may be necessary to solve the issue.
Ran into the same thing trying to install brew's python3 under Mac OS! The issue here is that in Mac OS, homebrew puts the "real" python a whole layer deeper than you think. You would think from the homebrew output that
$ echo $PYTHONHOME
/usr/local/Cellar/python3/3.6.2/
$ echo $PYTHONPATH
/usr/local/Cellar/python3/3.6.2/bin
would be correct, but invoking $PYTHONPATH/python3 immediately crashes with the abort 6 "can't find encodings." This is because although that $PYTHONHOME looks like a complete installation, having a bin, lib etc, it is NOT the actual Python, which is in a Mac OS "Framework". Do this:
PYTHONHOME=/usr/local/Cellar/python3/3.x.y/Frameworks/Python.framework/Versions/3.x
PYTHONPATH=$PYTHONHOME/bin
(substituting version numbers as appropriate) and it will work fine.
From python3k, the startup need the encodings module, which can be found in PYTHONHOME\Lib directory.
In fact, the API Py_Initialize () do the init and import the encodings module.
Make sure PYTHONHOME\Lib is in sys.path and check the encodings module is there.
I had this issue with python 3.5, anaconda 3, windows 7 32 bit. I solved it by moving my pythonX.lib and pythonX.dll files into my working directory and calling
Py_SetPythonHome(L"C:\\Path\\To\\My\\Python\\Installation");
before initialize so that it could find the headers that it needed, where my path was to "...\Anaconda3\". The extra step of calling Py_SetPythonHome was required for me or else I'd end up getting other strange errors where python import files.
I just ran into the exact same problem (same Python version, OS, code, etc).
You just have to copy Python's Lib/ directory in your program's working directory ( on VC it's the directory where the .vcproj is )
There appears to be something going wrong with the release build either failing to include the appropriate codecs or else misidentifying the codec to use for system APIs. Since the python_d executable is working, what does that return for os.getfsencoding()? (Use the C API to call that between your Initialize/Finalize calls)
I had the same issue and found this question. However from the answers here I was not able to solve my problem. I started debugging the cpython code and thought that I might be discovered a bug. Therefore I opened a issue on the python issue tracker.
My mistake was that I did not understand that Py_SetPath clears all inferred paths.
So one needs to set all paths when calling this function.
Link to the issue conversation
For completion I also copied the most important part of the conversation below.
My original issue text
I compiled the source of CPython 3.7.3 myself on Windows with Visual Studio 2017 together with some packages like e.g numpy. When I start the Python Interpreter I am able to import and use numpy. However when I am running the same script via the C-API I get an ModuleNotFoundError.
So the first thing I did, was to check if numpy is in my site-packages directory and indeed there is a folder named numpy-1.16.2-py3.7-win-amd64.egg. (Makes sense because the python interpreter can find numpy)
The next thing I did was to get some information about the sys.path variable created when running the script via the C-API.
#### sys.path content ####
C:\Work\build\product\python37.zip
C:\Work\build\product\DLLs
C:\Work\build\product\lib
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\PROFESSIONAL\COMMON7\IDE\EXTENSIONS\TESTPLATFORM
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages
Examining the content of sys.path I noticed two things.
C:\Work\build\product\python37.zip has the correct path 'C:\Work\build\product\'. There was just no zip file. All my files and directory were unpacked. So I zipped the files to an archive named python37.zip and this resolved the import error.
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages is wrong it should be C:\Work\build\product\Lib\site-packages but I dont know how this wrong path is created.
The next thing I tried was to use Py_SetPath(L"C:/Work/build/product/Lib/site-packages") before calling Py_Initialize(). This led to
Fatal Python Error 'unable to load the file system encoding'
ModuleNotFoundError: No module named 'encodings'
I created a minimal c++ project with exact these two calls and started to debug Cpython.
int main()
{
Py_SetPath(L"C:/Work/build/product/Lib/site-packages");
Py_Initialize();
}
I tracked the call of Py_Initialize() down to the call of
static int
zipimport_zipimporter___init___impl(ZipImporter *self, PyObject *path)
inside of zipimport.c
The comment above this function states the following:
Create a new zipimporter instance. 'archivepath' must be a path-like
object to a zipfile, or to a specific path inside a zipfile. For
example, it can be '/tmp/myimport.zip', or
'/tmp/myimport.zip/mydirectory', if mydirectory is a valid directory
inside the archive. 'ZipImportError' is raised if 'archivepath'
doesn't point to a valid Zip archive. The 'archive' attribute of the
zipimporter object contains the name of the zipfile targeted.
So for me it seems that the C-API expects the path set with Py_SetPath to be a path to a zipfile. Is this expected behaviour or is it a bug?
If it is not a bug is there a way to changes this so that it can also detect directories?
PS: The ModuleNotFoundError did not occur for me when using Python 3.5.2+, which was the version I used in my project before. I also checked if I had set any PYTHONHOME or PYTHONPATH environment variables but I did not see one of them on my system.
Answer
This is probably a documentation failure more than anything else. We're in the middle of redesigning initialization though, so it's good timing to contribute this feedback.
The short answer is that you need to make sure Python can find the Lib/encodings directory, typically by putting the standard library in sys.path. Py_SetPath clears all inferred paths, so you need to specify all the places Python should look. (The rules for where Python looks automatically are complicated and vary by platform, which is something I'm keen to fix.)
Paths that don't exist are okay, and that's the zip file. You can choose to put the stdlib into a zip, and it will be found automatically if you name it the default path, but you can also leave it unzipped and reference the directory.
A full walk through on embedding is more than I'm prepared to type on my phone. Hopefully that's enough to get you going for now.
I had the problem and was tinkering with different solutions mentioned here. Since I was running my project from Visual Studio, apparently, I needed to set the environment path inside Visual Studio and not the system path.
Adding a simple PYTHONHOME=PATH\TO\PYTHON\DIR in the project solution\properties\environment solved the problem.
For me this happened when I updated Python 64 bit from 3.6.4 to 3.6.5. It threw some error like "unable to extract python.dll. Do you have permissions."
Pycharm also failed to load interpreter, even though I reloaded it in settings. Running python command gave same error, with and without administrator mode.
Reason
There was error in installation of Python, include folder in python installation directory C:\Users\USERNAME\AppData\Local\Programs\Python\Python36 was missing
Reinstalling Python also dint solve the issue.(Not removal and install)
Solution
Uninstall Python and Install of Python again.
Because running installer was just extracting same files excluding include folder
In my cases, for windows, if you have multiple python versions installed, if PYTHONPATH is pointing to one version the other ones didn't work. I found that if you just remove PYTHONPATH, they all work fine
For those working in Visual Studio simply add the include, Lib and libs directories to the Include Directories and Library Directories under
Projects Properties -> Configuration Properties > VC++ Directories :
For example I have Anaconda3 on my system and working with Visual Studio 2015 This is how the settings looks like (note the Include and Library directories) :
Edit:
As also pointed out by bossi setting PYTHONPATH in your user Environment Variables section seems necessary.
a sample input can be like this (in my case):
C:\Users\Master\Anaconda3\Lib;C:\Users\Master\Anaconda3\libs;C:\Users\Master\Anaconda3\Lib\site-packages;C:\Users\Master\Anaconda3\DLLs
is necessary it seems.
Also, you need to restart Visual Studio after you set up the PYTHONPATH in your user Environment Variables for the changes to take effect.
Also note that :
Make sure the PYTHONHOME environment variable is set to the Python
interpreter you want to use. The C++ projects in Visual Studio rely on
this variable to locate files such as python.h, which are used when
creating a Python extension.
So, for some reason the python dll fails to locate the encodings module. The python.exe executable apparently finds it because it has the expected relative path. Modifying the search path works.
The reason for all of this? Don't know but at least it works. I highly suspect a typo on my part somewhere, that's usually the reason for odd bugs it seems.