Code jumps into unreferenced shared object - c++

Question

My statically linked CryptoPP code (when called via Matlab mex on Linux) jumps into the libmwflcryptocryptopp.so binary (and then it freezes).
How can my code jump into a foreign .so instead of the statically linked library?
from function CryptoPP::SourceTemplate<CryptoPP::FileStore>::PumpAll2 in my vcpkg_installed/.../cryptopp/filters.h:1443 it jumps into CryptoPP::BufferedTransformation::TransferAllTo2 from Matlabs own libmwflcryptocryptopp.so file.
the gdb stack trace
#0 0x00007ffff02e6cab in CryptoPP::BufferedTransformation::Peek(unsigned char&) const () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#1 0x00007ffff02e6c19 in CryptoPP::BufferedTransformation::AnyRetrievable() const () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#2 0x00007ffff02e6ffe in CryptoPP::BufferedTransformation::TransferMessagesTo2(CryptoPP::BufferedTransformation&, unsigned int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#3 0x00007ffff02e7129 in CryptoPP::BufferedTransformation::TransferAllTo2(CryptoPP::BufferedTransformation&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#4 0x00007fff9758aa4d in CryptoPP::SourceTemplate<CryptoPP::FileStore>::PumpAll2 (this=0x7fffdd87fbc0, blocking=<optimized out>)
at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1443
#5 0x00007fff9758a130 in CryptoPP::Source::PumpAll (this=0x7fffdd87fbc0) at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1420
#6 CryptoPP::Source::SourceInitialize (parameters=warning: RTTI symbol not found for class 'CryptoPP::AlgorithmParameters'
..., pumpAll=true, this=0x7fffdd87fbc0) at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1420
#7 CryptoPP::FileSource::FileSource (attachment=0x7fffd6063160, pumpAll=true, in=..., this=0x7fffdd87fbc0) at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/files.h:102
#8 mexFunction (nlhs=<optimized out>, plhs=<optimized out>, nrhs=<optimized out>, prhs=<optimized out>) at /home/keinkoenig/src/CryptoppMinimal/CryptoppMinimal.cpp:18
minmal example code
#include <mex.h>
#include <cryptopp/files.h>
#include <cryptopp/aes.h>
#include <cryptopp/modes.h>
#include <fstream>
unsigned char Key[CryptoPP::AES::DEFAULT_KEYLENGTH] = { 0x00,0x11,0x22,0x33,0x44,0x55,0x66,0x77,0x88,0x99,0xAA,0xBB,0xCC,0xDD,0xEE,0xFF };
unsigned char IV[CryptoPP::AES::BLOCKSIZE] = { 0x00,0x11,0x22,0x33,0x44,0x55,0x66,0x77,0x88,0x99,0xAA,0xBB,0xCC,0xDD,0xEE,0xFF };
extern "C"
{
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
std::ifstream instream(mxArrayToString(prhs[0]));
std::string decryptedString;
CryptoPP::CFB_Mode<CryptoPP::AES>::Decryption cfbDecryption(Key, sizeof(Key), IV);
CryptoPP::FileSource(instream, true, new CryptoPP::StreamTransformationFilter( cfbDecryption, new CryptoPP::StringSink( decryptedString)));
}
}
and the CMakeLists linking
set(PROJECT_NAME CryptoppMinimal)
...
target_link_libraries(${PROJECT_NAME} PRIVATE cryptopp-static)
starting console matlab with gdb
/usr/local/MATLAB/R2020b/bin/matlab -nojvm -Dgdb
and running the test mex function in Matlab
addpath ~/build-o1/bin
CryptoppMinimal('/home/keinkoenig/src/encrypted.dat')

Answer 1

TL;DR; your cryptocpp is getting LD_PRELOADed by Matlab, even if it is being linked statically.
A big hint towards what is going on is the path of the .so being loaded:
#3 0x00007ffff02e7129 [...] /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
Compared to the headers that were used during compilation:
#5 [...] /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1420
What this shows is that not only is cryptocpp running from a dynamic library, it's running from a different .so than the one you compiled against!
How could this happen? If a static library is compiled with position independent code, calls to functions in the library will still be dispatched in the same way they would for a dynamic library: with a jump table.
And much like dynamic libraries, if the jump table is already populated when the code loaded, the existing function pointers will be reused.
So, if:
crypto-cpp was compiled with -fPIC
Matlab happens to have a crypto-cpp .so loaded prior to your code being loaded.
Then the version of crypto-cpp embedded within your project will be ignored, and Matlab's will be used instead.
Having a quick look at the crypto-cpp project, we find, in the Makefile:
# Add -fPIC for targets *except* X86, X32, Cygwin or MinGW
ifeq ($(IS_X86)$(IS_CYGWIN)$(IS_MINGW),000)
ifeq ($(findstring -fpic,$(CXXFLAGS))$(findstring -fPIC,$(CXXFLAGS)),)
CRYPTOPP_CXXFLAGS += -fPIC
endif
endif
Which appears to confirm this is what's going on.
You can confirm that by loading and running your library from outside Matlab, and it should be calling the static library's code as expected.