When using GCC, given that I compile the same library sometimes in release and sometimes in debug, is the ABI guaranteed to be compatible?
(while using the same compiler)
I have an executable and some shared objects (some depend on others), I want to be able to swap out release/debug shared objects without recompiling everything but
only the shared objects in interest.
Is this possible, or is there some scenario where I might get some undefined behavior this way? (Assuming my code is strictly packed, and padded in both release and debug)
EDIT:
I'll elaborate on the problem we're seeing. We have a custom version of intrusive_ptr, in debug mode we have our own intrusive_ptr which has a single member that is a boost::intrusive_ptr, and in release we simply use boost::intrusive_ptr. The API of our intrusive_ptr is the same of boost::intrusive_ptr, and we don't have any virtual functions in the class.
What we are seeing is this:
If we use all debug libs or all release libs all works well. If we mix debug executable with release libs, there is a memory leak from the intrusive_ptr and it does not release the object.
The sizeof our intrusive_ptr and boost::intrusive_ptr are identical, both in debug and release (our class does not add any size overhead on top).
So I am wondering what could be causing the leak, ABI difference are the only things that come to mind.
Ideas?
I have known several compilers that generate incompatible code for release and debug (though these compilers are long since deprecated). In face I would not trust object modules to fully compatible unless they have been compiled with Exactly the same flags.
This is why makefiles (following GNU principles) and IDE like Eclipse build release/debug/profile objects into different directories. To make sure that they can never be mixed up.
Typically no, because the usual difference between a release and a debug build are just options to make debug stepping works better. But unexpected behavior is still possible, if you define other options differently (such as integer size, target architecture, ??) between the release and debug builds, the caller parameters may not match the callee expectations. There could also be issues with exception safety, and const-ness that the run time linker may not check.
Related
I'm aware that mixing debug and release libraries that pass STL containers to each other causes big problems. But what exactly in 'debug' or 'release' causes this?
I have a QT project that gets built as 'release', but it has /DEBUG added to the compiler flags.
If I build another QT project under 'debug' (which also has the /DEBUG flag), are they compatible?
Or is there an optimization flag or some other flag that makes them incompatible?
Basically, is there a way I can look at the compilation line of the 2 libraries and see something that says "DON'T MIX THESE!"?
This is undefined behaviour, so it might work but more probably it will crash your app. It also depends on stl implementation, in debug version it might enable additional checks and allocate additional memory making data layout different in release modes. In the end this violates ODR rule (one definition rule) once again causing undefined behaviour.
I have a QT project that gets built as 'release', but it has /DEBUG added to the compiler flags. If I build another QT project under 'debug' (which also has the /DEBUG flag), are they compatible?
If your Release contains exactly the same compiler flags as Debug then I would say they are compatible.
Or is there an optimization flag or some other flag that makes them incompatible?
you should be able to see those flags, I dont think optimization flags should cause UB problems, its rather different macros used during compilation phase that cause ODR violation. Maybe there are optimization flags which changes alignment in structures...
Basically, is there a way I can look at the compilation line of the 2 libraries and see something that says "DON'T MIX THESE!"?
no idea here.
Why are you mixing differently build libraries in the first place? this is asking for trouble.
Not really. Concerning Visual C++'s STL implementation, there are data members of the containers corresponding to the iterator checking that is only compiled in if some preprocessor variables are set. These variables are default valued base on the NDEBUG preprocessor variable that is quasi standard for "no debug". But these could be also set directly from the command line, or from header files, or Visual Studio property pages, etc.
E.g.: all containers are derived from _Container_base, while that is a typedef depending on _ITERATOR_DEBUG_LEVEL. The different _Container_base implementations have different memory layout, and that is what causes incompatibilities between the debug and release version.
The /DEBUG compiler flag tells the compiler whether to generate debug information or not, and might also affect default values for optimization settings, but I am not sure about that, and of course that is compiler dependent.
Just like there could be an STL implementation that does not depend on any preprocessor directives, and in that case it would not matter how you compile it, debug or release, the memory layout would be identical, and so it could be passed around between modules compiled the different ways.
Firstly the /DEBUG flag doesn't actually create a 'debug' build. It just tells the linker to generate debugging information and create a .pdb file along with the resulting binary.
As to the difference between the debug and release MSVC++ runtimes the issue is to do with the fact that different runtimes can have different sizes for the same object. e.g. in debug extra information might be placed in iterators to ensure their validity at run time. If code has been compiled against the release version of these structures then corruption is likely. Another issue would be if an object if allocated from the heap of one runtime and is attempted to be freed on the heap of another runtime.
I am confused about the binary compatibility of compiled libraries between VS2010 and VS2012. I would like to migrate to VS2012, however many closed-source binary-only SDKs are only out for VS2010, for example SDKs for interfacing hardware devices.
Traditionally, as far as I know Visual Studio was extremely picky about compiler versions, and in VS2010 you could not link to libraries which have been compiled for VS2008.
The reason I'm confused now, is that I'm in the process of migrating to VS2012, and I have tried a few projects, and for my biggest surprise, many of them work cross-versions with no problems.
Note: I'm not talking about the v100 mode, which as far as I know is just a VS2012 GUI over the VS2010 compiling engine.
I'm talking about opening a VS2010 solution in VS2012, click update, and seeing what happens.
When linking to some bigger libraries, like boost, the compiling didn't work, as there are checks for compiler version and they raise an error and abort compilation. Some other libraries just abort at missing functions. This is the behaviour what I was expecting.
On the other hand, many libraries work fine with no errors or additional warnings.
How is it possible? Was VS2012 made in a special way to keep binary compatibility with VS2010 libraries? Does it depend on dynamic vs. static linking?
And the most important question: even though there is no error raised at compile time, can I trust the compiler that there won't be any bugs when linking a VS2012 project to VS2010 compiled libraries?
“Many libraries work fine. How is it possible?”
1) the lib is compiled to use static RTL, so the code won't pull in a second RTL DLL that clashes.
2) the code only calls functions (and uses structures, etc.) that is completely in the header files
so doesn't cause a linker error, or
calls functions that are still present in the new RTL so doesn't cause a linker error,
3) doesn't call anything with structures that have changed layout or meaning so it doesn't crash.
#3 is the one to worry about. You can use imports to see what it uses and make a complete list, but there is no documentation or guarantee as to which ones are compatible. Just because it seems to run doesn't mean it doesn't have a latent bug.
There is also the possibility of
4) the driver SDK or other code that's fairly low level was written to avoid using standard library calls completely.
also (not your situation I think) DLLs can be isolated and have their own RTL and not pass stuff back and forth (like memory allocation and freeing) between different regimes. In-proc COM servers work this way. DLLs can do this in general if you're careful about what you pass and return and what you do with things like pointers. Crypto++ for example is initialized with the memory routines from the enclosing program and doesn't expose malloc'ed memory from the RTL version it was compiled with.
I have this job of implementing a library that provides a file-sharing feature.
This has already happened twice:
First, in a string in an if-else path, only the if path is being executed, but when i change a spelling in the else path, the software after a few minutes crashes in an std library. I verified with a debug attached that the lined changed was never being touched. When i reversed the change, it works nicely again.
Second, my software crashes on a std library again with the out-of-array check into a standard basic_string destructor.
I did everything, all library matched the _HAS_ITERATOR_DEBUGGING.
After 4 hours I discovered that the problematic file is TorrentFile.cpp/h.
If i add a function ( even though it is never called ), the program crashes at the end of that file, but if its not there, there's no bug. The code causing the problem:
std::vector<TorrentFileListPacket> TorrentFile::GetFileMap()
{
std::vector<TorrentFileListPacket> vFiles;
return vFiles;
};
If i comment this code out, the crash is gone.
This is really driving me crazy!
I've been a developer for 8 years, and I've never seen something like this before!
Additional Information
My memory is OK, I'm using Visual Studio 2010 with SP1 in Windows 7. The library is libTorrent from RasterBar and it links to boost. The software is using MFC.
This smells strongly of memory corruption in a totally different location from where you would expect from the crashes. Most likely adding and removing functions is changing the memory layout in such a way that causes the memory corruption's effects to be immediately visible or not.
Your best hope is something like Purify or Valgrind to hunt it down.
You probably want to make sure that all your object files and libraries are ABI compatible with each other.
Numerous compiler settings will change the ABI. Especially debug and release builds and iterator debugging. The struct layout for standard containers typically change when you enable iterator debugging (which I believe is on by default for all debug builds in msvc, and off for release builds).
So, if a single object file, static library or DLL that you link against is built with an incompatible configuration, you typically see very odd behaviors. With libtorrent you need to make sure you build the library with the same configuration as you link against it with. Many of the TORRENT_* defines will actually change some aspect of some struct layout or function call. Make sure you define the exact same set of those in your client as when building the library. One simple way of dealing with this problem is to simply pull all source files into your project and build everything together.
If you are using libtorrent as a DLL (or boost for that matter), are they compiled against the same C Run-Time?
Often when I run into this type of issue it is because I make a call into a library that was compiled with MinGW (which uses the CRT from VS6.0) or an older version of Visual Studio. If memory is allocated by the library and then free'd by your application, you will often get these types of errors in the destructor.
If you aren't sure, you can open the DLL in question in a tool like the Dependency Walker. Look for the dependency MSVCRT.DLL, MSVCR100.DLL, etc.
I'm using some 3rd party. I'm using it's shared library version, since the library is big (~60MB) and is used by several applications.
Is there a way at application startup to find out that release/debug version of library is used respectively for release/debug version of my application?
Longer description
The library which exposes C++ interface. One of API methods return std::vector<std::string>.
The problem when I compile my application in debug mode, debug version of the library should be used. Same for release. If incorrect version of the library is used application is crashed.
According to gcc (see http://gcc.gnu.org/onlinedocs/libstdc++/manual/bk01pt03ch17s04.html)
but with a mixed mode standard library
that could be using either debug-mode
or release-mode basic_string objects,
things get more complicated
P.S. 1
It looks like proposal of Timbo is a possible solution - use different soname for debug and release libraries. So, what should be passed to ./configure script to change library soname?
P.S. 2
My problem is not at link time, but rather at run time.
P.S. 3
Here is question demonstrating problem I is facing with.
The debug mode referenced here has nothing to do with debug or release build of your application. The STL debug mode is activated with -D_GLIBCXX_DEBUG and is a special checking mode.
It is very unlikely that the 3rd party library was in fact compiled with STL checking mode, but if it was, it would likely very promptly mention that your code should also be compiled with -D_GLIBCXX_DEBUG.
If the 3rd party library was not built with checking STL, then it is compatible with your code regardless of whether you are doing optimized or debug build.
Since you state that debug build of your code linked with optimized build of 3rd party library causes a crash, that crash is most likely caused by a bug in your code (or possibly by a bug in 3rd party library).
Valgrind and GDB are your friends.
I believe that you have misread the documentation at the link you provide. In particular, you've misunderstood its purpose -- that section is entitled "Goals", and describes a number of hypothetical designs for a C++ debug library and the consequences of those designs in order to explain the actual design choices that were made. The bits of text that follow the lines you quoted are describing the chaos that would result from a hypothetical implementation that had separate designs for release-mode and debug-mode strings. It goes on to say:
For this reason we cannot easily provide safe iterators for the std::basic_string class template, as it is present throughout the C++ standard library.
(Or, rephrasing that, providing a special "debug" version of string iterators is impossible.)
...
With the design of libstdc++ debug mode, we cannot effectively hide the differences between debug and release-mode strings from the user. Failure to hide the differences may result in unpredictable behavior, and for this reason we have opted to only perform basic_string changes that do not require ABI changes. The effect on users is expected to be minimal, as there are simple alternatives (e.g., __gnu_debug::basic_string), and the usability benefit we gain from the ability to mix debug- and release-compiled translation units is enormous.
In other words, the design of the debug and release modes in GCC's libstdc++ has rejected this hypothetical implementation with separate designs for the strings, specifically in order to allow cross-mode linking of the sort that you are worrying about how to avoid.
Thus, you should not have problems with compiling your library once, without -D_GLIBCXX_DEBUG (or with it, if for some reason you prefer), and linking it with either mode of your application. If you do have problems, it is due to a bug somewhere. [But see edit below! This is specific to std::string, not other containers!]
Edit: After this answer was accepted, I followed up in answering the follow-up question at std::vector<std::string> crash, and realized that the conclusion of this answer is incorrect. GCC's libstdc++ does clever things with strings to support "Per-use recompilation" (in which all uses of a given container object must be compiled with the same flags, but uses of the same container class within a program need not be compiled with the same flags), but that is not the same thing as complete "Per-unit compilation" that would provide the cross-linking ability you need. In particular, the documentation says of that cross-linking ability,
We believe that this level of recompilation is in fact not possible if we intend to supply safe iterators, leave the program semantics unchanged, and not regress in performance under release mode....
Thus, if you're passing containers across your library interface, you will need two separate libraries. Honestly, for this situation I've found that the easiest solution is just to install the two libraries into different directories (one for each variant -- and you'll want both to be separate from your main library directory). Alternately, you can rename the debug library file and then install it manually.
As a further suggestion -- you're presumably not running this in debug mode very often. It may be worth only compiling and linking the debug version statically into your application, so you don't have to worry about installing multiple dynamic libraries and keeping them straight at runtime.
Give the debug and release versions of the DLL different names and link the correct one through library dependency. Your application then wont start unless it finds the correct DLL.
This is the sort of check you should be doing in your build system. In your build script,
if you're building for release then link against the release library.
if you're building for debug then link against the debug library.
For instance, if you're using make:
release: $(OBJ)
$(CC) $(CXXFLAGS_RELEASE) $(foreach LIB,$(LIBS_RELEASE),-l$(LIB))
debug: $(OBJ)
$(CC) $(CXXFLAGS_DEBUG) $(foreach LIB,$(LIBS_DEBUG),-l$(LIB))
I am writing an C++ application that could be compiled under Linux (gcc 4.3) and Windows (MS VS08 Express).
My application uses third-party libraries,
On Linux , they are compiled as shared libraries, while
on Windows, there are two versions "Debug" and "Release".
I know that debug version provides extra support for debugging ( just like using -ggdb option in linux gcc, right? )
But I found that if my application is in debug version , the libraries must also be in debug version, otherwise the application will crash.
Why is there such a limit? It seems that there are no such limits in the linux world
Thank you very much!
The Debug configuration of your
program is compiled with full symbolic
debug information and no optimization.
Optimization complicates debugging,
because the relationship between
source code and generated instructions
is more complex.
The Release configuration of your
program contains no symbolic debug
information and is fully optimized.
Debug information can be generated in
PDB Files (C++) depending on the
compiler options used. Creating PDB
files can be very useful if you later
need to debug your release version.
Debug vs Release
It is also possible to debug your release build with the compiler flags.
Debugging Release Builds
Heap Layout - Guard Bytes to prevent overwriting
Compilation - Removing Assert/Debug Info
Pointer Support - Buffers around pointers to prevent seg faults
Optimization - Inline Functions
Common Problems When Creating Release Builds
To elaborate on Martin Tornwall. The various libraries linked when in Debug or Release
LIBCPMT.LIB, Multithreaded, static link, /MT, _MT
LIBCPMTD.LIB, Multithreaded, static link, /MTd, _DEBUG, _MT
CRT Libraries
Most likely, the Release and Debug versions are linked against different versions of the C++ Runtime library. Usually, a Debug build links against the "Multithreaded Debug DLL" runtime, whereas a Release build will generally link against "Multithreaded DLL". Loading DLLs whose runtime libraries are mismatched with that of the application will often lead to mysterious crashes.
You could try verifying that all your DLLs build against the same runtime library as your application, regardless of which configuration (Debug or Release) is active. Whether or not this is desirable is another question entirely.
The ability to choose which runtime library to link with enables the application developer to select the best feature set given her application's requirements. For example, a single-threaded application might incur performance penalties as a result of unnecessary thread synchronization if it is linked with a runtime library that is designed with thread safety in mind. Likewise, the consequences of linking a multi-threaded application against a single-threaded runtime could potentially be disastrous.
While I never intentionally link libraries that were built with different compiler settings, there isn't much point in doing so, I only know of the STL classes in the Dinkumware implementation (the one MSFT uses) to cause this problem.
They support a feature called 'iterator debugging' which is turned on by default in the Debug configuration. This adds members to the classes to aid the diagnostic code. Making them larger. This goes bad when you create the object in a chunk of code that was compiled with one setting and pass it to code that was compiled with the opposite setting. You can turn this off by setting the _HAS_ITERATOR_DEBUGGING macro to 0. Which is rather a major loss, the feature is excellent to diagnose mistakes in the way you use STL classes.
Passing objects or pointers between different libraries is always a problem if you don't carefully control the compile settings. Mixing and matching the CRT version and flavor gets you in trouble when you do so. This normally generates a warning from the linker, not sure what you did to not see it. There won't be one if the code lives in a DLL.