Yesterday, I got bit by a rather annoying crash when using DLLs compiled with GCC under Cygwin. Basically, as soon as you run with a debugger, you may end up landing in a debugging trap caused by RtlFreeHeap() receiving an address to something it did not allocate.
This is a known bug with GCC 3.4 on Cygwin. The situation arises because the libstdc++ library includes a "clever" optimization for empty strings. I spare you the details (see the references throughout this post), but whenever you allocate memory in one DLL for an std::string object that "belongs" to another DLL, you end up giving one heap a chunk to free that came from another heap. Hence the SIGTRAP in RtlFreeHeap().
There are other problems reported when exceptions are thrown across DLL boundaries.
This makes GCC 3.4 on Windows an unacceptable solution as soon as your project is based on DLLs and the STL. I have a few options to move past this option, many of which are very time-consuming and/or annoying:
I can patch my libstdc++ or rebuild it with the --enable-fully-dynamic-string configuration option
I can use static libraries instead, which increases my link time
I cannot (yet) switch to another compiler either, because of some other tools I'm using. The comments I find from some GCC people is that "it's almost never reported, so it's probably not a problem", which annoys me even more.
Does anyone have some news about this? I can't find any clear announcement that this has been fixed (the bug is still marked as "assigned"), except one comment on the GNU Radio bug tracker.
Thanks!
The general problem you're running into is that C++ was never really meant as a component language. It was really designed to be used to create complete standalone applications. Things like shared libraries and other such mechanisms were created by vendors on their own. Think of this example: suppose you created a C++ component that returns a C++ object. How is the C++ component know that it will be used by a C++ caller? And if the caller is a C++ application, why not just use the library directly?
Of course, the above information doesn't really help you.
Instead, I would create the shared libraries/DLLs such that you follow a couple of rules:
Any object created by a component is also destroyed by the same component.
A component can be safely unloaded when all of its created objects are destroyed.
You may have to create additional APIs in your component to ensure these rules, but by following these rules, it will ensure that problems like the one described won't happen.
Related
I recently posted a question about stack segmentation and boost coroutines but it seems like the -fsplit-stack approach only works with source files that are compiled with that flag, the runtime breaks down when you branch to another function that has not been compiled with -fsplit-stack. For example
This implies that the runtime uses a function local technique to detect when the current stack has been surpassed. And not a "guard page signal" trick, where the end of the stack always has a guard page which will raise a signal on write or read, telling the runtime to allocate a new stack frame and branch to that.
Then what is the use of this flag? If I link to any other library that has not been built with this, code will break (even libstdc++ and libc), then how is this something people use practically with big projects?
From reading the gcc wiki about split stacks it seems like calling a non split stack function from a split stack function results in an allocation of a 64KB stack frame. Good.
But it seems like calling a non split stack function from a function pointer has not yet been implemented to follow the above scheme.
What use is this flag then? If I proceed to call any virtual function will my program break?
Further from the answer below it seems like clang has not implemented split stacks?
You have to compile boost (at least boost.context and boost.coroutine) with segmeented-stacks support AND your application.
compile boost (boost.context and boost.coroutine) with b2 property segmented-stacks=on (enables special code inside boost.coroutine and boost.context).
your app has to be compiled with -DBOOST_USE_SEGMENTED_STACKS and -fsplit-stack (required by boost.coroutines headers).
see boost.coroutine documentation
boost.coroutine contains an example that demonstrates segmented stacks (in directory coroutine/example/asymmetric/ call b2 toolset=gcc segmented-stacks=on).
regarding your last question GCC Wiki states:
For calls from split-stack code to non-split-stack code, the linker
will change the initial instructions in the split-stack (caller)
function. This means that the linker will have to have special
knowledge of the instructions that the compiler emits. The effect of
the changes will be to increase the required framesize by a number
large enough to reasonably work for a non-split-stack. This will be a
target dependent number; the default will be something like 64K. Note
that this large stack will be released when the split-stack function
returns. Note that I'm disregarding the case of split-stack code in a
shared library calling non-split-stack code in the main executable;
that seems like an unlikely problem.
please note: while llvm supports segmented stacks, clang seams not to provide the __splitstack_<xyz> functions.
First I'd say split stack support is somewhat experimental in nature to begin with. It is not a widely supported thing nor has a single implementation become accepted as the way to go. As such, part of the purpose of it existing in the compiler is to enable research in real use.
That said, one generally wants to use such a feature to enable lots of threads with small stacks, but which can get bigger if they need to. In some applications, the code that runs in these threads can be tightly controlled. E.g. fairly specialized request handlers that do not call general purpose libraries such as Boost. High performance systems work often involves tightening down the constraints on what code is used in a given path and this would be an example thereof. It certainly limits the applicability of the feature, but I wouldn't be surprised if someone is using it in production this way.
Note that similar issues exist with flags such as -fno-exceptions and -fno-rtti . Generally C++ requires compiling everything that goes into an executable with a compatible set of flags. Sometimes one can mix and match, but it is often fragile. This is part of the motivation of building everything from source and hermetic build tools like bazel. Other languages have different approaches to non-source components, especially virtual machine based languages such as Java and the .NET family. In those worlds things like split stacks are decided at a lower-level of compilation, but typically one would not have any control over or awareness of them at the source code level.
I am facing a rather peculiar issue: I have a Qt C++ application that used to work fine. Now, suddenly I cannot start it anymore. No error is thrown, no nothing.
Some more information:
Last line of output when application is started in debug mode with Visual Studio 2012:
The program '[4456] App.exe' has exited with code -1 (0xffffffff).
Actual application code (= first line in main()) is never called or at least no breakpoints are triggered, so debugging is not possible.
The executable process for a few seconds appears in the process list and then disappears again.
Win 7 x64 with latest Windows updates.
The issues simultaneously appeared on two separate machines.
Application was originally built with Qt 5.2.1. Today I test-wise switched to Qt 5.4.1. But as expected no change.
No changes to source code were made. The issue also applies to existing builds of the application.
Running DependencyWalker did not yield anything of interest from my point of view.
I am flat out of ideas. Any pointers on what to try or look at? How can an executable suddenly stop working at all with no error?
I eventually found the reason for this behavior...sort of. The coding (e. g. my singletons) were never the problem (as I expected since the code always worked). Instead an external library (SAP RFC SDK) caused the troubles.
This library depends on the ICU Unicode libraries and apparently specific versions at that. Since I wasn't aware of that fact, I only had the ICU libraries that my currently used Qt version needs in my application directory. The ICU libraries for the SAP RFC SDK must have been loaded from a standard windows path so far.
In the end some software changes (Windows updates, manual application uninstalls, etc.) must have removed those libraries which resulted in that described silent fail. Simply copying the required ICU library version DLLs into my application folder, solved the issue.
The only thing I am not quite sure about, is why this was not visible when tracing the loaded DLLs via DependencyWalker.
"Actual application code (= first line in main()) is never called. So debugging is not possible."
You probably have some static storage initialization failing, that's applied before main() is called.
Do you use any interdependent singletons in your code? Consolidate them to a single singleton if so (remember, there shouldn't be more than one singleton).
Also note, debugging still is possible well for such situation, the trap is ,- for such case as described in my answer -, main()'s bodies' first line is set as the first break point as default, when you start up your program in the debugger.
Nothing hinders you to set breakpoints, that are hit before starting up the code reaches main() actually.
As for your clarification from comments:
"I do use a few singletons ..."
As mentioned above, if you are really sure you need to use a singleton, use actually a single one.
Otherwise you may end up, struggling with undefined order of initialization of static storage.
Anyway, it doesn't matter that much if static storage data depends on each other, provide a single access point to it throughout your code, to avoid cluttering it with heavy coupling to a variety of instances.
Coupling with a single instance, makes it easier to refactor the code to go with an interface, if it turns out singleton wasn't one.
I've just started trying to integrate the Poco C++ Library with our game engine, however every time I link /usr/lib/libPocoFoundation.so my program suddenly has 51 memory leaks. Removing the link option gets rid of all the leaks (none of them are from my code). This happens even if I remove all Poco #includes from my c++ files.
I doubt there are really 51 memory leaks with Poco's Foundation (core) methods - searching their forums didn't reveal anything, and I'm sure other users would notice something this glaring. I think it is more likely a problem with how I'm linking to Poco?
I'm on Ubuntu 11, using Code::Blocks for an IDE, g++ 4.5.2 to build, and fetched Poco 1.3.6p1-1build1 from the ubuntu ppa (sudo apt-get install libpoco-dev libpoco-doc).
Any suggestions on what the problem could be are most welcome! The only thing I can think of is that I'm not linking Poco correctly.
Possibly related: The OP at c-poco-lib-linking-error-when-trying-static-linking-vs9-express mentions something about "Added Preprocessor flags Foundation_EXPORTS POCO_STATIC PCRE_STATIC" - what would that do? I can't see any information on the Poco reference pages about how to correctly link Poco.
You don't say what tool is telling you that you have memory leaks, but I'm going to presume that it is valgrind since you are on Ubuntu.
If the leaks occur even if you aren't calling any POCO methods, then it is likely that these are one-off allocations occurring during the static initialization of the POCO library that for whatever reason are not being torn down later.
While that isn't particularly great behavior on the part of a library, it is not fatal. The leaks that you should be worried about are ones that will occur repeatedly and will gradually consume memory.
I'd recommend using valgrind --gen-suppressions=all to generate suppressions for the one time leaks for now (especially good that you aren't calling any POCO methods). Then, have a look at the POCO library and see if you can figure out why these allocations aren't being unwound at .fini time. If you can, great, let the POCO people have your fixes, and then you can take down your suppression entries. If not, leave them in so these 'false positives' don't interfere with finding truly harmful memory leaks in your code.
Some objects must be deallocated by calling method "release"
Below is an extract from a doc poco
"
DOMObject defines the rules for memory management in this implementation of the DOM. Violation of these rules, which are outlined in the following, results in memory leaks or dangling pointers.
Every object created by new or by a factory method (for example, Document::create*) must be released with a call to release() or autoRelease() when it is no longer needed. "
I know that E&C is a controversial subject and some say that it encourages a wrong approach to debugging, but still - I think we can agree that there are numerous cases when it is clearly useful - experimenting with different values of some constants, redesigning GUI parameters on-the-fly to find a good look... You name it.
My question is: Are we ever going to have E&C on GDB? I understand that it is a platform-specific feature and needs some serious cooperation with the compiler, the debugger and the OS (MSVC has this one easy as the compiler and debugger always come in one package), but... It still should be doable. I've even heard something about Apple having it implemented in their version of GCC [citation needed]. And I'd say it is indeed feasible.
Knowing all the hype about MSVC's E&C (my experience says it's the first thing MSVC users mention when asked "why not switch to Eclipse and gcc/gdb"), I'm seriously surprised that after quite some years GCC/GDB still doesn't have such feature. Are there any good reasons for that? Is someone working on it as we speak?
It is a surprisingly non-trivial amount of work, encompassing many design decisions and feature tradeoffs. Consider: you are debugging. The debugee is suspended. Its image in memory contains the object code of the source, and the binary layout of objects, the heap, the stacks. The debugger is inspecting its memory image. It has loaded debug information about the symbols, types, address mappings, pc (ip) to source correspondences. It displays the call stack, data values.
Now you want to allow a particular set of possible edits to the code and/or data, without stopping the debuggee and restarting. The simplest might be to change one line of code to another. Perhaps you recompile that file or just that function or just that line. Now you have to patch the debuggee image to execute that new line of code the next time you step over it or otherwise run through it. How does that work under the hood? What happens if the code is larger than the line of code it replaced? How does it interact with compiler optimizations? Perhaps you can only do this on a specially compiled for EnC debugging target. Perhaps you will constrain possible sites it is legal to EnC. Consider: what happens if you edit a line of code in a function suspended down in the call stack. When the code returns there does it run the original version of the function or the version with your line changed? If the original version, where does that source come from?
Can you add or remove locals? What does that do to the call stack of suspended frames? Of the current function?
Can you change function signatures? Add fields to / remove fields from objects? What about existing instances? What about pending destructors or finalizers? Etc.
There are many, many functionality details to attend to to make any kind of usuable EnC work. Then there are many cross-tools integration issues necessary to provide the infrastructure to power EnC. In particular, it helps to have some kind of repository of debug information that can make available the before- and after-edit debug information and object code to the debugger. For C++, the incrementally updatable debug information in PDBs helps. Incremental linking may help too.
Looking from the MS ecosystem over into the GCC ecosystem, it is easy to imagine the complexity and integration issues across GDB/GCC/binutils, the myriad of targets, some needed EnC specific target abstractions, and the "nice to have but inessential" nature of EnC, are why it has not appeared yet in GDB/GCC.
Happy hacking!
(p.s. It is instructive and inspiring to look at what the Smalltalk-80 interactive programming environment could do. In St80 there was no concept of "restart" -- the image and its object memory were always live, if you edited any aspect of a class you still had to keep running. In such environments object versioning was not a hypothetical.)
I'm not familiar with MSVC's E&C, but GDB has some of the things you've mentioned:
http://sourceware.org/gdb/current/onlinedocs/gdb/Altering.html#Altering
17. Altering Execution
Once you think you have found an error in your program, you might want to find out for certain whether correcting the apparent error would lead to correct results in the rest of the run. You can find the answer by experiment, using the gdb features for altering execution of the program.
For example, you can store new values into variables or memory locations, give your program a signal, restart it at a different address, or even return prematurely from a function.
Assignment: Assignment to variables
Jumping: Continuing at a different address
Signaling: Giving your program a signal
Returning: Returning from a function
Calling: Calling your program's functions
Patching: Patching your program
Compiling and Injecting Code: Compiling and injecting code in GDB
This is a pretty good reference to the old Apple implementation of "fix and continue". It also references other working implementations.
http://sources.redhat.com/ml/gdb/2003-06/msg00500.html
Here is a snippet:
Fix and continue is a feature implemented by many other debuggers,
which we added to our gdb for this release. Sun Workshop, SGI ProDev
WorkShop, Microsoft's Visual Studio, HP's wdb, and Sun's Hotspot Java
VM all provide this feature in one way or another. I based our
implementation on the HP wdb Fix and Continue feature, which they
added a few years back. Although my final implementation follows the
general outlines of the approach they took, there is almost no shared
code between them. Some of this is because of the architectual
differences (both the processor and the ABI), but even more of it is
due to implementation design differences.
Note that this capability may have been removed in a later version of their toolchain.
UPDATE: Dec-21-2012
There is a GDB Roadmap PDF presentation that includes a slide describing "Fix and Continue" among other bullet points. The presentation is dated July-9-2012 so maybe there is hope to have this added at some point. The presentation was part of the GNU Tools Cauldron 2012.
Also, I get it that adding E&C to GDB or anywhere in Linux land is a tough chore with all the different components.
But I don't see E&C as controversial. I remember using it in VB5 and VB6 and it was probably there before that. Also it's been in Office VBA since way back. And it's been in Visual Studio since VS2005. VS2003 was the only one that didn't have it and I remember devs howling about it. They intended to add it back anyway and they did with VS2005 and it's been there since. It works with C#, VB, and also C and C++. It's been in MS core tools for 20+ years, almost continuous (counting VB when it was standalone), and subtracting VS2003. But you could still say they had it in Office VBA during the VS2003 period ;)
And Jetbrains recently added it too their C# tool Rider. They bragged about it (rightly so imo) in their Rider blog.
OK..... I've done all the reading on related questions, and a few MSDN articles, and about a day's worth of googling.
What's the current "state of the art" answer to this question:
I'm using VS 2008, C++ unmanaged code. I have a solution file with quite a few DLLs and quite a few EXEs. As long as I completely control the build environment, such that all pieces and parts are built with the same flags, and use the same runtime libaries, and no one has a statically linked CRT library, am I ok to pass STL objects around?
It seems like this should be OK, but depending on which article you read, there's lots of Fear, Uncertainty, and Doubt.
I know there's all sorts of problems with templates that produce static data behind the scenes (every dll would get their own copy, leading to heartache), but what about regular old STL?
As long as they ALL use the exact same version of runtime DLLs, there should be no problem with STL. But once you happen to have several around, they will use for instance different heaps - leading to no end of troubles.
We successfully pass STL objects around in our application which is made up from dozens of DLLs. To ensure it works one of our automated tests that runs at every build is to verify the settings for all projects. If you add a new project and misconfigure it, or break the configuration of an existing project, the build fails.
The settings we check are as follows. Note not all of these will cause issues, but we check them for consistency.
#defines
_WIN32_WINNT
STRICT
_WIN32_IE
NDEBUG
_DEBUG
_SECURE_SCL
Compiler options
DebugInformationFormat
WholeProgramOptimization
RuntimeLibrary
We use stl collections in our application and pass them to and from methods in different dlls (usually as references). This doesn't cause any trouble.
The only area where we have had trouble is where one dll allocates memory and another dll tries to delete it. This only is reported as bad, but I am not sure why. However it only seems to be a problem on Debug builds (where it is reported), but still works on release builds. Having said that where ever I come across this I do fix it.
If I was writing a 3rd party library I would think twice about using stl parameters in the api. Previously (VC6) we had to use the OCI (Oracles C api) as opposed to OCCI (Oracles C++ api) because it only worked with the Microsoft STL implementation and we were using stlport. Of course if you enable your clients to build the library with their own stl implementation this is not an issue.