RtlValidateHeap Problem - c++

I have a Windows DLL that at one point, returns a pointer to a class that is new'ed in the DLL's code. The class itself is a very thin wrapper around another class private to the DLL.
The calling executable has no problem working with this class, everything works fine, with the exception that when the calling executable tries to delete this class, I get a RtlValidateHeap error.
The reasons behind the error makes sense, the exe is trying to free memory on the DLL's heap, which is normally a bad thing.
I've come up with a few possible solutions:
Override the class's new operator to
allocate its memory off of the
executable's heap (provided I can
even get at that heap space). The wrapper is very thin, so I would only be allocating a few bytes from the exe's heap.
Provide a special destruction
function for this class (yuck).
Tell the user not to destroy the
class and live with leaks (no way!)
Is there a 'normal' way of doing this?

Most libraries that use this sort of functionality, whether C or C++, actually go with your second option - specific functions to get and release objects. It allows distribution of binary DLLs that don't have to be compiled identically to the calling executable.

I've seen this behaviour is when one of the binaries was built with debug options (which uses a debug heap that, amongst other things, uses RtlValidateHeap) and the other binary was built as a release binary. It might be worth checking that you only use like with like...

By far the easiest solution is (for the Visual Studio users) to use /MD or /MDd everywhere. If the CRT itself is in a DLL, there's only one heap. Your app and your DLL will share it.

Are you loading the DLL dynamically at runtime (by calling LoadLibrary), or are you linking to using a .lib file during the link phase of your exe? If you're dynamically loading/unloading the DLL then when the delete happens is the DLL still loaded? This is a classic problem for plug-in type designs and the typical solution is something like your second proposal - specific functions to create and delete the objects in question.

Option 2 is pretty normal, eg, a Release() method that just does a "delete this".
Note that normally the DLL doesn't have it's own heap as such - most DLLs will share the process heap (see GetProcessHeap), but often the language runtime will decorate or add padding around allocated memory - so the pointer your dll ends up with is not the same pointer the heap will expect in a call to free.
The general rule is always release allocated memory with the matching free function, from within the same module.
There are some exceptions though - memory you know has been allocated directly with HeapAlloc from a known heap can be freed from another module. GlobalAlloc/GlobalFree works across modules. Strings allocated with SysAllocString can be freed from another module. Memory you know has been allocated by the same DLL version of the runtime can be freed from another module, but I'd stay away from this.

On the DLL side, I would use something along these lines:
dll.h: /* all exported, included by dll user */
class dll_class {
public:
virtual void foo() = 0;
}
dll_class* create_class();
void delete_class(dll_class*);
dll.cpp: /* implement the creator and destroyer */
dll_class* create_class() { return new dll_class_imp(); }
void delete_class(dll_class* pClass) { delete pClass; }
dll_imp.h: /* implement the actual dll_class logic */
class dll_class_imp : public dll_class {
public:
virtual void foo() {...}
}
Now, on the DLL user's side, dll.h will be included, and used like so:
dll_class* pClass = create_class();
pClass->foo();
delete_class(pClass);
this way, there's a clear distinction between the DLL's part and the user's part. Being an abstract class, the user cannot create a dll_class using new; and it only makes sense you delete object similarly to the way you've created them, i.e. with the exported functions.

What looks like will work for us was to override the new and delete operators for the problem class. It turned out that there were 3 situations that could happen:
EXE new's object, EXE delete's object
DLL new's object, DLL delete's object
DLL new's object, EXE delete's object
The new and delete operators ensure that the DLL would be allocating and freeing all memory allocations. So far, so good - testing shows that the operators are always being used and the memory is coming from the DLL's heap.

Related

Pass data between EXE/DLL

I have DLL and EXE app. DLL is registered in EXE. Then in EXE instance of class A is created as shared_ptr. Next, this instance is passed to the DLL and method example of this instance is called on DLL side.
class Result
{
//Some attributes e.g std::vector
}
class A
{
public:
A(){}
~A(){}
Result example(){
Result r;
//Fill data in r object....
return r;
}
}
The problem is that when example is called somewhere in DLL :
void someDLLMethod()
{
//a is a shared pointer of A class
{
Result r = a->example();
}//PROBLEM IS HERE DURING DEALLOCATING Result OBJECT
//some stuff...
}
It seems that the problem occurs during deleting r object. I understand when I will try to allocate some memory in EXE and then try to deallocate this memory in DLL then memory issues can occur, but in this case, Result object is copied from EXE to DLL. Also, vector stored in Result object doesn't contain any pointers so it should be deep copy of vector.
One solution is to allocate Result object in DLL, pass it as reference to called method and fill all necessary data, but I want to get copy as result.
Why this problem occurs? How it is possible to fix it? EXE and DLL are in this same solution of Visual studio and must be compiled with \MT or \MTd and there is no possibility to change it.
Memory for the vector is being allocated in the DLL and free'd in the exe as the vector is probably using move semantics in its return (or even just RVO) which just moves the pointer to the allocated internals. As you're using a static CRT this means the memory is being allocated with one crt and free'd with a different instance which leads to badness.
Short story - don't do this. Exposing C++ objects across DLL boundaries is bad. You should use a C interface or an interface that is designed for this - e.g. WinRT.
If you do choose to expose c++ objects across a DLL interface you must make sure that both the EXE and the DLL use the same version of the compiler and you link with /MD. If you can't use /MD you'll never get this to work reliably.
Sure you could pre-allocate the vector and fill it in in the call but that's just masking the issue until things inevitably change down the line.

Dll exporting a constructor risks causing heap corruption

I am working on a program with a plugin system which allows users to develop their own modules in the form of DLL files.
The modules are supposed to use objects which are defined in a DLL imported by all the components of the application. Here is what a sample object would look like:
#include <boost/system/api_config.hpp>
#if defined BOOST_WINDOWS_API
#ifdef EXPORT
#define API __declspec(dllexport)
#else
#define API __declspec(dllimport)
#endif
#else
#define API
#endif
class A
{
public:
API A();
API virtual ~A();
};
All DLLs are built statically (with their own CRT) and with the exact same compilation flags. I know that exchanging objects through DLL boundaries can get hairy, so I use boost::shared_ptrs almost everywhere. There is one difficulty around the object constructors though: if I create the object on the stack (from a different DLL), everything works as expected. But if I use the operator new, the heap gets corrupted whet the object is deleted.
A a; // Works fine, no problem when the object goes out of scope.
A* b = new A();
delete b; // Causes heap corruption!
What is the proper way around this? I feel that the code would be less readable if I had to define a method in the object's DLL such as A* A::create() { return new A(); }. In the worst scenario, I'm thinking of making the new operator private, to make sure that the user won't use it.
Based on your responsed to my comments: /MT and /MTd don't
work well with DLLs (even if they are the default). If you want
to use DLLs and dynamic allocation, you must use /MD or
/MDd. When you use /MT or /MTd, you effectively tell the
system to use a separate heap for each DLL. Which means that
allocating in one, and deleting in another will corrupt the
heap. And when the destructor is virtual, the actual delete
will be in the destructor, not in the delete expression.
(The actual issue is with malloc and free, called by the
operator new() and operator delete() functions.)
The classical way of working around this is to use factory
methods for dynamic allocation and a static or a member function
for delete. An alternative (not tried, but I think it would
work) is to defined non-inline operator new() and operator
delete() members, which just forward to malloc and free.
(In the case of operator new, of course, you have to check
that the pointer you get from malloc isn't null, and throw an
std::bad_alloc if it is.)
But these are just work-arounds for a situation which shouldn't
exist. Everything works fine with /MD or /MDd, and this is
what you should be using (even if it means you can't legally
deploy the debug version on machines which don't have a license
for Visual Studios).
I think the problem could be related to the fact that you are mixing heaps. When you build your DLL with all CRT statically linked in it means that it maintains its own heap. However the commands new and delete from your host process are using the process heap. I think this configuratin can cause issues. I think the best way is to add two methods to your DLL: CreateA() and DestroyA(), and use only them for allocation/destruction of heap objects from DLL A.
The solution is to simply never export a constructor from a DLL. Export a factory function instead. Never export a concrete class while you're at it. Export pure abstract classes only. Which is a good style anyway, for reduced coupling and all associated goodies.

Mixing versions of the MSVCRT

So, I have a C++ library with a statically linked copy of the MSVCRT. I want for anyone to be able to use my library with any version of the MSVC Runtime. What is the best way to accomplish this goal?
I'm already being quite careful with how things are done.
Memory never passes the DLL barrier to be freed
Runtime C++ objects aren't passed across barriers (ie, vectors, maps, etc.. unless they were created on that side of the barrier)
No file handles or resource handles are passed between barriers
Yet, I still have some simple code that causes heap corruption.
I have an object like so in my library:
class Foos
{
public: //There is an Add method, but it's not used, so not relevant here
DLL_API Foos();
DLL_API ~Foos();
private:
std::map<std::wstring, Foo*> map;
};
Foos::~Foos()
{
// start at the begining and go to the end deleting the data object
for(std::map<std::wstring, Foo*>::iterator it = map.begin(); it != map.end(); it++)
{
delete it->second;
}
map.clear();
}
And then I use it from my application like so:
void bar() {
Foos list;
}
After I call this function from anywhere, I get a debug warning about stack corruption. And If I actually let it run out, it actually does corrupt the stack and segfault.
My calling application is compiled with Visual Studio 2012 platform tools. The library is compiled using Visual Studio 2010 platform tools.
Is this just something I should absolutely not be doing, or am I actually violating the rules for using multiple runtimes?
Memory never passes the DLL barrier
But, it does. Many times in fact. Your application created the storage for the class object, in this case on the stack. And then passes a pointer to the methods in the library. Starting with the constructor call. That pointer is this inside the library code.
What goes wrong in a scenario like this one is that it didn't create the correct amount of storage. You got the VS2012 compiler to look at your class declaration. It uses the VS2012 implementation of std::map. Your library however was compiled with VS2010, it uses a completely different implementation of std::map. With an entirely different size. Huge changes thanks to C++11.
This is just complete memory corruption at work, code in your application that writes stack variables will corrupt the std::map. And the other way around.
Exposing C++ classes across module boundaries are filled with traps like that. Only ever consider it when you can guarantee that everything is compiled with the exact same compiler version and the exact same settings. No shortcuts on that, you can't mix Debug and Release build code either. Crafting the library so no implementation details are exposed is certainly possible, you have to abide by these rules:
Only expose pure interfaces with virtual methods, argument types must be simple types or interface pointers.
Use a class factory to create the interface instance
Use reference counting for memory management so it is always the library that releases.
Nail down core details like packing and calling convention with hard rules.
Never allow exceptions to cross the module boundary, only use error codes.
You'd be well on you way to write COM code by then, also the style you see used in for example DirectX.
map member variable is still created by application with some internal data allocated by application and not DLL (and they may use different implementations of map). As a rule of thumb don't use stack objects from DLLs, add something like Foos * CreateFoos() in your DLL.
Runtime C++ objects aren't passed across barriers (ie, vectors, maps,
etc.. unless they were created on that side of the barrier)
You are doing exactly that. Your Foos object is being created by the main program on the stack and then being used in the library. The object contains a map as a part of it...
When you compile the main program it looks at the header files etc to determine how much stack space to allocate for the Foos object. And the calls the constructor which is defined in the library... Which might have been expecting an entirely different layout/size of the object
It may not fit your needs, but don't forgot that implementing the whole thing in header files simplifies the problem (sort of) :-)

operator "new" returning a non-local heap pointer for only one class?

Language : C++
Platform : Windows Server 2003
I have an exe calling a DLL.
EDIT :
(exe is not doing anything, it calls few global function which does everything related to DLL within DLL. It does not explicitly new any of DLL classes)
I allocate (new) the memory for class A within the DLL, it returns me a non-local heap pointer.
I tried to new other classes within the DLL (which are in DLL), "new" returns a valid heap pointer for them, its only Class A which is not being allocated properly.
I am on windows and validating the heap by this function call :
EDIT:
ClassA* pA = new ClassA();
_CrtIsValidHeapPointer ( (const void *) pA )
I am seriously confused why this only happens with new-ing Class A and no other class ?
(All Native Code)
FINAL EDIT :
it turned out to be a bad build. can't get it to reproduce in the new one... wasted 5 days on this :'(
Export only global functions from DLLs and call only exported functions and through v-tables. Your problem is only one of many caused by trying to export entire classes. DLLs aren't the same as .so libraries.
EDIT: Since the question now reveals that the class isn't exported, but all of this is observed inside the single DLL, going to suggest other possible causes.
Is the same pointer returned from new being tested, or could it be a pointer to a base sub-object? Did the allocation use new or new []? Either of these could cause the pointer to point into the middle of a heap block instead of the start.
If Class A overrides operator new then the memory for that class could be allocated in the DLL's copy of the C runtime. This would cause _CrtIsValidHeapPointer to return false - because your version of the C runtime library is using a different heap than the one in the DLL.
The value returned from _CrtIsValidHeapPointer is generally not reliable anyway (certainly, even though it may return TRUE it doesn't necessarily mean you can use the pointer). Why do you do this anyway?
Perhaps the class declares its own operator new that get storage from some mystery location?
e.g. The author of the class may have written:
class MyClass {
public:
void * operator new (size_t amt);
void operator delete(void *);
};
Then their operator new beats your operator new.
Either that or the DLL is built with /MTd while the executable uses /MD.

Using shared_ptr in dll-interfaces

I have an abstract class in my dll.
class IBase {
protected:
virtual ~IBase() = 0;
public:
virtual void f() = 0;
};
I want to get IBase in my exe-file which loads dll.
First way is to create following function
IBase * CreateInterface();
and to add the virtual function Release() in IBase.
Second way is to create another function
boost::shared_ptr<IBase> CreateInterface();
and no Release() function is needed.
Questions.
1) Is it true that the destructor and memory deallocation is called in the dll (not in exe-file) in the second case?
2) Does the second case work well if exe-file and dll was compiled with different compilers (or different settings).
An answer to your first question: The virtual destructor in your dll is called - the information about its location is embedded in your object (in the vtable). In the case of memory deallocation it depends how disciplined the users of your IBase are. If they know they have to call Release() and consider that exception can bypass the control flow in an surprising direction, the right one will be used.
But if CreateInterface() returns shared_ptr<IBase> it can bind the right deallocation function right to this smart pointer. Your library may look like this:
Destroy(IBase* p) {
... // whatever is needed to delete your object in the right way
}
boost::shared_ptr<IBase> CreateInterface() {
IBase *p = new MyConcreteBase(...);
...
return shared_ptr<IBase>(p, Destroy); // bind Destroy() to the shared_ptr
} // which is called instead of a plain
// delete
Thus every user of your DLL is easily prevented against resource leaks. They never have to bother about calling Release() or pay attention to exceptions bypassing surprisingly their control flow.
To answer your second question: The downside of this approach is clearly stated by the other answers: You're audience has to use the same compiler, linker, settings, libraries as you. And if they could be quite a lot this can be major drawback for your library. You have to choose: Safety vs. larger audience
But there's a possible loophole: Use shared_ptr<IBase>in your application, i.e.
{
shared_ptr<IBase> p(CreateInterface(), DestroyFromLibrary);
...
func();
...
}
Thus no implementation specific object is passed across the DLL boundary. Nevertheless your pointer is safely hidden behind the shared_ptr, who's calling DestroyFromLibrary at the right time even if func()'s throwing an exception or not.
I would advise against using shared_ptr in the interface. Even using C++ at all in the interface of a DLL (as opposed to "extern C" only routines) is problematic because name-mangling will prevent you from using the DLL with a different compiler. Using shared_ptr is particularly problematic because, as you have already identified, there is no guarantee that the client of the DLL will use the same implementation of shared_ptr as the caller. (This is because shared_ptr is a template class and the implementation is contained entirely in the header file.)
To answer your specific questions:
I'm not quite sure what you are asking here... I'm assuming that your DLL will contain implementations of classes derived from IBase. The code for their destructors (as well as the rest of the code) will, in both of your cases, be contained in the DLL. However, if the client initiates the destruction of the object (by calling delete in the first case or by letting the last instance of the shared_ptr go out of scope in the second case), then the destructor will be called from client code.
Name-mangling will usually prevent your DLL from being used with a different compiler anyway... but the implementation of shared_ptr may change even in a new release of the same compiler, and that can get you into trouble. I would shy away from using the second option.
Using shared_ptr will make sure the resource releasing function will be called in the DLL.
Have a look at the answers to this question.
A way out of this problem is to create a pure C interface and a thin fully inlined C++ wrapper around it.
On your first question: I'm taking an educated guess and not speaking from experience, but it seems to me that the second case the memory deallocation will be called "in the .exe". There are two things that happen when you call delete object;: first, destructors are called and second, the memory for the object is freed. The first part, destructor calling, will definitely work as you expect, calling the right destructors in your dll. However, since shared_ptr is class template, its destructor is generated in your .exe, and therefore it will call operator delete() in your exe and not the one in the .dll. If the two were linked against different runtime versions (or even statically linked against the same runtime version) this should lead to the dreaded undefined behavior (this is the part I'm not entirely sure about, but it seems logical to be that way). There's a simple way to verify if what I said is true - override the global operator delete in your exe, but not your dll, put a breakpoint in it and see what's called in the second case (I'd do that myself, but I have this much time for slacking off, unfortunately).
Note that the same gotcha exist for the first case (you seem to realize that, but just in case). If you do this in the exe:
IBase *p = CreateInterface();
delete p;
then you are in the same trap - calling operator new in the dll and calling operator delete in the exe. You'll either need a corresponding DeleteInterface(IBase *p) function in your dll or a Release() method in IBase (which doesn't have to be virtual, just not make it inline) for the sole purpose of calling the right memory deallocation function.