I am opening an audio file to read it and I get an abandoned malloc block from this caller each time.
In a loop I set data like this (which is marked as the memory usage in instruments as 99.7%) data = (short*)malloc(kSegmentSize*sizeof(short));
and free it like this free(data); at the end of each iteration.
Im not really sure what is happening here and would appreciate any help.
EDIT: KSegmentSize varies in the thousands, from minimum 6000 - max 50000 (speculative)
Instruments trace:
Not having the exact code:
Pretty sure you're having this problem b/c something between the malloc and free is throwing (and you're probably catching it already so you don't exit the loop). Depending on if this is happening in C (or objective-C) or C++ code, you have slightly different methods of resolution.
In C++, wrap the malloc/free in the RAII pattern so that when the stack is unwound the free is called.
class MyData {
public:
A(size_t numShorts) : dataPtr(0) { dataPtr = malloc(numShorts * sizeof(short)); }
~A() { free(dataPtr); }
operator short*() { return dataPtr; }
private:
short* dataPtr;
}
MyData data(numShorts);
// do your stuff, you can still use data as you were before due the 'operator short*'
// allow the dtor to be called when you go out of scope
In Objective-C you need to use a finally block:
void* myPtr = 0;
#try { myPtr = malloc(...); }
#catch {}
#finally { free(myPtr); }
Suggest that you start by simplifying, for example comment out (preferably using #if 0) all of the code except the malloc/free. Run the code and ensure no abandoned heap blocks. Then gradually re-introduce the remaining code and re-run until you hit the problem, then debug.
Sorry to answer my own question, but after commenting out code back up the stack trace the actual issue was to do with the file not be disposed.
Calling ExtAudioFileDispose(audioFile); solved this hidden bug. Instruments was not entirely clear and marked mallocs as the leak. To be fair the mallocs where from data that was within the file referenced by the ExtAudioOpenFile method, not disposing the file reference left a leak.
Related
what happens to data created in local scope of thread if thread is terminated, memory leak?
void MyThread()
{
auto* ptr = new int[10];
while (true)
{
// stuff
}
// thread is interrupted before this delete
delete[] ptr;
}
Okay, my perspective.
If the program exits, the threads exit wherever they are. They don't clean up. But in this case you don't care. You might care if it's an open file and you want it flushed.
However, I prefer a way to tell my threads to exit cleanly. This isn't perfect, but instead of while (true) you can do while (iSHouldRun) and set the field to false when it's time for the thread to exit.
You can also set a flag that says, iAmExiting at the end, then myThread.join() once the flag is set. That gives your exit code a chance to clean up nicely.
Coding this from the beginning helps when you write your unit tests.
The other thing -- as someone mentioned in comments -- use RAII. Pretty much if you're using raw pointers, you're doing something you shouldn't do in modern C++.
That's not an absolute. You can write your own RAII classes. For instance:
class MyIntArray {
MyArray(int sizeIn) { ... }
~MyArray() { delete array; }
private:
int * array = nullptr;
int size = 0;
};
You'll need a few more methods to actually get to the data, like an operator[]. Now, this isn't any different than using std::vector, so it's only an example of how to implement RAII for your custom data, for instance.
But your functions should NEVER call new like this. It's old-school. If your method pukes somehow, you have a memory leak. If it pukes on exit(), no one cares. But if it pukes for another reason, it's a problem. RAII is a much, much better solution than the other patterns.
In googletest you can use ASSERT_THROW to test that a certain function throws an error. For instance
ASSERT_THROW(PhysicalPropertyResource p("other/identifier72652"), InappropriateResourceException);
How would you explicitely call another method for releasing some memory used by PhysicalPropertyResource? Normally used like so:
PhysicalPropertyResource p("other/identifier72652");
p.free();
Or should I just not worry about the memory leak since its only in a test and therefore benign. (Its just my OCD to want to keep valgrind completly happy).
I've tried this:
ASSERT_THROW(
PhysicalPropertyResource p("other/identifier72652");
p.free();
, InappropriateResourceException);
}
which doesn't free the memory.
The proper syntax for executing multiple instructions would to wrap them in a block:
ASSERT_THROW({PhysicalPropertyResource p("other/identifier72652");
p.free();},
InappropriateResourceException);
However, this won't help it's not the last line that throws, because then p.free() will not be called. You could create your own "assertion"
void shouldThrow() {
std::unique_ptr<PhysicalPropertyResource> p;
try {
p = std::make_unique<PhysicalPropertyResource>("other/identifier72652");
} catch (const InappropriateResourceException&) {
if (p) //note that this is not possible if it is really constructor that throws
p->free();
return; //test succeeded
} catch (...) {
if (p)
p->free();
FAIL();
}
if (p)
p->free;
FAIL();
}
Note: If you are writing C++ code that uses C library, then your best solution is creation of a RAII wrapper that will take care of resources. If you create a wrapper class, it can free resources in destructor and you will never leak them. You can even write such wrapper for unit tests only, just to avoid this ugly assertion-like code.
I developed a WDM filter driver on disk driver. I want to send an asynchronous request to write data on disk. The windows will crash when I delete the writeBuffer memory in WriteDataIRPCompletion function.
My question is: How can I safely free the writeBuffer memory without crashing?
This my send request code:
#pragma PAGEDCODE
NTSTATUS WriteToDeviceRoutine() {
PMYDRIVER_WRITE_CONTEXT context = (PMYDRIVER_WRITE_CONTEXT)ExAllocatePool(NonPagedPool,sizeof(PMYDRIVER_WRITE_CONTEXT));
context->writeBuffer = new(NonPagedPool) unsigned char[4096];
PIRP pNewIrp = IoBuildAsynchronousFsdRequest(IRP_MJ_WRITE,
pdx->LowerDeviceObject,
context->writeBuffer,(wroteRecordNodeCount<<SHIFT_BIT),
&startingOffset,NULL);
IoSetCompletionRoutine(pNewIrp,WriteDataIRPCompletion,context,TRUE,TRUE,TRUE);
IoCallDriver(pdx->LowerDeviceObject,pNewIrp);
}
This is my completion routine code:
#pragma LOCKEDCODE
NTSTATUS WriteDataIRPCompletion(IN PDEVICE_OBJECT DeviceObject,IN PIRP driverIrp,IN PVOID Context) {
PMDL mdl,nextMdl;
KdPrint((" WriteDataIRPCompletion \n"));
PMYDRIVER_WRITE_CONTEXT writeContext = (PMYDRIVER_WRITE_CONTEXT) Context;
if(driverIrp->MdlAddress!=NULL){
for(mdl=driverIrp->MdlAddress;mdl!=NULL;mdl = nextMdl) {
nextMdl = mdl->Next;
MmUnlockPages(mdl);
IoFreeMdl(mdl);
KdPrint(("mdl clear\n"));
}
driverIrp->MdlAddress = NULL;
}
delete [] writeContext->writeBuffer;
if(Context)
ExFreePool(Context);
KdPrint(("leave WriteDataIRPCompletion \n"));
return STATUS_CONTINUE_COMPLETION;
}
you error in next line
context = ExAllocatePool(NonPagedPool,sizeof(PMYDRIVER_WRITE_CONTEXT));
when must be
context = ExAllocatePool(NonPagedPool,sizeof(MYDRIVER_WRITE_CONTEXT));
not sizeof(PMYDRIVER_WRITE_CONTEXT) but sizeof(MYDRIVER_WRITE_CONTEXT) you allocate not structure but pointer to it.
this not produce error only if your MYDRIVER_WRITE_CONTEXT containing single field writeBuffer and no more data. otherwise you overwrite allocated memory (which is only sizeof(PVOID)) and this create bug
and about completion for IoBuildAsynchronousFsdRequest. unfortunately documentation not very good. here sated that
Before calling IoFreeIrp, an additional step is required to free the
buffer for an IRP built by IoBuildAsynchronousFsdRequest if the
following are all true:
The buffer was allocated from system memory pool.
but then all attention for
The Irp->MdlAddress field is non-NULL.
however we must check and for IRP_DEALLOCATE_BUFFER|IRP_BUFFERED_IO, without this we can leak Irp->AssociatedIrp.SystemBuffer. need next code
if (Irp->Flags & IRP_BUFFERED_IO)
{
if (Irp->Flags & IRP_INPUT_OPERATION)
{
if (!NT_ERROR(Irp->IoStatus.Status) && Irp->IoStatus.Information)
{
memcpy( Irp->UserBuffer, Irp->AssociatedIrp.SystemBuffer, Irp->IoStatus.Information );
}
}
if (Irp->Flags & IRP_DEALLOCATE_BUFFER)
{
ExFreePool(Irp->AssociatedIrp.SystemBuffer);
Irp->AssociatedIrp.SystemBuffer = 0;
}
Irp->Flags &= ~(IRP_DEALLOCATE_BUFFER|IRP_BUFFERED_IO);
}
and check for if (writeContext) after use writeContext->writeBuffer already senseless and nosense. really you need do check for context != NULL yet in WriteToDeviceRoutine()
I'm not too familiar with the specifics of what you're working with, so here're a few details that caught my attention.
In WriteDataIRPCompletion function
PMYDRIVER_WRITE_CONTEXT writeContext = (PMYDRIVER_WRITE_CONTEXT) Context;
// ...
delete [] writeContext->writeBuffer;
if(Context)
ExFreePool(Context);
Notice that your writeContext originates from your Context argument. However, you seem to be deleting/freeing the allocated memory twice.
The ExFreePool function docs state:
Specifies the address of the block of pool memory being deallocated.
It looks like the delete [] writeContext->writeBuffer; line might be causing the problem and it just needs to be removed.
As it is right now, part of the memory that should be freed by the function has already been manually deleted by the time you invoke ExFreePool, but not set to NULL, which in turn causes ExFreePool to receive a now-invalid pointer (i.e. a non-null pointer pointing to de-allocated memory) in its Context argument, causing the crash.
In WriteToDeviceRoutine function
The documentation for ExFreePool explicitly states that it deallocates memory that has been allocated with other functions, such as ExAllocatePool and other friends.
However, your code is trying to allocate/deallocate the writeContext->writeBuffer directly using the new/delete operators respectively. It seems like you should be allocating your memory with ExAllocatePool and then deallocating with ExFreePool instead of trying to do things manually like that.
These functions may be organizing the memory in a specific way and if/when this pre-condition is not met in ExFreePool, it could end up in a crash.
On a separate note, it seems odd that you check if(Context) is null before invoking ExFreePool, but not above before you try to type-cast for your local writeContext variable and use it.
Maybe you should also check at that first point of use? If Context is always non-null, then the check might also be unnecessary prior to invoking ExFreePool.
I have overloaded new function but unfortunetly never been able to execute global handler for requesting more memory access on my compiler. I also don't understand as per below code snippet if we invoke the
global handler for requesting more memory how it is gling to allocate to P.
I appreciate if anybody can through some light on this
void * Pool:: operator new ( size_t size ) throw( const char *)
{
int n=0;
while(1)
{
void *p = malloc (100000000L);
if(p==0)
{
new_handler ghd= set_new_handler(0);//deinstall curent handler
set_new_handler(ghd);// install global handler for more memory access
if(ghd)
(*ghd)();
else
throw "out of memory exception";
}
else
{
return p;
}
}
}
To have any effect, some other part of the program must have installed a global handler previously. That handler must also have some kind of memory to release when the handler is called (perhaps some buffers or cache that can be discarded).
The default new_handler is just a null pointer, so your code is very likely to end up throwing an exception.
Also, I would have thrown a bad_alloc exception to be consistent with other operator new overloads.
Here are two things to discuss, the first is using new_handler, the second is overloading operator new.
set_new_handler()
When you want use a new_handler, you have to register it. It is typically the first thing to do after entering main(). The handler should also be provided by you.
#include <iostream>
#include <new>
void noMemory() throw()
{
std::cout << "no memory" << std::endl;
exit(-1);
}
int main()
{
set_new_handler(noMemory);
// this will probably fail and noMemory() will be called
char *c = new char[100000000L];
std::cout << "end" << std::endl;
}
When no memory can be allocated, your registered handler will be called, and you have the chance to free up some memory. When the handler returns, operator new will give another try to allocate the amount of memory you requested.
operator new
The structure of the default operator new is something similar you presented. From the point of the new_handler the important part is the while(1) loop, since it is responsible for trying to get memory after called the new_handler.
There is two way out of this while(1) loop:
getting a valid pointer
throwing an exception
You have to have this in mind when you provide a new_handler, because if you can not do anything to free up memory you should deinstall the handler (or terminating or throwing an exception), otherwise you can stuck in an endless loop.
I guess omitting parameter size in your code is just for test purpose.
Also see Scott Meyers' Effective C++ Item 7 for details. As operator new must return a valid pointer even with parameter size = 0, the first thing to do in your operator new should be overwriting size to 1 in case of the user want to allocate 0 number of bytes. This trick is simple and fairly effective.
I have a leak in my application and I've come to reduce my code to the following and it's leaking about 12kb per iteration. I cannot see if this is a problem with my code or a problem with the xerces library itself. But looking at the Private Bytes in Perfmon I can only see growth and no shrinkage, so it's obviously leaking.
Can someone please advice what could be wrong with the following code that causes it to leak at such an incredible rate?:
(single threaded test app)
for (int x = 0; x < 1000000; x++){
DataSerializer* ds = new DataSerializer();
ds->test(request);
ds->releasedocument();
ds->destroy_xml_lib();
delete ds;
}
void DataSerializer::test(std::string& request)
{
impl = initialize_impl();
}
DOMImplementation* DataSerializer::initialize_impl()
{
try
{
boost::mutex::scoped_lock init_lock(impl_mtx);
XMLPlatformUtils::Initialize();
return DOMImplementationRegistry::getDOMImplementation(XConv("Core"));
}
catch(const XMLException& toCatch)
{
char *pMsg = XMLString::transcode(toCatch.getMessage());
std::string msg(pMsg);
XMLString::release(&pMsg);
}
return NULL;
}
void DataSerializer::destroy_xml_lib()
{
boost::mutex::scoped_lock terminate_lock (impl_mtx); //is being used in MT app
XMLPlatformUtils::Terminate();
}
void DataSerializer::releasedocument()
{
if (document){
document->release();
document = NULL;
}
}
I don't understand how this could possibly leak? What have I missed?
Where does impl get deleted?
I know nothing more about the API than googling the docs, but they suggest to me that you should not be calling Terminate() - in a real program, other code elsewhere, possibly in other threads, may still be using the xerces library.
The DOMImplementation is returned as a pointer and has a destructor - clear indications you have to manage its lifetime. It seems a really likely story that that is your memory leak.
Furthermore, that the DOMImplementationRegistry::getDOMImplementation() can return NULL so you have to guard against that.
If you can run this on Linux,use Valgrind (Purify is a commercial equivalent for windows)
Not sure where you allocate document.
In the ReleaseDocument() function you don't delete it. All you do is set it to zero after clearing its content.
PS: don't know xerces either.