Heap (HeapAlloc) Corruption in release mode only
I can just guess but why wouldn't you check the result of HeapFree? Because according the documentation it could be the reason. Try to use HEAP_NO_SERIALIZE flag when you allocate heap.
You should not refer in any way to memory that has been freed by HeapFree. After that memory is freed, any information that may have been in it is gone forever. If you require information, do not free memory containing the information. Function calls that return information about memory (such as HeapSize) may not be used with freed memory, as they may return bogus data. Calling HeapFree twice with the same pointer can cause heap corruption, resulting in subsequent calls to HeapAlloc returning the same pointer twice.
Serialization ensures mutual exclusion when two or more threads attempt to simultaneously allocate or free blocks from the same heap. There is a small performance cost to serialization, but it must be used whenever multiple threads allocate and free memory from the same heap. Setting the HEAP_NO_SERIALIZE value eliminates mutual exclusion on the heap. Without serialization, two or more threads that use the same heap handle might attempt to allocate or free memory simultaneously, likely causing corruption in the heap. The HEAP_NO_SERIALIZE value can, therefore, be safely used only in the following situations:
The process has only one thread.
The process has multiple threads, but only one thread calls the heap functions for a specific heap.
The process has multiple threads, and the application provides its own mechanism for mutual exclusion to a specific heap.
Related
I have this code:
#include <malloc.h>
int main()
{
int x = 1000;
//~600kb memory at this point
auto m = (void**)malloc(x * sizeof(void*));
for (int i = 0; i < x; i++)
m[i] = malloc(x * sizeof(void*));
for (int i = 0; i < x; i++)
free(m[i]);
free(m);
//~1700kb memory at this point
return 0;
}
When program starts memory consumption is about ~600kb, and when it ends ~1700kb. Is it memory leak or what?
malloc() acquires memory from the system using a variety of platform-specific methods. But free() does not necessarily always give memory back to the system.
One reason the behavior you're seeing might exist is that when you first call malloc() it will ask the system for a "chunk" of memory, say 1 MB as a hypothetical example. Subsequent calls will be fulfilled from that same 1 MB chunk until it is exhausted, then another chunk will be allocated.
There is little reason to immediately return all allocated memory to the system just because the application is no longer using it. Indeed, the most likely possibilities are that (a) the application requests more memory, in which case the recently freed pieces can be doled out again, or (b) the application terminates, in which case the system can efficiently clean up all its allocated memory in a single operation.
Is it memory leak or what?
No. You have each malloc matching a free and if I dont miss something you have no memory leak.
Note that what you see in the process manager is the memory assigned to the process. This is not necessarily equal to the memory actually in use by the process. The OS does not immediately reclaim it when you call free.
If you have a more complex code you can use a tool like valgrind to inspect it for leaks. Though, you better dont use manual memory managment at all, but use std containers instead (eg std::vector for dynamic arrays).
free() and malloc() are part of your Standard Library and their behavior is implementation dependent. The standard does not require free() to release once acquired memory back to your Operating System.
How the memory is reserved is plattform specific. On POSIX systems the mmap() and munmap() system calls can be used.
Please Note that most Operating Systems implement Paging, allocating memory in chunks to processes anyway. So releasing each single byte would only pose a performance overhead.
When you are running your application under management of operating system, the process of memory allocation in languages like C/C++ is two-fold. First, there is a business of actually requesting memory to be mapped into process from operating system. This is done through OS-specific call, and an application needs not bothering about it. On Linux this call is either sbrk or mmap, depending on several factors. However, this OS-requested memory is not directly accessible to C++ code, it has no control over it. Instead, C++ runtime manages it.
The second thing is actually providing a usable pointer to C/C++ code. This is also done by runtime when C++ code asks for it.
When C/C++ code calls malloc(or new), C++ runtime first determines, if it has enough continuous memory under it's management to provide a pointer to it to application code. This process is further complicated by efficiency optimizations inside malloc/new, which usually provide for several allocation arenas, used depending on object sizes, but we won't talk about it. If it has, the memory region will be marked as used by application, and a pointer will be returned to the code.
If there is no memory available, a chunk of memory would be requested from the OS. The size of the chunk normally will be way bigger than what was requested - because requesting from OS is expensive!
When the memory is deleted/freed, C++ runtime reclaims it. It might decide to return memory back to OS (for example, if there is a pressure in terms of total memory consumption on the machine), or it can just store it for future use. Many times memory will never be returned to the OS until process exits, so any OS-level tool will show process consuming memory, even though the code delete'd/freed it all!
It is also worth noting that usually application can request memory from OS bypassing C++ runtime. For example, on Linux it could be done through mmap call or, in bizarre cases, through sbrk - but in latter case you won't be able to use runtime memory management at all after this. If you use those mechanisms, you will immediately see process memory consumption growing or decreasing with every memory request or return.
When I use the top terminal program at Linux, I can't see the result of free.
My expectation is:
free map and list.
The memory usage that I can see at the top(Linux function) or /proc/meminfo
get smaller than past.
sleep is start.
program exit.
But
The usage of memory only gets smaller when the program ends.
Would you explain the logic of free function?
Below is my code.
for(mapIter = bufMap->begin(); mapIter != bufMap -> end();mapIter++)
{
list<buff> *buffList = mapIter->second;
list<buff>::iterator listIter;
for(listIter = buffList->begin(); listIter != buffList->end();listIter++)
{
free(listIter->argu1);
free(listIter->argu2);
free(listIter->argu3);
}
delete buffList;
}
delete bufMap;
printf("Free Complete!\n");
sleep(10);
printf("endend\n");
Thanks you.
Memory is allocated onto a heap.
When you request some memory in your program (with a new() or malloc() etc.) Your program requests some memory from its heap, which in turn requests it from the operating system{1}. Since this is an expensive operation, it gets a chunk of memory from the OS, not just what you ask for. The memory manager puts everything it gets into the heap, just returning to you the perhaps small amount you asked for. When you free() or delete() this memory, it simply gets returned to the heap, not the OS.
It's absolutely normal for that memory to not be returned to the operating system until your program exits, as you may request further memory later on.
If your program design relies on this memory be recycled, it may be achievable using multiple copies of your program (by fork()~ing) which run and exit.
{1} The heap is probably non-empty on program start, but assuming it's not illustrates my point.
The heap typically uses operating system functions to manage its memory. The heap’s
size may be fixed when the program is created, or it may be allowed to grow. However,
the heap manager does not necessarily return memory to the operating system when
the free function is called. The deallocated memory is simply made available for subsequent use by the application. Thus, when a program allocates and then frees up memory, the deallocation of memory is not normally reflected in the application’s memory
usage as seen from the operating system perspective.
Imagine that I use C++11 threads. The thread will run a function that do malloc. After that I will use join without free (the memory). So, I killed the thread. It is expected that the memory frees automatically?
No, it is not. The memory is freed only after the whole application is terminated. The whole benefit of using multiple threads (as opposed to processes) is that they share the same memory, so they collectively own all the memory allocated in one of them.
How can I free the virtual memory that is left up after calling TerminateThread? Can it be done via VirtualFree and how of course. I fully understand the "Dangers" of TerminateThread.
In an unmanaged process, there's no realistic way to tidy up memory from the outside.
Memory can be allocated in many different ways. Ultimately it all starts with calls to VirtualAlloc, VirtualAllocEx etc. But in practice runtime libraries invariably use sub allocating heap managers. These heap allocators will get memory by calls to VirtualAlloc, but then will hand out sub-blocks. And heap managers are generally shared between threads in a process. So you've no way from the outside of knowing how to free those sub-blocks.
And even if we did not have sub-allocators, how could you know which blocks handed out by VirtualAlloc you were allowed to destroy? A thread may allocate memory with a call to VirtualAlloc and require that the memory out lives the allocating thread and is destroyed by another thread.
But if you are happy to let all of that go, and just want the stack to be destroyed (as per your comments), then this article shows you how to do so with RtlFreeUserThreadStack: http://www.nicklowe.org/2012/01/thread-termination-dont-leak-the-stack/
I am using malloc_stats() to print malloc related statistics in which I am finding "Arena 0" for some programs and "Arena 0 and Arena 1" for some other programs.
What do these arenas represent?
The heap code resides inside the glibc component, and is packaged in the libc.so.x shared library. The current implementation of the heap uses multiple independent sub-heaps called arenas. Each arena has its own mutex for concurrency protection. Thus if there are sufficient arenas within a process' heap, and a mechanism to distribute the threads' heap accesses evenly between them, then the potential for contention for the mutexes should be minimal. It turns out that this works well for allocations. In malloc(), a test is made to see if the mutex for current target arena for the current thread is free (trylock). If so then the arena is now locked and the allocation proceeds. If the mutex is busy then each remaining arena is tried in turn and used if the mutex is not busy. In the event that no arena can be locked without blocking, a fresh new arena is created. This arena by definition is not already locked, so the allocation can now proceed without blocking. Lastly, the ID of the arena last used by a thread is retained in thread local storage, and subsequently used as the first arena to try when malloc() is next called by that thread. Therefore all calls to malloc() will proceed without blocking.
See link text. It looks like heap is a collection of arenas ("sub-heaps") to handle memory allocation between several threads, thus reducing contention.
In certain malloc implementations, an "arena" is a pool of memory from which individual allocations are made. The algorithms to determine which arena is used will differ between implementations, so it's not possible for us to explain why you see a difference. One common factor is allocation size.
Everything is there: http://www.gnu.org/software/libc/manual/html_node/Statistics-of-Malloc.html
int arena
This is the total size of memory allocated with sbrk by malloc, in bytes.