I am trying to implement classes implementing the wav playing, as explained in this example. The relevant code part is here :
/* Setup for conversion */
wav_cvt.buf = malloc(wav_len * wav_cvt.len_mult);
wav_cvt.len = wav_len;
memcpy(wav_cvt.buf, wav_buf, wav_len);
/* We can delete to original WAV data now */
SDL_FreeWAV(wav_buf);
/* And now we're ready to convert */
SDL_ConvertAudio(&wav_cvt);
When a wav file finishes playing (I am not going to play it again), do I need to free the memory buffer that is malloc()-ed above? Or is it done automatically somewhere?
No, nothing is done automatically. You must free it.
Remember that C (and anything of it's implementation) doesn't manage dynamic memory allocation automatically, whenever you have allocated some pieces of memory (mark the memory offset as USED), you should free() it when you are done to remark that offset as UNUSED. But that's not MUST!!!.
Any malloc is generally free'd elsewhere by the same module. I say generally because you may never intend to give the memory back for performance or persistence reasons. Furthermore, memory allocations will be reclaimed by the operating system when the process terminates regardless You're not endangering the system.
Since you malloced the buf, you should actually free it yourself. Save SDL_FreeWav for wave buffers passed to you by SDL that you are done with (such as from SDL_LoadWav).
Internal to SDL_LoadWav, will be a malloc call by SDL. SDL_FreeWav is a wrapper around the corresponding free. This allocate/deallocate function pairing is common as some libraries may implement custom memory management routines that resemble or wrap malloc and free. They may even open up new heap contexts that are not accessible from the standard functions, and intended to be private. There's not even a requirement that the memory be allocated on a heap, but this is orthogonal to your question.
It's likely that SDL_FreeWav is just a straight free, but when a library provides deallocation functions you should prefer those in case the behaviour differs.
When in doubt, always call the deallocation routine if you believe you're done with the memory resource. Double free errors are noisy and generally generate stack traces that will let you quickly identify the problem. Other libraries such as glib will usually have built-in diagnostics that will alert you to overzealous deallocation. Deallocating aggressively also aids in locating logical errors: If you think you're done with the memory, but some other part of the program isn't, the resource use will need to be re-examined.
Related
How to detect programmatically count of bytes allocated by process on Heap?
This test should work from process itself.
I think mallinfo() is what you want:
#include <malloc.h>
struct mallinfo *info;
info = mallinfo();
printf ("total allocated space: %llu bytes\n", info->uordblks);
printf ("total free space: %llu bytes\n", info->fordblks);
The struct mallinfo structure is technical, and specific to the malloc() implementation. But the information you want is in there. Here is how I report the values:
mallinfo.arena = "Total Size (bytes)"
mallinfo.uordblks = "Busy size (bytes)"
mallinfo.fordblks = "Free size (bytes)"
mallinfo.ordblks = "Free blocks (count)"
mallinfo.keepcost = "Top block size (bytes)"
mallinfo.hblks = "Blocks mapped via mmap() (count)"
mallinfo.hblkhd = "Bytes mapped via mmap() (bytes)"
These two are allegedly not used, but they seem to change on my system, and thus might be valid:
mallinfo.smblks = "Fast bin blocks (count)"
mallinfo.fsmblks = "Fast bin bytes (bytes)"
And the other interesting value is returned by "sbrk (0)"
There are a number of possibilities.
How accurate do you need it to be? You can get some useful data via cat /proc/${PID}/status | grep VmData.
You can #define your own malloc(), realloc(), calloc(), and free() functions, wrapping the real functions behind your own counter. You can do really cool things here with __FILE__, __LINE__, & __func__ to facilitate identifying core leaks in simple tests. But it will only instrument your own code!
(Similarly, you can also redefine the default operator new and operator delete methods, both array and non-array variants, and both throwing std::bad_alloc and std::nothrow_t variants. Again, this will only instrument your own code!)
(Be aware: On most C++ systems, new ultimately invokes malloc(). It doesn't have to. Especially with in-place new! But typically new does make use of malloc(). (Or it operates on a region of memory that has previously been malloc()'ed.) Otherwise you'd get into really funky stuff with multiple heap managers...)
You can use sbrk(0) to see where the data segment is currently set. That's not so great. It's a very coarse measurement, and it doesn't account for holes (unused memory regions) in the heap. (You're much better off with the VmData line from /proc/${PID}/status.) But if you're just looking for a general idea...
You can trap malloc()/free()/etc by writing your own shared library and forcing your process to use it instead of the real versions via LD_PRELOAD. You can use dlopen()/dlsym() to load & invoke the *real* malloc()/free()/etc. This works quite beautifully. The original code is unmodified, not even recompiled. But be aware of re-entrant situations when coding this library, and that your process will initially invoke malloc()/calloc()/realloc() before dlopen()/dlsym() can complete loading the real functions.
You might check out tools like Valgrind, though that's really aimed more at memory leaks.
Then again, perhaps mtrace() is what you want? Or __malloc_hook? Very proprietary (GNU) & nonstandard... But you are tagged "Linux"...
There's no easy, automatic way to do it, if that's what you're asking. You basically have to manually keep track of heap allocations yourself using a counter variable. The problem is that it's difficult to control which parts of your program are allocating memory on the heap, especially if you're using a lot of libraries out of your control. To complicate things further, there's two ways a program might allocate heap memory: new or malloc. (Not to mention direct OS calls like sbrk.)
You can override global operator new, and have each call to new increase a global tally. However, this won't necessarily include times when your program calls malloc, or when your program uses some class-specific new override. You can also override malloc using a macro, but this is not necessarily portable. And you'd also have to override all the variations of malloc, like realloc, calloc, etc. All of this is further complicated by the fact that on some implementations, new itself may call malloc.
So, essentially, it's very difficult to do this properly from within your program. I'd recommend using a memory profiler tool instead.
A speculative solution: redefine new and delete operators.
On each new operator call, a number of bytes to allocate is passed. Allocate a bit more memory and store the amount of bytes allocated within. Add this amount to global variable that holds the heap size.
On delete operator call, check the value you stored before you dispose the memory. Subtract it from that global variable.
If you're on windows, you can use GetProcessHeap(), HeapQueryInfo() to retrieve information about the processes heap. An example of walking the heap from MSDN
Since you've tagged your question 'linux' it might help to look at some of the information provided in the /proc directory. I haven't researched this a lot so I can only give you a starting point.
/proc/<your programs pid> contains files with some information about your process from the viewpoint of the kernel. There is a symlink /proc/self that will always about the process your investigating this from.
The files you might be most interested in are stat, statm and status. The latter is more human-readable, whereas the former two give the same info in a more machine-readable format.
A starting point about how to interpret the content of those files is available in the proc(5) manpage.
Other then keeping track of all your memory allocations I don't believe there is a way to calculate the heap size or usage
Use malloc_info(). Have fun with the XML aspect of it. mallinfo() (as shown in another answer) does a similar thing but is restricted to 32-bit values... a foolish thing to rely on in 2010+.
When specifying a VkAllocationCallbacks struct to vkCreate* functions, I would like to use only the vulkan notification without overwriting the real allocators, but I can't find how.
From https://www.khronos.org/registry/vulkan/specs/1.1-extensions/man/html/VkAllocationCallbacks.html
pfnAllocation must be a valid pointer to a valid user-defined PFN_vkAllocationFunction
pfnReallocation must be a valid pointer to a valid user-defined PFN_vkReallocationFunction
pfnFree must be a valid pointer to a valid user-defined PFN_vkFreeFunction
if either of pfnInternalAllocation or pfnInternalFree is not NULL, both must be valid callbacks
It seems weird to me that I can't just pass a nullptr to use the default allocations, is there any rationals for this ?
Maybe that it is possible to query the default allocators at runtime, but I haven't found any way to do it, I'd be glad to know if it is possible in a portable way.
You may not be fully understanding the difference between these sets of functions.
The first set of functions, the allocation functions, are used by Vulkan to allocate CPU memory... most of the time. The other two functions, the internal notification functions, are for the other times.
See, there are times when the Vulkan implementation needs to make OS-specific system calls to allocate memory. The Vulkan specification recognizes such a case: to allocate "executable memory": memory containing opcodes for the CPU's instruction set. For security reasons, OS's don't let you just execute random memory addresses; modern OS's require most apps to allocate memory in a special way in order for it to be executable. Therefore:
The application is not expected to handle allocating memory that is intended for execution by the host due to the complexities of differing security implementations across multiple platforms. The implementation will allocate such memory internally and invoke an application provided informational callback when these internal allocations are allocated and freed.
Normal memory allocation functions like ::operator new or malloc can't do that. And the Vulkan specification does not expect the client code to be able to make those system calls either.
However, the client code may need to be able to track such allocations, so that it can know how much memory the Vulkan implementation is keeping around. Therefore, when "internal allocations" are made/freed, the internal notification functions are called.
This is the only time such functions are called.
So if your goal is just to track when implementation allocations are being made/freed, the internal notification functions alone are not going to get the job done. You have to override them all, which means you'll need to actually do the allocation/reallocation/freeing.
As it seems that it's not possible, I would simply use a hand written allocator fulfilling the required alignement (with for exemple ::operator new in C++17).
This allocator is very likely to be less performant than the Vulkan default one, but if it is only needed for debugging purposes it should do the trick.
How do memory leak detectors actually work? What are the underlying concepts in general? Can take C++ as the language to explain this.
There are a couple of different ways that leak detectors work. You can replace the implementation of malloc and free with ones that can track more information during allocation and are not concerned with performance. This is similar to how dmalloc works. In general, any address that is malloc'ed but not free'd is leaked.
The basic implementation is actually pretty simple. You just maintain a lookup table of every allocation and its line number, and remove the entry when it is freed. Then when the program is done you can list all leaked memory. The hard part is determining when and where the allocation should have been freed. This is even harder when there are multiple pointers to the same address.
In practice, you'll probably want more than just the single line number, but rather a stack trace for the lost allocations.
Another approach is how valgrind works which implements an entire virtual machine to keep track of addresses and memory references and associated bookkeeping. The valgrind approach is much more expensive, but also much more effective as it can also tell you about other types of memory errors like out of bounds reads or writes.
Valgrind essentially instruments the underlying instructions and can track when a given memory address has no more references. It can do this by tracking assignments of addresses, and so it can tell you not just that a piece of memory was lost, but exactly when it became lost.
C++ makes things a little harder for both types of leak detectors because it adds the new and delete operators. Technically new can be a completely different source of memory than malloc. However, in practice many real C++ implementations just use malloc to implement new or have an option to use malloc instead of the alternate approach.
Also higher level languages like C++ tend to have alternative higher level ways of allocating memory like std::vector or std::list. A basic leak detector would report the potentially many allocations made by the higher level modes separately. That's much less useful than saying the entire container was lost.
Here's a published technical paper on how our CheckPointer tool works.
Fundamentally it tracks the lifetimes of all values (heap and stack), and their sizes according their types as defined by the language. This allows CheckPointer to find not only leaks, but out-of-array bound accesses, even for arrays in the stack, which valgrind won't do.
In particular, it analyzes the source code to find all pointer uses.
(This is quite the task just by itself).
It keeps track of pointer meta data for each pointer, consisting of
A reference to the object meta data for the heap-allocated object or global or local variable orfunction pointed to by the pointer and
The address range of the (sub)object of the object that the pointer may currently access. This may be smaller than the address range of the
whole object; e.g. if you take the address of a struct member, the instrumented source code will only allow access to that member when using the resulting pointer.
It also tracks the kind and location of each object, i.e. whether
it is a function, a global, thread-local or local variable, heap-allocated memory, or a string literal constant:
The address range of the object that may be safely accessed, and
For each pointer stored in the heap-allocated object or variable, a reference to the pointer metadata for that pointer.
All this tracking is accomplished by transforming the original program source, into a program which does what the original program does, and interleaves various meta-data checking or updating routines. The resulting program is compiled and run. Where a meta-data check fails at runtime, a backtrace is provided with a report of the type of failure (invalid pointer, pointer outside valid bounds, ...)
This is tagged C and C++ and no operating system is mentioned. This answer is for Windows.
C
Windows has the concept of virtual memory. Any memory a process can get is virtual memory. This is done through VirtualAlloc() [MSDN]. You can imagine the leak detector to put a breakpoint on that function and whenever it is called, it gets the callstack and saves it somewhere. Then it can do similar for VirtualFree()[MSDN].
The difference can then be identified and shown along with the callstacks that have been saved.
C++
C++ has a different concept: it takes the large 64kb blocks which it gets from VirtualAlloc() and splits it into smaller pieces, called the Heap. The C++ heap manager comes from Microsoft and offers new methods HeapAlloc() [MSDN] and HeapFree()[MSDN].
Then, you could do the same as before, but actually, that feature is already built-in. Microsoft's GFlags [MSDN] tool can enable the tracking:
In this case it will save up to 50 MB of callstack information for C++ heap manager calls.
Since that settings can also be enabled via the Windows Registry, a memory leak detector can make use of it easily.
General concept
As you can see, the general concept is to keep track of allocations and deallocations, compare them and show the callstacks of the difference.
This question is as in title:
Is it possible to produce a memory leak without using any kernel specific means like malloc, new, etc?
What if I will make a linked list inside a function with lot of elements in there, and after it I'll exit from this function without cleaning a list. The list will be created without using any malloc calls, i.e.
struct list_head {
struct list_head *next, *prev;
}
Can it be guaranteed that all resources will be freed after exiting from this function? So I can freely execute it a million times and nothing will be leaked?
Subject: If you not using any particular malloc or new calls you won't get a heap memory leak. Never. Is that right?
A leak is always connected to a resource. A resource is by definition something that you acquire manually, and that you must release manually. Memory is a prime example, but there are other resources, too (file handles, mutex locks, network connections, etc.).
A leak occurs when you acquire a resource, but subsequently lose the handle to the resource so that nobody can release it. A lesser version of a leak is a "still-reachable" kind of situation where you don't release the resource, but you still have the handle and could release it. That's mostly down to laziness, but a leak by contrast is always a programming error.
Since your code never acquires any resources, it also cannot have any leaks.
The variables you applied without malloc or new is located at stack
space in the memory. So when the function returned, the variable is
taken back.
On the other hand, the memory you applied with malloc or new is
located at heap space. The system doesn't care whether you release the
space or not. In this situation, if you don't use free or delete,
memory leak will happen.
Subject: If you not using any particular malloc or new calls you won't get a heap memory leak. Never. Is that right?
That assumption is not entirely correct. The problem is that the operating system itself (or other third party components you have to rely on) can have memory leaks as well. In that case you might not actively call malloc, but call other (operating system) functions which could leak.
So your assumption depends on how strongly you consider such a thing. You can argue that the OS/third party implementation is outside your domain, then this assumption would be correct. If you have a well defined system and your requirements are such that you have to garuantee a certain uptime, something like this may have to be considered as well.
So the answer to this question ...
Is it possible to make memory leak without using malloc?
... is:
Yes, it is possible.
malloc() allocates memory from the heap, while space for string and struct literals (string1, string2 and those list_head's) will be reserved at compile time at the stack.
Actually any memory allocated for a program (heap or stack) will be reclaimed by the kernel when the process exits (at *nix system at least).
I would define memory leak as allocating memory on heap and without freeing it when your program exits. This definition actually answers your question.
There are standard functions (like strdup) that will allocate memory on heap, beware of them.
Another example of a resource that you can allocate and forget to free:
If you're using OpenGL, and you call glGenBuffers() a million times without the corresponding glDeleteBuffers calls, it's extremely likely that you will run out of VRAM and your graphics driver will start leaking to system memory.
I just had this happen. Fortunately, Visual Studio's memory profiler made it pretty easy to find. It showed up as a large number of allocations made by the external process nvoglv32.dll, which is my NVIDIA OpenGL driver.
Where can I read about sbrk() in some detail?
How does it exactly work?
In what situations would I want to use sbrk() instead of the cumbersome malloc() and new()?
btw, what is the expansion for sbrk()?
Have a look at the specification for brk/sbrk.
The call basically asks the OS to allocate some more memory for the application by incrementing the previous "break value" by a certain amount. This amount (the first parameter) is the amount of extra memory your application then gets.
Most rudimentary malloc implementations build upon the sbrk system call to get blocks of memory that they split up and track. The mmap function is generally accepted as a better choice (which is why mallocs like dlmalloc support both with an #ifdef).
As for "how it works", an sbrk at its most simplest level could look something like this:
uintptr_t current_break; // Some global variable for your application.
// This would probably be properly tracked by the OS for the process
void *sbrk(intptr_t incr)
{
uintptr_t old_break = current_break;
current_break += incr;
return (void*) old_break;
}
Modern operating systems would do far more, such as map pages into the address space and add tracking information for each block of memory allocated.
sbrk is pretty much obsolete, these days you'd use mmap to map some pages out of /dev/zero. It certainly isn't something you use instead of malloc and friends, it's more a way of implementing those. Also, of course, it exists only on posix-based operating systems that care about backwards compatibility to ancient code.
If you find Malloc and New too cumbersome, you should look into garbage collection instead... but beware, there is a potential performance cost to that, so you need to understand what you are doing.
You never want to use sbrk instead of malloc or free. It is non-portable and is typically used only by implementers of the standard C library or in cases where it's not available. It's described pretty well in its man page:
Description
brk() sets the end of the
data segment to the value specified by
end_data_segment, when that value is
reasonable, the system does have
enough memory and the process does not
exceed its max data size (see
setrlimit(2)).
sbrk() increments the program's data
space by increment bytes. sbrk() isn't
a system call, it is just a C library
wrapper. Calling sbrk() with an
increment of 0 can be used to find the
current location of the program break.
Return Value
On success, brk() returns
zero, and sbrk() returns a pointer to
the start of the new area. On error,
-1 is returned, and errno is set to ENOMEM.
Finally,malloc and free are not cumbersome - they are the standard way to allocate and release memory in C. Even if you want to implement your own memory allocator, it's best to just use malloc and free as the basis - a common approach is to allocate a large chunk at a time with malloc and provide memory allocation from it (this is what suballocators, or pools, usually implement)
Re the origin of the name sbrk (or its cousin brk), it may have something to do with the fact that the end of the heap is marked by a pointer known as the "break". The heap starts right after the BSS segments and typically grows up towards the stack.
You've tagged this C++ so why would you use 'cumbersome' malloc() rather than new? I am not sure what is cumbersome about malloc in any case; internally maybe so, but why would you care? And if you did care (for reasons of determinism for example), you could allocate a large pool and implement your own allocator for that pool. In C++ of course you can overload the new operator to do that.
sbrk is used to glue the C library to the underlying system's OS memory management. So make OS calls rather than using sbrk(). As to how it works, that is system dependent. If for example you are using the Newlib C library (commonly used on 'bare-metal' embedded systems with the GNU compiler), you have to implement sbrk yourself, so how it works in those circumstances is up to you so long as it achieves its required behaviour of extending the heap or failing.
As you can see from the link, it does not do much and would be extremely cumbersome to use directly - you'd probably end-up wrapping it in all the functionality that malloc and new provide in any case.
This depends on what you mean by malloc being "Cumbersome". sbrk is typically not used directly anymore, unless you're implementing your own memory allocator: IE, operator overriding "new". Even then I'd possibly use malloc to give me my initial memory.
If you'd like to see how to to implement malloc() on top of sbrk(), check out http://web.ics.purdue.edu/~cs354/labs/lab6/ which is an exercise going through that.
On a modern system you shouldn't touch this interface, though. Since you're calling malloc and new cumbersome, I suspect you don't have all the requisite experience to safely and properly use sbrk for your code.