When, if ever, can delete and free be used interchangeably in C++?
My concern is as follows: Say there is an incorrect mixup in the use of malloc/ free and
new/ delete (not to mention new[]/ delete[]). However delete and free doing the same thing;
Fortuitously so this goes uncaught in testing. Later this may lead to a crash in production.
How can I enforce some kind of check to prevent this? Can I be warned if the two are mixed up?
If not at compile time, perhaps some code instrumentation at run time? How would I approach
this?
The intention of this question is to find ways to avoid inadvertent mix up in the usages.
The easy way to not get them mixed up is never to use malloc(), then you will never be tempted to call free(). The infrastructure to create to avoid this problem is called "code review", though in this case a quick "grep malloc(" or "grep free(" on the codebase will probably suffice.
Never. If it works it's by sheer accident of implementation. Do not rely on this behavior.
To answer the second question, if you control both malloc/free and operator new/delete, you can stash extra information to associate with pointers returned by both that tell you how they were allocated. When a pointer is passed to free or operator delete, check to see that it was allocated by the appropriate function. If not, assert or raise an exception or do whatever it is you do to report the mismatch.
Usually this is done by allocating extra memory, e.g., given malloc(size) or operator new(size), you allocate size + additional space and shove extra information in there.
The only way you can ensure you never get them mixed up is by either:
Never using malloc/free in the first place, or
Rely on RAII for your memory allocations. Protect every memory allocation in a RAII object which ensures the memory get correctly and consistently freed when it goes out of scope, or wrap the allocation in a smart pointer.
Manually calling delete or free is just an invitation for bugs.
You should always use new and delete in C++, as only new and delete will call the object's constructor and destructor.
If you find you must use both (for instance, if you're interfacing with a C library), thorough code reviews should carefully scrutinize any uses of free() to determine whether or not they correspond to a malloc(), and whether or not they are being used in a C context.
If I had to codify it, I'd put in the style guide something like this:
free() may be called only on a private pointer field of an object.
malloc()ed buffers (or buffers returned from a C API which caller must free()) must be assigned to a private pointer field of an object.
private pointer fields which hold free()-able buffers must only be used for that purpose.
if you use hungarian notation, add a letter for it (and if you don't, don't).
generally free() will be called only in a destructor, with exceptions when the free()-able buffer is replaced during the lifetime of the object. In that case you can call free() on a value recently copied out of a private field during replacement, rather than on the field value directly.
In other words, stick a wrapper around anything that uses malloc/free. This wrapper could be a single template everyone uses, or you could allow smart pointers with the deletor function set to free(). Then in code review, if you see a call to malloc/free anywhere else, it's wrong.
Basically the best way to stop this being a problem is to be on top of your resource handling in general. In C, people do not have a major problem with accidentally calling free() on streams instead of fclose().
You should always use delete or delete[] when freeing things allocated with new. The same goes for malloc and free.
If using free for deleting new:ed classes the destructor won't be properly called.
Also new and delete doesn't necessarily use malloc/free for its allocations so you might end up corrupting your heap as well.
always use delete for things allocated with new, delete [] for things allocated with new [] and free() for things allocated using malloc().
new and new[] allocate from different heaps than malloc() and using the wrong free()/delete will attempt to deallocate from the wrong heap.
Never mix new/delete with new[]/delete[] or with malloc()/free(). Atop of this, the use of malloc()/free() in C++ is questionable at least.
The easiest way to make sure you never do this accidentally is to not to manage memory manually. In C++ there strings and other containers as well as smart pointers to take care of memory management. There simply isn't a need to do this manually. (I'm not sure whether the last time I remember me typing delete really was the last time I typed it. But if my memory doesn't fail me, this must have been 2001 or 2002.)
You could write your own version of the functions which allocate some extra memory in new/new[]/malloc to track which one did the allocation. Then check in delete/delete[]/free that the right function was used to re-claim the memory. You can also check for things like mixing new[] and delete (without the []) this way. You might want to only use these versions in the debug build.
You could try running valgrind on your application to see if it catches anything. The manual specifically mentions its ability to catch uses of the wrong deallocation function for a block of memory. I don't think there's a way to do this at compile time, however.
One way to avoid misusage of malloc/free or new/delete is that DO NOT call these function directly, call them by a wrapper layer. In the wrapper layer, you can manage the pointers distributed by your own hands and guarantee that misusage will get an error explicitly.
Why dont you just count the total number of malloc statments and tally that with the total count of free? Do the same for New and delete. The process can be automated with regular expressions.
One thing I've done relating to this is to reimplement malloc and free
in C++, so that it calls new/delete under the hood. It's not
optimal, but it serves well enough on a smaller project.
Related
I have a background in Java and I'm still not fully used to the concept of pointers and scope, so sorry if the question seems silly.
I read somewhere that I should delete pointers that I have allocated on the Heap. I understand that but should I also delete a pointer that is given to me like this:
#include<dirent.h>
DIR* dir;
struct dirent* entries;
dir= opendir("D:/DIR")
entries= readdir(entries)
// Should I delete the pointers after I'm done with them?
delete entries;
delete dir;
Should I delete the pointers which are assigned from somewhere else or just going out of scope deletes them automatically?
Or is it even right to delete them since I haven't assigned them using new? But then if it's wrong how can I make sure that the memory assigned from the other methods would be deleted after I'm finished using them?
Not necessarily.
The unavoidable rule in C++ is that every new needs to be paired with a delete, and every new[] with a delete[], and that malloc, free, &c. are still available.
So it's tempting to suppose that the memory you get back from readdir is required to be released with an explicit call to delete, but that might not be the case:
It might have been allocated with a new[], or even, malloc!
The library might provide a function that you need to call that releases the memory.
If the library has a persuasion to working well with the C++ standard library, it might provide you with a deleter: which you can pass on construction to a smart pointer such as std::unique_ptr.
I think that (2) is most likely and is the most sensible since different C++ runtime environments might perform new and delete differently. (3) is an extension to this concept and if they support it then use that.
The golden rule is to check the documentation and do what it tells you.
There's no definitive answer to the question, because it always depends on the semantics by which the memory was allocated. For example in the very code example you gave, you must not use delete for deallocation, because opendir is not a C++ function (but a POSIX one) and to properly close it you call closedir. The pointer itself then can be discarded (no deletion requierd, closedir internally does the cleanup). Just make sure you don't use it after free (see: use-after-free-bug).
In general you always have to consult the manual of the function that gives you a pointer, where it's also specified exactly how to deallocate it.
Just to give you the idea:
malloc/calloc/realloc → free
fopen → fclose
X… → XFree
C is not the same as C++, notably for this aspect.
When using some external C (or C++) function provided by some external library, you should read its documentation and follow the "ownership" rules and conventions. For example if you use getline you understand that you need to free like here. If you use opendir you should use closedir. If you use sqlite3_prepare_v2 you'll need to sqlite3_finalize it, etc... Sometimes you'll think in terms of some abstract data type (like here) with a destructor-like function.
When you develop your own (public or "private") function in C, you need to document if it returns heap allocated memory and who (and how) is responsible of free (or releasing) it.
With C++, you also have smart pointers and RAII; so you generally can avoid manual new and delete. Smart pointers are helpful, but not a silver bullet.
So you should explicit and document conventions (and follow the conventions of external libraries and APIs) about ownership. Understanding and defining wisely such conventions is an important task.
Circular references are difficult to handle (consider weak pointers when applicable). I recommend reading about garbage collection concepts and techniques (e.g. the GC handbook), to at least be able to name appropriately your approaches and understand the limitations and power of reference counting. In some cases, you might even explicitly use a garbage collector library or code your own allocator.
Manual memory management is a whole program property, and that is why it is hard. There are even cases (long-living processes doing a lot of allocation) where you need to be afraid of fragmentation.
Tools like valgrind and the address sanitizer (with GCC's instrumentation options or Clang's ones) are practically very helpful to hunt memory leaks and some other memory bugs. Be also aware of ASLR. Take care to understand the virtual address space of your process. On Linux, read proc(5) then try cat /proc/$$/maps and cat /proc/self/maps in some terminal for a useful insight.
Ultimately you should consult the vendor's manual and see if their functions do the cleaning themselves or you need to call another function to do the cleaning etc. In general when talking about raw pointers in C++ you should explicitly release the allocated memory where appropriate. Whether it comes from a function or a new / new[] operator does not make a difference. To avoid this new / delete combination you can utilize smart pointers and the RAII technique.
I know free() won't call the destructor, but what else will this cause besides that the member variable won't be destructed properly?
Also, what if we delete a pointer that is allocated by malloc?
It is implementation defined whether new uses malloc under the hood. Mixing new with free and malloc with delete could cause a catastrophic failure at runtime if the code was ported to a new machine, a new compiler, or even a new version of the same compiler.
I know free() won't call the destructor
And that is reason enough not to do it.
In addition, there's no requirement for a C++ implementation to even use the same memory areas for malloc and new so it may be that you're trying to free memory from a totally different arena, something which will almost certainly be fatal.
Many points:
It's undefined behaviour, and hence inherently risky and subject to change or breakage at any time and for no reason at all.
(As you know) delete calls the destructor and free doesn't... you may have some POD type and not care, but it's easy for someone else to add say a string to that type without realising there are weird limitations on its content.
If you malloc and forget to use placement new to construct an object in it, then invoke a member function as if the object existed (including delete which calls the destructor), the member function may attempt operations using pointers with garbage values
new and malloc may get memory from different heaps.
Even if new calls malloc to get its memory, there may not be a 1:1 correspondence between the new/delete and underlying malloc/free behaviour.
e.g. new may have extra logic such as small-object optimisations that have proven beneficial to typical C++ programs but harmful to typical C programs.
Someone may overload new, or link in a debug version of malloc/realloc/free, either of which could break if you're not using the functions properly.
Tools like ValGrind, Purify and Insure won't be able to differentiate between the deliberately dubious and the accidentally.
In the case of arrays, delete[] invokes all the destructors and free() won't, but also the heap memory typically has a counter of the array size (for 32-bit VC++2005 Release builds for example, the array size is in the 4 bytes immediately before the pointer value visibly returned by new[]. This extra value may or may not be be there for POD types (not for VC++2005), but if it is free() certainly won't expect it. Not all heap implementations allow you to free a pointer that's been shifted from the value returned by malloc().
An important difference is that new and delete also call the constructor and destructor of the object. Thus, you may get unexpected behavior. That is the most important thing i think.
Because it might not be the same allocator, which could lead to weird, unpredictable behaviour. Plus, you shouldn't be using malloc/free at all, and avoid using new/delete where it's not necessary.
It totally depends on the implementation -- it's possible to write an implementation where this actually works fine. But there's no guarantee that the pool of memory new allocates from is the same pool that free() wants to return the memory to. Imagine that both malloc() and new use a few bytes of extra memory at the beginning of each allocated block to specify how large the block is. Further, imagine that malloc() and new use different formats for this info -- for example, malloc() uses the number of bytes, but new uses the number of 4-byte long words (just an example). Now, if you allocate with malloc() and free with delete, the info delete expects won't be valid, and you'll end up with a corrupted heap.
How to check if memory to which pointer p points has been succesfully deallocated?
In few words: you can't.
Check out tools like Valgrind to help you debugging memory leaks issues.
Some other things you should consider:
Use smart pointers so that you don't have do think about memory management,
Set your pointers to 0 after you free them, so that a further delete has no effect,
Use standard classes (vector, ...) instead of rolling your own,
Finally, don't use pointers (actually you almost can)
Sorry, very short answer "You can't"
Use IBM rational purify tool to check correct deallocation of memory.
Define successfully! Define deallocated!
After deallocating memory (whether it is free or delete) you must not use that pointer again. All other assumptions are irrelevant.
After all, you call the C/C++ runtime to deallocate memory, but the C/C++ runtime also calls functions of the operating system to free the page. You could even have a custom memory allocator on top of the C/C++ runtime that e.g. uses caching to implement a faster memory allocation algorithm.
All of these layers may keep the deallocated memory for themselves (because of fragmentation or just because they like to keep it to themselves) or may tell the underlying layer to deallocate it. Anything can happen, just don't use that pointer anymore.
Some tools which are doing static code analysis can point some problems regarding the memory deallocation.
Use valgrind to check whether you have memory leaks
Avoid raw pointers - use smart pointers instead
Exception handling. I.e. try/catch blocks.
In C++, you can safely assume deallocation never fails. Destructors must not throw exceptions, and the actual memory reserved should never fail to be released, so given those two points, nothing can go wrong.
However, if you delete a pointer that has already been deleted, your program will probably crash. This isn't a problem with deallocation itself though - the original delete worked successfully. It's a problem with your program's memory management if it tries to delete a pointer twice, but that's rarely necessary with modern STL and smart pointers like std::vector, std::unique_ptr, etc...
I am working on modifying a relatively large C++ program, where unfortunately it is not always clear whether someone before me used C or C++ syntax (this is in the electrical engineering department at a university, and we EEs are always tempted to use C for everything, and unfortunately in this case, people can actually get away with it).
However, if someone creates an object:
Packet* thePacket = new Packet();
Does it matter whether it is destroyed with delete thePacket; or free(thePacket); ?
I realize that delete calls the destructor while free() does not, but Packet does not have a destructor. I am having a terrible time stuck in a memory management swamp here and I'm thinking this may be one of the many problems.
Yes it does matter.
For memory obtained using new you must use delete.
For memory obtained using malloc you must use free.
new and malloc may use different data structures internally to keep track of what and where it has allocated memory. So in order to free memory, you have to call that corresponding function that knows about those data structures. It is however generally a bad idea to mix these two types of memory allocation in a piece of code.
If you call free(), the destructor doesn't get called.
Also, there's no guarantee that new and free operate on the same heap.
You can also override new and delete to operate specially on a particular class. If you do so, but call free() instead of the custom delete, then you miss whatever special behavior you had written into delete. (But you probably wouldn't be asking this question if you had done that, because you'd know what behaviors you were missing..)
Packet has a destructor, even if you haven't explicitly declared one. It has a default destructor. The default destructor probably doesn't actually do much, but you can't count on that being the case. It's up to the compiler what it does.
new and malloc also may have wildly different implementations. For example, delete is always called in a context where it has perfect information about the size of the data structure it's deleting at compile time. free does not have this luxury. It's possible that the allocator that new is using may not store the bytes at the beginning of the memory area stating how many bytes it occupies. This would lead free to do entirely the wrong thing and crash your program when freeing something allocated with new.
Personally, if getting people to do the right thing or fixing the code yourself is completely impossible, I would declare my own global operator new that called malloc so then free would definitely not crash, even though it would still not call the destructor and be generally really ugly.
In short, it is as bad as undefined behavior.
This is quiet self explanatory.
C Standard ($7.20.3.2/2) - "The free
function causes the space pointed to
by ptr to be deallocated, that is,
made available for further allocation.
If ptr is a null pointer, no action
occurs. Otherwise, if the argument
does not match a pointer earlier
returned by the calloc, malloc, or
realloc function, or if the space has
been deallocated by a call to free or
realloc, the behavior is undefined."
You are absolutely right, it is NOT correct. As you said yourself, free won't call the destructor. Even if Packet doesn't have an explicit destructor, it's using an inherited one.
Using free on an object created with new is like destroying only what a shallow-copy would reach. Deep-destroying NEEDS the destructor function.
Also, I'm not sure objects created with new() are on the same memory map as malloc()'d memory. They are not guaranteed to be, I think.
if someone creates an object:
Packet* thePacket = new Packet();
Does it matter whether is is destroyed with delete thePacket; or free(thePacket); ?
Yes it does matter. free (thePacket) would invoke Undefined Behaviour but delete thePacket would not and we all know Undefined Behaviour may have disastrous consequences.
Additional thanks extend to Daniel Newby for answering my memory usage question (and Martin York for explaining it a bit more). It is definitely the answer I was looking for, but more of my other questions were answered by others.
Thanks everyone
for clearing up all of my concerns. Very pleased to see things running how I expect them to run.
I've run into something that I'm not exactly sure about.
In my program, I'm not using malloc() or free(). I'm making instances of my classes with new and I've made sure each one runs it's destructor when it's delete'd, however, there are no free() calls or even setting their pointers (to things inside a global scope, or other classes) to NULL or 0.
What I mean by "I've made sure", is not that I call each destructor. I only use delete to call on the destructor to run, but I have variables that increase by 1 everytime an object is created, and everytime it's destructor is run. This is how I've made sure the amount of objects I created are equal to the amount of destructors called.
Should I be using malloc() and free() anyway? Should I be NULLing pointers to things that I still want to exist?
A second question is why, when I look at my task manager, does my process never "drop" memory? It used to never stop gaining, and then I started deleting everything properly. Or so I thought.
Wouldn't free() or delete make the memory usage go down?
What practices should I pursue about malloc'ing and free'ing memory with linked lists?
There's rarely a reason to use malloc() and free() in a C++ program. Stick with new and delete. Note that unlike languages with garbage collection, setting a pointer to NULL or 0 in C++ has nothing to do with deallocating the memory.
you should be using delete with a new and free with a malloc. delete will call the class' destructor so you don't have to explicitly call it. The purpose of the destructor is to release and resources the class might have and delete will free the memory as well.
The only time you should explicetly use the destructor is when you have initialized your object through placement new. You should put yourself in a position where the compiler generated code releases your resources -- read this article on the C++ idiom : resource acquisition is initialization.
Also setting the pointer of a class to null does nothing, there is no garbage collector in the background cleaning up your memory. If you don't free dynamic memory in C++ it will be "leaked" memory -- i.e., there are no links to the memory and it will never be reclaimed till the process exits.
p.s., once again do not mix the pairs of the memory allocation functions.
edt: don't implement linked lists, use the containers provided by the Standard template library. If you feel you need better performance use the intrusive containers from boost.
You should use new and delete in preference to malloc()/calloc()/realloc() and free().
If you're creating linked lists you should use std::list. Also, look into std::vector
As far as the apparent memory usage of your application: it's quite likely that memory is not returned to the system until the application exits.
new and delete can more or less to be considered the C++ versions of malloc and free. So stick within one pair or another, i.e. if a pointer was created with new, it should be released with delete, which ensures the destructor call you mentioned. The malloc/free pair are not C++ aware, and just allocate and release a block of memory with no attached constructor/destructor semantics.
Yes indeed I consider it good form to set pointers to NULL or zero when they've been freed or deleted. During debugging I also sometimes set them to something characteristic like 0xdeadbeef so they stand out.
It's likely that the OS "memory usage" is reflecting the entire size of your process' heap, rather than your memory manager's idea of how much memory is allocated. When the allocator discovers that it doesn't have enough heap space, it grows the heap, and this will be reflected in the "memory usage" you're looking at. In theory it would be possible to shrink the heap accordingly on memory release, but this doesn't always happen. Thus you may only see the memory usage grow, and never shrink.
"Should I be using malloc() and free() anyway?"
No, in most cases. Stick with one form of memory management only. Trying to use both means that you will inevitably screw up and delete a malloc()ed item, or free() a newed item, giving a subtle bug. Stick with one form and you fix these bugs in advance.
"Wouldn't free() or delete make the memory usage go down?"
The OS allocates memory in pages, often 4 kiB in size. As long as a single byte of the page is still in use, it will not be returned to the OS. You are probably allocating many small items, but not deallocating all of them. (There are ways to help with a custom new/delete, but they are generally not worth the effort.)
In my program, I'm not using malloc()or free(). I'm making instances of my classes with new and I've made sure each one runs it's destructor when it's delete'd,
That is scary. You should not need to make anything run its destructor. It is allocated (new) it is destroyed (delete) the constructor is run atomatically on new and the destructor is run automatically on delete.
however, there are no free() calls or even setting their pointers (to things inside a global scope, or other classes) to NULL or 0.
There is no need to use malloc/free in C++ code (there are a few situations where you are using C libs that require malloced memory but they are explicitly documented and few).
Technically there is no need to set a pointer to NULL after you delete it.
It is good technique for a variable to go out of scope just after it is deleted so it can not accidently be re-used. If for some reason the pointer variable lives (ie does not go out of scope) for a long time after you call delete then it is usefull to set it to NULL so that it is no accidently re-used.
Should I be using malloc() and free() anyway? Should I be NULLing pointers to things that I still want to exist?
No and No.
Note: C++ unlike Java does not keep track of how many pointers point at an object.
If you have more than one pointer pointing at an object you need to use smart pointers (you should be using smart pointers anyway).
A second question is why, when I look at my task manager, does my process never "drop" memory? It used to never stop gaining, and then I started deleting everything properly. Or so I thought.
The application never releases back to the OS (on most OS's in normal situations).
So the memory will never go down (until the application exits).
Internally the memory management tracks all the frees so that the memory can be re-used.
But if it runs out it will ask the OS for more and thus in the task manager the memory allocation will go up (this will not be returned to the OS).
Wouldn't free() or delete make the memory usage go down?
No.
What practices should I pursue about malloc'ing and free'ing memory with linked lists?
You should use Smart Pointers so you don't need to worry about when to delete an object.
They also make your code exception safe.
But if you using pointers. call delete to remove an element then set is value to NULL (the pointer scope is alive long after the delete is called).
Note: A boost:shared_pointer is very similar to a Java pointer. It tracks the number of pointers and deletes the object when the last reference to the object is destroyed. No need for you to do any deleting (just like Java) and all you actually need to do is call new (just like Java).