Is there a way to allocate memory on stack instead of heap? I can't find a good book on this, anyone here got an idea?
Use alloca() (sometimes called _alloca() or _malloca() ), but be very careful about it — it frees its memory when you leave a function, not when you go out of scope, so you'll quickly blow up if you use it inside a loop.
For example, if you have a function like
int foo( int nDataSize, int iterations )
{
for ( int i = 0; i < iterations ; ++i )
{
char *bytes = alloca( nDataSize );
// the memory above IS NOT FREED when we pass the brace below!
}
return 0;
} // alloca() memory only gets freed here
Then the alloca() will allocate an additional nDataSize bytes every time through the loop. None of the alloca() bytes get freed until you return from the function. So, if you have an nDataSize of 1024 and an iterations of 8, you'll allocate 8 kilobytes before returning. If you have an nDataSize= 65536 and iterations = 32768, you'll allocate a total 65536×32768=2,147,483,648 bytes, almost certainly blowing your stack and causing a crash.
anecdote: You can easily get into trouble if you write past the end of the buffer, especially if you pass the buffer into another function, and that subfunction has the wrong idea about the buffer's length. I once fixed a rather amusing bug where we were using alloca() to create temporary storage for rendering a TrueType font glyph before sending it over to GPU memory. Our font library didn't account for the diacritic in the Swedish Å character when calculating glyph sizes, so it told us to allocate n bytes to store the glyph before rendering, and then actually rendered n+128 bytes. The extra 128 bytes wrote into the call stack, overwriting the return address and inducing a really painful nondeterministic crash!
Since this is tagged C++, typically you just declare the objects you need in the correct scope. They are allocated on the stack, and guaranteed to be released on scope exit. This is RAII, and a critical advantage of C++ over C. No mallocs or news, and especially no allocas, required.
You can declare a local char[1024] or whatever number of bytes you'd like (up to a point), then take the address of the local for a pointer to this block of memory on the stack. Not exactly dynamic, but you could then wrap up this memory with your own memory manager if desired.
See _malloca.
Article discussing about dynamic memory allocation
We can allocate variable length space dynamically on stack memory by
using function
_alloca. This function allocates memory from the program stack. It simply takes number of bytes to be allocated and return void* to the
allocated space just as malloc call. This allocated memory will be
freed automatically on function exit.
So it need not to be freed explicitly. One has to keep in mind about
allocation size here, as stack overflow exception may occur. Stack
overflow exception handling can be used for such calls. In case of
stack overflow exception one can use _resetstkoflw() to restore it
back.
So our new code with _alloca would be :
int NewFunctionA()
{
char* pszLineBuffer = (char*) _alloca(1024*sizeof(char));
…..
// Program logic
….
//no need to free szLineBuffer
return 1;
}
When/if C++ allows the use of (non-static) const values for array bounds, it will be easier.
For now, the best way I know of is via recursion. There are all kinds of clever tricks that can be done, but the easiest I know of is to have your routine declare a fixed-sized array, and fill and operate on what it has. When its done, if it needs more space to finish, it calls itself.
You could use the BDE C++ library, e.g.
const int BUFFER_SIZE = 1024;
char buffer[BUFFER_SIZE];
bdlma::BufferedSequentialAllocator allocator(buffer, BUFFER_SIZE);
bsl::vector<int> dataVector(&allocator);
dataVector.resize(50);
BDE supplies comprehensive allocator options along with collections like bsl::vector that can use polymorphic allocators without changing the type of the container.
You might also consider:
https://github.com/facebook/folly/blob/master/folly/docs/small_vector.md
http://www.boost.org/doc/libs/1_55_0/doc/html/container/non_standard_containers.html#container.non_standard_containers.static_vector
http://llvm.org/docs/doxygen/html/classllvm_1_1SmallVector.html
Related
I have 4 gig RAM and the following lines that throws a stackoverflow exception:
int main()
{
double X[4096*512], Y[4096*512], Z[4096*512];
return 0;
}
each double takes 8 bytes space, so my three arrays should be 3*4096*512*8/1024/1024 = 48 Mbyte big, can somebody explain the error or is 48 Mbyte too much to handle?
You are declaring in the stack, normally the stack in OS are limited (eg: 1MB), you could expanded when compiling (eg: in GCC use -Wl,stack_size,134217728 128Mb) but don't recommend.
Better use std::vector<double>.
#include <vector>
int main() {
std::vector<double> X(4096*512), Y(4096*512), Z(4096*512);
return 0;
}
If you want to avoid the overhead of std::vector, you can allocate the arrays on the heap
double *myArray = new double[SIZE];
and remember to free them.
delete [] myArray;
Expanding my comment (not a thorough description, you should do more research), there are two types of memory, the "stack" and the "heap". The stack is the local working memory of a piece of software, while the heap is the big pool. The stack holds all the local variables of a function call. Anything local declared within the function will be stored on the stack. The stack had a nifty property that it can be "pushed": when we call the next function, we go further down the stack and start over. But this means that the amount of stack memory needs to be generally reserved for the lifetime of a program, so we limit it to a small amount: general 1 to a few megabytes.
When you run out of stack memory, you get a "Stack Overflow" ("Now thats what that means!")
The heap is kindof the rest of the memory. The heap is where the program stores any dynamic memory and (possibly, it can be more complicated) global variables. The heap is where things like malloc and new puts its memory.
Whenever you declare a local variable, it is stored on the stack. This isn't a problem for small variables, but arrays like the ones you have get HUGE easy.
If you don't want to worry about new or malloc you can use things like std::vector, which will put the large amount of data on the heap while giving you local variable semantics.
Again, this is "basic programming" so you should get really familiar with this subject.
Recently I was asked a question to implement a very simple malloc with the following restrictions and initial conditions.
#define HEAP_SIZE 2048
int main()
{
privateHeap = malloc(HEAP_SIZE + 256); //extra 256 bytes for heap metadata
void* ptr = mymalloc( size_t(750) );
myfree( ptr );
return 0;
}
I need to implement mymalloc and myfree here using the exact space provided. 256 bytes is nicely mapping to 2048 bits, and I can have a bit array storing if a byte is allocated or if it is free. But when I make a myfree call with ptr, I cannot tell how much size was allocated to begin with. I cannot use any extra bits.
I don't seem to think there is a way around this, but I've been reiterated that it can be done. Any suggestions ?
EDIT 1:
Alignment restrictions don't exist. I assumed I am not going to align anything.
There was a demo program that did a series of mallocs and frees to test this, and it didn't have any memory blocks that were small. But that doesn't guarantee anything.
EDIT 2:
The guidelines from the documentation:
Certain Guidelines on your code:
Manage the heap metadata in the private heap; do not create extra linked lists outside of the provided private heap;
Design mymalloc, myrealloc, myFree to work for all possible inputs.
myrealloc should do the following like the realloc in C++ library:
void* myrealloc( void* C, size_t newSize ):
If newSize is bigger than the size of chunk in reallocThis:
It should first try to allocate a chunk of size newSize in place so that new chunk's base pointer also is reallocThis;
If there is no free space available to do in place allocation, it should allocate a chunk of requested size in a different region;
and then it should copy the contents from the previous chunk.
If the function failed to allocate the requested block of memory, a NULL pointer is returned, and the memory block pointed to
by argument reallocThis is left unchanged.
If newSize is smaller, realloc should shrink the size of the chunk and should always succeed.
If newSize is 0, it should work like free.
If reallocThis is NULL, it should work like malloc.
If reallocThis is pointer that was already freed, then it should fail gracefully by returning NULL
myFree should not crash when it is passed a pointer that has already been freed.
A common way malloc implementations keep track of the size of memory allocations so free knows how big they are is to store the size in the bytes before pointer return by malloc. So say you only need two bytes to store the length, when the caller of malloc requests n bytes of memory, you actually allocate n + 2 bytes. You then store the length in the first two bytes, and return a pointer to the byte just past where you stored the size.
As for your algorithm generally, a simple and naive implementation is to keep track of unallocated memory with a linked list of free memory blocks that are kept in order of their location in memory. To allocate space you search for a free block that's big enough. You then modify the free list to exclude that allocation. To free a block you add it back to the free list, coalescing adjacent free blocks.
This isn't a good malloc implementation by modern standards, but a lot of old memory allocators worked this way.
You seem to be thinking of the 256 bytes of meta-data as a bit-map to track free/in-use on a byte-by-byte basis.
I'd consider the following as only one possible alternative:
I'd start by treating the 2048-byte heap as a 1024 "chunks" of 2 bytes each. This gives you 2 bits of information for each chunk. You can treat the first of those as signifying whether that chunk is in use, and the second as signifying whether the following chunk is part of the same logical block as the current one.
When your free function is called, you use the passed address to find the correct beginning point in your bitmap. You then walk through bits marking each chunk as free until you reach one where the second bit is set to 0, indicating the end of the current logical block (i.e., that the next 2 byte chunk is not part of the current logical block).
[Oops: just noticed that Ross Ridge already suggested nearly the same basic idea in a comment.]
What is the advantage of allocating a memory for some data. Instead we could use an array of them.
Like
int *lis;
lis = (int*) malloc ( sizeof( int ) * n );
/* Initialize LIS values for all indexes */
for ( i = 0; i < n; i++ )
lis[i] = 1;
we could have used an ordinary array.
Well I don't understand exactly how malloc works, what is actually does. So explaining them would be more beneficial for me.
And suppose we replace sizeof(int) * n with just n in the above code and then try to store integer values, what problems might i be facing? And is there a way to print the values stored in the variable directly from the memory allocated space, for example here it is lis?
Your question seems to rather compare dynamically allocated C-style arrays with variable-length arrays, which means that this might be what you are looking for: Why aren't variable-length arrays part of the C++ standard?
However the c++ tag yields the ultimate answer: use std::vector object instead.
As long as it is possible, avoid dynamic allocation and responsibility for ugly memory management ~> try to take advantage of objects with automatic storage duration instead. Another interesting reading might be: Understanding the meaning of the term and the concept - RAII (Resource Acquisition is Initialization)
"And suppose we replace sizeof(int) * n with just n in the above code and then try to store integer values, what problems might i be facing?"
- If you still consider n to be the amount of integers that it is possible to store in this array, you will most likely experience undefined behavior.
More fundamentally, I think, apart from the stack vs heap and variable vs constant issues (and apart from the fact that you shouldn't be using malloc() in C++ to begin with), is that a local array ceases to exist when the function exits. If you return a pointer to it, that pointer is going to be useless as soon as the caller receives it, whereas memory dynamically allocated with malloc() or new will still be valid. You couldn't implement a function like strdup() using a local array, for instance, or sensibly implement a linked representation list or tree.
The answer is simple. Local1 arrays are allocated on your stack, which is a small pre-allocated memory for your program. Beyond a couple thousand data, you can't really do much on a stack. For higher amounts of data, you need to allocate memory out of your stack.
This is what malloc does.
malloc allocates a piece of memory as big as you ask it. It returns a pointer to the start of that memory, which could be treated similar to an array. If you write beyond the size of that memory, the result is undefined behavior. This means everything could work alright, or your computer may explode. Most likely though you'd get a segmentation fault error.
Reading values from the memory (for example for printing) is the same as reading from an array. For example printf("%d", list[5]);.
Before C99 (I know the question is tagged C++, but probably you're learning C-compiled-in-C++), there was another reason too. There was no way you could have an array of variable length on the stack. (Even now, variable length arrays on the stack are not so useful, since the stack is small). That's why for variable amount of memory, you needed the malloc function to allocate memory as large as you need, the size of which is determined at runtime.
Another important difference between local arrays, or any local variable for that matter, is the life duration of the object. Local variables are inaccessible as soon as their scope finishes. malloced objects live until they are freed. This is essential in practically all data structures that are not arrays, such as linked-lists, binary search trees (and variants), (most) heaps etc.
An example of malloced objects are FILEs. Once you call fopen, the structure that holds the data related to the opened file is dynamically allocated using malloc and returned as a pointer (FILE *).
1 Note: Non-local arrays (global or static) are allocated before execution, so they can't really have a length determined at runtime.
I assume you are asking what is the purpose of c maloc():
Say you want to take an input from user and now allocate an array of that size:
int n;
scanf("%d",&n);
int arr[n];
This will fail because n is not available at compile time. Here comes malloc()
you may write:
int n;
scanf("%d",&n);
int* arr = malloc(sizeof(int)*n);
Actually malloc() allocate memory dynamically in the heap area
Some older programming environments did not provide malloc or any equivalent functionality at all. If you needed dynamic memory allocation you had to code it yourself on top of gigantic static arrays. This had several drawbacks:
The static array size put a hard upper limit on how much data the program could process at any one time, without being recompiled. If you've ever tried to do something complicated in TeX and got a "capacity exceeded, sorry" message, this is why.
The operating system (such as it was) had to reserve space for the static array all at once, whether or not it would all be used. This phenomenon led to "overcommit", in which the OS pretends to have allocated all the memory you could possibly want, but then kills your process if you actually try to use more than is available. Why would anyone want that? And yet it was hyped as a feature in mid-90s commercial Unix, because it meant that giant FORTRAN simulations that potentially needed far more memory than your dinky little Sun workstation had, could be tested on small instance sizes with no trouble. (Presumably you would run the big instance on a Cray somewhere that actually had enough memory to cope.)
Dynamic memory allocators are hard to implement well. Have a look at the jemalloc paper to get a taste of just how hairy it can be. (If you want automatic garbage collection it gets even more complicated.) This is exactly the sort of thing you want a guru to code once for everyone's benefit.
So nowadays even quite barebones embedded environments give you some sort of dynamic allocator.
However, it is good mental discipline to try to do without. Over-use of dynamic memory leads to inefficiency, of the kind that is often very hard to eliminate after the fact, since it's baked into the architecture. If it seems like the task at hand doesn't need dynamic allocation, perhaps it doesn't.
However however, not using dynamic memory allocation when you really should have can cause its own problems, such as imposing hard upper limits on how long strings can be, or baking nonreentrancy into your API (compare gethostbyname to getaddrinfo).
So you have to think about it carefully.
we could have used an ordinary array
In C++ (this year, at least), arrays have a static size; so creating one from a run-time value:
int lis[n];
is not allowed. Some compilers allow this as a non-standard extension, and it's due to become standard next year; but, for now, if we want a dynamically sized array we have to allocate it dynamically.
In C, that would mean messing around with malloc; but you're asking about C++, so you want
std::vector<int> lis(n, 1);
to allocate an array of size n containing int values initialised to 1.
(If you like, you could allocate the array with new int[n], and remember to free it with delete [] lis when you're finished, and take extra care not to leak if an exception is thrown; but life's too short for that nonsense.)
Well I don't understand exactly how malloc works, what is actually does. So explaining them would be more beneficial for me.
malloc in C and new in C++ allocate persistent memory from the "free store". Unlike memory for local variables, which is released automatically when the variable goes out of scope, this persists until you explicitly release it (free in C, delete in C++). This is necessary if you need the array to outlive the current function call. It's also a good idea if the array is very large: local variables are (typically) stored on a stack, with a limited size. If that overflows, the program will crash or otherwise go wrong. (And, in current standard C++, it's necessary if the size isn't a compile-time constant).
And suppose we replace sizeof(int) * n with just n in the above code and then try to store integer values, what problems might i be facing?
You haven't allocated enough space for n integers; so code that assumes you have will try to access memory beyond the end of the allocated space. This will cause undefined behaviour; a crash if you're lucky, and data corruption if you're unlucky.
And is there a way to print the values stored in the variable directly from the memory allocated space, for example here it is lis?
You mean something like this?
for (i = 0; i < len; ++i) std::cout << lis[i] << '\n';
What is the difference between this:
somefunction() {
...
char *output;
output = (char *) malloc((len * 2) + 1);
...
}
and this:
somefunction() {
...
char output[(len * 2) + 1];
...
}
When is one more appropriate than the other?
thanks all for your answers. here is a summary:
ex. 1 is heap allocation
ex. 2 is stack allocation
there is a size limitation on the stack, use it for smaller allocations
you have to free heap allocation, or it will leak
the stack allocation is not accessible once the function exits
the heap allocation is accessible until you free it (or the app ends)
VLA's are not part of standard C++
corrections welcome.
here is some explanation of the difference between heap vs stack:
What and where are the stack and heap?
The first allocates memory on the heap. You have to remember to free the memory, or it will leak. This is appropriate if the memory needs to used outside the function, or if you need to allocate a huge amount of memory.
The second allocates memory on the stack. It will be reclaimed automatically when the function returns. This is the most convenient if you don't need to return the memory to your caller.
Use locals when you only have a small amount of data, and you are not going to use the data outside the scope of the function you've declared it in. If you're going to pass the data around, use malloc.
Local variables are held on the stack, which is much more size limited than the heap, where arrays allocated with malloc go. I usually go for anything > 16 bytes being put on the heap, but you have a bit more flexibility than that. Just don't be allocating locals in the kb/mb size range - they belong on the heap.
The first example allocates a block of storage from the heap. The second one allocates storage from the stack. The difference becomes visible when you return output from somefunction(). The dynamically allocated storage is still available for your use, but the stack-based storage in the second example is, um, nowhere. You can still write into this storage and read it for awhile, until the next time you call a function, at which time the storage will get overwritten randomly with return addresses, arguments, and such.
There's a lot of other weird stuff going on with the code in this question too. First off, it this is a c++ program, you'd want to use new instead of malloc() so you'd say
output = new char[len+1];
And what's with the len*2 + 1 anyway? Maybe this is something special in your code, but I'm guessing you want to allocate unicode characters or multibyte characters. If it's unicode, the null termination takes two bytes as well as each character does, and char is the wrong type, being 8 bit bytes in most compilers. If it's multibyte, then hey, all bets are off.
First some terminology:
The first sample is called heap allocation.
The second sample is called stack allocation.
The general rule is: allocate on the stack, unless:
The required size of the array is unknown at compile time.
The required size exceeds 10% of the total stack size. The default stack size on Windows and Linux is usually 1 or 2 MB. So your local array should not exceed 100,000 bytes.
You tagged your question with both C++ and C, but the second solution is not allowed in C++. Variable length arrays are only allowed in C(99).
If you were to assume 'len' is a constant, both will work.
malloc() (and C++'s 'new') allocate the memory on the heap, which means you have to free() (or if you allocated with 'new', 'delete') the buffer, or the memory will never be reclaimed (leak).
The latter allocates the array on the stack, and will be gone when it goes out of scope. This means that you can't return pointers to the buffer outside the scope it's allocated in.
The former is useful when you want to pass the block of memory around (but in C++, it's best managed with an RAII class, not manually), while the latter is best for small, fixed-size arrays that only need to exist in one scope.
Lastly, you can mark the otherwise stack-allocated array with 'static' to take it off the stack and into a global data section:
static char output[(len * 2) + 1];
This enables you to return pointers to the buffer outside of its scope, however, all calls to such a function will refer to the same piece of global data, so don't use it if you need a unique block of memory every time.
Lastly, don't use malloc in C++ unless you have a really good reason (i.e, realloc). Use 'new' instead, and the accompanying 'delete'.
So I have a function that creates a dynamic array, I then delete the array before I leave the function (as I thought I am supposed to), however I am getting a 'Heap Corruption Detected' warning in VS2008. If I remove the line that deallocates the memory everything works fine:
void myFunc()
{
char* c = new char[length];
memset(c, 0, length);
//.. do somsething with array
delete[] c; //this line throws an error??
}
Thanks for any advice
Most likely you are doing something else bad (like under/overflowing your buffer) and corrupting the heap at that point, but it isn't detected until you call delete[] and try to interpret the now corrupted heap structures.
Post the 'do something' section if you need more assistance.
I think you have a problem with your //.. do something with array code (or even some other code) since the rest of what you have is okay.
Often, memory arena corruption is only detected when freeing the memory which is probably why removing the line seems to fix it.
But rest assured, the arena is still corrupted whether or not you're detecting it. You need to fix it.
One way this might happen is if your memory allocation routines actually allocate extra bits before and possibly after what you're given. For example, if you ask for 1024 bytes, what might actually be allocated from the heap is a 1040-byte block of which you're given the address of the 16th byte. This gives the arena manager 16 bytes at the start to store housekeeping information (including a sentinal value at the very start of it).
Then, when the block is deleted, the arena manager knows that its housekeeping is in the 16 bytes before the address you're trying to free and can check the sentinal value (or all the sentinals in the arena or just the ones on either side of the one you're freeing - this is all implementation detail) to make sure they're unchanged. If they have been changed, that's detected as corruption.
As I said earlier, the corruption couuld be caused by your //.. do something with array code or it could be somewhere totally different - all that matters is that the arena is being trashed.
You're probably underflowing the buffer, actually - the VC heap (and most heap implementations) keep book-keeping information immediately before the allocation they hand out. It includes some data validation (sentinel bytes, etc), which is it doesn't pass, this error is thrown.
Every time you allocate memory using new, you will need to free that memory using a matching delete. The code you quote should work.
C++ memory manager implementations typically interleave their control data structures with areas of memory you allocate. C++ does not bounds-check arrays for you. If your code writes data off the end or before the start of an array, this will corrupt the heap. It is very likely that this is what is happening here. Carefully examine the code that performs work on the array.
First, to answer the title, no dynamically allocated memory (with new, malloc, etc) is not freed when function exits. You are responsible for freeing it.
Second, just an advice that might help you debug your problem.
One great option is to use a free tool from Microsoft, called Application Verifier. It's a great tool to have in your toolbox, it's really great at helping you finding bugs in your applications.
Another option, not involving use of other tools would be, instead of your allocated array, you could try using std::vector, it might help detecting your heap corruption in debug mode. It has a huge amount of various checks in debug mode, which would likely cause it to break into debugger at the right time. Here's what you could try:
{
const size_t size_of_array = 64;
// use constructor with size and value
// do _not_ use memset on this object
std::vector<char> your_array(size_of_array, 0);
// do something here with it e.g.:
snprintf(&your_array[0], your_array.size(), "hello");
// do whatever you do with your array
// use debug build and run it under debugger,
// likely you will spot your problem pretty soon
// no need to delete anything here
}
This warning means you probably wrote to memory you don't own, perhaps by overrunning a buffer, freeing memory more than once, or forgetting to initialize a pointer before using it.
Good luck
delete does not throw. This is guaranteed. If you are allocating for some "length", and if you are using the entire char array without having '/0' at the end, then you would get this error. Eg:
char* arr = new char[5];
strcpy(arr, "Jagan");
delete[] arr;
Instead, allocate arr of length 6 in this case.