I have 4 gig RAM and the following lines that throws a stackoverflow exception:
int main()
{
double X[4096*512], Y[4096*512], Z[4096*512];
return 0;
}
each double takes 8 bytes space, so my three arrays should be 3*4096*512*8/1024/1024 = 48 Mbyte big, can somebody explain the error or is 48 Mbyte too much to handle?
You are declaring in the stack, normally the stack in OS are limited (eg: 1MB), you could expanded when compiling (eg: in GCC use -Wl,stack_size,134217728 128Mb) but don't recommend.
Better use std::vector<double>.
#include <vector>
int main() {
std::vector<double> X(4096*512), Y(4096*512), Z(4096*512);
return 0;
}
If you want to avoid the overhead of std::vector, you can allocate the arrays on the heap
double *myArray = new double[SIZE];
and remember to free them.
delete [] myArray;
Expanding my comment (not a thorough description, you should do more research), there are two types of memory, the "stack" and the "heap". The stack is the local working memory of a piece of software, while the heap is the big pool. The stack holds all the local variables of a function call. Anything local declared within the function will be stored on the stack. The stack had a nifty property that it can be "pushed": when we call the next function, we go further down the stack and start over. But this means that the amount of stack memory needs to be generally reserved for the lifetime of a program, so we limit it to a small amount: general 1 to a few megabytes.
When you run out of stack memory, you get a "Stack Overflow" ("Now thats what that means!")
The heap is kindof the rest of the memory. The heap is where the program stores any dynamic memory and (possibly, it can be more complicated) global variables. The heap is where things like malloc and new puts its memory.
Whenever you declare a local variable, it is stored on the stack. This isn't a problem for small variables, but arrays like the ones you have get HUGE easy.
If you don't want to worry about new or malloc you can use things like std::vector, which will put the large amount of data on the heap while giving you local variable semantics.
Again, this is "basic programming" so you should get really familiar with this subject.
Related
I am using Dev C++ to write a simulation program. For it, I need to declare a single dimensional array with the data type double. It contains 4200000 elements - like double n[4200000].
The compiler shows no error, but the program exits on execution. I have checked, and the program executes just fine for an array having 5000 elements.
Now, I know that declaring such a large array on the stack is not recommended. However, the thing is that the simulation requires me to call specific elements from the array multiple times - for example, I might need the value of n[234] or n[46664] for a given calculation. Therefore, I need an array in which it is easier to sift through elements.
Is there a way I can declare this array on the stack?
No there is no(we'll say "reasonable") way to declare this array on the stack. You can however declare the pointer on the stack, and set aside a bit of memory on the heap.
double *n = new double[4200000];
accessing n[234] of this, should be no quicker than accessing n[234] of an array that you declared like this:
double n[500];
Or even better, you could use vectors
std::vector<int> someElements(4200000);
someElements[234];//Is equally fast as our n[234] from other examples, if you optimize (-O3) and the difference on small programs is negligible if you don't(+5%)
Which if you optimize with -O3, is just as fast as an array, and much safer. As with the
double *n = new double[4200000];
solution you will leak memory unless you do this:
delete[] n;
And with exceptions and various things, this is a very unsafe way of doing things.
You can increase your stack size. Try adding these options to your link flags:
-Wl,--stack,36000000
It might be too large though (I'm not sure if Windows places an upper limit on stack size.) In reality though, you shouldn't do that even if it works. Use dynamic memory allocation, as pointed out in the other answers.
(Weird, writing an answer and hoping it won't get accepted... :-P)
Yes, you can declare this array on the stack (with a little extra work), but it is not wise.
There is no justifiable reason why the array has to live on the stack.
The overhead of dynamically allocating a single array once is neglegible (you could say "zero"), and a smart pointer will safely take care of not leaking memory, if that is your concern.
Stack allocated memory is not in any way different from heap allocated memory (apart from some caching effects for small objects, but these do not apply here).
Insofar, just don't do it.
If you insist that you must allocate the array on the stack, you will need to reserve 32 megabytes of stack space first (preferrably a bit more). For that, using Dev-C++ (which presumes Windows+MingW) you will either need to set the reserved stack size for your executable using compiler flags such as -Wl,--stack,34000000 (this reserves somewhat more than 32MiB), or create a thread (which lets you specify a reserved stack size for that thread).
But really, again, just don't do that. There's nothing wrong with allocating a huge array dynamically.
Are there any reasons you want this on the stack specifically?
I'm asking because the following will give you a construct that can be used in a similar way (especially accessing values using array[index]), but it is a lot less limited in size (total max size depending on 32bit/64bit memory model and available memory (RAM and swap memory)) because it is allocated from the heap.
int arraysize= 4200000;
int *heaparray= new int[arraysize];
...
k= heaparray[456];
...
delete [] heaparray;
return;
I am trying to better understand items on a stack and how they are addressed. The article I found here seems to indicate that when MIPS stack is initialized, a fixed amount of memory is allocated and the stack grows down to the stack limit which would appear to be smaller addresses. I would assume that based on this logic a stack overflow would occur when 0x0000 was traversed?
I realize MIPS is big endian, but does that change how the stack grows? I wrote what I believed would be a quick way to observe this on an x86_64 machine, but the stack appears to grow up, as I originally assumed it did.
#include <iostream>
#include <vector>
int main() {
std::vector<int*> v;
for( int i = 0; i < 10; i++ ) {
v.push_back(new int);
std::cout << v.back() << std::endl;
}
}
I'm also confused by the fact that not all of the memory address's do not appear to be contiguous, which makes me think I did something stupid. Could somebody please clarify?
The stack on x86 machines also grows downwards. Endianness is unrelated to the direction in which the stack grows.
The stack of a machine has absolutely nothing to do with std::vector<>. Also, new int allocates heap memory, so it tells you absolutely nothing about the stack.
In order to see in which direction the stack grows, you need to do something like this:
recursive( 5 );
void recursive( int n )
{
if( n == 0 )
return;
int a;
printf( "%p\n", &a );
recursive( n - 1 );
}
(Note that if your compiler is smart enough to optimize tail recursion, then you will need to tell it to not optimize it, otherwise the observations will be all wrong.)
essentially there are 3 types of memory you use in programming: static, dynamic/heap and stack.
Static memory is pre-allocated by the compiler and consist of the constants and variables declared statically in your program.
Heap is the memory which you can freely allocate and release
Stack is the memory which gets allocated for all local variables declared in a function. This is important because every time you call the function a new memory for its variables is allocated. So, that every call to a function will assure that it has its own unique copy of the variables. And every time you return from the function the memory gets freed.
It absolutely does not matter how the stack is managed as soon as it follows the above rules. It is convenient however to have program memory to be allocated in the lower address space and grow up, and the stack to start from a top memory space and grow down. Most systems implement this scheme.
In general there is a stack pointer register/variable which points so the current stack address. when a function gets called it decrease this address by the number of bytes it needs for its variables. when it calls the next function, this new one will start with the new pointer already decreased by the caller. When the function returns it restores the pointer which it started from.
There could be different schemes but as far as I know, mips and i86 follow this one.
And essentially there is only one virtual memory space in the program. This is up to the operating system and/or compiler how to use it. The compiler will split the memory in the logical regions for its own use and handle them, hopefully, according to the calling conventions defined in the platform documents.
So, in our example, v and i are allocated on the function stack. cout is static. every new int allocate space in heap. v is not a simple variable but a struct which contains fields which it needs to manage the list. it needs space for all these internals. So, every push_back modifies those fields to point to the allocated 'int' in some way. push_back() and back() are function calls and allocate their own stacks for internal variables to not interfere with the top function.
What is the advantage of allocating a memory for some data. Instead we could use an array of them.
Like
int *lis;
lis = (int*) malloc ( sizeof( int ) * n );
/* Initialize LIS values for all indexes */
for ( i = 0; i < n; i++ )
lis[i] = 1;
we could have used an ordinary array.
Well I don't understand exactly how malloc works, what is actually does. So explaining them would be more beneficial for me.
And suppose we replace sizeof(int) * n with just n in the above code and then try to store integer values, what problems might i be facing? And is there a way to print the values stored in the variable directly from the memory allocated space, for example here it is lis?
Your question seems to rather compare dynamically allocated C-style arrays with variable-length arrays, which means that this might be what you are looking for: Why aren't variable-length arrays part of the C++ standard?
However the c++ tag yields the ultimate answer: use std::vector object instead.
As long as it is possible, avoid dynamic allocation and responsibility for ugly memory management ~> try to take advantage of objects with automatic storage duration instead. Another interesting reading might be: Understanding the meaning of the term and the concept - RAII (Resource Acquisition is Initialization)
"And suppose we replace sizeof(int) * n with just n in the above code and then try to store integer values, what problems might i be facing?"
- If you still consider n to be the amount of integers that it is possible to store in this array, you will most likely experience undefined behavior.
More fundamentally, I think, apart from the stack vs heap and variable vs constant issues (and apart from the fact that you shouldn't be using malloc() in C++ to begin with), is that a local array ceases to exist when the function exits. If you return a pointer to it, that pointer is going to be useless as soon as the caller receives it, whereas memory dynamically allocated with malloc() or new will still be valid. You couldn't implement a function like strdup() using a local array, for instance, or sensibly implement a linked representation list or tree.
The answer is simple. Local1 arrays are allocated on your stack, which is a small pre-allocated memory for your program. Beyond a couple thousand data, you can't really do much on a stack. For higher amounts of data, you need to allocate memory out of your stack.
This is what malloc does.
malloc allocates a piece of memory as big as you ask it. It returns a pointer to the start of that memory, which could be treated similar to an array. If you write beyond the size of that memory, the result is undefined behavior. This means everything could work alright, or your computer may explode. Most likely though you'd get a segmentation fault error.
Reading values from the memory (for example for printing) is the same as reading from an array. For example printf("%d", list[5]);.
Before C99 (I know the question is tagged C++, but probably you're learning C-compiled-in-C++), there was another reason too. There was no way you could have an array of variable length on the stack. (Even now, variable length arrays on the stack are not so useful, since the stack is small). That's why for variable amount of memory, you needed the malloc function to allocate memory as large as you need, the size of which is determined at runtime.
Another important difference between local arrays, or any local variable for that matter, is the life duration of the object. Local variables are inaccessible as soon as their scope finishes. malloced objects live until they are freed. This is essential in practically all data structures that are not arrays, such as linked-lists, binary search trees (and variants), (most) heaps etc.
An example of malloced objects are FILEs. Once you call fopen, the structure that holds the data related to the opened file is dynamically allocated using malloc and returned as a pointer (FILE *).
1 Note: Non-local arrays (global or static) are allocated before execution, so they can't really have a length determined at runtime.
I assume you are asking what is the purpose of c maloc():
Say you want to take an input from user and now allocate an array of that size:
int n;
scanf("%d",&n);
int arr[n];
This will fail because n is not available at compile time. Here comes malloc()
you may write:
int n;
scanf("%d",&n);
int* arr = malloc(sizeof(int)*n);
Actually malloc() allocate memory dynamically in the heap area
Some older programming environments did not provide malloc or any equivalent functionality at all. If you needed dynamic memory allocation you had to code it yourself on top of gigantic static arrays. This had several drawbacks:
The static array size put a hard upper limit on how much data the program could process at any one time, without being recompiled. If you've ever tried to do something complicated in TeX and got a "capacity exceeded, sorry" message, this is why.
The operating system (such as it was) had to reserve space for the static array all at once, whether or not it would all be used. This phenomenon led to "overcommit", in which the OS pretends to have allocated all the memory you could possibly want, but then kills your process if you actually try to use more than is available. Why would anyone want that? And yet it was hyped as a feature in mid-90s commercial Unix, because it meant that giant FORTRAN simulations that potentially needed far more memory than your dinky little Sun workstation had, could be tested on small instance sizes with no trouble. (Presumably you would run the big instance on a Cray somewhere that actually had enough memory to cope.)
Dynamic memory allocators are hard to implement well. Have a look at the jemalloc paper to get a taste of just how hairy it can be. (If you want automatic garbage collection it gets even more complicated.) This is exactly the sort of thing you want a guru to code once for everyone's benefit.
So nowadays even quite barebones embedded environments give you some sort of dynamic allocator.
However, it is good mental discipline to try to do without. Over-use of dynamic memory leads to inefficiency, of the kind that is often very hard to eliminate after the fact, since it's baked into the architecture. If it seems like the task at hand doesn't need dynamic allocation, perhaps it doesn't.
However however, not using dynamic memory allocation when you really should have can cause its own problems, such as imposing hard upper limits on how long strings can be, or baking nonreentrancy into your API (compare gethostbyname to getaddrinfo).
So you have to think about it carefully.
we could have used an ordinary array
In C++ (this year, at least), arrays have a static size; so creating one from a run-time value:
int lis[n];
is not allowed. Some compilers allow this as a non-standard extension, and it's due to become standard next year; but, for now, if we want a dynamically sized array we have to allocate it dynamically.
In C, that would mean messing around with malloc; but you're asking about C++, so you want
std::vector<int> lis(n, 1);
to allocate an array of size n containing int values initialised to 1.
(If you like, you could allocate the array with new int[n], and remember to free it with delete [] lis when you're finished, and take extra care not to leak if an exception is thrown; but life's too short for that nonsense.)
Well I don't understand exactly how malloc works, what is actually does. So explaining them would be more beneficial for me.
malloc in C and new in C++ allocate persistent memory from the "free store". Unlike memory for local variables, which is released automatically when the variable goes out of scope, this persists until you explicitly release it (free in C, delete in C++). This is necessary if you need the array to outlive the current function call. It's also a good idea if the array is very large: local variables are (typically) stored on a stack, with a limited size. If that overflows, the program will crash or otherwise go wrong. (And, in current standard C++, it's necessary if the size isn't a compile-time constant).
And suppose we replace sizeof(int) * n with just n in the above code and then try to store integer values, what problems might i be facing?
You haven't allocated enough space for n integers; so code that assumes you have will try to access memory beyond the end of the allocated space. This will cause undefined behaviour; a crash if you're lucky, and data corruption if you're unlucky.
And is there a way to print the values stored in the variable directly from the memory allocated space, for example here it is lis?
You mean something like this?
for (i = 0; i < len; ++i) std::cout << lis[i] << '\n';
Is there a way to allocate memory on stack instead of heap? I can't find a good book on this, anyone here got an idea?
Use alloca() (sometimes called _alloca() or _malloca() ), but be very careful about it — it frees its memory when you leave a function, not when you go out of scope, so you'll quickly blow up if you use it inside a loop.
For example, if you have a function like
int foo( int nDataSize, int iterations )
{
for ( int i = 0; i < iterations ; ++i )
{
char *bytes = alloca( nDataSize );
// the memory above IS NOT FREED when we pass the brace below!
}
return 0;
} // alloca() memory only gets freed here
Then the alloca() will allocate an additional nDataSize bytes every time through the loop. None of the alloca() bytes get freed until you return from the function. So, if you have an nDataSize of 1024 and an iterations of 8, you'll allocate 8 kilobytes before returning. If you have an nDataSize= 65536 and iterations = 32768, you'll allocate a total 65536×32768=2,147,483,648 bytes, almost certainly blowing your stack and causing a crash.
anecdote: You can easily get into trouble if you write past the end of the buffer, especially if you pass the buffer into another function, and that subfunction has the wrong idea about the buffer's length. I once fixed a rather amusing bug where we were using alloca() to create temporary storage for rendering a TrueType font glyph before sending it over to GPU memory. Our font library didn't account for the diacritic in the Swedish Å character when calculating glyph sizes, so it told us to allocate n bytes to store the glyph before rendering, and then actually rendered n+128 bytes. The extra 128 bytes wrote into the call stack, overwriting the return address and inducing a really painful nondeterministic crash!
Since this is tagged C++, typically you just declare the objects you need in the correct scope. They are allocated on the stack, and guaranteed to be released on scope exit. This is RAII, and a critical advantage of C++ over C. No mallocs or news, and especially no allocas, required.
You can declare a local char[1024] or whatever number of bytes you'd like (up to a point), then take the address of the local for a pointer to this block of memory on the stack. Not exactly dynamic, but you could then wrap up this memory with your own memory manager if desired.
See _malloca.
Article discussing about dynamic memory allocation
We can allocate variable length space dynamically on stack memory by
using function
_alloca. This function allocates memory from the program stack. It simply takes number of bytes to be allocated and return void* to the
allocated space just as malloc call. This allocated memory will be
freed automatically on function exit.
So it need not to be freed explicitly. One has to keep in mind about
allocation size here, as stack overflow exception may occur. Stack
overflow exception handling can be used for such calls. In case of
stack overflow exception one can use _resetstkoflw() to restore it
back.
So our new code with _alloca would be :
int NewFunctionA()
{
char* pszLineBuffer = (char*) _alloca(1024*sizeof(char));
…..
// Program logic
….
//no need to free szLineBuffer
return 1;
}
When/if C++ allows the use of (non-static) const values for array bounds, it will be easier.
For now, the best way I know of is via recursion. There are all kinds of clever tricks that can be done, but the easiest I know of is to have your routine declare a fixed-sized array, and fill and operate on what it has. When its done, if it needs more space to finish, it calls itself.
You could use the BDE C++ library, e.g.
const int BUFFER_SIZE = 1024;
char buffer[BUFFER_SIZE];
bdlma::BufferedSequentialAllocator allocator(buffer, BUFFER_SIZE);
bsl::vector<int> dataVector(&allocator);
dataVector.resize(50);
BDE supplies comprehensive allocator options along with collections like bsl::vector that can use polymorphic allocators without changing the type of the container.
You might also consider:
https://github.com/facebook/folly/blob/master/folly/docs/small_vector.md
http://www.boost.org/doc/libs/1_55_0/doc/html/container/non_standard_containers.html#container.non_standard_containers.static_vector
http://llvm.org/docs/doxygen/html/classllvm_1_1SmallVector.html
What is the difference between this:
somefunction() {
...
char *output;
output = (char *) malloc((len * 2) + 1);
...
}
and this:
somefunction() {
...
char output[(len * 2) + 1];
...
}
When is one more appropriate than the other?
thanks all for your answers. here is a summary:
ex. 1 is heap allocation
ex. 2 is stack allocation
there is a size limitation on the stack, use it for smaller allocations
you have to free heap allocation, or it will leak
the stack allocation is not accessible once the function exits
the heap allocation is accessible until you free it (or the app ends)
VLA's are not part of standard C++
corrections welcome.
here is some explanation of the difference between heap vs stack:
What and where are the stack and heap?
The first allocates memory on the heap. You have to remember to free the memory, or it will leak. This is appropriate if the memory needs to used outside the function, or if you need to allocate a huge amount of memory.
The second allocates memory on the stack. It will be reclaimed automatically when the function returns. This is the most convenient if you don't need to return the memory to your caller.
Use locals when you only have a small amount of data, and you are not going to use the data outside the scope of the function you've declared it in. If you're going to pass the data around, use malloc.
Local variables are held on the stack, which is much more size limited than the heap, where arrays allocated with malloc go. I usually go for anything > 16 bytes being put on the heap, but you have a bit more flexibility than that. Just don't be allocating locals in the kb/mb size range - they belong on the heap.
The first example allocates a block of storage from the heap. The second one allocates storage from the stack. The difference becomes visible when you return output from somefunction(). The dynamically allocated storage is still available for your use, but the stack-based storage in the second example is, um, nowhere. You can still write into this storage and read it for awhile, until the next time you call a function, at which time the storage will get overwritten randomly with return addresses, arguments, and such.
There's a lot of other weird stuff going on with the code in this question too. First off, it this is a c++ program, you'd want to use new instead of malloc() so you'd say
output = new char[len+1];
And what's with the len*2 + 1 anyway? Maybe this is something special in your code, but I'm guessing you want to allocate unicode characters or multibyte characters. If it's unicode, the null termination takes two bytes as well as each character does, and char is the wrong type, being 8 bit bytes in most compilers. If it's multibyte, then hey, all bets are off.
First some terminology:
The first sample is called heap allocation.
The second sample is called stack allocation.
The general rule is: allocate on the stack, unless:
The required size of the array is unknown at compile time.
The required size exceeds 10% of the total stack size. The default stack size on Windows and Linux is usually 1 or 2 MB. So your local array should not exceed 100,000 bytes.
You tagged your question with both C++ and C, but the second solution is not allowed in C++. Variable length arrays are only allowed in C(99).
If you were to assume 'len' is a constant, both will work.
malloc() (and C++'s 'new') allocate the memory on the heap, which means you have to free() (or if you allocated with 'new', 'delete') the buffer, or the memory will never be reclaimed (leak).
The latter allocates the array on the stack, and will be gone when it goes out of scope. This means that you can't return pointers to the buffer outside the scope it's allocated in.
The former is useful when you want to pass the block of memory around (but in C++, it's best managed with an RAII class, not manually), while the latter is best for small, fixed-size arrays that only need to exist in one scope.
Lastly, you can mark the otherwise stack-allocated array with 'static' to take it off the stack and into a global data section:
static char output[(len * 2) + 1];
This enables you to return pointers to the buffer outside of its scope, however, all calls to such a function will refer to the same piece of global data, so don't use it if you need a unique block of memory every time.
Lastly, don't use malloc in C++ unless you have a really good reason (i.e, realloc). Use 'new' instead, and the accompanying 'delete'.