I would like to use a large array on the stack. However I keep on getting stack overflow although I changed the stack reserve size.
For example:
int main()
{
int a[5000000];
return 0;
}
and in visual studio 2012: Properties -> Linker -> System -> Stack reserve size: 10000000
What could cause it?
The size of an int is probably 4 bytes, so the array is larger in bytes than in the number of elements. sizeof(int)*5000000 is what you need. (at least)
Related
I was just going through this Wikipedia entry. Out of curiosity to find the stack size allocated to a simple process, i tried this
int main()
{
static int count = 0;
cout<<" Count = "<<count++<<endl;
main();
return 0;
}
Compiler DevC++
I got this :-
Till this point everything is fine, understandable, by the way from last digit i.e. 43385 can i guess the maximum stack size - on 32 bit machine (what if i say 4 bytes(4 bytes for return address on stack for each call), i may sound silly on this.
Now if i modify my program to :-
void foo()
{
static int count = 0;
cout<<" Count = "<<count++<<endl;
foo();
}
int main()
{
foo();
return 0;
}
In this i get Stack Over flow at count :- 130156 (ok, fine)
But my question if, i add one function in between main and foo, i get this count decrements by 1(130155), if 2 functions in b/w foo and main count is decremented by 2(130154) and so on. Why is this behavior? Is it because 1 space is being consumed for each function address.
Firstly correct your program by adding Count++ (silly).
Stack size is not fixed, most compilers let you specify the Stack size. Stack size is also dependent on some factors like Platform, toolchain, ulimit,and other parameters.There are many static and dynamic properties that can influence it.
There are three kinds of memory limits:
for 32-bit (windows)
Static data - 2GB
Dynamic data - 2GB
Stack data - 1GB (the stack size is set by the linker, the default is 1MB. This can be increased using the Linker property System > Stack Reserve Size).
by using your program you can guess the current stack size.
The memory-limits-applications-windows Stack Overflow Recursion in C c-maximum-stack-size-of-program will help you.
I’d like to understand the call-stack a bit better, and in an effort to do so, I’d like to work with it some more.
What will I do with this information? I have no idea! It’s fun to learn new things though, and this is my new thing for today.
I don’t like the call-stack in my programs being this mystical entity that I know nothing about. How big is it? How much memory is available on my current call-stack? Where is it? I don't know! And, I would like to know.
My current way to deal with the call-stack is to be aware of it, but to not knowingly interact with it until I hit a stack overflow error. That's just not good enough!
So in my quest, I’d like to figure out how to do the following:
I'd like to know the total size of the current call-stack that I'm operating on.
I'd like to calculate the total available memory available on my current call-stack.
I’d like to be able to find the address of where my call-stack starts.
The declaration could look like this, and be filled in for various platforms:
class call_stack
{
inline static void* base_address();
inline static void* end_address();
inline static std::size_t size();
inline static std::size_t remaining();
};
How is stack information defined on current desktop and mobile platforms, and how can I access it? How can I code this at compile time, or determine it at runtime?
Looking into this reminds me of a talk that Herb Sutter gave about atomics. He said something along the lines of “as C++ programmers, we like to push the big red buttons that say don’t touch.”
I realize there are many questions about the call-stack, and I believe that mine is both meaningful and unique.
Other questions on the call-stack topic don't ask about the call-stack in a similarly broad manner to what I'm asking here.
The Stack
In modern platforms, the stack structure is simplified to using a pointer to where the next item should be inserted. And most platforms use a register to do this.
+---+
| |
+---+
| |
+---+
| |
+---+
| | <-- Next available stack slot
+---+
An item is written at the present location of the stack pointer, then the stack pointer is incremented.
The CALL Stack
The minimum that needs to be pushed onto the stack is the return address or address of next instruction after the call. This is about as common as you get. Anything else that is pushed onto the stack depends on the protocol set up by the compiler or operating system.
For example, the compiler my choose to place parameters into registers rather than pushing them onto the stack.
The order of the parameters depends ultimately on the compiler (and language). The compiler may push the leftmost value first or the last parameter first.
Stack Starting Address
The Stack Starting Address is usually determined by the operating systems, or on embedded systems, at a fixed location. There is no guarantee that the operating system will place your program's stack in the same location for every invocation.
Stack Size
There are two terms here: capacity and content. Stack size can refer to the number of elements in the stack (content) or the capacity that the stack can hold.
There is no fixed, common limit here. Platforms differ. Usually the OS is involved in allocating the capacity of the stack. And on many platforms, the OS doesn't check to see if you have exceeded the capacity.
Some Operating Systems provide mechanisms so that the executable can adjust the capacity of the stack. Most OS providers usually provide an average amount.
Sharing Memory with Heap
A common setup is to have the stack grow towards the heap. So all the extra memory for a process is allocated so that, for example, the stack grows from the beginning of the extra memory down and the heap allocates from the bottom up. This allows for programs that use little dynamic memory to have more stack space and those that use little stack space to have more dynamic memory. The big issue is when they cross. No notification, just undefined behavior.
Accessing Stack Information
Most of the time, I never look at the values of the stack pointer or register. This is usually on performance critical systems that have restricted memory capacities (like embedded systems).
The stack is usually reviewed by the debugger to provide a call trace.
None of what you ask can be done in standard c++ since the stack is not specified in the standard.
You may be able to access that information by reading cpu registers in assembly language. How to do that depends on the cpu architecture, the OS and possibly the calling convention used by the compiler. Best place to find the information you're looking for is the manual for the architecture, OS, etc. The platfrom may also provide the information through system calls or virtual filesystems.
As an example, here's a quick look at the wikipedia page for the common x86 architecture
SP/ESP/RSP: Stack pointer for top address of the stack.
BP/EBP/RBP: Stack base pointer for holding the address of the current stack frame.
You can unwind the stack and find find the top of the stack in the first call frame. How to unwind the stack is again, specific to calling convention used. Subtracting the base of the first stack frame with the current stack pointer would give you the current size of the stack. Also remember that each thread has their own call stack so you must subtract from the bottom of the stack of the correct thread.
But keep in mind that:
Although the main registers (with the exception of the instruction pointer) are "general-purpose" in the 32-bit and 64-bit versions of the instruction set and can be used for anything...
You should check the manual of your target platform before assuming what the registers are used for.
Getting the remaining / total maximum stack space can be a bit trickier. In Linux the stack size is typically limited during runtime. The limit can be accessed through the /proc/ filesystem or using system calls. In Windows, the maximum stack size can be set at linktime and should be accessible in the executable file headers.
Below is an example program that works on Linux. I read the start of stack from /proc/<pid>/stat. I also provide an example for unwinding and for that I use a library that abstracts away all the OS / architecture specific assembly code. The stack is unwound all the way up to the initialization code before main and the stack space used by that is accounted for.
I use SP register instead of BP to get the bottom of the stack in the fisrt call frame because BP does not exist in some architectures and on my platform it was zero in the initialization frames. That means that the bottom is off by the size of the first call frame and is therefore just an approximation. See it on coliru, unfortunately access to /proc/<pid>/stat is denied there.
#include <iostream>
using namespace std;
#include <fstream>
#include <sstream>
#include <unistd.h>
// read bottom of stack from /proc/<pid>/stat
unsigned long bottom_of_stack() {
unsigned long bottom = 0;
ostringstream path;
path << "/proc/" << getpid() << "/stat";
ifstream stat(path.str());
// possibly not the best way to parse /proc/pid/stat
string ignore;
if(stat.is_open()) {
// startstack is the 28th field
for(int i = 1; i < 28; i++)
getline(stat, ignore, ' ');
stat >> bottom;
}
return bottom;
}
#include <sys/resource.h>
rlim_t get_max_stack_size() {
rlimit limits;
getrlimit(RLIMIT_STACK, &limits);
return limits.rlim_cur;
}
#define UNW_LOCAL_ONLY
#include <libunwind.h>
// using global variables for conciseness
unw_cursor_t frame_cursor;
unw_context_t unwind_context;
// approximate bottom of stack using SP register and unwinding
unw_word_t appr_bottom_of_stack() {
unw_word_t bottom;
unw_getcontext(&unwind_context);
unw_init_local(&frame_cursor, &unwind_context);
do {
unw_get_reg(&frame_cursor, UNW_REG_SP, &bottom);
} while(unw_step(&frame_cursor) > 0);
return bottom;
}
// must not inline since that would change behaviour
unw_word_t __attribute__((noinline)) current_sp() {
unw_word_t sp;
unw_getcontext(&unwind_context);
unw_init_local(&frame_cursor, &unwind_context);
unw_step(&frame_cursor); // step to frame before this function
unw_get_reg(&frame_cursor, UNW_REG_SP, &sp);
return sp;
}
// a little helper for absolute delta of unsigned integers
#include <algorithm>
template<class UI>
UI abs_udelta(UI left, UI right) {
return max(left,right) - min(left,right);
}
unsigned long global_bottom;
rlim_t global_max;
// a test function to grow the call stack
int recurse(int index) {
if(index < 2 ) {
auto stack_size = abs_udelta(current_sp(), global_bottom);
cout << "Current stack size: " << stack_size
<< "\tStack left: " << global_max - stack_size << '\n';
return index;
}
return recurse(index - 1) + recurse(index - 2); // do the fibonacci
}
int main() {
global_max = get_max_stack_size();
global_bottom = bottom_of_stack();
auto appr_bottom = appr_bottom_of_stack();
cout << "Maximum stack size: "
<< global_max << '\n';
cout << "Approximate bottom of the stack by unwinding: "
<< (void*)appr_bottom << '\n';
if(global_bottom > 0) {
cout << "Bottom of the stack in /proc/<pid>/stat: "
<< (void*)global_bottom << '\n';
cout << "Approximation error: "
<< abs_udelta(global_bottom, appr_bottom) << '\n';
} else {
global_bottom = appr_bottom;
cout << "Could not parse /proc/<pid>/stat" << '\n';
}
// use the result so call won't get optimized out
cout << "Result of recursion: " << recurse(6);
}
Output:
Maximum stack size: 8388608
Approximate bottom of the stack by unwinding: 0x7fff64570af8
Bottom of the stack in /proc/<pid>/stat: 0x7fff64570b00
Approximation error: 8
Current stack size: 640 Stack left: 8387968
Current stack size: 640 Stack left: 8387968
Current stack size: 576 Stack left: 8388032
Current stack size: 576 Stack left: 8388032
Current stack size: 576 Stack left: 8388032
Current stack size: 576 Stack left: 8388032
Current stack size: 576 Stack left: 8388032
Current stack size: 512 Stack left: 8388096
Current stack size: 576 Stack left: 8388032
Current stack size: 576 Stack left: 8388032
Current stack size: 512 Stack left: 8388096
Current stack size: 512 Stack left: 8388096
Current stack size: 512 Stack left: 8388096
Result of recursion: 8
linux: start here http://man7.org/linux/man-pages/man3/backtrace.3.html
windows: start here: http://msdn.microsoft.com/en-us/library/windows/desktop/bb204633(v=vs.85).aspx
Here's a way to get a rough estimate of the size of the current call stack.
In main(), save the address of argc to a global variable. This is somewhat close to where your stack starts. Then when you want to check the current size, take the address of the first argument to your current function and subtract it from your saved value. This will be less accurate if you have a large amount of data in automatic variables in main() or the current function (I'm not sure which one - it may vary by platform).
There are probably cases where the stack grows dynamically by chaining blocks, in which case this technique will not be accurate.
In a multithreaded program, you would have to track the start of each thread's stack separately. Similarly to main, get the address of one of the parameters to the top-level thread function when the thread starts up.
I wish I had thought to save the link to the page where I found someone doing something similar. They got the address of one of the automatic variables instead of one of the parameters, which would still give a very similar result.
I want to allocate a vector of size 1765880295 and so i used resize(1765880295) but the program stops running.The adjact problem would be code block not responding.
what is wrong?
Although the max_size gives 4294967295 which is greater than 1765880295 the problem is still the same even without resizing the vector.
Depending on what is stored in the vector -- for example, a 32-bit pointer or uint32, the size of the vector (number of elements * size of each element) will exceed the maximum addressable space on a 32-bit system.
The max_size is dependent on the implementation (some may have 1073741823 as their max_size). But even if your implementation supports a bigger number, the program will fail if there is not enough memory.
For example: if you have a vector<int>, and the sizeof(int) == 4 // bytes, we do the math, and...
1765880295 * 4 bytes = 7063521180 bytes ≈ 6.578 gygabytes
So you would require around 6.6GiB of free memory to allocate that enormous vector.
I'm using Visual Studio 2010 Win 8. I have a class where I'm making a 2D array to hold game data for a game.
Create a blank console app and make main.cpp and add this code. Using 360 for MAP_SIZE causes stack overflow using 359 doesn't. Why would this be? I'm looking to have a much larger size array. I'd like something like 2000 - 10,000 ideally.
#define MAP_SIZE 360
typedef unsigned short ushort;
typedef unsigned long ulong;
struct Tile
{
ushort baseLayerTileID;
ulong ownerID;
};
class Server
{
private:
Tile _map[MAP_SIZE][MAP_SIZE];
};
int main()
{
Server s;
return 0;
}
My estimates put sizeof(Tile) at 8 or more. That means sizeof(Server) is at least 360*360*8 = 1036800, which is 0.99 MB. The stack is usually small, and 1MB is a common default size. You should allocate the tiles on the heap instead, perhaps using std::vector.
class Server
{
public:
Server() : _map(MAP_SIZE * MAP_SIZE) {}
private:
std::vector<Tile> _map; // position [i][j] is at [i*MAP_SIZE+j]
};
You're allocating an array of 360 x 360 Tile objects on the stack. This is a bad idea from the get go. You are allocated a very large block of memory on the stack. The stack isn't intended for this type of usage.
This memory should either be static, if you only need one instance and know in advance the size, or you should allocate it from the heap (using new or even malloc()).
Consider having the constructor for Server allocate the memory using new instead of doing it how you are doing it.
The stack has limited size. If you need to hold a big array, use dynamic allocation.
You've created a type which requires ~1MB of stack space per instance, which apparently is larger than your stack can accommodate.
The portable option is to change from a fixed array to dynamically allocated or to a vector type.
The non-portable option is to increase the stack size in your application (which in turn increases the size of the stack for all threads)
The default stack size is 1MB. your struct size = ushort(2bytes) + ulong (4byte)= 6 bytes which the compiler converts to 8 bytes for struct alignment .
so 8*360*360 =1036800 Bytes , marginally above 1MB
There are 3 solutions:
1- force stop alignment :
#pragma pack(push) /* push current alignment to stack */
#pragma pack(1) /* set alignment to 1 byte boundary */
struct Tile
{
ushort baseLayerTileID;
ulong ownerID;
};
#pragma pack(pop) /* restore original alignment from stack */
This will allow for a maximum MAP_SIZE= sqrt(1024*1024/6)=418 ,so this allows for a larger mapsize but not the size you wish
2-You can change visual studio settings to allow compiler and linker to use more than 1 MB in stack:
you need to change it to be larger the maximum mapsize you need which is 8*10000*10000 ~800MB
right click project, and choose properties from the menu .
go to configuration properties->C/C++-> Commandline, add this parameter:
/F801000000
go to Configuration properties->Linker->Commandline, add this parameter
/STACK:801000000
Done!
3- the third solution is dynamic array to allocate over the heap, instead of static array , as all have said.
I am doing some extremely large array processing. I do a global declaration of:
`float array[200][1600][811];`
When I build my solution in MS Visual Studio 2010, I get the following error
LINK : fatal error LNK1248: image size (F85C8000) exceeds maximum allowable size (80000000)
Now, I am aware this amounts to about 1 GB of program memory. But this declaration worked for a declaration of float [50][1600][811] which amounts to 250 MB. I know the default stack size is very limited.
There are a couple of things I have already tried. I increased the stack size in VS through Properties -> Linker -> Stack reserved size. This didnt help. I changed my executable to run in x64 mode (which is said to address upto 2GB memory!). This didnt help either.
I do not wish to do a malloc on the array because I know for sure I need them in my code. I had to make them global declarations so that I can avail the stack/heap memory. If I declare them inside my Main (), it gives me error of memory overflow.
Any pointers would be greatly appreciated.
Thanks.
It appears that even when you're building an x64 executable, the linker has limits more appropriate for an x86 build. Not much you can do about that.
The only solution is to allocate it from the heap. This should be usable in the same way as your original declaration.
typedef float partial_array[1600][811];
std::unique_ptr<partial_array> array = new partial_array[200];
If you are malloc adverse, you have two immediately obvious possibilties. C++11 has a nice array type which might help:
std::array<std::array<std::array<float, 50>, 1600>, 811> matrix;
Or you could consider using std::vector with a loop to initialise all the values correctly:
std::vector<std::vector<std::vector<float>>> matrix;
matrix.reserve(50);
for (size_t i = 0; i < 50; i++)
{
std::vector<std::vector<float>> submatrix;
submatrix.reserve(1600);
for (size_t j = 0; j < 1600; j++)
{
std::vector<float> row;
row.resize(811);
submatrix.push_back(row);
}
matrix.push_back(submatrix);
}